Robots tag

Alex_smith

New member
Joined
Dec 19, 2012
Messages
385
Points
0
How to add the Robots tag to the website, I want to hide my particular page from search engines. Please explain about the same ?
 

imcd

New member
Joined
Jan 16, 2013
Messages
4
Points
0
Open a notepad and name it as robots. So it would be a robots.txt file. There are certain protocols to write a robots.txt file. First you need to adree the user agent or the robot you don't want to crawl and index certain pages of your site. And then you are to disallow just like below.

user agent : * (covers all the search engine robots)
user agent: google bot (or any particular bot you want to mention)

Disallow : /thankyou.php
Disallow: /ebook.pdf

(here you don't need to write the full url)

Save the file and upload it to the root directory of your site. You are done. I suggest you to read some guide before you create your robots.txt.
 

Alex_smith

New member
Joined
Dec 19, 2012
Messages
385
Points
0
You have explained nicely about Robots.txt file, I needed this info, and I have saved your post. Thank you.
 

HostNurse

New member
Joined
Aug 9, 2012
Messages
376
Points
0
By default the search engines robots will index and follow your web site, if you do not wish a certain page to be indexed or/and followed by the search engines, you can use the following meta tags:

To allow indexing and following of a certain page:
<meta name="robots" content="noindex,follow" />

Another way of declaring to (Index and follow):
<meta name="robots" CONTENT="all">

This will disallow indexing and following of the page:
<meta name="robots" content="noindex,nofollow" />

Another way to declaring to (Do not index and do not follow):
<meta name="robots" content="none">

This will disallow indexing of the page, lets the crawler go on and follow/crawl links contained within the page:
<meta name="robots" content="noindex,follow" />

This allows indexing of the page, but instructs the crawler to not follow/crawl links withing the page:
<meta name="robots" content="index,nofollow" />

You can also use the robots.txt file as mentioned by imcd, which is effective specially with dynamic websites, where you do not have control over every page of your website and where you are unable to include the meta tag on a certain page.
 

OWF

New member
Joined
Feb 25, 2013
Messages
69
Points
0
Better suggestion, but it would be better if you know how to use URL parameter in GWT
 

PankajS

New member
Joined
Mar 5, 2013
Messages
79
Points
0
Thanks for sharing this information about robot.txt
 
Last edited by a moderator:
Older threads
Replies
12
Views
4,518
Replies
22
Views
26,715
Replies
12
Views
9,094
Replies
2
Views
2,765
Newer threads
Replies
14
Views
7,861
Replies
3
Views
5,226
Replies
14
Views
6,079
Replies
7
Views
4,684
Replies
3
Views
3,302
Latest threads
Replies
2
Views
126
Replies
1
Views
189
Replies
6
Views
431
Replies
11
Views
546
Replies
2
Views
238
Recommended threads
Replies
7
Views
3,134
Replies
5
Views
805
Replies
2
Views
3,105
Replies
46
Views
16,007

Latest postsNew threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top