Robots tag

[Alex_smith]

How to add the Robots tag to the website, I want to hide my particular page from search engines. Please explain about the same ?

 

[imcd]

Open a notepad and name it as robots. So it would be a robots.txt file. There are certain protocols to write a robots.txt file. First you need to adree the user agent or the robot you don't want to crawl and index certain pages of your site. And then you are to disallow just like below.

user agent : * (covers all the search engine robots)
user agent: google bot (or any particular bot you want to mention)

Disallow : /thankyou.php
Disallow: /ebook.pdf

(here you don't need to write the full url)

Save the file and upload it to the root directory of your site. You are done. I suggest you to read some guide before you create your robots.txt.

 

[Alex_smith]

You have explained nicely about Robots.txt file, I needed this info, and I have saved your post. Thank you.

 

[trueman1]

You can use online tools like: http://www.yellowpipe.com/yis/tools/robots.txt/

 

[HostNurse]

By default the search engines robots will index and follow your web site, if you do not wish a certain page to be indexed or/and followed by the search engines, you can use the following meta tags:

To allow indexing and following of a certain page:
<meta name="robots" content="noindex,follow" />

Another way of declaring to (Index and follow):
<meta name="robots" CONTENT="all">

This will disallow indexing and following of the page:
<meta name="robots" content="noindex,nofollow" />

Another way to declaring to (Do not index and do not follow):
<meta name="robots" content="none">

This will disallow indexing of the page, lets the crawler go on and follow/crawl links contained within the page:
<meta name="robots" content="noindex,follow" />

This allows indexing of the page, but instructs the crawler to not follow/crawl links withing the page:
<meta name="robots" content="index,nofollow" />

You can also use the robots.txt file as mentioned by imcd, which is effective specially with dynamic websites, where you do not have control over every page of your website and where you are unable to include the meta tag on a certain page.

 

[OWF]

Better suggestion, but it would be better if you know how to use URL parameter in GWT

 

[benedepil]

The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in search results.

 

[dagnyjbarber]

I agree with Trueman.
Great! I agree with it. You can use such type of tools.

 

[PankajS]

Thanks for sharing this information about robot.txt