The difference between Robots.txt and Robot Meta Tag

[Kal K]

Hi Guys,

Can you tell me know what is the major difference between robot meta tag and robot.txt?
I am confused in both. When should I use robots.txt or robot meta tag?

Kal

 

[elcidofaguy]

They basically do the same thing... Robots.txt is a single file where you can consolidate bot access or alternatively you can use the robot meta tag on specific pages.... Saying that a lot of bots do not respect robots.txt so alternatively if you are seeking to block bots you could also use .htaccess which enforces it at server access level...

 

[MattRobinson]

The robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

Whereas Robot meta tag, you have to provide allow!/disallow to read the content of your web page !!bu default allow exists

 

[sweetsdailyupdates]

The role of robots.txt file & Robot meta tag are same. Robots.txt file is specific text file where you can restrict some parts of website. If you have ids in URL in some pages of your website and you don't want to crawl them so you can restrict them in robots.txt file. If you don't want use robots.txt file so you have to add robots meta tag in each page like as given below.

<meta name="robots" content="noindex,nofollow">

 

[hoangvu]

They basically do the same thing... Robots.txt is a single file where you can consolidate bot access or alternatively you can use the robot meta tag on specific pages.... Saying that a lot of bots do not respect robots.txt so alternatively if you are seeking to block bots you could also use .htaccess which enforces it at server access level...

I don't think it is the same thing, if you want to block some webpages or whole a website you should add into robots.txt. Otherwise google is still caching pages using robot meta tags.

The role of robots.txt file & Robot meta tag are same. Robots.txt file is specific text file where you can restrict some parts of website. If you have ids in URL in some pages of your website and you don't want to crawl them so you can restrict them in robots.txt file. If you don't want use robots.txt file so you have to add robots meta tag in each page like as given below.

<meta name="robots" content="noindex,nofollow">

I had have this tag in my webpages but Google made cached versions for them, seem google ignored meta robot tags hence to avoid your pages are cached by Google spiders, you should go with robots.txt