The difference between Robots.txt and Robot Meta Tag

Kal K

Member
Joined
Feb 2, 2015
Messages
50
Points
8
Hi Guys,

Can you tell me know what is the major difference between robot meta tag and robot.txt?
I am confused in both. When should I use robots.txt or robot meta tag?

Kal
 

elcidofaguy

Well-known member
Joined
Jan 13, 2015
Messages
1,281
Points
113
They basically do the same thing... Robots.txt is a single file where you can consolidate bot access or alternatively you can use the robot meta tag on specific pages.... Saying that a lot of bots do not respect robots.txt so alternatively if you are seeking to block bots you could also use .htaccess which enforces it at server access level...
 

MattRobinson

New member
Joined
Jun 29, 2016
Messages
4
Points
0
The robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

Whereas Robot meta tag, you have to provide allow!/disallow to read the content of your web page !!bu default allow exists
 

sweetsdailyupdates

New member
Joined
Jul 5, 2016
Messages
41
Points
0
The role of robots.txt file & Robot meta tag are same. Robots.txt file is specific text file where you can restrict some parts of website. If you have ids in URL in some pages of your website and you don't want to crawl them so you can restrict them in robots.txt file. If you don't want use robots.txt file so you have to add robots meta tag in each page like as given below.

<meta name="robots" content="noindex,nofollow">
 

hoangvu

New member
Joined
Jun 6, 2012
Messages
1,843
Points
0
They basically do the same thing... Robots.txt is a single file where you can consolidate bot access or alternatively you can use the robot meta tag on specific pages.... Saying that a lot of bots do not respect robots.txt so alternatively if you are seeking to block bots you could also use .htaccess which enforces it at server access level...
I don't think it is the same thing, if you want to block some webpages or whole a website you should add into robots.txt. Otherwise google is still caching pages using robot meta tags.

The role of robots.txt file & Robot meta tag are same. Robots.txt file is specific text file where you can restrict some parts of website. If you have ids in URL in some pages of your website and you don't want to crawl them so you can restrict them in robots.txt file. If you don't want use robots.txt file so you have to add robots meta tag in each page like as given below.

<meta name="robots" content="noindex,nofollow">
I had have this tag in my webpages but Google made cached versions for them, seem google ignored meta robot tags hence to avoid your pages are cached by Google spiders, you should go with robots.txt
 
Older threads
Replies
11
Views
4,824
Replies
0
Views
2,302
Replies
4
Views
2,768
Recommended threads
Replies
5
Views
2,626
Replies
29
Views
7,508
  • Locked
Replies
0
Views
5,561
Replies
30
Views
6,211

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top