Difference between meta robots and robots.txt

Riviera

New member
Joined
Aug 24, 2015
Messages
55
Points
0
I have very detail articles here for you, Sir. You can research it here:

robotstxt.org/meta.html
and
robotstxt.org/robotstxt.html

Hope that i can help.
 

Rob Whisonant

Moderator
Joined
May 24, 2016
Messages
2,481
Points
113
Personally I prefer to use meta tags to block a single page and robots.txt to block an entire directory.

Also note if you have links on a page that you want to send link juice but not have the page indexed, use the meta noindex tag.
 

Aegis software

New member
Joined
Feb 1, 2017
Messages
26
Points
0
If you don't want to index or follow some pages so you can use robots meta tag. But if you don't want to index a whole directory so you can use robots.txt file.

So you can restrict a whole directory or any URL format from indexing.
 

kinexmedia

New member
Joined
Feb 14, 2017
Messages
6
Points
0
I like your query . Both does the same function

If you are using Robots tags it instruct the google in different way depends on your pattern like if you write follow index it will allow spider to cache and index both . if you will write no index . follow It will cache that page but not index in google. index, no follow allow index but not cache.. index noodp - to disallow google to fetch data from directory listing


Same Robots.txt allow you to allow disallow particular page/directory/folder or entire site.


In my opinion You Should Go With Only Robots.txt file
 

vps9

New member
Joined
Apr 22, 2015
Messages
64
Points
0
Robots.txt dictates site or directory wide crawl behavior, whereas meta robots can dictate indexation behavior at the individual page level.
 

carlbrewster

New member
Joined
May 10, 2017
Messages
6
Points
0
Well, Via Robot.txt file you can instruct the search engine bot how to crawl your website, you can stop the bot to crawl you private page by mentioning them in your robot.txt file, this file exists in your website root folder so, whenever bot comes to your website it will frist read instructions from robot.txt file on the other hand robot meta has to update on a particular webpage it is of 3 types

<meta name="robots" content="noindex, nofollow">
<meta name="robots" content="nofollow, Index">
<meta name="robots" content="index, nofollow">
 

nickchernets

New member
Joined
Mar 14, 2017
Messages
30
Points
0
There are slightly different uses for each.

personally, I try and use the meta robots where possible, search engines seem to follow these directives more strongly than the robots.txt, you can also choose options such as noindex,follow which is really helpful for certain areas.

The problem I've got with the robots.txt is that it's too easy to make a mistake and harm a lot of your site, and you're completely blocking off areas of the site, so if there's links in there that you want to get indexed but forget about the robots.txt you'll have a problem, whereas if you use noindex,follow that's not an issue.
 
Newer threads
Replies
30
Views
8,632
Replies
1
Views
1,669
Latest threads
Replies
1
Views
74
Replies
1
Views
172
Replies
4
Views
378
Replies
11
Views
522
Replies
2
Views
227
Recommended threads
Replies
2
Views
2,242
Replies
10
Views
4,090
Replies
9
Views
4,931
Replies
1
Views
3,127

Latest postsNew threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top