Robots.txt for your website ?

bob

New member
Joined
Apr 22, 2014
Messages
203
Points
0
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
 

suzykenedy

New member
Joined
Mar 10, 2014
Messages
16
Points
0
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.
 
Older threads
Replies
46
Views
19,823
Replies
12
Views
14,369
Replies
14
Views
8,185
Replies
10
Views
5,658
bob
Replies
9
Views
5,676
Newer threads
Replies
94
Views
115,934
Replies
12
Views
7,289
Replies
4
Views
3,994
Latest threads
Recommended threads
Replies
2
Views
2,393
Replies
2
Views
4,287
Replies
5
Views
1,483

Latest postsNew threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top