How to check robots.txt on site?

hoangvu

New member
Joined
Jun 6, 2012
Messages
1,835
Points
0
Hi Friends..


If I would like to check whether a website has robots.txt, how would I check it?
 

Top Niche Websites

New member
Joined
Jul 11, 2013
Messages
81
Points
0
Just type this into your browser and see if you get a result

[URLnf=http://domain/robots.txt]http://domain/robots.txt[/URLnf]

just change domain to whatever site you are checking

Michael
 

psavsnacks

New member
Joined
Mar 11, 2014
Messages
15
Points
0
simple way to check robots.txt file....

type home page website url/robots.txt
 

TejasWebHosting

New member
Joined
Nov 9, 2014
Messages
33
Points
0
It is important to remember that just because you can browse the site, doesn't always mean everything is okay....

I would recommend using either Bing or Google's Webmaster Tools service to check it. They have a tool that allows you to confirm if it is visible or not with the click of a button.
 
Older threads
Replies
1
Views
3,274
Replies
35
Views
20,137
Newer threads
Latest threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top