Robots.txt for your website ?

Jack London

New member
Joined
Jul 9, 2012
Messages
386
Points
0
Hi webmasters,

As I knew if we want Google crawl all our website then use this code in robots.txt file putted in website root directory
Code:
User-agent: *
Allow: /
but I checked some sites and they didn't put robots.txt file for their websites. Thus I want to know that it's necessary to have a robots file for my website or not ? and why ?

please share your suggestions ?
 

Joseph_Hill

New member
Joined
Jan 18, 2013
Messages
192
Points
0
Thanks for sharing this code, but initially I want to know why Robot.txt file should be their in our website?And why it's important in terms of SEO strategy?
 
  • Like
Reactions: William

ketul

New member
Joined
Jul 16, 2014
Messages
108
Points
0
ketul
If you want to hide your sites some portion like directory and file that very secret for you, you can specify it in robots.txt, so SE crawler does not crawl and index that file and user don't see that particular page or file in search result.
Simply it tells to SE crawler that which page is allow for crawl and index and which is not.
 

sachingoyal

New member
Joined
Aug 1, 2013
Messages
7
Points
0
The robots.txt file is a very essential file if you wanna increase your site ranking.

Robots.txt file

When a search engine crawler comes to your website, it will look for a special root file on your website. That file is called robots.txt and it tells the search engine spider, which Web pages of your website should be indexed and which Web pages should be ignored.

Like : http://www.yourwebsite.com/robots.txt
 

aishasony

New member
Joined
May 7, 2013
Messages
101
Points
0
yes its necessary to place Robot.txt file because it get rid of all the 404 errors simply create an empty file called robots.txt and upload that to your site.

You should be able to tell if it works by examining site stats for periods before and after uploading as the number of 404 errors on the site should be dramatically reduced.
 

PankajS

New member
Joined
Mar 5, 2013
Messages
79
Points
0
if you are confused that some website not using robot.txt then i would like to describe that robot file is only use for preventing any webpage from search engine. not to crawl webpage.
 

subin

New member
Joined
Aug 29, 2013
Messages
47
Points
0
Robots.txt is an important tool to identify crawl all websites, faster to index them on all top search engines.
 

novandak

New member
Joined
Aug 27, 2013
Messages
111
Points
0
Hi,
I think robots.txt is most important for Search Engine.
When a search engine crawler comes to your site, it will look for a specific file on your site. That is called robots.txt. It's a simple text file.
It tells the search engine spider that which web pages should be ignored by Search Engine and which should be indexed.
 

dagnyjbarber

New member
Joined
Mar 5, 2013
Messages
3
Points
0
disallow term only prevent from crawling not from indexing. for prevent from indexing, you have to type noindex.
 

kathleenortan

New member
Joined
Jan 28, 2013
Messages
47
Points
0
Robots.txt is not always necessary, you can choose not to use them on your websites depending on the content of your website. You can use them occasionally for some websites. Some SEOs not happening practical knowledge may suggest you that it is important because they have read like this online.
 

BloodMaster

New member
Joined
Aug 5, 2013
Messages
346
Points
0
Through the robot.txt file Google bot/spiders/crawlers know what is in site. Crawlers get familiar with site.
 

hoangvu

New member
Joined
Jun 6, 2012
Messages
1,844
Points
0
It's necessary for you to create robots.txt when you need to protect some files or folders on your server from crawling by Google.

I found one good article on How to use Robots.txt on our webmaster sun.

Hope this helps!
 

Deepak Kaushik

New member
Joined
Dec 14, 2013
Messages
114
Points
0
Main purpose of uploading robots.txt file is that if you want to block any webpage, folder, file in the Google's eye then by the use of robots.txt you can do this.
 

jasonquey

New member
Joined
Jan 25, 2013
Messages
50
Points
0
robots.txt file in your website. The robots.txt file which presents in webmaster's will be updated automatically... No need to do any changes in webmaster's robots.txt.
 

deepakrajput

New member
Joined
Mar 28, 2014
Messages
43
Points
0
Robots.txt is the main file for Search Engine caching your site to rank them. It is set of some instructions which admin want to Allow or disallow to Search Engine.
 

nathancs

New member
Joined
Mar 7, 2014
Messages
62
Points
0
Robots.txt is most important for search engine spider to tell the search engine crawl the web pages.
 

seoindia

New member
Joined
Apr 21, 2014
Messages
7
Points
0
There is no way that can prevent search engines or say Google to index a website and/or pages. Google will surely fetch your webpage information in any case whether you put or do not put robot.txt in your URL. Search engines will get your site information from the back links that are distributed throughout the web.
 
Older threads
Replies
46
Views
19,340
Replies
12
Views
14,055
Replies
14
Views
7,544
Replies
10
Views
5,514
bob
Replies
9
Views
5,564
Newer threads
Replies
94
Views
112,220
Replies
12
Views
7,105
Replies
4
Views
3,792
Latest threads
Replies
2
Views
127
Replies
1
Views
190
Replies
6
Views
438
Replies
11
Views
547
Replies
2
Views
238
Recommended threads

Latest postsNew threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top