Crawl-Delay, what does it mean?

Marc van Leeuwen

Premium Member
Joined
May 29, 2016
Messages
1,132
Points
63
I have asked a SEO question on a forum and I get answered like this

Code:
What you're looking for is the robots.txt file though, and you want to add some lines like this:

User-agent: *
Disallow:
Crawl-Delay: 10

User-agent: ia_archiver
Disallow: /

User-agent: Ask Jeeves
Crawl-Delay: 120

User-agent: Teoma
Disallow: /html/
Crawl-Delay: 120
or

Code:
User-agent: Googlebot
Crawl-delay: 10

Can someone explain me what is the meaning above lines?

Thanks
Marc
 

Intuz

New member
Joined
Nov 24, 2016
Messages
44
Points
0
This is instruction for search engine bots to do not crawl your site too frequently, to prevent server overwhelming. It means having a 10 second delay between each. The idea is to prevent an overload of the servers by all the bots crawling at the same time.
 

Liam

New member
Joined
Jan 25, 2017
Messages
75
Points
0
Liam
10 seconds, is that a normal delay?

I don't really know much about this but 10 seconds for a delay doesn't seem like it would have that much of an impact.
 

Unknown

New member
Joined
Jun 28, 2017
Messages
151
Points
0
Well, I have read that Googlebot actually ignores the crawl-delay but generally speaking, crawl-delay is just used to prevent a server from getting overwhelmed with the crawls... that's what the delay is for. I don't know how needed it is for an average website but I think that sites like Facebook/Twitter have it set up because the content there constantly changes.

So, unless you have an extremely big website with tons of new content and it changing, I don't think that crawl-delay is really needed ;) I hope that this helps a bit!
 

extremebizopp

New member
Joined
Mar 22, 2015
Messages
10
Points
0
I agree with @Unknown and that's why I let Google decides my crawl delay rate for almost my websites.
 
Older threads
Replies
7
Views
5,480
Replies
4
Views
2,430
Replies
7
Views
6,016
Replies
12
Views
8,075
Latest threads
Replies
2
Views
710
Replies
0
Views
650
Replies
4
Views
1,769
Replies
2
Views
770
Recommended threads
Replies
11
Views
5,078
Replies
13
Views
20,238
Replies
0
Views
2,244
Replies
4
Views
3,325

Latest postsNew threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top