Crawl-Delay, what does it mean?

Marc van Leeuwen

Premium Member
Joined
May 29, 2016
Messages
1,127
Points
63
I have asked a SEO question on a forum and I get answered like this

Code:
What you're looking for is the robots.txt file though, and you want to add some lines like this:

User-agent: *
Disallow:
Crawl-Delay: 10

User-agent: ia_archiver
Disallow: /

User-agent: Ask Jeeves
Crawl-Delay: 120

User-agent: Teoma
Disallow: /html/
Crawl-Delay: 120
or

Code:
User-agent: Googlebot
Crawl-delay: 10

Can someone explain me what is the meaning above lines?

Thanks
Marc
 

Intuz

New member
Joined
Nov 24, 2016
Messages
44
Points
0
This is instruction for search engine bots to do not crawl your site too frequently, to prevent server overwhelming. It means having a 10 second delay between each. The idea is to prevent an overload of the servers by all the bots crawling at the same time.
 

Liam

New member
Joined
Jan 25, 2017
Messages
75
Points
0
Liam
10 seconds, is that a normal delay?

I don't really know much about this but 10 seconds for a delay doesn't seem like it would have that much of an impact.
 

Unknown

New member
Joined
Jun 28, 2017
Messages
151
Points
0
Well, I have read that Googlebot actually ignores the crawl-delay but generally speaking, crawl-delay is just used to prevent a server from getting overwhelmed with the crawls... that's what the delay is for. I don't know how needed it is for an average website but I think that sites like Facebook/Twitter have it set up because the content there constantly changes.

So, unless you have an extremely big website with tons of new content and it changing, I don't think that crawl-delay is really needed ;) I hope that this helps a bit!
 

extremebizopp

New member
Joined
Mar 22, 2015
Messages
10
Points
0
I agree with @Unknown and that's why I let Google decides my crawl delay rate for almost my websites.
 
Older threads
Replies
7
Views
5,265
Replies
4
Views
2,291
Replies
7
Views
5,551
Replies
12
Views
7,395
Latest threads
Replies
1
Views
50
Replies
1
Views
50
Replies
1
Views
81
Replies
1
Views
74
Replies
2
Views
107

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top