Crawl-Delay, what does it mean?

[Marc van Leeuwen]

I have asked a SEO question on a forum and I get answered like this

What you're looking for is the robots.txt file though, and you want to add some lines like this:

User-agent: *
Disallow:
Crawl-Delay: 10

User-agent: ia_archiver
Disallow: /

User-agent: Ask Jeeves
Crawl-Delay: 120

User-agent: Teoma
Disallow: /html/
Crawl-Delay: 120

or

User-agent: Googlebot
Crawl-delay: 10

Can someone explain me what is the meaning above lines?

Thanks
Marc

 

[Intuz]

This is instruction for search engine bots to do not crawl your site too frequently, to prevent server overwhelming. It means having a 10 second delay between each. The idea is to prevent an overload of the servers by all the bots crawling at the same time.

 

[Liam]

Liam

10 seconds, is that a normal delay?

I don't really know much about this but 10 seconds for a delay doesn't seem like it would have that much of an impact.

 

[Unknown]

Well, I have read that Googlebot actually ignores the crawl-delay but generally speaking, crawl-delay is just used to prevent a server from getting overwhelmed with the crawls... that's what the delay is for. I don't know how needed it is for an average website but I think that sites like Facebook/Twitter have it set up because the content there constantly changes.

So, unless you have an extremely big website with tons of new content and it changing, I don't think that crawl-delay is really needed I hope that this helps a bit!

 

[extremebizopp]

I agree with @Unknown and that's why I let Google decides my crawl delay rate for almost my websites.