Do you use any kind of bot control method via htaccess file method or some other method or limit crawler traffic using robots.txt or anything?
You can either use .htaccess or robots.txt to block them or tell them some other message like do one ya nosey git.
The question you have to ask yourself is, should you?
And what are the pros and cons to blocking crawler bots from accessing, crawling and indexing your site?
Can it effect your SERP ranking by blocking bots and can it have an adverse or positive effect on your site?
My answer to the first question is one that's based on experience of managing large sites and that is an explicit yes, you should.
Other than the genuine crawlers you want on your site, all these others do is use up server resources and bloat your Analytics with duff data logs.
So naturally some kind of control is needed. And for that you can use htacess file which can be a VERY POWERFUL beast of a file.
Now obviously you don't want to block every single crawler bot from visiting your site including Google. For obvious reasons.
If Google can't access and crawl your site, it can't index your content and thus you can't rank for content that can't be accessed.
Thusly your rankings would be effected if Google can't even access/crawl and index your site to show to searchers.
It's the same for Bing or any other search engine crawler bot too.
However one can certainly apply some limitation and can indeed block certain bots from accessing/crawling your site and for good reason too.
Especially if that bot is a bot that is for a search engine or country that your website is not even served to.
And let's face it, there are only 2 or 3 search engines you want to be indexed in Google, Bing/Yahoo.
Unless you're a Russian site then you might want to be indexed by Yandex too.
Or perhaps its not even a search engine bot and is up to something much more sinister like looking for vulnerabilities or places to spam or just sites to rip.
And in these cases where you suddenly see lots of bot activity its not too hard to find out what they are or where they're from.
Check your Apache logfiles for starters as that will show everything you need to know including the bots string in which to use to block it with in htaccess or robots file.
By the way, if you're looking for "
A Close to perfect .htaccess file" then there's a useful block you can enter
here which blocks most known bots and rippers I like to use when this starts happening sometimes.
But don't think just putting them in the htaccess file is all you need to do. Well it is but sometimes you only need to block them for a few weeks or something and eventually they stop trying and go somewhere else haha
You can then remove them from it or keep them in up to you but its good to have a slimline htaccess file.
Good luck!
