Thread: Why Bing why?
04-04-2016, 02:55 PM #1
Why Bing why?
My web-host told me to keep track of bots that keep using up my resources....and it seems Bing constantly has an army of bots constantly crawling my website!! Google crawls my websites less often and gives better results. 30 Bing bots vs 1 Google bot....
The Following User Says Thank You to denvercardonations For This Useful Post:
04-04-2016, 03:34 PM #2
- Join Date
- Jan 2015
- Blog Entries
- Thanked 883 Times in 562 Posts
Thumbs Up/Down Received: 162/0
I'm somewhat surprised at that and my initial reaction is that a bot is attempting to masquerade as a bing bot given it is pretty easy to do so with declared user agent type. I would investigate this further with looking into your server logs with getting the IP data. You can check if it is a fake bing bot with use of bings own checker tool here.
04-05-2016, 04:02 AM #3
This definitely doesn't sound like it's a true bingbot, not at that crawl rate ratio.
I'd definitely do as elcidofaguy stated and check to make sure it's a true bot from Bing.
I was getting something similar about a year or so ago and it turned out to come from some Russian source.
The Following User Says Thank You to ulterios For This Useful Post:
04-05-2016, 05:45 PM #4
Do you use any kind of bot control method via htaccess file method or some other method or limit crawler traffic using robots.txt or anything?
You can either use .htaccess or robots.txt to block them or tell them some other message like do one ya nosey git.
The question you have to ask yourself is, should you?
And what are the pros and cons to blocking crawler bots from accessing, crawling and indexing your site?
Can it effect your SERP ranking by blocking bots and can it have an adverse or positive effect on your site?
My answer to the first question is one that's based on experience of managing large sites and that is an explicit yes, you should.
Other than the genuine crawlers you want on your site, all these others do is use up server resources and bloat your Analytics with duff data logs.
So naturally some kind of control is needed. And for that you can use htacess file which can be a VERY POWERFUL beast of a file.
Now obviously you don't want to block every single crawler bot from visiting your site including Google. For obvious reasons.
If Google can't access and crawl your site, it can't index your content and thus you can't rank for content that can't be accessed.
Thusly your rankings would be effected if Google can't even access/crawl and index your site to show to searchers.
It's the same for Bing or any other search engine crawler bot too.
However one can certainly apply some limitation and can indeed block certain bots from accessing/crawling your site and for good reason too.
Especially if that bot is a bot that is for a search engine or country that your website is not even served to.
And let's face it, there are only 2 or 3 search engines you want to be indexed in Google, Bing/Yahoo.
Unless you're a Russian site then you might want to be indexed by Yandex too.
Or perhaps its not even a search engine bot and is up to something much more sinister like looking for vulnerabilities or places to spam or just sites to rip.
And in these cases where you suddenly see lots of bot activity its not too hard to find out what they are or where they're from.
Check your Apache logfiles for starters as that will show everything you need to know including the bots string in which to use to block it with in htaccess or robots file.
By the way, if you're looking for "A Close to perfect .htaccess file" then there's a useful block you can enter here which blocks most known bots and rippers I like to use when this starts happening sometimes.
But don't think just putting them in the htaccess file is all you need to do. Well it is but sometimes you only need to block them for a few weeks or something and eventually they stop trying and go somewhere else haha
You can then remove them from it or keep them in up to you but its good to have a slimline htaccess file.
08-23-2016, 01:08 PM #5
Replies: 0 | Views: 665Last post by Omer Farooq, 04-05-2016, 10:03 AM
Replies: 0 | Views: 884Last post by Hawker, 04-05-2016, 01:44 PM
By Joe B. Ray in forum ServicesReplies: 0 | Views: 660Last post by Joe B. Ray, 04-05-2016, 02:24 PM
By Hawker in forum Link SalesReplies: 2 | Views: 1341Last post by yellowmed, 06-03-2016, 08:38 AM
By OWF in forum Buy and Sell WebsitesReplies: 10 | Views: 5970Last post by sumit_sharma, 05-30-2016, 11:28 AM
Replies: 8 | Views: 1354Last post by Hawker, 04-08-2016, 01:14 PM
Replies: 8 | Views: 912Last post by SEOPub, 04-04-2016, 01:34 PM
Replies: 4 | Views: 1338Last post by Adam Yunker, 04-15-2016, 05:02 PM
Replies: 4 | Views: 809Last post by ElixantTechnology, 04-06-2016, 12:13 AM
Replies: 7 | Views: 1409Last post by Ron Killian, 05-21-2016, 09:17 AM
Replies: 1 | Views: 16Last post by Marc van Leeuwen, Today, 01:01 PM
By dexxterr in forum Online Internet MarketingReplies: 0 | Views: 14Last post by dexxterr, Today, 10:11 AM
Replies: 0 | Views: 21Last post by PenguinManiac, Today, 09:19 AM
Replies: 0 | Views: 13Last post by PenguinManiac, Today, 09:08 AM
Replies: 9 | Views: 2534Last post by markbatham, 01-30-2017, 01:17 PM
Replies: 1 | Views: 879Last post by seoking, 10-28-2015, 01:20 PM
Replies: 16 | Views: 6214Last post by talukdar, 06-07-2015, 11:05 AM
Replies: 3 | Views: 2064Last post by ericplotz1, 10-20-2014, 04:26 AM
Replies: 3 | Views: 2814Last post by zebrathemes, 03-29-2013, 01:08 PM