Thread: Why Bing why?
04-04-2016, 02:55 PM #1
Why Bing why?
My web-host told me to keep track of bots that keep using up my resources....and it seems Bing constantly has an army of bots constantly crawling my website!! Google crawls my websites less often and gives better results. 30 Bing bots vs 1 Google bot....
04-04-2016, 03:34 PM #2
- Join Date
- Jan 2015
- Thanked 880 Times in 560 Posts
- Blog Entries
Thumbs Up/Down Received: 156/0
I'm somewhat surprised at that and my initial reaction is that a bot is attempting to masquerade as a bing bot given it is pretty easy to do so with declared user agent type. I would investigate this further with looking into your server logs with getting the IP data. You can check if it is a fake bing bot with use of bings own checker tool here.
The Following User Says Thank You to elcidofaguy For This Useful Post:
04-05-2016, 04:02 AM #3
This definitely doesn't sound like it's a true bingbot, not at that crawl rate ratio.
I'd definitely do as elcidofaguy stated and check to make sure it's a true bot from Bing.
I was getting something similar about a year or so ago and it turned out to come from some Russian source.
04-05-2016, 05:45 PM #4
Do you use any kind of bot control method via htaccess file method or some other method or limit crawler traffic using robots.txt or anything?
You can either use .htaccess or robots.txt to block them or tell them some other message like do one ya nosey git.
The question you have to ask yourself is, should you?
And what are the pros and cons to blocking crawler bots from accessing, crawling and indexing your site?
Can it effect your SERP ranking by blocking bots and can it have an adverse or positive effect on your site?
My answer to the first question is one that's based on experience of managing large sites and that is an explicit yes, you should.
Other than the genuine crawlers you want on your site, all these others do is use up server resources and bloat your Analytics with duff data logs.
So naturally some kind of control is needed. And for that you can use htacess file which can be a VERY POWERFUL beast of a file.
Now obviously you don't want to block every single crawler bot from visiting your site including Google. For obvious reasons.
If Google can't access and crawl your site, it can't index your content and thus you can't rank for content that can't be accessed.
Thusly your rankings would be effected if Google can't even access/crawl and index your site to show to searchers.
It's the same for Bing or any other search engine crawler bot too.
However one can certainly apply some limitation and can indeed block certain bots from accessing/crawling your site and for good reason too.
Especially if that bot is a bot that is for a search engine or country that your website is not even served to.
And let's face it, there are only 2 or 3 search engines you want to be indexed in Google, Bing/Yahoo.
Unless you're a Russian site then you might want to be indexed by Yandex too.
Or perhaps its not even a search engine bot and is up to something much more sinister like looking for vulnerabilities or places to spam or just sites to rip.
And in these cases where you suddenly see lots of bot activity its not too hard to find out what they are or where they're from.
Check your Apache logfiles for starters as that will show everything you need to know including the bots string in which to use to block it with in htaccess or robots file.
By the way, if you're looking for "A Close to perfect .htaccess file" then there's a useful block you can enter here which blocks most known bots and rippers I like to use when this starts happening sometimes.
But don't think just putting them in the htaccess file is all you need to do. Well it is but sometimes you only need to block them for a few weeks or something and eventually they stop trying and go somewhere else haha
You can then remove them from it or keep them in up to you but its good to have a slimline htaccess file.
08-23-2016, 01:08 PM #5
Replies: 0 | Views: 479Last post by Omer Farooq, 04-05-2016, 10:03 AM
Replies: 0 | Views: 566Last post by Hawker, 04-05-2016, 01:44 PM
By Joe B. Ray in forum ServicesReplies: 0 | Views: 449Last post by Joe B. Ray, 04-05-2016, 02:24 PM
By Hawker in forum Link SalesReplies: 2 | Views: 992Last post by yellowmed, 06-03-2016, 08:38 AM
By OWF in forum Buy and Sell WebsitesReplies: 10 | Views: 4283Last post by sumit_sharma, 05-30-2016, 11:28 AM
Replies: 8 | Views: 988Last post by Hawker, 04-08-2016, 01:14 PM
Replies: 8 | Views: 715Last post by SEOPub, 04-04-2016, 01:34 PM
Replies: 4 | Views: 1017Last post by Adam Yunker, 04-15-2016, 05:02 PM
Replies: 4 | Views: 636Last post by ElixantTechnology, 04-06-2016, 12:13 AM
Replies: 7 | Views: 1106Last post by Ron Killian, 05-21-2016, 09:17 AM
Replies: 0 | Views: 9Last post by Sam, Today, 08:11 PM
Replies: 13 | Views: 62Last post by hbailey, Today, 08:59 PM
Replies: 1 | Views: 30Last post by Sam, Today, 08:33 PM
Replies: 9 | Views: 1931Last post by jitendrasurve, 02-03-2016, 01:12 PM
Replies: 1 | Views: 685Last post by seoking, 10-28-2015, 01:20 PM
Replies: 16 | Views: 5608Last post by talukdar, 06-07-2015, 11:05 AM
Replies: 3 | Views: 1740Last post by ericplotz1, 10-20-2014, 04:26 AM
Replies: 3 | Views: 2554Last post by zebrathemes, 03-29-2013, 01:08 PM