Why Bing why?

denvercardonations

New member
Joined
Mar 1, 2016
Messages
117
Points
0
My web-host told me to keep track of bots that keep using up my resources....and it seems Bing constantly has an army of bots constantly crawling my website!! Google crawls my websites less often and gives better results. 30 Bing bots vs 1 Google bot....
bingbots.jpg
 

elcidofaguy

Well-known member
Joined
Jan 13, 2015
Messages
1,281
Points
113
I'm somewhat surprised at that and my initial reaction is that a bot is attempting to masquerade as a bing bot given it is pretty easy to do so with declared user agent type. I would investigate this further with looking into your server logs with getting the IP data. You can check if it is a fake bing bot with use of bings own checker tool here.
 

ulterios

New member
Joined
Nov 25, 2015
Messages
1,267
Points
0
This definitely doesn't sound like it's a true bingbot, not at that crawl rate ratio.

I'd definitely do as elcidofaguy stated and check to make sure it's a true bot from Bing.

I was getting something similar about a year or so ago and it turned out to come from some Russian source.
 

Hawker

Active member
Joined
Dec 22, 2015
Messages
504
Points
43
Do you use any kind of bot control method via htaccess file method or some other method or limit crawler traffic using robots.txt or anything?

You can either use .htaccess or robots.txt to block them or tell them some other message like do one ya nosey git.

The question you have to ask yourself is, should you?

And what are the pros and cons to blocking crawler bots from accessing, crawling and indexing your site?

Can it effect your SERP ranking by blocking bots and can it have an adverse or positive effect on your site?

My answer to the first question is one that's based on experience of managing large sites and that is an explicit yes, you should.

Other than the genuine crawlers you want on your site, all these others do is use up server resources and bloat your Analytics with duff data logs.

So naturally some kind of control is needed. And for that you can use htacess file which can be a VERY POWERFUL beast of a file.

Now obviously you don't want to block every single crawler bot from visiting your site including Google. For obvious reasons.

If Google can't access and crawl your site, it can't index your content and thus you can't rank for content that can't be accessed.

Thusly your rankings would be effected if Google can't even access/crawl and index your site to show to searchers.

It's the same for Bing or any other search engine crawler bot too.

However one can certainly apply some limitation and can indeed block certain bots from accessing/crawling your site and for good reason too.

Especially if that bot is a bot that is for a search engine or country that your website is not even served to.

And let's face it, there are only 2 or 3 search engines you want to be indexed in Google, Bing/Yahoo.

Unless you're a Russian site then you might want to be indexed by Yandex too.

Or perhaps its not even a search engine bot and is up to something much more sinister like looking for vulnerabilities or places to spam or just sites to rip.

And in these cases where you suddenly see lots of bot activity its not too hard to find out what they are or where they're from.

Check your Apache logfiles for starters as that will show everything you need to know including the bots string in which to use to block it with in htaccess or robots file.

By the way, if you're looking for "A Close to perfect .htaccess file" then there's a useful block you can enter here which blocks most known bots and rippers I like to use when this starts happening sometimes.

But don't think just putting them in the htaccess file is all you need to do. Well it is but sometimes you only need to block them for a few weeks or something and eventually they stop trying and go somewhere else haha

You can then remove them from it or keep them in up to you but its good to have a slimline htaccess file.

Good luck!
 

markbatham

Member
Joined
Jul 29, 2016
Messages
53
Points
8
My web-host told me to keep track of bots that keep using up my resources....and it seems Bing constantly has an army of bots constantly crawling my website!! Google crawls my websites less often and gives better results. 30 Bing bots vs 1 Google bot....
This is strange and totally different of Bing Crawler nature, this is definitely by some fake crawler designed same like Bing.
Cross check in bing Webmasters tools. Hope you'll find some way.
 
Older threads
Replies
8
Views
4,643
Replies
8
Views
3,177
Replies
4
Views
3,537
Replies
4
Views
3,183
Replies
7
Views
3,492
Latest threads
Recommended threads
Replies
6
Views
2,300
Replies
6
Views
2,867
Replies
8
Views
4,447
Replies
7
Views
3,930
Similar threads
Replies
2
Views
1,413
Replies
4
Views
2,891
Replies
6
Views
4,398
Replies
2
Views
2,945

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top