Do you block spiders?

Harry P

New member
Joined
Feb 3, 2015
Messages
331
Points
0
Hey guys,
I have seen any features on my web hosting control panel to block spiders from indexing my sites. Of course, I have never really used them. However, more people suggested to block any bots/spiders to increase performance and wonder if I should start blocking some search spiders? what do you think about blocking spiders? What are the advantages and disadvantages of it?
 

EpicGlobalWeb

New member
Joined
Jan 24, 2016
Messages
467
Points
0
You can and should block spiders from specific sections of your site. There is no need for them to index an admin panel for example. There are some fun, hacky ways to mess with spiders like making them "eat" a cookie but the standard way, if you don't want them to index you site you make a robot.txt file and type Disallow * in it.

But I don't think that is a good marketing move. And won't necessarily stop any search engine from scraping your site either. Your site is public so they may do it anyway. This is where you can get more creative and block them by other means.
 

elcidofaguy

Well-known member
Joined
Jan 13, 2015
Messages
1,281
Points
113
Blocking spiders should not have any impact to performance... in addition you'll want web crawlers for search engines to visit in order to get content indexed... The only situation when you want to keep certain spiders out such as Moz, Majestic, Ahrefs etc - is when you are operating PBNs (private networks) as then you hide your backlinks from competitors and with that your ranking strategy...
 

Hawker

Active member
Joined
Dec 22, 2015
Messages
504
Points
43
Blocking spiders should not have any impact to performance...
Yes for small sites / new sites, this shouldn't really be something you have to think about. However, sometimes you can get a chock full of thousands of spiders every second of every day as a big site and that can really reduce and effect performance for your actual visitors. That's about the only real time you'd want to block some spiders from visiting your site is when they are hogging resources and not providing anything in return. Also, some search engine bots are malware and there are known malware bots which are blocked by most good WP plugins like Wordfence if you're a WP user. Otherwise you can just use the robots.txt, htaccess or Meta Robots tag.
in addition you'll want web crawlers for search engines to visit in order to get content indexed...
Yes true that. But as much as we may wish for all the world’s search engines to take notice of our Websites - when they’ve actually managed to crash your system a few times you may be pardoned for having second thoughts. Equally, when they’re hitting your servers with such a task load, your visitors may easily get the impression that viewing your pages is akin to plodding through treacle — impeding sales and your company’s reputation, not to mention the fact that it’s anything but a great user experience.

So what to do about it? Well, how about simply blocking them? After all, as always in business, it’s a question of what kind of a trade-off you’re actually stuck with here.

Not all spiders are created equal, and only your specific online business model should govern your decision to either bear with them accessing your pages regularly or telling them to get lost. After all, bandwidth doesnt come cheap and losing sales due to poor server performance isnt particularly funny either.

Are you targeting the Russian market at all? If not, all that traffic created by Yandex search engine crawlers is something you may very well do without.

How about China? Japan? Korea? Chinese search engines such as Baidu, SoGou and Youdao will merrily spider your sites to oblivion if you let them. In Japan it’s Goo, and in South Korea it’s Naver that can mutate into performance torpedoes once they’ve started to fancy your website.

Nor is that all, because the search engines arent the only culprits in this field.
The only situation when you want to keep certain spiders out such as Moz, Majestic, Ahrefs etc - is when you are operating PBNs (private networks) as then you hide your backlinks from competitors and with that your ranking strategy...
Not the only situation as explained but definitely one of the reasons. That's definitely a smart move. :)

You have to ask yourself. Are you happy with your competition sussing out your entire linking strategy (both incoming and outgoing)? A number of services around will help them do exactly that. Fortunately, at least one major contender, namely Majestic-SEO is perfectly open about things and lets you block their crawlers gracefully. (No such luck with most other setups…)

The other thing to consider is, if those other search engine spiders are visiting your site, and you are getting sales/customers/clicks from that, then do you really want to block that spider from crawling/indexing your site? If it is simply just crawling your site and eating up your logs and not providing anything in return not even clicks then that is when you should probably block them. That is what you need to know. :)

Some Tips on Blocking Spiders To Think About

  • If you’re considering blocking search engine spiders, make sure you’re doing it for the right reasons and not just because you've heard you can.
  • Dont try to use any methods of tricking the spiders such as using agent detection and redirection. Be up front by using the robots.txt file or Meta Robots tag.
  • Dont forget that just because you’re using the recommended methods to block content you’re safe. Understand how blocking content will make your site appear to the bots.
Reasons to block bots:

  • Less bots on your site and more bandwidth/performance/speed for your real visitors.
  • Helps to keep you safe against malware bots which search for vulnerabilities.
  • log size
Reasons to not to block bots:

  • Search engine bots can increase your traffic by indexing your website on more search engines.
 

postcd

Member
Joined
Jul 8, 2012
Messages
34
Points
8
just monitor your access logs and see if there are any excessive visit spiders and you may then gogole it and consider blocking them.
 

cre8ivemedia

New member
Joined
Mar 14, 2017
Messages
3
Points
0
Blocking Googlebot is not a good idea as you might know why but you can certainly block other spiders if you want to hide your website data from indexing from other crawlers/bots like Majestic, SEM Rush etc.
 
Older threads
Newer threads
Replies
9
Views
8,104
Replies
9
Views
3,540
Replies
4
Views
3,876
Replies
13
Views
6,099
Replies
16
Views
7,831
Latest threads
Replies
3
Views
1,608
Replies
2
Views
1,023
Replies
4
Views
1,825
Replies
2
Views
815
Recommended threads

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top