Block all bots except Google?

Dopani

Active member
Joined
Mar 11, 2014
Messages
324
Points
28
I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
 

SEOPub

Well-known member
Joined
Mar 15, 2015
Messages
1,656
Points
83
I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.

I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
 

SEOPub

Well-known member
Joined
Mar 15, 2015
Messages
1,656
Points
83

Judas2018

New member
Joined
Sep 4, 2017
Messages
185
Points
0
BIG MISTAKE! The more search engines that crawl your site the more search results your site will appear in. People still use search engines besides google. While ranking high in google is certainly a worthy goal, don't ignore or completely take yourself out of Yahoo search, BING and others!
 

nesir

Member
Joined
Jul 23, 2017
Messages
34
Points
8
I know im late to the party here i had a issue with backlink bots a while back. I was hell bent on blocking them. First off what seo pub says is right. Rbt text means nothing to bot if they dont want to listen to it. Secondly regarding ht access , bot don NEED to follow a proper name, while you may block say XBot with ht access via lets say its user agent string , it can also call itself google bot if it wants to idetify like that , bypassing your ht access.

Now to come back to your situation. I made a white list eventually blocked everything exept the major bots eg googlebot from my website. Then i realised my ranking was going to heck and i was losing a ton of traffic. Why? well because i blocked them there are just too many sources that provide traffic that you dont even know about.... i also later read .. and may seo pub can elaborate on this but google bot and other search engines have say mobile bot ( i forgot the name of googles one) secondly major browsers like google from time to time will crawl your site under a different user agent string ( this is to see what you are up to behind your ht access ). Thirdly google checks bing and vice cersa from time to time to see what you crawling looks like there.

The benefits of such a wide range of blocking imo cannot out weight the cons of what you are facing right now. Perhaps this will help somebody wit the same issue.
 

archivarix

New member
Joined
Nov 7, 2017
Messages
5
Points
0
I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
I think you want to block not ALL bots, but only bad ones like refspams, mail collectors and other trash. All of them ignore robots.txt so setting disallow is useless. The better way I think is to use plugin for CMS to block all unwanted bots and spammers, you can easily find them for any popular CMS.
 

jackychen

New member
Joined
Aug 9, 2017
Messages
16
Points
0
I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
wordpress: you can go to the settings tick prevent google search
 
Older threads
Replies
0
Views
1,573
Replies
1
Views
5,107
Replies
19
Views
21,535
Replies
5
Views
1,605
Newer threads
Replies
11
Views
3,250
Replies
3
Views
24,872
Replies
3
Views
1,649
Replies
23
Views
5,179
Latest threads
Replies
1
Views
50
Replies
1
Views
50
Replies
1
Views
81
Replies
1
Views
74
Replies
2
Views
107
Recommended threads
Replies
6
Views
3,000
Replies
1
Views
3,822
Replies
1
Views
2,544

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top