Thread: Block all bots except Google?
10-08-2017, 02:55 PM #1
Block all bots except Google?
I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
10-08-2017, 05:50 PM #2
I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.
I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
10-08-2017, 08:25 PM #3
You could use a Robots .txt file
10-09-2017, 12:05 AM #4
See, what did I tell you? I knew someone would say that.
No, you cannot reliably block them with robots.txt. Bots can ignore robots.txt files.
10-09-2017, 02:03 AM #5
BIG MISTAKE! The more search engines that crawl your site the more search results your site will appear in. People still use search engines besides google. While ranking high in google is certainly a worthy goal, don't ignore or completely take yourself out of Yahoo search, BING and others!
The Following User Says Thank You to Judas2018 For This Useful Post:
11-13-2017, 01:23 PM #6
I know im late to the party here i had a issue with backlink bots a while back. I was hell bent on blocking them. First off what seo pub says is right. Rbt text means nothing to bot if they dont want to listen to it. Secondly regarding ht access , bot don NEED to follow a proper name, while you may block say XBot with ht access via lets say its user agent string , it can also call itself google bot if it wants to idetify like that , bypassing your ht access.
Now to come back to your situation. I made a white list eventually blocked everything exept the major bots eg googlebot from my website. Then i realised my ranking was going to heck and i was losing a ton of traffic. Why? well because i blocked them there are just too many sources that provide traffic that you dont even know about.... i also later read .. and may seo pub can elaborate on this but google bot and other search engines have say mobile bot ( i forgot the name of googles one) secondly major browsers like google from time to time will crawl your site under a different user agent string ( this is to see what you are up to behind your ht access ). Thirdly google checks bing and vice cersa from time to time to see what you crawling looks like there.
The benefits of such a wide range of blocking imo cannot out weight the cons of what you are facing right now. Perhaps this will help somebody wit the same issue.
11-20-2017, 11:07 AM #7
02-10-2018, 03:25 PM #8
02-10-2018, 03:37 PM #9
02-11-2018, 08:28 AM #10
02-11-2018, 01:14 PM #11
Replies: 11 | Views: 673Last post by premiumproxies, 10-23-2017, 06:58 AM
Replies: 1 | Views: 396Last post by Rob Whisonant, 10-08-2017, 07:45 PM
Replies: 37 | Views: 1479Last post by marblank1313, 12-13-2017, 03:19 PM
Replies: 3 | Views: 225Last post by ulterios, 10-10-2017, 03:15 AM
Replies: 23 | Views: 872Last post by kapil005, 01-06-2018, 07:14 AM
Replies: 0 | Views: 292Last post by Nemanja, 10-08-2017, 02:52 PM
Replies: 0 | Views: 207Last post by Nemanja, 10-08-2017, 02:47 PM
Replies: 17 | Views: 2090Last post by itarachiu, 04-18-2018, 09:33 AM
Replies: 0 | Views: 393Last post by harrygreen90, 10-08-2017, 02:41 PM
Replies: 5 | Views: 273Last post by vishwa, 10-08-2017, 09:03 AM
By prolimehost in forum Dedicated Hosting OffersReplies: 0 | Views: 9Last post by prolimehost, Yesterday, 05:58 PM
Replies: 1 | Views: 35Last post by Rob Whisonant, Yesterday, 07:50 AM
By sunny_pro in forum Web Development - Programming & CodingReplies: 0 | Views: 24Last post by sunny_pro, 05-20-2018, 09:00 PM
Replies: 5 | Views: 73Last post by DarranD, Yesterday, 02:17 PM
By theprb in forum Search Engine OptimizationReplies: 2 | Views: 76Last post by theprb, 05-20-2018, 04:14 PM
Replies: 5 | Views: 1299Last post by cre8ivemedia, 03-14-2017, 02:54 PM
Replies: 6 | Views: 9559Last post by elcidofaguy, 03-27-2016, 06:48 PM
Replies: 2 | Views: 1045Last post by hostslim, 11-19-2015, 11:02 AM
By Shirley_Sharp in forum Search Engine OptimizationReplies: 12 | Views: 2832Last post by Alby, 10-21-2015, 09:02 AM
Replies: 7 | Views: 1911Last post by sunilgupta, 08-08-2015, 06:02 AM