Webmaster Sun Login
Not a member yet? Sign up

Block all bots except Google?

  • HOME
  • FORUMS
  • BLOGS
  • MARKETPLACE
  • ADVERTISING
  • SPECIAL OFFERS
  • WEB HOSTING
  • QUICK MENU
  • REGISTER HERE - Join us for FREE
Results 1 to 11 of 11
    Stick this thread
  1. #1
    Join Date
    Mar 2014
    Posts
    197
    Thumbs Up/Down
    Received: 11/0
    Given: 159/0
    Thanks
    87
    Thanked 18 Times in 16 Posts

    Block all bots except Google?

    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?

  2. #2
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,502
    Thumbs Up/Down
    Received: 393/14
    Given: 64/35
    Thanks
    29
    Thanked 412 Times in 294 Posts
    I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.

    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.

  3. #3
    Join Date
    Aug 2016
    Location
    UK
    Posts
    70
    Thumbs Up/Down
    Received: 37/1
    Given: 15/0
    Thanks
    5
    Thanked 12 Times in 10 Posts

  4. #4
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,502
    Thumbs Up/Down
    Received: 393/14
    Given: 64/35
    Thanks
    29
    Thanked 412 Times in 294 Posts
    Quote Originally Posted by SEOPub View Post
    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
    Quote Originally Posted by Mick M View Post
    You could use a Robots .txt file

    http://www.robotstxt.org/robotstxt.html

    See, what did I tell you? I knew someone would say that.

    No, you cannot reliably block them with robots.txt. Bots can ignore robots.txt files.

  5. #5
    Join Date
    Sep 2017
    Location
    New York
    Posts
    182
    Thumbs Up/Down
    Received: 14/0
    Given: 7/0
    Thanks
    0
    Thanked 4 Times in 4 Posts
    BIG MISTAKE! The more search engines that crawl your site the more search results your site will appear in. People still use search engines besides google. While ranking high in google is certainly a worthy goal, don't ignore or completely take yourself out of Yahoo search, BING and others!

  6. The Following User Says Thank You to Judas2018 For This Useful Post:
    Hometutor (02-13-2018)

  7. #6
    Join Date
    Jul 2017
    Location
    south africa
    Posts
    33
    Thumbs Up/Down
    Received: 5/0
    Given: 10/0
    Thanks
    11
    Thanked 2 Times in 2 Posts
    I know im late to the party here i had a issue with backlink bots a while back. I was hell bent on blocking them. First off what seo pub says is right. Rbt text means nothing to bot if they dont want to listen to it. Secondly regarding ht access , bot don NEED to follow a proper name, while you may block say XBot with ht access via lets say its user agent string , it can also call itself google bot if it wants to idetify like that , bypassing your ht access.

    Now to come back to your situation. I made a white list eventually blocked everything exept the major bots eg googlebot from my website. Then i realised my ranking was going to heck and i was losing a ton of traffic. Why? well because i blocked them there are just too many sources that provide traffic that you dont even know about.... i also later read .. and may seo pub can elaborate on this but google bot and other search engines have say mobile bot ( i forgot the name of googles one) secondly major browsers like google from time to time will crawl your site under a different user agent string ( this is to see what you are up to behind your ht access ). Thirdly google checks bing and vice cersa from time to time to see what you crawling looks like there.

    The benefits of such a wide range of blocking imo cannot out weight the cons of what you are facing right now. Perhaps this will help somebody wit the same issue.

  8. #7
    Join Date
    Nov 2017
    Location
    Prague
    Posts
    5
    Thumbs Up/Down
    Received: 1/0
    Given: 0/0
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Dopani View Post
    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
    I think you want to block not ALL bots, but only bad ones like refspams, mail collectors and other trash. All of them ignore robots.txt so setting disallow is useless. The better way I think is to use plugin for CMS to block all unwanted bots and spammers, you can easily find them for any popular CMS.

  9. #8
    Join Date
    Mar 2014
    Posts
    197
    Thumbs Up/Down
    Received: 11/0
    Given: 159/0
    Thanks
    87
    Thanked 18 Times in 16 Posts
    Quote Originally Posted by SEOPub View Post
    I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.

    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
    Have you read an article about websites could be overloaded by bots when they crawl websites. I remember that could be Google bots and some people had to set the ratio of crawling for their sites from Google webmaster tool.

  10. #9
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,502
    Thumbs Up/Down
    Received: 393/14
    Given: 64/35
    Thanks
    29
    Thanked 412 Times in 294 Posts
    Quote Originally Posted by Dopani View Post
    Have you read an article about websites could be overloaded by bots when they crawl websites. I remember that could be Google bots and some people had to set the ratio of crawling for their sites from Google webmaster tool.
    I'm aware of that possibility, but honestly 99.9999% of internet marketers are never ever going to have to worry about something like that.

  11. #10
    Join Date
    Aug 2017
    Location
    us
    Posts
    14
    Thumbs Up/Down
    Received: 0/1
    Given: 2/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dopani View Post
    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
    wordpress: you can go to the settings tick prevent google search

  12. #11
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,502
    Thumbs Up/Down
    Received: 393/14
    Given: 64/35
    Thanks
    29
    Thanked 412 Times in 294 Posts
    Quote Originally Posted by jackychen View Post
    wordpress: you can go to the settings tick prevent google search
    No. That is not what they are looking to do. They want to do the opposite of that.

Newer Threads

  1. Dopani
    premiumproxies
    Replies: 11 | Views: 673
    Last post by premiumproxies, 10-23-2017, 06:58 AM
  2. Marc van Leeuwen
    cPanel default password?
    By Marc van Leeuwen in forum Web Hosting
    Rob Whisonant
    Replies: 1 | Views: 396
    Last post by Rob Whisonant, 10-08-2017, 07:45 PM
  3. Gizamulake07
    marblank1313
    Replies: 37 | Views: 1479
    Last post by marblank1313, 12-13-2017, 03:19 PM
  4. HazelDJ
    ulterios
    Replies: 3 | Views: 225
    Last post by ulterios, 10-10-2017, 03:15 AM
  5. Gizamulake07
    kapil005
    Replies: 23 | Views: 872
    Last post by kapil005, 01-06-2018, 07:14 AM

Older Threads

  1. Nemanja
    Nemanja
    Replies: 0 | Views: 292
    Last post by Nemanja, 10-08-2017, 02:52 PM
  2. Nemanja
    Nemanja
    Replies: 0 | Views: 207
    Last post by Nemanja, 10-08-2017, 02:47 PM
  3. harrygreen90
    .Info vs .Com?
    By harrygreen90 in forum Domain Names
    itarachiu
    Replies: 17 | Views: 2090
    Last post by itarachiu, 04-18-2018, 09:33 AM
  4. harrygreen90
    harrygreen90
    Replies: 0 | Views: 393
    Last post by harrygreen90, 10-08-2017, 02:41 PM
  5. KamFatz
    vishwa
    Replies: 5 | Views: 273
    Last post by vishwa, 10-08-2017, 09:03 AM

Latest Threads

  1. prolimehost
    prolimehost
    Replies: 0 | Views: 9
    Last post by prolimehost, Yesterday, 05:58 PM
  2. ORLOVA
    Rob Whisonant
    Replies: 1 | Views: 35
    Last post by Rob Whisonant, Yesterday, 07:50 AM
  3. sunny_pro
    sunny_pro
    Replies: 0 | Views: 24
    Last post by sunny_pro, 05-20-2018, 09:00 PM
  4. MSP_Rebecca
    Hi, i'm Rebecca
    By MSP_Rebecca in forum New Member Introductions
    DarranD
    Replies: 5 | Views: 73
    Last post by DarranD, Yesterday, 02:17 PM
  5. theprb
    theprb
    Replies: 2 | Views: 76
    Last post by theprb, 05-20-2018, 04:14 PM

Similar Threads

  1. Harry P
    Do you block spiders?
    By Harry P in forum Web Hosting
    cre8ivemedia
    Replies: 5 | Views: 1299
    Last post by cre8ivemedia, 03-14-2017, 02:54 PM
  2. ElixantTechnology
    elcidofaguy
    Replies: 6 | Views: 9559
    Last post by elcidofaguy, 03-27-2016, 06:48 PM
  3. Tshepo
    hostslim
    Replies: 2 | Views: 1045
    Last post by hostslim, 11-19-2015, 11:02 AM
  4. Shirley_Sharp
    Alby
    Replies: 12 | Views: 2832
    Last post by Alby, 10-21-2015, 09:02 AM
  5. lkovnih226
    Google bots or spiders?
    By lkovnih226 in forum Google
    sunilgupta
    Replies: 7 | Views: 1911
    Last post by sunilgupta, 08-08-2015, 06:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Internet Marketing Forum

Webmaster Sun is a forum where you’ll find in-depth discussions and resources to help you succeed on the web whether you are new or experienced. You’ll find it all here. With topics ranging from internet marketing, search engine optimization, social networking, make money online, web hosting, affiliate marketing as well as hands-on technical support for web design, programming and more. We are a growing community of like-minded people that is keen to help and support each other with ambitions and online endeavors. Learn and grow, make friends and contacts for life.

Come Hang Out With Us

    Facebook Twitter Webmaster Forum Google+

    Webmaster Sun Logo
Copyright ©2012 - 2018, WebmasterSun.com. All rights reserved. Internet marketing forum for internet marketers, webmasters, web hosting providers, designers and affiliate marketers.

Welcome to Webmaster Sun

The World's Number 1 Webmaster Community, SEO and Marketplace

Log in!

Continue with Facebook
Continue With Email. By signing up you indicate that you have read and agree to the Terms of Service and Privacy Policy.

Sign in Manually

Need an account? Sign up now!