Webmaster Sun Login
Not a member yet? Sign up

Block all bots except Google?

  • HOME
  • FORUMS
  • BLOGS
  • MARKETPLACE
  • ADVERTISING
  • SPECIAL OFFERS
  • WEB HOSTING
  • QUICK MENU
  • REGISTER HERE - Join us for FREE
Results 1 to 11 of 11
    Stick this thread
  1. #1
    Join Date
    Mar 2014
    Posts
    219
    Thumbs Up/Down
    Received: 15/0
    Given: 175/0
    Thanks
    90
    Thanked 19 Times in 17 Posts

    Block all bots except Google?

    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?

  2. #2
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,535
    Thumbs Up/Down
    Received: 411/14
    Given: 67/36
    Thanks
    29
    Thanked 415 Times in 297 Posts
    I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.

    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.

  3. #3
    Join Date
    Aug 2016
    Location
    UK
    Posts
    70
    Thumbs Up/Down
    Received: 37/1
    Given: 15/0
    Thanks
    5
    Thanked 12 Times in 10 Posts
    You could use a Robots .txt file

    http://www.robotstxt.org/robotstxt.html

  4. #4
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,535
    Thumbs Up/Down
    Received: 411/14
    Given: 67/36
    Thanks
    29
    Thanked 415 Times in 297 Posts
    Quote Originally Posted by SEOPub View Post
    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
    Quote Originally Posted by Mick M View Post
    You could use a Robots .txt file

    http://www.robotstxt.org/robotstxt.html

    See, what did I tell you? I knew someone would say that.

    No, you cannot reliably block them with robots.txt. Bots can ignore robots.txt files.

  5. #5
    Join Date
    Sep 2017
    Location
    New York
    Posts
    184
    Thumbs Up/Down
    Received: 17/0
    Given: 8/0
    Thanks
    0
    Thanked 4 Times in 4 Posts
    BIG MISTAKE! The more search engines that crawl your site the more search results your site will appear in. People still use search engines besides google. While ranking high in google is certainly a worthy goal, don't ignore or completely take yourself out of Yahoo search, BING and others!

  6. The Following User Says Thank You to Judas2018 For This Useful Post:
    Hometutor (02-13-2018)

  7. #6
    Join Date
    Jul 2017
    Location
    south africa
    Posts
    33
    Thumbs Up/Down
    Received: 5/0
    Given: 10/0
    Thanks
    11
    Thanked 2 Times in 2 Posts
    I know im late to the party here i had a issue with backlink bots a while back. I was hell bent on blocking them. First off what seo pub says is right. Rbt text means nothing to bot if they dont want to listen to it. Secondly regarding ht access , bot don NEED to follow a proper name, while you may block say XBot with ht access via lets say its user agent string , it can also call itself google bot if it wants to idetify like that , bypassing your ht access.

    Now to come back to your situation. I made a white list eventually blocked everything exept the major bots eg googlebot from my website. Then i realised my ranking was going to heck and i was losing a ton of traffic. Why? well because i blocked them there are just too many sources that provide traffic that you dont even know about.... i also later read .. and may seo pub can elaborate on this but google bot and other search engines have say mobile bot ( i forgot the name of googles one) secondly major browsers like google from time to time will crawl your site under a different user agent string ( this is to see what you are up to behind your ht access ). Thirdly google checks bing and vice cersa from time to time to see what you crawling looks like there.

    The benefits of such a wide range of blocking imo cannot out weight the cons of what you are facing right now. Perhaps this will help somebody wit the same issue.

  8. #7
    Join Date
    Nov 2017
    Location
    Prague
    Posts
    5
    Thumbs Up/Down
    Received: 2/0
    Given: 0/0
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Dopani View Post
    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
    I think you want to block not ALL bots, but only bad ones like refspams, mail collectors and other trash. All of them ignore robots.txt so setting disallow is useless. The better way I think is to use plugin for CMS to block all unwanted bots and spammers, you can easily find them for any popular CMS.

  9. #8
    Join Date
    Mar 2014
    Posts
    219
    Thumbs Up/Down
    Received: 15/0
    Given: 175/0
    Thanks
    90
    Thanked 19 Times in 17 Posts
    Quote Originally Posted by SEOPub View Post
    I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.

    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
    Have you read an article about websites could be overloaded by bots when they crawl websites. I remember that could be Google bots and some people had to set the ratio of crawling for their sites from Google webmaster tool.

  10. #9
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,535
    Thumbs Up/Down
    Received: 411/14
    Given: 67/36
    Thanks
    29
    Thanked 415 Times in 297 Posts
    Quote Originally Posted by Dopani View Post
    Have you read an article about websites could be overloaded by bots when they crawl websites. I remember that could be Google bots and some people had to set the ratio of crawling for their sites from Google webmaster tool.
    I'm aware of that possibility, but honestly 99.9999% of internet marketers are never ever going to have to worry about something like that.

  11. #10
    Join Date
    Aug 2017
    Location
    us
    Posts
    16
    Thumbs Up/Down
    Received: 0/1
    Given: 3/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dopani View Post
    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
    wordpress: you can go to the settings tick prevent google search

  12. #11
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,535
    Thumbs Up/Down
    Received: 411/14
    Given: 67/36
    Thanks
    29
    Thanked 415 Times in 297 Posts
    Quote Originally Posted by jackychen View Post
    wordpress: you can go to the settings tick prevent google search
    No. That is not what they are looking to do. They want to do the opposite of that.

Newer Threads

  1. Dopani
    premiumproxies
    Replies: 11 | Views: 893
    Last post by premiumproxies, 10-23-2017, 06:58 AM
  2. Marc van Leeuwen
    cPanel default password?
    By Marc van Leeuwen in forum Web Hosting
    dineshsma
    Replies: 3 | Views: 1113
    Last post by dineshsma, 06-24-2018, 03:21 PM
  3. Gizamulake07
    hoainam1102
    Replies: 43 | Views: 2870
    Last post by hoainam1102, 09-17-2018, 07:49 AM
  4. HazelDJ
    ulterios
    Replies: 3 | Views: 329
    Last post by ulterios, 10-10-2017, 03:15 AM
  5. Gizamulake07
    kapil005
    Replies: 23 | Views: 1114
    Last post by kapil005, 01-06-2018, 07:14 AM

Older Threads

  1. Nemanja
    Nemanja
    Replies: 0 | Views: 381
    Last post by Nemanja, 10-08-2017, 02:52 PM
  2. Nemanja
    Nemanja
    Replies: 0 | Views: 285
    Last post by Nemanja, 10-08-2017, 02:47 PM
  3. harrygreen90
    .Info vs .Com?
    By harrygreen90 in forum Domain Names
    Salmaan
    Replies: 18 | Views: 3098
    Last post by Salmaan, 06-23-2018, 02:09 PM
  4. harrygreen90
    harrygreen90
    Replies: 0 | Views: 467
    Last post by harrygreen90, 10-08-2017, 02:41 PM
  5. KamFatz
    vishwa
    Replies: 5 | Views: 353
    Last post by vishwa, 10-08-2017, 09:03 AM

Latest Threads

  1. jhow
    Fredrick
    Replies: 2 | Views: 13
    Last post by Fredrick, Today, 06:00 PM
  2. CyberAlchemist
    Rob Whisonant
    Replies: 2 | Views: 44
    Last post by Rob Whisonant, Yesterday, 06:00 PM
  3. Dopani
    Rob Whisonant
    Replies: 2 | Views: 41
    Last post by Rob Whisonant, Yesterday, 05:56 PM
  4. Nemanja
    Nemanja
    Replies: 0 | Views: 21
    Last post by Nemanja, Yesterday, 03:27 AM
  5. Fredrick
    Fredrick
    Replies: 0 | Views: 27
    Last post by Fredrick, 09-17-2018, 06:41 AM

Similar Threads

  1. Harry P
    Do you block spiders?
    By Harry P in forum Web Hosting
    cre8ivemedia
    Replies: 5 | Views: 1427
    Last post by cre8ivemedia, 03-14-2017, 02:54 PM
  2. ElixantTechnology
    elcidofaguy
    Replies: 6 | Views: 10221
    Last post by elcidofaguy, 03-27-2016, 06:48 PM
  3. Tshepo
    hostslim
    Replies: 2 | Views: 1141
    Last post by hostslim, 11-19-2015, 11:02 AM
  4. Shirley_Sharp
    Alby
    Replies: 12 | Views: 3011
    Last post by Alby, 10-21-2015, 09:02 AM
  5. lkovnih226
    Google bots or spiders?
    By lkovnih226 in forum Google
    sunilgupta
    Replies: 7 | Views: 2111
    Last post by sunilgupta, 08-08-2015, 06:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Internet Marketing Forum

Webmaster Sun is a forum where you’ll find in-depth discussions and resources to help you succeed on the web whether you are new or experienced. You’ll find it all here. With topics ranging from internet marketing, search engine optimization, social networking, make money online, web hosting, affiliate marketing as well as hands-on technical support for web design, programming and more. We are a growing community of like-minded people that is keen to help and support each other with ambitions and online endeavors. Learn and grow, make friends and contacts for life.

Come Hang Out With Us

    Facebook Twitter Webmaster Forum Google+

    Webmaster Sun Logo
Copyright ©2012 - 2018, WebmasterSun.com. All rights reserved. Internet marketing forum for internet marketers, webmasters, web hosting providers, designers and affiliate marketers.

Welcome to Webmaster Sun

The World's Number 1 Webmaster Community, SEO and Marketplace

Log in!

Continue with Facebook
Continue With Email. By signing up you indicate that you have read and agree to the Terms of Service and Privacy Policy.

Sign in Manually

Need an account? Sign up now!