Webmaster Sun Login
Not a member yet? Sign up

Block all bots except Google?

  • HOME
  • FORUMS
  • BLOGS
  • MARKETPLACE
  • ADVERTISING
  • SPECIAL OFFERS
  • WEB HOSTING
  • QUICK MENU
  • REGISTER HERE - Join us for FREE
Results 1 to 7 of 7
    Stick this thread
  1. #1
    Join Date
    Mar 2014
    Posts
    189
    Thumbs Up/Down
    Received: 11/0
    Given: 137/0
    Thanks
    87
    Thanked 17 Times in 15 Posts

    Block all bots except Google?

    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?

  2. #2
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,416
    Thumbs Up/Down
    Received: 372/11
    Given: 58/27
    Thanks
    28
    Thanked 403 Times in 286 Posts
    I wouldn't worry much about it unless there is a bot eating up a ton of bandwidth, but you can do it with the .htaccess file. That is the best way to do it.

    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.

  3. #3
    Join Date
    Aug 2016
    Location
    UK
    Posts
    70
    Thumbs Up/Down
    Received: 36/1
    Given: 15/0
    Thanks
    5
    Thanked 10 Times in 9 Posts

  4. #4
    Join Date
    Mar 2015
    Location
    York, PA
    Posts
    1,416
    Thumbs Up/Down
    Received: 372/11
    Given: 58/27
    Thanks
    28
    Thanked 403 Times in 286 Posts
    Quote Originally Posted by SEOPub View Post
    I'm sure someone is going to say you can do it with your robots.txt file, but that is incorrect. Bots can ignore the robots.txt file, but cannot ignore .htaccess.
    Quote Originally Posted by Mick M View Post
    You could use a Robots .txt file

    http://www.robotstxt.org/robotstxt.html

    See, what did I tell you? I knew someone would say that.

    No, you cannot reliably block them with robots.txt. Bots can ignore robots.txt files.

  5. #5
    Join Date
    Sep 2017
    Location
    New York
    Posts
    151
    Thumbs Up/Down
    Received: 7/0
    Given: 4/0
    Thanks
    0
    Thanked 3 Times in 3 Posts
    BIG MISTAKE! The more search engines that crawl your site the more search results your site will appear in. People still use search engines besides google. While ranking high in google is certainly a worthy goal, don't ignore or completely take yourself out of Yahoo search, BING and others!

  6. #6
    Join Date
    Jul 2017
    Location
    south africa
    Posts
    33
    Thumbs Up/Down
    Received: 4/0
    Given: 10/0
    Thanks
    11
    Thanked 2 Times in 2 Posts
    I know im late to the party here i had a issue with backlink bots a while back. I was hell bent on blocking them. First off what seo pub says is right. Rbt text means nothing to bot if they dont want to listen to it. Secondly regarding ht access , bot don NEED to follow a proper name, while you may block say XBot with ht access via lets say its user agent string , it can also call itself google bot if it wants to idetify like that , bypassing your ht access.

    Now to come back to your situation. I made a white list eventually blocked everything exept the major bots eg googlebot from my website. Then i realised my ranking was going to heck and i was losing a ton of traffic. Why? well because i blocked them there are just too many sources that provide traffic that you dont even know about.... i also later read .. and may seo pub can elaborate on this but google bot and other search engines have say mobile bot ( i forgot the name of googles one) secondly major browsers like google from time to time will crawl your site under a different user agent string ( this is to see what you are up to behind your ht access ). Thirdly google checks bing and vice cersa from time to time to see what you crawling looks like there.

    The benefits of such a wide range of blocking imo cannot out weight the cons of what you are facing right now. Perhaps this will help somebody wit the same issue.
    My spammy lookink Signature Web and graphic design east london

  7. #7
    Join Date
    Nov 2017
    Location
    Prague
    Posts
    5
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Dopani View Post
    I see a list of bots crawling my website but I am looking for the best way to block all bots/crawlers except Google. If yes then how can I do that?
    I think you want to block not ALL bots, but only bad ones like refspams, mail collectors and other trash. All of them ignore robots.txt so setting disallow is useless. The better way I think is to use plugin for CMS to block all unwanted bots and spammers, you can easily find them for any popular CMS.

Newer Threads

  1. Dopani
    premiumproxies
    Replies: 11 | Views: 319
    Last post by premiumproxies, 10-23-2017, 06:58 AM
  2. Marc van Leeuwen
    cPanel default password?
    By Marc van Leeuwen in forum Web Hosting
    Rob Whisonant
    Replies: 1 | Views: 108
    Last post by Rob Whisonant, 10-08-2017, 07:45 PM
  3. Gizamulake07
    kleo
    Replies: 32 | Views: 668
    Last post by kleo, Yesterday, 04:28 AM
  4. HazelDJ
    ulterios
    Replies: 3 | Views: 99
    Last post by ulterios, 10-10-2017, 03:15 AM
  5. Gizamulake07
    HazelDJ
    Replies: 14 | Views: 257
    Last post by HazelDJ, 11-23-2017, 05:06 AM

Older Threads

  1. Nemanja
    Nemanja
    Replies: 0 | Views: 102
    Last post by Nemanja, 10-08-2017, 02:52 PM
  2. Nemanja
    Nemanja
    Replies: 0 | Views: 56
    Last post by Nemanja, 10-08-2017, 02:47 PM
  3. harrygreen90
    .Info vs .Com?
    By harrygreen90 in forum Domain Names
    ecole11j
    Replies: 7 | Views: 392
    Last post by ecole11j, 11-13-2017, 03:37 AM
  4. harrygreen90
    harrygreen90
    Replies: 0 | Views: 149
    Last post by harrygreen90, 10-08-2017, 02:41 PM
  5. KamFatz
    vishwa
    Replies: 5 | Views: 123
    Last post by vishwa, 10-08-2017, 09:03 AM

Latest Threads

  1. HaruXHaru
    Rob Whisonant
    Replies: 2 | Views: 16
    Last post by Rob Whisonant, Yesterday, 07:30 PM
  2. trendjing
    Rob Whisonant
    Replies: 1 | Views: 29
    Last post by Rob Whisonant, Yesterday, 07:32 PM
  3. Castro_Mike
    Rob Whisonant
    Replies: 1 | Views: 37
    Last post by Rob Whisonant, 11-23-2017, 11:48 PM
  4. Hugo E.
    vishwa
    Replies: 3 | Views: 39
    Last post by vishwa, 11-23-2017, 01:09 PM
  5. Hugo E.
    Ciprian Bitter
    Replies: 8 | Views: 103
    Last post by Ciprian Bitter, 11-23-2017, 03:54 PM

Similar Threads

  1. Harry P
    Do you block spiders?
    By Harry P in forum Web Hosting
    cre8ivemedia
    Replies: 5 | Views: 1131
    Last post by cre8ivemedia, 03-14-2017, 02:54 PM
  2. ElixantTechnology
    elcidofaguy
    Replies: 6 | Views: 8719
    Last post by elcidofaguy, 03-27-2016, 06:48 PM
  3. Tshepo
    hostslim
    Replies: 2 | Views: 929
    Last post by hostslim, 11-19-2015, 11:02 AM
  4. Shirley_Sharp
    Alby
    Replies: 12 | Views: 2063
    Last post by Alby, 10-21-2015, 09:02 AM
  5. lkovnih226
    Google bots or spiders?
    By lkovnih226 in forum Google
    sunilgupta
    Replies: 7 | Views: 1700
    Last post by sunilgupta, 08-08-2015, 06:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Internet Marketing Forum

Webmaster Sun is a forum where you’ll find in-depth discussions and resources to help you succeed on the web whether you are new or experienced. You’ll find it all here. With topics ranging from internet marketing, search engine optimization, social networking, make money online, web hosting, affiliate marketing as well as hands-on technical support for web design, programming and more. We are a growing community of like-minded people that is keen to help and support each other with ambitions and online endeavors. Learn and grow, make friends and contacts for life.

Come Hang Out With Us

    Facebook Twitter Webmaster Forum Google+

    Webmaster Sun Logo
Copyright ©2012 - 2017, WebmasterSun.com. All rights reserved. Internet marketing forum for internet marketers, webmasters, web hosting providers, designers and affiliate marketers.

Welcome to Webmaster Sun

The World's Number 1 Webmaster Community, SEO and Marketplace

Log in!

Continue with Facebook
Continue With Email. By signing up you indicate that you have read and agree to the Terms of Service and Privacy Policy.

Sign in Manually

Need an account? Sign up now!