Webmaster Sun Login
Not a member yet? Sign up

Robots.txt for your website ?

  • HOME
  • FORUMS
  • WHAT'S NEW
  • BLOGS
  • MARKETPLACE
  • ADVERTISING
  • SPECIAL OFFERS
  • WEB HOSTING
  • QUICK MENU
  • REGISTER HERE - Join us for FREE
Page 1 of 2 12 LastLast
Results 1 to 25 of 28
    Stick this thread
  1. #1
    Join Date
    Jul 2012
    Posts
    389
    Thumbs Up/Down
    Received: 4/0
    Given: 56/0
    Thanks
    95
    Thanked 23 Times in 22 Posts

    Robots.txt for your website ?

    Hi webmasters,

    As I knew if we want Google crawl all our website then use this code in robots.txt file putted in website root directory
    Code:
    User-agent: *
    Allow: /
    but I checked some sites and they didn't put robots.txt file for their websites. Thus I want to know that it's necessary to have a robots file for my website or not ? and why ?

    please share your suggestions ?

  2. #2
    Join Date
    Jan 2013
    Location
    USA
    Posts
    192
    Thumbs Up/Down
    Received: 0/0
    Given: 8/0
    Thanks
    28
    Thanked 20 Times in 12 Posts
    Thanks for sharing this code, but initially I want to know why Robot.txt file should be their in our website?And why it's important in terms of SEO strategy?

  3. The Following User Says Thank You to Joseph_Hill For This Useful Post:
    William (09-23-2013)

  4. #3
    Join Date
    Aug 2013
    Posts
    7
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    The robots.txt file is a very essential file if you wanna increase your site ranking.

    Robots.txt file

    When a search engine crawler comes to your website, it will look for a special root file on your website. That file is called robots.txt and it tells the search engine spider, which Web pages of your website should be indexed and which Web pages should be ignored.

    Like : http://www.yourwebsite.com/robots.txt

  5. #4
    Join Date
    May 2013
    Posts
    101
    Thumbs Up/Down
    Received: 1/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    yes its necessary to place Robot.txt file because it get rid of all the 404 errors simply create an empty file called robots.txt and upload that to your site.

    You should be able to tell if it works by examining site stats for periods before and after uploading as the number of 404 errors on the site should be dramatically reduced.

  6. #5
    Join Date
    Aug 2013
    Posts
    84
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    2
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by aishasony View Post
    yes its necessary to place Robot.txt file because it get rid of all the 404 errors simply create an empty file called robots.txt and upload that to your site.

    You should be able to tell if it works by examining site stats for periods before and after uploading as the number of 404 errors on the site should be dramatically reduced.
    Really a good information. But I would like to know more about Robot.txt that how it actually helps in indexing.

  7. #6
    Join Date
    Mar 2013
    Location
    Delhi
    Posts
    79
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    2
    Thanked 1 Time in 1 Post
    if you are confused that some website not using robot.txt then i would like to describe that robot file is only use for preventing any webpage from search engine. not to crawl webpage.

  8. #7
    Join Date
    Aug 2013
    Posts
    47
    Thumbs Up/Down
    Received: 1/0
    Given: 0/0
    Thanks
    0
    Thanked 1 Time in 1 Post
    Robots.txt is an important tool to identify crawl all websites, faster to index them on all top search engines.

  9. #8
    Join Date
    Aug 2013
    Posts
    27
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by PankajS View Post
    if you are confused that some website not using robot.txt then I would like to describe that robot file is only used for preventing any web page from a search engine. not to crawl webpage.
    Really a good information. I was unaware of that. Can you please tell some more about robot.txt?

  10. #9
    Join Date
    Aug 2013
    Location
    United States
    Posts
    67
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    6
    Thanked 2 Times in 2 Posts
    Mr.Jordan , you can read about robot.txt in details and for that you just have to Google it.
    Name:  2013-09-20_1706.png
Views: 60
Size:  28.6 KB

  11. #10
    Join Date
    Aug 2013
    Location
    AU
    Posts
    111
    Thumbs Up/Down
    Received: 1/0
    Given: 14/0
    Thanks
    23
    Thanked 13 Times in 10 Posts
    Hi,
    I think robots.txt is most important for Search Engine.
    When a search engine crawler comes to your site, it will look for a specific file on your site. That is called robots.txt. It's a simple text file.
    It tells the search engine spider that which web pages should be ignored by Search Engine and which should be indexed.

  12. #11
    Join Date
    Jul 2013
    Posts
    32
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    3
    Thanked 2 Times in 2 Posts
    Quote Originally Posted by Joseph_Hill View Post
    Thanks for sharing this code, but initially I want to know why Robot.txt file should be their in our website?And why it's important in terms of SEO strategy?
    Its a very good question, I have same doubt and still searching for perfect answer.

  13. #12
    Join Date
    Mar 2013
    Location
    India
    Posts
    3
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    disallow term only prevent from crawling not from indexing. for prevent from indexing, you have to type noindex.

  14. #13
    Join Date
    Jan 2013
    Location
    Tustin, CA
    Posts
    47
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 1 Time in 1 Post
    Robots.txt is not always necessary, you can choose not to use them on your websites depending on the content of your website. You can use them occasionally for some websites. Some SEOs not happening practical knowledge may suggest you that it is important because they have read like this online.

  15. #14
    Join Date
    Sep 2013
    Location
    India
    Posts
    37
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by aishasony View Post
    yes its necessary to place Robot.txt file because it get rid of all the 404 errors simply create an empty file called robots.txt and upload that to your site.

    You should be able to tell if it works by examining site stats for periods before and after uploading as the number of 404 errors on the site should be dramatically reduced.
    Thanks But can you tell me in detail why we need robots.txt ??

  16. #15
    Join Date
    Aug 2013
    Location
    In everyones Heart
    Posts
    346
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    7
    Thanked 87 Times in 59 Posts
    Through the robot.txt file Google bot/spiders/crawlers know what is in site. Crawlers get familiar with site.

  17. #16
    Join Date
    Aug 2013
    Location
    In everyones Heart
    Posts
    346
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    7
    Thanked 87 Times in 59 Posts
    Quote Originally Posted by guma View Post
    Thanks But can you tell me in detail why we need robots.txt ??
    Robots.txt was originally conceived as a way for Webmasters to prevent robots from consuming excess bandwidth, and to keep them from executing cgi scripts. However, now that the Web has gone commercial, there are many other good reasons to Disallow spiders from fetching various URLs.

  18. #17
    Join Date
    Aug 2012
    Posts
    327
    Thumbs Up/Down
    Received: 12/0
    Given: 60/0
    Thanks
    28
    Thanked 44 Times in 38 Posts
    It's necessary for you to create robots.txt when you need to protect some files or folders on your server from crawling by Google.

    I found one good article on How to use Robots.txt on our webmaster sun.

    Hope this helps!

  19. #18
    Join Date
    Dec 2013
    Location
    Gurgaon
    Posts
    114
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Main purpose of uploading robots.txt file is that if you want to block any webpage, folder, file in the Google's eye then by the use of robots.txt you can do this.

  20. #19
    Join Date
    Jan 2013
    Posts
    50
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    18
    Thanked 0 Times in 0 Posts
    robots.txt file in your website. The robots.txt file which presents in webmaster's will be updated automatically... No need to do any changes in webmaster's robots.txt.

  21. #20
    Join Date
    Jan 2013
    Location
    Tustin, CA
    Posts
    47
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 1 Time in 1 Post
    It is must for eCommerce business websites so as to hide their financial details.

  22. #21
    Join Date
    Mar 2014
    Location
    Auraiya, India
    Posts
    43
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 11 Times in 9 Posts
    Robots.txt is the main file for Search Engine caching your site to rank them. It is set of some instructions which admin want to Allow or disallow to Search Engine.

  23. #22
    Join Date
    Mar 2014
    Posts
    62
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    8
    Thanked 1 Time in 1 Post
    Robots.txt is most important for search engine spider to tell the search engine crawl the web pages.

  24. #23
    Join Date
    Apr 2014
    Location
    Connecticut, USA
    Posts
    7
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Robots file use in many case to disallow confidential information of your website.

  25. #24
    Join Date
    Apr 2014
    Location
    Kolkata, India
    Posts
    7
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    There is no way that can prevent search engines or say Google to index a website and/or pages. Google will surely fetch your webpage information in any case whether you put or do not put robot.txt in your URL. Search engines will get your site information from the back links that are distributed throughout the web.

  26. #25
    Join Date
    Apr 2014
    Location
    websiteeonline
    Posts
    18
    Thumbs Up/Down
    Received: 0/0
    Given: 0/0
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Robots.txt is an important tool to identify crawl all websites, faster to index them on all top search engines.

Newer Threads

  1. BleepingDesign
    Developer
    Replies: 4 | Views: 4244
    Last post by Developer, 03-01-2016, 05:10 AM
  2. Jack London
    Marc van Leeuwen
    Replies: 77 | Views: 20056
    Last post by Marc van Leeuwen, 06-03-2019, 03:53 PM
  3. Peter
    JuanRoth
    Replies: 12 | Views: 5244
    Last post by JuanRoth, 12-02-2013, 12:21 PM
  4. seizarkaelan
    Hi Friends
    By seizarkaelan in forum New Member Introductions
    azhosty
    Replies: 4 | Views: 2519
    Last post by azhosty, 01-10-2014, 12:01 AM
  5. Dmoz
    Dmoz
    Replies: 0 | Views: 2161
    Last post by Dmoz, 09-16-2013, 02:56 AM

Older Threads

  1. Jack London
    dether
    Replies: 46 | Views: 14440
    Last post by dether, 02-20-2014, 08:59 AM
  2. Britto15
    Selvass
    Replies: 12 | Views: 12060
    Last post by Selvass, 11-08-2013, 05:43 AM
  3. Alish shrestha
    How to earn money from SEO?
    By Alish shrestha in forum Make Money Online
    nacyreyes
    Replies: 14 | Views: 5103
    Last post by nacyreyes, 12-04-2018, 06:14 AM
  4. Alish shrestha
    bob
    Replies: 10 | Views: 3561
    Last post by bob, 06-07-2014, 01:39 AM
  5. Alish shrestha
    Facebook Name
    By Alish shrestha in forum Facebook
    jayjacobs
    Replies: 9 | Views: 4332
    Last post by jayjacobs, 07-04-2014, 03:40 PM

Latest Threads

  1. hostnetindia
    hostnetindia
    Replies: 0 | Views: 36
    Last post by hostnetindia, Today, 05:46 AM
  2. moaaz
    Rob Whisonant
    Replies: 3 | Views: 30
    Last post by Rob Whisonant, Today, 06:41 AM
  3. ClicksCrowd
    ClicksCrowd
    Replies: 0 | Views: 39
    Last post by ClicksCrowd, Yesterday, 08:06 PM
  4. moaaz
    Rob Whisonant
    Replies: 2 | Views: 56
    Last post by Rob Whisonant, Yesterday, 05:56 PM
  5. andyweaver
    New here
    By andyweaver in forum New Member Introductions
    Rob Whisonant
    Replies: 2 | Views: 54
    Last post by Rob Whisonant, Yesterday, 05:57 PM

Similar Threads

  1. Mark Levin
    TejasWebHosting
    Replies: 3 | Views: 3437
    Last post by TejasWebHosting, 12-07-2014, 02:16 PM
  2. Devid Kenny
    WHUK
    Replies: 2 | Views: 2822
    Last post by WHUK, 04-29-2014, 12:06 PM
  3. Stephan
    Jack London
    Replies: 1 | Views: 3311
    Last post by Jack London, 12-24-2013, 07:05 AM
  4. marshal23
    zebrathemes
    Replies: 2 | Views: 1746
    Last post by zebrathemes, 03-29-2013, 01:06 PM
  5. Alex_smith
    Robots tag
    By Alex_smith in forum Search Engine Optimization
    PankajS
    Replies: 8 | Views: 3470
    Last post by PankajS, 03-07-2013, 12:56 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Internet Marketing Forum

Webmaster Sun is a forum where you’ll find in-depth discussions and resources to help you succeed on the web whether you are new or experienced. You’ll find it all here. With topics ranging from internet marketing, search engine optimization, social networking, make money online, web hosting, affiliate marketing as well as hands-on technical support for web design, programming and more. We are a growing community of like-minded people that is keen to help and support each other with ambitions and online endeavors. Learn and grow, make friends and contacts for life.

Come Hang Out With Us

    Facebook Twitter Webmaster Forum Google+

    Webmaster Sun Logo
Copyright ©2012 - 2019, WebmasterSun.com. All rights reserved. Internet marketing forum for internet marketers, webmasters, web hosting providers, designers and affiliate marketers.

Welcome to Webmaster Sun

The World's Number 1 Webmaster Community, SEO and Marketplace

Log in!

Continue with Facebook
Continue With Email. By signing up you indicate that you have read and agree to the Terms of Service and Privacy Policy.

Sign in Manually

Need an account? Sign up now!