{"id":397,"date":"2014-02-05T08:00:54","date_gmt":"2014-02-05T08:00:54","guid":{"rendered":"http:\/\/www.webmastersun.com\/blog\/?p=397"},"modified":"2014-02-10T13:18:35","modified_gmt":"2014-02-10T13:18:35","slug":"what-is-robots-txt-and-how-to-use-it","status":"publish","type":"post","link":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/","title":{"rendered":"What is Robots.txt and How to use it"},"content":{"rendered":"<p><strong><a href=\"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg\"><img loading=\"lazy\" class=\"alignright\" alt=\"robotstxt\" src=\"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg\" width=\"255\" height=\"217\" \/><\/a>Robots.txt<\/strong> is used to instruct search engines to automatically discover your pages and then search the index page. Most sites have folders and files that do not need the search engine robots to visit, so creating a robots.txt file can help you with SEO.<\/p>\n<p><strong>Robots.txt<\/strong> file is a simple file format that can be created with Notepad and is located in the root directory of the website<br \/>\nfor example : <a href=\"https:\/\/www.webmastersun.com\/robots.txt\">https:\/\/www.webmastersun.com\/robots.txt<\/a><\/p>\n<p><strong>User-agent : *<\/strong>\u00a0 means all search robots from Google, Yahoo and MSN should use this guide to find your site.<\/p>\n<p><strong>Disallow : \/your- folder\/<\/strong>\u00a0 This line of code tells the search engines that it should not &#8220;rummage&#8221; in the files of the website beginning with \/your-folder\/.<\/p>\n<p><strong>Instructions on how to create a robots.txt file<\/strong><br \/>\nTo create a robots.txt file open Notepad and type the command, then save it as robots.txt and copy it to the root directory of your website.<\/p>\n<p>Three of the most basic commands for a robots.txt file are as follows:<\/p>\n<pre>User-agent : *\r\nAllow :\/folder-name\/\r\nDisallow :\/folder-name\/<\/pre>\n<p><strong>User-agent<\/strong>: bot used to determine the search engines.<br \/>\n<strong>Allow<\/strong>: Allows bot to discover folders on any page.<br \/>\n<strong>Disallow<\/strong>: blocks the bot from finding folders on any page.<\/p>\n<p><strong>Some things should avoid when using robots.txt<\/strong><\/p>\n<p>Do not use footnotes in the robots.txt file, it can confuse the search engine spiders.<br \/>\nDo not leave any white space at the top of the command line<\/p>\n<pre>User-agent : *\r\nDisallow :\/folder-name\/<\/pre>\n<p>Do not change the order of the command line.<\/p>\n<pre>Disallow :\/foldername\r\nUser-agent : *<\/pre>\n<p>Do not use more than one folder in the Disallow line.= like this:<\/p>\n<pre>User-agent : *\r\nDisallow :\/foldername1\/foldername2\/foldername3\/<\/pre>\n<p>It should instead be written like this:<\/p>\n<pre>User-agent : *\r\nDisallow :\/foldername1\r\nDisallow :\/foldername2\/\r\nDisallow :\/foldername3\/<\/pre>\n<p>I hope this will help you create a robots.txt file suitable for your site!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt is used to instruct search engines to automatically discover your pages and then search the index page. Most sites have folders and files that do not need the search engine robots to visit, so creating a robots.txt file can help you with SEO. Robots.txt file is a simple file format that can be created [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":398,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[52],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v17.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is Robots.txt and How to use it - Webmaster Sun Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is Robots.txt and How to use it - Webmaster Sun Blog\" \/>\n<meta property=\"og:description\" content=\"Robots.txt is used to instruct search engines to automatically discover your pages and then search the index page. Most sites have folders and files that do not need the search engine robots to visit, so creating a robots.txt file can help you with SEO. Robots.txt file is a simple file format that can be created [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/\" \/>\n<meta property=\"og:site_name\" content=\"Webmaster Sun Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-02-05T08:00:54+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-02-10T13:18:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"255\" \/>\n\t<meta property=\"og:image:height\" content=\"217\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Tommy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.webmastersun.com\/blog\/#website\",\"url\":\"https:\/\/www.webmastersun.com\/blog\/\",\"name\":\"Webmaster Sun Blog\",\"description\":\"Blogging Tips to help you make money online\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.webmastersun.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg\",\"contentUrl\":\"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg\",\"width\":255,\"height\":217},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#webpage\",\"url\":\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/\",\"name\":\"What is Robots.txt and How to use it - Webmaster Sun Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.webmastersun.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#primaryimage\"},\"datePublished\":\"2014-02-05T08:00:54+00:00\",\"dateModified\":\"2014-02-10T13:18:35+00:00\",\"author\":{\"@id\":\"https:\/\/www.webmastersun.com\/blog\/#\/schema\/person\/441faf493750eb1507f97d50ea9cd742\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.webmastersun.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is Robots.txt and How to use it\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.webmastersun.com\/blog\/#\/schema\/person\/441faf493750eb1507f97d50ea9cd742\",\"name\":\"Tommy\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.webmastersun.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b17c25a7f242f8dc0328ed5929c512fd?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b17c25a7f242f8dc0328ed5929c512fd?s=96&d=mm&r=g\",\"caption\":\"Tommy\"},\"url\":\"https:\/\/www.webmastersun.com\/blog\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is Robots.txt and How to use it - Webmaster Sun Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/","og_locale":"en_US","og_type":"article","og_title":"What is Robots.txt and How to use it - Webmaster Sun Blog","og_description":"Robots.txt is used to instruct search engines to automatically discover your pages and then search the index page. Most sites have folders and files that do not need the search engine robots to visit, so creating a robots.txt file can help you with SEO. Robots.txt file is a simple file format that can be created [&hellip;]","og_url":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/","og_site_name":"Webmaster Sun Blog","article_published_time":"2014-02-05T08:00:54+00:00","article_modified_time":"2014-02-10T13:18:35+00:00","og_image":[{"width":255,"height":217,"url":"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg","type":"image\/jpeg"}],"twitter_misc":{"Written by":"Tommy","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/www.webmastersun.com\/blog\/#website","url":"https:\/\/www.webmastersun.com\/blog\/","name":"Webmaster Sun Blog","description":"Blogging Tips to help you make money online","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.webmastersun.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#primaryimage","inLanguage":"en-US","url":"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg","contentUrl":"https:\/\/www.webmastersun.com\/blog\/wp-content\/uploads\/2014\/02\/robotstxt.jpg","width":255,"height":217},{"@type":"WebPage","@id":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#webpage","url":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/","name":"What is Robots.txt and How to use it - Webmaster Sun Blog","isPartOf":{"@id":"https:\/\/www.webmastersun.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#primaryimage"},"datePublished":"2014-02-05T08:00:54+00:00","dateModified":"2014-02-10T13:18:35+00:00","author":{"@id":"https:\/\/www.webmastersun.com\/blog\/#\/schema\/person\/441faf493750eb1507f97d50ea9cd742"},"breadcrumb":{"@id":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.webmastersun.com\/blog\/what-is-robots-txt-and-how-to-use-it\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.webmastersun.com\/blog\/"},{"@type":"ListItem","position":2,"name":"What is Robots.txt and How to use it"}]},{"@type":"Person","@id":"https:\/\/www.webmastersun.com\/blog\/#\/schema\/person\/441faf493750eb1507f97d50ea9cd742","name":"Tommy","image":{"@type":"ImageObject","@id":"https:\/\/www.webmastersun.com\/blog\/#personlogo","inLanguage":"en-US","url":"https:\/\/secure.gravatar.com\/avatar\/b17c25a7f242f8dc0328ed5929c512fd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b17c25a7f242f8dc0328ed5929c512fd?s=96&d=mm&r=g","caption":"Tommy"},"url":"https:\/\/www.webmastersun.com\/blog\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/posts\/397"}],"collection":[{"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/comments?post=397"}],"version-history":[{"count":0,"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/posts\/397\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/media\/398"}],"wp:attachment":[{"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/media?parent=397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/categories?post=397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webmastersun.com\/blog\/wp-json\/wp\/v2\/tags?post=397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}