Robots.txt is used to instruct search engines to automatically discover your pages and then search the index page. Most sites have folders and files that do not need the search engine robots to visit, so creating a robots.txt file can help you with SEO.
Robots.txt file is a simple file...