Block folder in robot.txt

[SEOWarrior]

I want to block /page/ in robots.txt file.

My website url is www.mywebsite.com/blog/page/12/.

My current robots.txt file is:
User-agent: *
disallow: /page/

How to block /page/ in robots.txt, so that /page/1/, /page/2/ ect. should not get indexed?

 

[Marc van Leeuwen]

Using this code

User-agent: *
Allow: /
disallow: /page/*

Hope that helped!

 

[SEOWarrior]

SEOWarrior

I used this code & it worked. Thank You!

 

[MyLead]

I want to block /page/ in robots.txt file.

My website url is www.mywebsite.com/blog/page/12/.

My current robots.txt file is:
User-agent: *
disallow: /page/

How to block /page/ in robots.txt, so that /page/1/, /page/2/ ect. should not get indexed?

User-agent: *
disallow: /page/ *

Also if you don't want to index some site, add it on disavow tool or do not index it on google with wordpress plugin

 

[SEOWarrior]

Thanks @Marc & @MyLead will apply this to robots, hope this help.

 

[A_Gramm]

I recommend use meta tags noindex and others, if you want blocking Googlebot.

 

[emililadjet]

emililadjet

The robots.txt file is more powerful than meta tags noindex

 

[A_Gramm]

A_Gramm

I prefer to use Google's recommendations and my experience.

The instructions in robots.txt files cannot enforce crawler behavior to your site; instead, these instructions act as directives to the crawlers accessing your site.
A robotted page can still be indexed if linked to from from other sites
While Google won't crawl or index the content blocked by robots.txt, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google search results. To properly prevent your URL from appearing in Google Search results, you should password-protect the files on your server or use the noindex meta tag or response header (or remove the page entirely).
https://support.google.com/webmaste...1&visit_id=636827838299160568-3600178748&rd=1

How to use metatags https://support.google.com/webmasters/answer/93710

 

[ademar]

I would like to know if I put this line in my robots file:

Code:
Disallow: / test /
this will prevent the engine from reading what is in a folder / test whether it is at the root or not

eg:

http://www.example.com/test/
but also:
http: // www .example.com / whatever / test /