Block folder in robot.txt

SEOWarrior

New member
Joined
Jan 19, 2018
Messages
5
Points
0
I want to block /page/ in robots.txt file.

My website url is www.mywebsite.com/blog/page/12/.

My current robots.txt file is:
User-agent: *
disallow: /page/

How to block /page/ in robots.txt, so that /page/1/, /page/2/ ect. should not get indexed?
 

MyLead

New member
Joined
Jul 27, 2018
Messages
26
Points
0
I want to block /page/ in robots.txt file.

My website url is www.mywebsite.com/blog/page/12/.

My current robots.txt file is:
User-agent: *
disallow: /page/

How to block /page/ in robots.txt, so that /page/1/, /page/2/ ect. should not get indexed?
User-agent: *
disallow: /page/ *

Also if you don't want to index some site, add it on disavow tool or do not index it on google with wordpress plugin :)
 

A_Gramm

New member
Joined
Jan 8, 2019
Messages
7
Points
0
I recommend use meta tags noindex and others, if you want blocking Googlebot.
 
  • Like
Reactions: SEOWarrior

emililadjet

New member
Joined
Dec 1, 2018
Messages
23
Points
0
emililadjet
The robots.txt file is more powerful than meta tags noindex
 

A_Gramm

New member
Joined
Jan 8, 2019
Messages
7
Points
0
A_Gramm
I prefer to use Google's recommendations and my experience.

The instructions in robots.txt files cannot enforce crawler behavior to your site; instead, these instructions act as directives to the crawlers accessing your site.
A robotted page can still be indexed if linked to from from other sites
While Google won't crawl or index the content blocked by robots.txt, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google search results. To properly prevent your URL from appearing in Google Search results, you should password-protect the files on your server or use the noindex meta tag or response header (or remove the page entirely).
https://support.google.com/webmaste...1&visit_id=636827838299160568-3600178748&rd=1

How to use metatags https://support.google.com/webmasters/answer/93710
 

ademar

New member
Joined
Jul 7, 2017
Messages
1
Points
0
I would like to know if I put this line in my robots file:

Code:
Disallow: / test /
this will prevent the engine from reading what is in a folder / test whether it is at the root or not

eg:

http://www.example.com/test/
but also:
http: // www .example.com / whatever / test /
 
Recommended threads
Replies
5
Views
2,314
Replies
3
Views
3,943
Replies
3
Views
2,604
Replies
1
Views
2,049

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top