Using robots.txt to block URLs with *

[Marc van Leeuwen]

Can you tell me how to block this type of URL using robots.txt

example.com/topic/this-is-web-page-1/reply
example.com/topic/this-is-web-page-2/reply
example.com/topic/this-is-web-page-3/reply
example.com/topic/this-is-web-page-4/reply

I mean it will block URLs with end of "reply"

but still allowing Google spiders to crawl these pages

example.com/topic/this-is-web-page-1/
example.com/topic/this-is-web-page-2/
example.com/topic/this-is-web-page-3/
example.com/topic/this-is-web-page-4/

What is your solution?

 

[Rob Whisonant]

One thing to keep in mind when using wildcards in robots.txt files. Not all crawlers and bots know how to interpret wildcards.

 

[Marc van Leeuwen]

Marc van Leeuwen

I understand because wildcards will only work with Google or a few search engines know it.
I tested and found out the answer for my question above, just putting this code into robots.txt and it worked.

Disallow: /threads/*/reply