Files robots.txt for WordPress – (War with Google)

– So Allow does not work in Google? Only Disallow?
– There is no such thing as Allow in the robots.txt standard. You can only Disallow things… BUT:

Google accepts some syntax which is not included in The standard and they spell out how to use it. Allow is an acceptable way to have google crawl a few specific files when all of them are disallowed as a default such as Disallow: /*.php. If it didn't work like that they would not be able to retrieve my php sitemaps which they are doing regularly.

This is how you can disallow "all" and allow "some" for Google, it needs to be in this order:
Disallow: /directory/*.asp
Allow: /directory/filename.asp
Allow: /directory/filename-2.php

I looked it up so you can go and read more about it if you like:

https://www.webmasterworld.com/r.cgi?f=93&d=4699148&url=http://support.google.com/webmasters/answer/6062596

Permalink version (Custom – /%category%/%postname%.php):

“Files robots.txt for WordPress – (War with Google)”Continue reading

Scroll to top