– So Allow does not work in Google? Only Disallow?
– There is no such thing as Allow in the robots.txt standard. You can only Disallow things… BUT:
Google accepts some syntax which is not included in The standard and they spell out how to use it. Allow is an acceptable way to have google crawl a few specific files when all of them are disallowed as a default such as Disallow: /*.php. If it didn't work like that they would not be able to retrieve my php sitemaps which they are doing regularly. This is how you can disallow "all" and allow "some" for Google, it needs to be in this order: Disallow: /directory/*.asp Allow: /directory/filename.asp Allow: /directory/filename-2.php
I looked it up so you can go and read more about it if you like:
https://www.webmasterworld.com/r.cgi?f=93&d=4699148&url=http://support.google.com/webmasters/answer/6062596
Permalink version (Custom – /%category%/%postname%.php):
User-agent: * Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: */trackback Disallow: */*/trackback Disallow: */*/feed/*/ Disallow: */feed Disallow: /*?* Disallow: /tag User-agent: Yandex Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: */trackback Disallow: */*/trackback Disallow: */*/feed/*/ Disallow: */feed Disallow: /*?* Disallow: /tag Host: upwork.link Sitemap: http://upwork.link/sitemap.xml.gz Sitemap: http://upwork.link/sitemap.xml
Without Permalink:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: */trackback Disallow: */*/trackback Disallow: */*/feed/*/ Disallow: */feed Disallow: /tag User-agent: Yandex Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: */trackback Disallow: */*/trackback Disallow: */*/feed/*/ Disallow: */feed Disallow: /tag Host: upwork.link Sitemap: http://upwork.link/sitemap.xml.gz Sitemap: http://upwork.link/sitemap.xml