UPWORK/ODESK guru: Tips and Tricks for clients

skype: odesk.by / upwork.link , email: info@odesk.by/upwork.link

Files robots.txt for WordPress – (War with Google)

without comments

– So Allow does not work in Google? Only Disallow?
– There is no such thing as Allow in the robots.txt standard. You can only Disallow things… BUT:

Google accepts some syntax which is not included in The standard and they spell out how to use it. Allow is an acceptable way to have google crawl a few specific files when all of them are disallowed as a default such as Disallow: /*.php. If it didn't work like that they would not be able to retrieve my php sitemaps which they are doing regularly.

This is how you can disallow "all" and allow "some" for Google, it needs to be in this order:
Disallow: /directory/*.asp
Allow: /directory/filename.asp
Allow: /directory/filename-2.php

I looked it up so you can go and read more about it if you like:

https://www.webmasterworld.com/r.cgi?f=93&d=4699148&url=http://support.google.com/webmasters/answer/6062596

Permalink version (Custom – /%category%/%postname%.php):

User-agent: *
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Disallow: /tag
 
User-agent: Yandex
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Disallow: /tag
Host: upwork.link       
 
Sitemap: http://upwork.link/sitemap.xml.gz
Sitemap: http://upwork.link/sitemap.xml

Without Permalink:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /tag

User-agent: Yandex
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /tag
Host: upwork.link 
Sitemap: http://upwork.link/sitemap.xml.gz
Sitemap: http://upwork.link/sitemap.xml

Written by spectre

July 8th, 2015 at 3:21 pm

Posted in Tips and Tricks,WP