Is your site performing poorly in AdSense? Then these AdSense tips may be of use.
AdSense Tips - Identifying AdSense Crawl Issues
It is time to check your "Site Diagnostics" under the "Reports" tab in your Google AdSense account. It might just be that Google AdSense is not able to gain crawler access to your web pages to be able to index same and display relevant ads. Your URL will thus be showing up as "blocked".
Please get it straight. Ads may currently be displayed but may be irrelevant ads or public service ads which of course will lower your click through rate.
It is important to understand that the crawler bots of Google AdSense is distinct from that of Google.com and AdSense's crawling of your site or otherwise do not affect your Google search ranking. Both these crawlers are also different from webmaster tools crawlers.
However, all three crawlers share a cache. They do this to help publishers conserve bandwidth which would not be the case if all crawlers were requesting the same pages.
AdSense Tips - Possible Causes Of AdSense Crawl Issues
Having determined that AdSense crawlers have no access to your site or specific posts, the next thing is to determine the possible reasons why this could be the case.
1. It could be that this is a temporary issue because your web host's server was down.
2. It could be of a more permanent nature such as crawler access being blocked as a result of incorrect specification in your robots .txt file or outright denial to bots.
Note that neither AdSense bots nor Google.com bots will violate the specifications in your robots .txt file by accessing a denied directory or file. In fact AdSense bots will only crawl content/posts populated with AdSense ads.
As a precaution however, please note that not all bots are this responsible and some will violate your instructions as specified in the robots .txt file and go ahead to crawl disallowed posts/pages. In cases of confidential information therefore, the best approach may be to specify passwords for such posts.
3. Where a post redirects from an original page and Google AdSense does not have access to the original page even though it has access to the page the original content redirects to, this can also be the genesis of the problem, as Google AdSense insists on sighting the redirect in place on the original page.
AdSense Tips - Resolving AdSense Crawl Issues
Identifying the causes of AdSense crawl issues will throw light on the appropriate method to solve these issues.
1. If caused by the server being down, then obviously, the solution is to contact your web host to have this sorted out soonest and if the fault is that of the web host and is recurring, it might just be time to change to a more reliable web host.
2. If the problem is caused by an absent or mis-specified robots .txt file content, then you should update as follows:
If you wish to have AdSense bots but no other bots to crawl your site, then you can insert this code in your robots.txt file
User-agent: Mediapartners-Google
Disallow:
If on the other hand you will want to grant all bots permission to access your site content, then you can add this code to the top of your robots.txt file in the root directory of your domain.
