WordPress and Google Adsense is now the most used combination to make money online. It is very easy to install the Google ads on WordPress powered site and also we can expect a good CTR on the pages. WordPress has a facility to preview the posts before publishing. This creates error of crawling for both Google search engine bot and Adsense page crawling bot.
What are Adsense Crawler and its errors?
Before going to discuss any types of error that are Google Adsense crawler generates we first look on what is Adsense crawler? Adsense crawler is very similar to Google, Yahoo, Bing search engine‘s BOT (Robot). It is a computer algorithm that continuously checks the pages of website on which Google ads are displaying.
What Google says?
“Our crawler regularly visits all pages on which Google ad code is displayed in order to determine the page content and deliver targeted ads. If our crawler encounters an error and can’t reach your page, we’ll report the error and suggest steps to fix it.”
You quit obviously ask why this BOT continuously visiting our page? The reason behind is that Google Adsense always display relevant ads to our page content. In other word they are known as contextual ads. For displaying such contextual ads on every page of your website Google Adsense needs to determine your page content continuously.
It is visible under account setting in Google Adsense Publisher dashboard.
Read that notice in the image that Adsense crawler is different than Google search engine crawler.
What is WordPress Preview post Page?
WordPress is best CMS and blogging platform that have being using by world. When we write any post inside the WordPress, there is an option to preview that post in our website before actually publishing it. The advantage is that we can detect and rectify the errors in the post very easily. When we preview any post see the image above to get idea about it.
Reasons behinds the generation of crawling errors:
Usually the crawling errors are generated for the following reasons:
· URL not found
· URL is unreachable due to hosting issues
· URL is blocked by robot.txt
· Some other reasons
The first one is most common reason behind it. That is page URL not found. When we preview any post inside WordPress it create a url that is temporary and it is valid only till post is unpublished. As soon as when we publish the post that URL gets destroyed. Hence when we publish the post that url is destroyed and crawler error created.
See image that WordPress URL structure for preview pages:
Method to Avoid the Adsense and Googlebot Crawling errors:
The simple way to avoid those errors and remove Adsense ads from your WordPress preview post pages is that to restrict Google Adsense BOT from preview pages. This can be done by editing the robots.txt file of the site. The robots.txt tells the BOTs about which pages to crawl and which to not.
Here is the method to add one line inside your robotx.txt file which will disallowed all bots to enter your WordPress preview page.
Add this line to robots.txt file:
[php]
Disallow: /?p=*</em>
[/php]
So the total file will be look like this:
[php]
User-agent: *
Allow: /
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: *?wptheme
Disallow: /?p=*
[/php]
To edit your robotx.txt file, first download it to pc by FTP or cPanel and edit it with notepad and upload it again so that original one will be replaced.
You can see my robots.txt here: https://xtendedview.com/robots.txt
Important:
If you are not familiar with editing and uploading/downloading the robotx.txt file then you can use this free WordPress plugin: PC Robots.txt which allows you to edit it inside your WordPress dashboard very easily. I will recommend you the manual method because you should keep minimum number of WordPress plugins on your website.