Close Menu
    Facebook X (Twitter) Instagram
    Facebook X (Twitter) Pinterest LinkedIn
    XtendedViewXtendedView
    • Home
    • Technology
      • How to
      • News
      • Computer
      • Windows
    • Internet
      • WordPress
      • Web
      • Google
      • Marketing
      • Social Media
    • Gadgets
      • iOS
      • Android
      • Games
    • About
      • Our Team
    • Contact us
    XtendedViewXtendedView
    Home»Google»How to avoid and remove Adsense Crawler errors and Ads form WordPress post preview page

    How to avoid and remove Adsense Crawler errors and Ads form WordPress post preview page

    Tushar ThakurBy Tushar ThakurUpdated:May 12, 202020 Comments4 Mins Read Google
    Facebook Twitter Pinterest LinkedIn Telegram Tumblr Email
    Share
    Facebook Twitter LinkedIn Pinterest Email

    WordPress and Google Adsense is now the most used combination to make money online. It is very easy to install the Google ads on WordPress powered site and also we can expect a good CTR on the pages. WordPress has a facility to preview the posts before publishing. This creates error of crawling for both Google search engine bot and Adsense page crawling bot.

    google crawler 

     

    What are Adsense Crawler and its errors?

    Before going to discuss any types of error that are Google Adsense crawler generates we first look on what is Adsense crawler?  Adsense crawler is very similar to Google, Yahoo, Bing search engine‘s BOT (Robot). It is a computer algorithm that continuously checks the pages of website on which Google ads are displaying.

     

    What Google says?

    “Our crawler regularly visits all pages on which Google ad code is displayed in order to determine the page content and deliver targeted ads. If our crawler encounters an error and can’t reach your page, we’ll report the error and suggest steps to fix it.”

     

     You quit obviously ask why this BOT continuously visiting our page? The reason behind is that Google Adsense always display relevant ads to our page content. In other word they are known as contextual ads. For displaying such contextual ads on every page of your website Google Adsense needs to determine your page content continuously.

    It is visible under account setting in Google Adsense Publisher dashboard.

    Read that notice in the image that Adsense crawler is different than Google search engine crawler.

     adsense crawler error

     

    What is WordPress Preview post Page?

    preview page problmes

    WordPress is best CMS and blogging platform that have being using by world. When we write any post inside the WordPress, there is an option to preview that post in our website before actually publishing it. The advantage is that we can detect and rectify the errors in the post very easily. When we preview any post see the image above to get idea about it.

     

     

     

    Reasons behinds the generation of crawling errors:

    Usually the crawling errors are generated for the following reasons:

    ·         URL not found

    ·         URL is unreachable due to hosting issues

    ·         URL is blocked by robot.txt

    ·         Some other reasons

    The first one is most common reason behind it. That is page URL not found. When we preview any post inside WordPress it create a url that is temporary and it is valid only till post is unpublished. As soon as when we publish the post that URL gets destroyed. Hence when we publish the post that url is destroyed and crawler error created.

    See image that WordPress URL structure for preview pages:

     

    previw page url wordpress

     

    Method to Avoid the Adsense and Googlebot Crawling errors:

     The simple way to avoid those errors and remove Adsense ads from your WordPress preview post pages is that to restrict Google Adsense BOT from preview pages. This can be done by editing the robots.txt file of the site. The robots.txt tells the BOTs about which pages to crawl and which to not.

     

    Here is the method to add one line inside your robotx.txt file which will disallowed all bots to enter your WordPress preview page.

    Add this line to robots.txt file:

    Disallow: /?p=*</em>

     

    So the total file will be look like this:

     

    User-agent:  *
    Allow: /
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: *?wptheme
    Disallow: /?p=*

     

    To edit your robotx.txt file, first download it to pc by FTP or cPanel and edit it with notepad and upload it again so that original one will be replaced.

    You can see my robots.txt here: https://xtendedview.com/robots.txt

     

     

    Important:

    If you are not familiar with editing and uploading/downloading the robotx.txt file then you can use this free WordPress plugin: PC Robots.txt  which allows you to edit it inside your WordPress dashboard very easily. I will recommend you the manual method because you should keep minimum number of WordPress plugins on your website. 

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Tushar Thakur
    • Website

    Tushar is founder of Xtendedview. He love to experiments on different gadgets, software/apps. He is professional blogger and Internet marketer. He is Interested in electronics and computers, Internet technology, Search Engine Optimization, Internet Marketing. Running online business and Blogs

    Related Posts

    The Ultimate Guide to Google Workspace

    Why Your Business Should Use White Hat SEO Strategies In Their Marketing Plan

    SEO in the Finest Choices Now

    Search Engine Optimization: Fact and Fiction

    4 Tips on How to Optimize your Blog for Google Assistant

    Solutions: Gmail retired the sender’s time zone from Labs – alternatives

    View 20 Comments

    20 Comments

    1. Kuldeep on February 29, 2012 4:25 pm

      I think that this post would really be a great help for all those who are using WP and suffering from this problem.
      Nice one. 🙂

      Reply
      • Tushar on February 29, 2012 4:31 pm

        Thankx for the comment .. 🙂

        Reply
    2. jaime on May 22, 2012 5:57 am

      just what i was looking for. excellent!

      Reply
      • Tushar on May 22, 2012 9:11 am

        Thanx for the visit.

        Reply
    3. Zarko on May 30, 2012 7:34 pm

      I wonder if disallowing anything with ?p=* can cause troubles with those whose WordPress permalinks (post urls) are configured with default “domainname.com/?p=xyz” (where xyz is the post number automatically assigned by WordPress)?

      My permalinks are different nowadays, but they were initially set like that by WordPress so I wonder whether there’s many blogs with this setup that could be affected. In that case wouldn’t this line in robots file block AdSense from virtually all the posts? Would it be more specific and less risky to look for and disallow “preview=true” instead and what would be the syntax for that?

      Reply
      • Tushar on May 30, 2012 9:32 pm

        You are right but that “domainname.com/?p=xyz” but that permalink structure is not recommended for SEO point of view. and about your second point “preview=true” is great idea for it I never tried that anyway I think the format should be like this disallow *preview=true* .

        Anyways thanx for sharing your knowledge here. I will try and tell you the result. 🙂

        Reply
        • Zarko on May 31, 2012 3:07 am

          Agreed on not recommended permalinks.
          Regarding the disallow statement, I already searched Google’s recommended syntax and set up my robots.txt to use:

          Disallow: /*preview=true$

          We’ll see how it works – I even put a post about it on my blog. Thanks for sharing ideas – your post showed me the way to this.

          Reply
    4. upender on October 18, 2012 11:05 am

      thanks for the great info.. i have resolve this error ..which is continuously coming my adsense account

      Reply
    5. Jeff on December 14, 2012 7:14 pm

      Don’t do this. Don’t use robots.txt to block previews or Google will index all of your preview pages with a “this page is blocked in robots.txt” description. Instead just use a conditional tag around your wordpress units.

      Reply
      • Tushar on December 14, 2012 10:43 pm

        No, I am using it since 1 year and it only avoid the adsense to display ads on the WordPress post preview page, not on actual WordPress page.

        Reply
    6. eamcet 2013 on February 19, 2013 8:02 am

      brillant piece of information, I had come to know about your web-page from my friend hardkik, chennai,i have read atleast 9 posts of yours by now, and let me tell you, your webpage gives the best and the most interesting information. This is just the kind of information that i had been looking for, i’m already your rss reader now and i would regularly watch out for the new posts, once again hats off to you! Thanx a million once again, Regards,

      Reply
    7. Rahul thial on March 10, 2013 8:47 pm

      well very helpful post i am dealing with the crawler issue in my ad sense account hopefully this post will show me the way out of it … thank you for sharing this valuable information … great post

      Reply
      • Tushar on March 10, 2013 11:09 pm

        Glad to know that it helped you. 🙂

        Reply
    8. Amal Rafeeq on April 3, 2013 9:49 am

      Hi Tushar,
      Hats off bro. I was fed up with this problem for over a month and this is the best explanation I have got till now. Thanks a lot buddy.

      And I’m having problems from other than preview pages. Like some URL’s which we accidentally type which are not valid. I’m really fed up but I think I can solve the preview shit again.

      Thanks again bro! 🙂

      Reply
    9. Kevin on August 17, 2013 1:41 am

      thanks! Easy to read and understand. Hopefully solves this problem for me – been driving me nuts for months! thanks again.

      Reply
    10. Batman on May 8, 2014 5:47 am

      Why is Google even looking at the preview pages in the first place? You’d think they’d have built something into their algo to avoid this, seeing as how WP is (one of) the largest blog CMS systems around.

      Reply
    11. Eric on October 14, 2014 6:53 pm

      This is great, this finally solve my problem!, thanks

      Reply
    12. Kriti on May 11, 2020 2:23 pm

      Hi Tushar,
      Recently I have faced same issue on my Adsense account. Thank you for providing details and solution for this error, but why you have used [php] [/php] for robots.txt file? I have never seen robots file like this. Thanks

      Reply
      • Tushar Thakur on May 12, 2020 7:54 pm

        Thanks for bringing that into our attention. It was WordPress formatting error. I have corrected that now.

        Reply
    13. hasan on August 24, 2021 1:52 am

      I am struggling with my robots.txt files last few days. now solved. thank you so much for the helpful article.

      Reply
    Leave A Reply Cancel Reply

    Stay In Touch
    • Facebook
    • Twitter
    • Pinterest
    • LinkedIn

    How To Change MAC Address Of Android With or Without Rooting

    How to Fix Mouse Lag in Windows 10

    Top 5 Grammar Checkers (Must Have Tools)

    8 Best SpeedFan Alternatives for Computers for Windows and Mac

    Recent Posts

    How To Change MAC Address Of Android With or Without Rooting

    How to Fix Mouse Lag in Windows 10

    Top 5 Grammar Checkers (Must Have Tools)

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    About

    At Xtendedview, we simplify tech and blogging for everyday users. Our goal is to share real, practical tips, while helping you avoid the mistakes we’ve already made. From gadgets to blogging hacks and money-making strategies, every article is written to actually help. Whether you're just starting out or looking to grow, we’re here to support your journey online.

    Facebook X (Twitter) Pinterest LinkedIn
    Some Rights Reserved. Xtendedview | © 2011 - 2025 | Site Map | Privacy Policy .

    Type above and press Enter to search. Press Esc to cancel.