robots.txt file not appearing in test

#622416
  • Resolved Anonymous
    Rank Math free

    I’ve input the following into my robots.txt editor in the rankmath plugin settings:
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Sitemap: classicbooksforall.com/sitemap_index.xml

    Before, it said (for some reason):
    User-agent: *
    Disallow: /

    But now when I go to this site to test:
    https://support.google.com/webmasters/answer/6062598?hl=en

    it’s still showing as:
    User-agent: *
    Disallow: /

Viewing 7 replies - 1 through 7 (of 7 total)
  • Hello,

    Thank you for contacting us and sorry for any inconvenience that might have been caused due to that.

    We’ve checked your robots.txt file and we can see that the new rules are appearing there properly.

    In this case, if you’re still facing the issue, please try to clear your website cache including any server-level cache, and test this on your GSC account again.

    Let us know how it goes. Looking forward to helping you.

    Thank you.

    Anonymous
    Rank Math free

    I’ve tried what you suggested, and the same results are coming up.

    Hello,

    We’ve also checked this on another Robots validator and it does seem to be working properly:
    https://prnt.sc/6n6dsWYMHTh_

    Also, please note that the warning you see on my screenshot is due to the sitemap not being the correct URL. I’ve added the correct structure in the Sensitive Data section. This is not really needed but it’s better to make sure.

    Would you kindly share a screenshot with us regarding the results you get whenever you try to validate your Robots through Google’s own tool?

    We really look forward to hearing back from you. Thank you.

    Anonymous
    Rank Math free

    Ok, I updated that sitemap info.

    How do I share a screenshot of the result I’m getting when I do a google search for “classic books for all?”

    It’s basically just coming up with the site url and the words “No information is available for this page. Learn why”

    Hello,

    You can use this tool to upload screenshots and add the generated link here so we can check.

    Looking forward to helping you.

    Thank you.

    Anonymous
    Rank Math free

    https://imgur.com/bFkedXM

    I’m seeing this result when typing in “classic books for all” into a Google search. It’s pretty far down the page, but that’s how it appears.

    Hello again

    I searched your site on Google with the site operator and I can see it is already indexed. Also, your sitemap and robots.txt are working fine. Here’s a screenshot for your reference:

    You can try checking using a tool like https://heymeta.com/ to see if the correct meta description is showing there.

    It is possible that you are seeing different results on Google due to the caching on your browser. Try clearing your browser cache or opening the website from a different device.

    I would also suggest you give Google some time to recrawl the affected page and see if it updates the SERP.

    We hope that helps and we are here should you have any follow-up questions.

    Thank you.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 7 replies - 1 through 7 (of 7 total)

The ticket ‘robots.txt file not appearing in test’ is closed to new replies.