robots.txt is not validLighthouse was unable to download a robots.txt file

#639956
  • Resolved Anonymous

    2 years ago

    Rank Math free

    Hi, i’ve followed your guide at https://rankmath.com/kb/fix-common-robots-txt-issues/ and added Disallow: /cdn-cgi/ in the robot.txt file on rankmath, but the problem still persists. This error is there for a while now, how do i fix it so googlebot can crawl my site smoothly? Thanks

Viewing 1 replies (of 1 total)
  • Miguel

    2 years ago

    Hello,

    Thank you for your query and we are so sorry about the trouble this must have caused.

    As we explained in the other ticket, this issue happens because we create a dynamic file and the Lighthouse tool tries to download a physical file which fails.

    However, that is perfectly acceptable by Google and if you use their tester you’ll see the file is detected: https://support.google.com/webmasters/answer/6062598?hl=en

    You can also see the same by accessing the file directly on your website.

    Don’t hesitate to get in touch if you have any other questions.

    RankBot

    1 year, 11 months ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 1 replies (of 1 total)

The ticket ‘robots.txt is not validLighthouse was unable to download a robots.txt file’ is closed to new replies.