Hello,
We apologize for the inconvenience you’re experiencing with the robots.txt file. The “Robots.txt unreachable” error can occur due to various reasons, such as a misconfigured robots.txt file or server-related issues.
To resolve this issue, you can follow these steps:
-
Ensure that your robots.txt file is accessible at https://yourdomain.com/robots.txt. If it’s available at this URL, you can safely ignore the error.
-
If you use a content delivery network (CDN) like Cloudflare, the error might be triggered due to the way Rank Math handles the robots.txt file. In this case, you can add the following rule to your robots.txt file and test it again in PageSpeed Insights:
Disallow: /cdn-cgi/
-
If you suspect that there might be a conflicting robots.txt file or if the URL returns a 404 error or a blank page, you can refer to this guide on how to fix common robots.txt issues for detailed troubleshooting steps.
If you have any further questions or need additional assistance, please feel free to reach out to our dedicated support team directly from here, and we’re always here to help: Contact Rank Math Support.
We hope this helps!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix the ‘Blocked Due to Other 4xx Issue’ Error » Rank Math
This is an automated reply from Rank Math Support Bot. If you need further assistance, please reply to this topic and a human support agent will reply to you shortly.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
