Hello,
Thanks for contacting us, and sorry for any inconvenience that might have been caused due to that.
Your robots.txt file looks to be set with the correct rules.
Could you please try clearing your site, plugin, or any server-level cache and try again to see if that works for you?
Looking forward to helping you.
I have done so and when I am using the google inspect tool to check the url its still saying its being disallowed, I cant work it out….. any ideas?
Hello,
Thank you for contacting Rank Math support and we apologize for the inconvenience.
We need some more information from you to troubleshoot the problem. Could you please tell us which URL on your website is showing this error? Is it your homepage or some other page?
This error is not caused by Rank Math but by your robots.txt file, which is a file that tells search engines which pages to crawl or not crawl on your website. You can find your robots.txt file by adding /robots.txt after your website domain name (for example, yourwebsite.com/robots.txt).
We noticed that you have a lot of Disallow rules in your robots.txt file, which means you are blocking many pages from being crawled by search engines. This could be the reason why some of your URLs are getting this error. You should check your robots.txt file and make sure that you are not blocking the URL that is showing the error. If you send us the URL that is having the issue, we can also check it for you and see if it is blocked by your robots.txt file.
We hope this helps you understand the issue better. Thank you for choosing Rank Math as your SEO plugin.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
