Hello,
Thank you for contacting Rank Math and bringing your concern to our attention.
This situation usually happens when there’s a problem with the file’s permissions or location on the server.
Please contact your hosting provider for assistance. They should be able to help you resolve the issue and grant the necessary permissions to the robots.txt file so that search engines can crawl your website.
You can also check the file path to ensure that it’s in the correct location and that the file name is written correctly as “robots.txt”.
If an actual robots.txt file is present in the root folder of your site, the option to edit the file from Rank Math > General Settings > Edit robots.txt won’t be available.
Please delete it from the root folder and edit it from the Rank Math Plugin if required.
Learn more about robots.txt here:
https://rankmath.com/kb/general-settings/#edit-robotstxt
We hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.
Thank you.
I had a word with the hosting provider and they told me there is no robots.txt file in the root directory as the file is generated by rank math (which is not a physical file) we can not make any changes from the server side. And they asked me to contact rank math support.
Also I am able to edit Robots.txt file from general setting of rankmatch. Please help me
Hello,
Can you please share a screenshot of your robots.txt file from Rank Math > General Settings > Edit robots.txt?
You may use a tool like imgur to share screenshots with us. That will help us to understand the situation better.
Looking forward to helping you.
Thank you.
Please check the Screen Shot using the link: https://imgur.com/a/Rwj6M2w
Hello,
We might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
Please do take a complete backup of your website before sharing the information with us.
It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:
https://wordpress.org/plugins/temporary-login-without-password/
You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):
https://wordpress.org/plugins/wp-security-audit-log/
Looking forward to helping you.
Thank you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Hey, have you guys checked what’s the problem?
Hello,
I ran the test on various robots.txt tester and everywhere it gives the same result i.e. allowed for Googlebot to crawl. One of the test link.
Would you please confirm if you’re still facing the issue with Google?
Looking forward to helping you.
RankMatch Support Team, I request you to please resolve my problem I am not able to index any new posts on my site.
Hello,
As we can access robots.txt from the browser, so everything is fine in terms of robots.txt file.
Now, when I run through mobile testing of the website, I get the error as you specified. So, normally it means your server is stopping the access of Googlebot.
So, I would ask hosting provider are they blocking any IPs? If they provide server and access log, would you please check if they mentioned any IP which are being blocked. Mostly, there are issues with server configuration of ModSecurity which causes blocking of bot IPs.
So, please contact your hosting provider to help you in this. Rank Math is creating robots.txt file on fly when requested by browser, IP or bot. If server doesn’t responds as expected, we are bound to receive the error.
Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.
Thank you
Okay,
Can you just let me know how to delete or replace robots.txt file generated by RankMath? so that at least Google can index new posts on my site? Google is not crawling the site because robots.txt is present but not accessible. If I delete the file google will crawl my site.
on checking I can see there are several blocked IPs as for security reasons
refer to this https://support-tools.com/?74a33d7b84841389#5ITgxMdrhDRazWtMRKaggAvLkWyNzXaqi23iP4xI9Xs=
How do I know which one to whitelist? No problem if you don’t know this but at lease let me know how to delete or replace robots.txt file created by RankMath
HEY WHY DON’T YOU REPLY ME????
Hello,
We are extremely sorry for the delay in our response but bumping a ticket only pushes it to the bottom of the queue because tickets are answered from oldest to newest. We do not recommend creating new replies when making a follow-up, instead, edit the last post you created to add more details about the concern.
We would love to help you with the issue but it seems you already marked this ticket as “Resolved”.
Do you still need our assistance regarding this issue?
Looking forward to helping you.
Thank you.
