Indexing And Robot.txt Issues

Hello,

Thank you for contacting Rank Math and bringing your concern to our attention. I’m sorry for any inconvenience this issue may have caused you.

I’ve checked your site, and your robots.txt is returning a white page.

Also, using the Health Check Plugin, the issue still persists even with Rank Math being only activated.

In this case, please check your website’s root directory for a static robots.txt file since Rank Math’s robots.txt are created virtually/dynamically.

Let us know how this goes.

Thank you.

Hello,

Yes, you should check if there’s a static robots.txt file in your website’s hosting cPanel (file manager).

If there’s a robots.txt file, you may need to delete or at least rename it so Rank Math can generate it on your site.

You can also get in touch with your web host for assistance for checking your website’s root directory.

Let us know how this goes.

Thank you.

Hello,

We’ve tested your Rank Math robots file on the following URL: https://ridersstage.com/?robots=1 and it’s working properly without any issues. It looks like the rewrite rules for the robots file might not be working on your site. Please add the following rewrite rules to your website:

# For Nginx server:

# START Nginx Rewrites for Rank Math Robots.txt File
rewrite ^/robots.txt$ /?robots=1 last;
rewrite ^/([^/]+?)-robots([0-9]+)?.txt$ /?robots=$1&robots_n=$2 last;
# END Nginx Rewrites for Rank Math Robots.txt File

Here’s an article you can check as well: https://ubiq.co/tech-blog/rewrite-url-parameters-nginx/

# For Apache server:

# START of Rank Math Robots.txt Rewrite Rules
RewriteEngine On
RewriteBase /
RewriteRule ^robots.txt$ /index.php?robots=1 [L]
# END of Rank Math Robots.txt Rewrite Rules

If you’re not sure how to apply this rule, you can get in touch with your hosting provider to do that for you.

Let us know how it goes. Looking forward to helping you.

Thank you.

Hello,

Your control panel includes data for a lot of different domains and it’s impossible to be sure what .htaccess we are editing.

Please get in touch with your hosting provider and ask them to add the relevant rules mentioned above and clear the caching after that.

Don’t hesitate to get in touch if you have any other questions.

Hello,

Upon checking your file manager, you have multiple .htaccess.txt files, but they should be saved as .htaccess only.

I have renamed them as backup and created a .htaccess file in your website’s root directory with the rewrite rules.

The next step should be clearing your server and website’s cache for the changes to be applied.

Let us know how this goes.

Thank you.

Hi there,

We appreciate your patience and cooperation. We have tried to fix your issue by updating your .htaccess file, which is a file that controls how your website interacts with the server. However, it seems that our solution did not work as expected. We apologize for the inconvenience.

To resolve this, you will need to contact your hosting provider and ask them for assistance. They should be able to create a rule that tells the server how to handle requests for your robots.txt file, which is a file that tells search engines what they can or cannot access on your website. The rule should look like this:

RewriteEngine On
RewriteBase /
RewriteRule ^robots.txt$ /index.php?robots=1 [L]

Please explain to them that we have already attempted to add this rule, but it did not have any effect. They might have some additional settings or configurations that are preventing the rule from working properly.

We hope that this will help you solve your problem. If you have any other questions or concerns, please feel free to reach out to us again. We are always happy to help.

Thank you for choosing Rank Math as your SEO plugin!