Robot.txt not accesible

#633567
  • Resolved Anonymous
    Rank Math free

    Hi,

    I get an 404 error when I try to reach my robot.txt. I tried to debug it using your articles but can’t seem to make it work. There is no physical file in the root of the site, DISALLOW_FILE_EDIT is false and also the really simple ssl site hardening thing is turned off. Cache is emptied aswell. Can you find out why my robot.txt isn’t accessible?

    https://www.pro-contact.nl/robot.txt

    Hope to hear from you soon.

    Kind regards,
    Colin Harzing
    Pro Contact

Viewing 6 replies - 1 through 6 (of 6 total)
  • Hello,

    Thank you for contacting us and sorry for any inconvenience that might have been caused due to this issue.

    If you have searched and your root directory does not contain a robots.txt file, then you may be experiencing redirection issues. To solve this, add the applicable rewrite rule to the .htaccess file on your site. You can refer to this guide on how to edit your .htaccess file: https://rankmath.com/kb/edit-htaccess/

    If your site runs on an Nginx server, add the below rewrite rule.

    # START Nginx Rewrites for Rank Math Robots.txt File
    rewrite ^/robots.txt$ /?robots=1 last;
    rewrite ^/([^/]+?)-robots([0-9]+)?.txt$ /?robots=$1&robots_n=$2 last;
    # END Nginx Rewrites for Rank Math Robots.txt File

    If your site runs on an Apache server, add the below rewrite rule.

    
    # START of Rank Math Robots.txt Rewrite Rules
    RewriteEngine On
    RewriteBase /
    RewriteRule ^robots.txt$ /index.php?robots=1 [L]
    # END of Rank Math Robots.txt Rewrite Rules

    Let us know how this goes. Please let us know if you have any further questions or concerns.

    Thank you.

    Anonymous
    Rank Math free

    Hi,

    Thanks for the quick reaction, I already added that piece of code to .htaccess aswell, doesn’t change anything for me.

    Are there any other fixes?

    Hello,

    We are sorry that you are still experiencing this issue.

    To help you resolve this issue, we might need to take a closer look at your website and settings. Please follow these steps:

    • Edit the first post on this ticket and include your WordPress and FTP logins in the Sensitive Data section. This is a secure section that only our support staff can access.

    We appreciate your patience. We are always here to help, please do not hesitate to communicate any concerns or questions you may have.

    Thank you for choosing Rank Math.

    Anonymous
    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Anonymous
    Rank Math free

    I Added the data, if you’re still missing something please let me know!

    Hope you can find the issue.

    Hello,

    Thank you for your patience.

    The reason why it is returning a 404 is because you’re trying to access https://www.pro-contact.nl/robot.txt, where the URL should be https://www.pro-contact.nl/robots.txt.

    You’ll notice that it should be robots with an s.

    Hope that clears up the confusion.

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this ticket.