Robots.txt – Add option to disabllow gptbot

Hello,

Thank you for contacting Rank Math support and we apologize for the inconvenience.

If you want to prevent GPTBot from crawling your website, you can do so by editing your robots.txt file. This is a text file that tells web crawlers which pages or files they can or can’t request from your site.

To edit your robots.txt file using Rank Math, you can follow these steps:

• Go to Rank Math > General Settings > Edit robots.txt in your WordPress dashboard.
• Add the following line to the end of the file to block GPTBot from accessing all of your site:

  User-agent: GPTBot
  Disallow: /

• Alternatively, if you only want to block GPTBot from specific pages, you can add this line instead, replacing /page with the URL of the page you want to block:

  User-agent: GPTBot
  Disallow: /page

• Save your changes and check if the robots.txt file is updated.

Learn more about editing your Robots.txt file with Rank Math here: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/

We hope this helps. If you have any other questions or feedback, please let us know. We’re always here to assist you.

Have a great day!

Hello,

Thank you for getting back to us.

If this is the first time you’re using the Rank Math robots.txt editor, it will overwrite your existing robots.txt file. To avoid this, you can copy your current robots.txt file and paste it into Rank Math’s robots.txt editor. Then, you can add the new rule to block GPTBot at the end of the file.

Here are the steps to do this:

1. Go to your website’s robots.txt page by typing your-domain.com/robots.txt in your browser’s address bar. You will see some rules for web crawlers.
2. Select all the text and copy it to your clipboard.
3. Go to Rank Math > General Settings > Edit robots.txt in your WordPress dashboard.
4. Paste the text you copied into the editor.
5. Add the following line at the end of your current rules (before the sitemap line):

   User-agent: GPTBot
   Disallow: /

6. Save your changes and check if your robots.txt file is updated.

After adding the rule, this is what your new robots.txt should look like (if you’re using the default Rank Math rules):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
Disallow: /

Sitemap: https://your-domain.com/sitemap_index.xml

We hope this helps. If you have any other questions or feedback, please let us know. We’re always here to assist you.

Have a great day!