Hello,
We apologize for the inconvenience you are facing. It seems that your website is experiencing issues with indexing and crawling. We understand that your robots.txt file does not disallow any bot and your sitemap settings are proper.
To address the issue, we recommend checking the status of the URLs using the HTTP Status Tester. This tool will display whether the URLs are being blocked by robots.txt or if they are allowed to be crawled. You can access the HTTP Status Tester by following this link: HTTP Status Tester.
Additionally, since your URLs are permanently redirected to https://rankmath.com/wp-content/uploads/2023/01/edit-robots-txt-1.jpg it’s important to ensure that the final URL is not being blocked by robots.txt even if it allows Google to crawl the other URLs that redirect to it. You can refer to the following guide to check and modify your robots.txt file: How to Edit Robots.txt with Rank Math.
If you have further questions or need assistance, please feel free to reach out to our dedicated support team. We are available 24/7, 365 days a year.
We hope this information helps resolve the issue you are experiencing.
Sources:
Discovered – Currently Not Indexed in Google Search Console » Rank Math
No Information is Available for This Page » Rank Math
How to Fix Submitted URL Marked ‘NoIndex’ Error » Rank Math
Common robots.txt Issues & How to Fix Them » Rank Math
Hi! HTTP Status Tester returns 200 status code for all the http and non http, www and non www URLs
Request URL
Status codes
Redirects
http://slashinsider.com
301
200
1
https://slashinsider.com
301
200
1
http://www.slashinsider.com
301
200
1
https://www.slashinsider.com
200
Also, the content of my Robots.txt file is as follows.
User-Agent: *
Crawl-Delay: 20
Disallow: /wp-content/plugins/
Disallow: *?s=*
Disallow: /cgi-bin/
Disallow: /recommend/
Disallow: /visit/
Disallow: /go/
Disallow: /wp-admin/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: /readme.html
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.slashinsider.com/sitemap_index.xml
Sitemap: https://www.slashinsider.com/post-sitemap.xml
Sitemap: https://www.slashinsider.com/page-sitemap.xml
Sitemap: https://www.slashinsider.com/news-sitemap.xml
And all the posts and pages are set to Index, whether its the setting while editing or writing the post, or in the Post Robots Meta
Hello,
Apologies for the inconvenience.
5xx errors are usually caused due to server misconfiguration or other server issues.
Can you please share the website error log with us? Having the error log file will give a clear idea of why this might be happening upon submitting your URLs.
Looking forward to helping you.
Hi Reinelle,
I resolved the issue. I updated the json and it resolved my issue.
However, still google is unable to crawl my website and it is not getting indexed instantly.
Hello,
Please note that the Instant Index API will only submit the URLs to the search engine to increase the chance for faster indexing. It will not guarantee actual indexing on the SERP.
Please share the affected page/post URL here that currently isn’t indexed on the SERP so we can take a closer look at them.
Also, you may submit the affected URL in the URL inspection tool of Google Search Console and share with us the report shown there.
Looking forward to helping you.
Hi! looks like all the links are not indexed on SERP.
Also, there is an error in the robots.txt tester https://search.google.com/search-console/settings/robots-txt?resource_id=sc-domain%3Aslashinsider.com
Below is my robots file. (Please note I have already used the deault robots file generated by Rank math)
So in the below robots file, the google’s robots.txt tester gives an error. The error is “Error – Invalid sitemap URL detected; syntax not understood (line 22)”
Line 22 being this one “Sitemap: https://www.slashinsider.com/sitemap_index.xml”
User-agent: *
Disallow: /wp-admin/ #block access to admin section
Disallow: /wp-login.php #block access to admin section
Disallow: /search/ #block access to internal search result pages
Disallow: *?s=* #block access to internal search result pages
Disallow: *?p=* #block access to pages for which permalinks fails
Disallow: *&p=* #block access to pages for which permalinks fails
Disallow: *&preview=* #block access to preview pages
Disallow: /tag/ #block access to tag pages
Disallow: /author/ #block access to author pages
Disallow: /404-error/ #block access to 404 page
Disallow: /wp-content/plugins/
Disallow: /recommend/
Disallow: /visit/
Disallow: /go/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: /readme.html
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.slashinsider.com/sitemap_index.xml
Hello,
We can’t see the error you shared with the robots.txt because that only shows for people with access to the Google Search Console for that property.
Can you please share a screenshot of that error so we can advise further?
We would also like to see the results of one of the pages that is not indexed passed through the URL Inspection Tool, so if you could also share a screenshot of that it would be helpful to pinpoint the issue.
Looking forward to hearing back from you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
