-
Google PageSpeed Insights is show up this error:
CRAWLING AND INDEXING
robots.txt is not valid 3 errors found
If your robots.txt file is malformed, crawlers may not be able to understand how you want your website to be crawled or indexed. Learn more about robots.txt.
Line #
Content
Error
1
<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.0 Transitional//EN” “http://www.w3.org/TR/REC-html40/loose.dtd”><html><head><meta http-equiv=”Content-Type” content=”text/html;charset=UTF-8″></head><body><p>User-agent: *
Unknown directive
2
Disallow: /wp-admin/
No user-agent specified
3
Allow: /wp-admin/admin-ajax.php
No user-agent specifiedIn RankMath I see:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://stevebaron.co.nz/sitemap_index.xmlFrom https://stevebaron.co.nz/robots.txt I see:
<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.0 Transitional//EN” “http://www.w3.org/TR/REC-html40/loose.dtd”><html><head><meta http-equiv=”Content-Type” content=”text/html;charset=UTF-8″></head><body><p>User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://stevebaron.co.nz/sitemap_index.xml</p></body></html>I do not know what to do to fix this.
Steve
-
Hello,
Thank you for your query and we are so sorry about the trouble this must have caused.
I see that you have tried editing the robots.txt via our plugin but it failed to display the correct format on the frontend so I went ahead and changed it a bit (like removing an extra line) just to update the robots.txt file and I also went to flush the permalink caches and the robots.txt is now working properly as intended. See screenshot below:
We would still recommend clearing any website caches and before you check it again on Google PageSpeed.
Hope that helps and please do not hesitate to let us know if you need my assistance with anything else.
The ticket ‘Robots.txt error’ is closed to new replies.