Blocked By Robots

Hello,

Thank you for contacting Rank Math support. We apologize for any inconvenience this might have caused.

If you have specified certain pages to be blocked in your robots.txt file and Google Search Console is reporting them as “Blocked by robots,” it typically means that Googlebot is not able to crawl those pages. This, in itself, is not a direct SEO issue, especially if those pages are not meant to be indexed.

The fact that Googlebot can’t crawl them won’t directly harm your SEO since the pages are not set to be indexed.

This warning in Google Search Console is more of an informational message. It’s letting you know that Googlebot can’t access those pages, and it’s not necessarily a problem if those pages aren’t supposed to be indexed.

Hope this helps and please do not hesitate to let us know if you need assistance with anything else.

Thank you.

Hello,

Are you able to access the Rank Math metabox on those pages to control the robots meta settings and set them to noindex following this guide? https://rankmath.com/kb/how-to-noindex-urls/

If you are not able to do that, you might need to apply the noindex dynamically by editing the following filter: https://rankmath.com/kb/filters-hooks-api-developer/#change-robots-meta

Don’t hesitate to get in touch if you have any other questions.

Hello,

Sure that’s totally fine.

But we should note that the robots.txt DISALLOW rule would only prevent Google from crawling the page. If Google has already indexed the page – which it has – this rule will not remove the page from Google, you will have to use the Google Removal tool to remove the URL from Google’s index.

Then you can apply the noindex tag dynamically to make sure the page is no longer indexed in the future. Here’s an example code to use the filter for multiple URLs at a time:

add_filter( 'rank_math/frontend/robots', function( $robots ) {
	// Get the current page URL
	$current_url = home_url( $_SERVER['REQUEST_URI'] );
	
	// Define an array of pages to change the robots tag to noindex
	$pages = array(
		'https://example.com/page1',
		'https://example.com/page2'
	);
	
	// Check if the current page is in the array
	if ( in_array( $current_url, $pages ) ) {
		// Change the robots tag to noindex
		$robots['index'] = 'noindex';
	}
	
	// Return the modified robots array
	return $robots;
});

Remember to change the example.com URLs to the URLs you want to dynamically set to noindex. You can also add more URLs. You can add the code to your site using any of the methods here.

We hope this helps you resolve the issue. If you have any other questions or concerns regarding Rank Math, don’t hesitate to get in touch with us again. We are always happy to help.

Thank you for choosing Rank Math!

Hello,

Sure, let us know how this goes.

Meanwhile, if you have any other concerns, please don’t hesitate to contact us anytime to assist you further.

Looking forward to helping you.

Thank you.

Hello,

This ticket will be open, so you can update us here.

However, if it has been closed by the bot, please don’t hesitate to create a new ticket and reference this ticket here.

Looking forward to helping you.

Thank you

Hello,

We’re delighted to hear that this issue has been resolved. We appreciate your kind words and feedback.

I’m closing this ticket now but if you ever have another question or need any help in the future, please don’t hesitate to create a new forum topic. We’ll be more than happy to assist you again.

Thank you for choosing Rank Math and have a wonderful day!

Cheers