blocking subdomain from indexing using robots.txt

#590686
  • Resolved Anonymous

    2 years, 3 months ago

    Rank Math free

    I am using Cloud front as a CDN for my website. My website has two versions. One is live website https://mydomain.com/ and second https://origin.mydomain.com/ as a CDN version. I want to ask how can I stop search engines to crawl and indexing my CDN version content.

    For SEO I am using Rank Math and added canonical URL for indexing master page on live website.

    Google page indexing report is showing issues with origin version pages like Crawled – currently not indexed and Duplicate, Google chose different canonical than user. And few pages are indexed.

    Is there any way I can stop crawling for my CDN version.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Md. Sakib Khandaker

    2 years, 3 months ago

    Rank Math agency

    Hello,

    Thank you for contacting Rank Math and bringing your concern to our attention.

    You can prevent Google from crawling your CDN version domain by adding a rule to your robots.txt file like the below:

    Disallow: https://origin.mydomain.com/*

    Here’s how you can edit the robots.txt file using Rank Math:
    https://rankmath.com/kb/add-sitemaps-to-robots-txt/#num-2-2-navigate-to-edit-robots-txt

    Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.

    Thank you.

    Anonymous

    2 years, 3 months ago

    Rank Math free

    Thanks for replying.

    I have added the specified rule in main domain robots.txt file. Can you please confirm this rule should be added in main domain robots.txt file or I have to create separate file origin version? And also let me know few of my origin URLs are also indexed. How can I remove origin version indexed URLs? I have tried link removal and it temporary removed URL from search engine but URL inspection is showing that URL is indexed. How can I deindexed it?

    Md. Sakib Khandaker

    2 years, 3 months ago

    Rank Math agency

    Hello,

    You should add the robots.txt rule to your CDN version domain.

    Can you please share your CDN version domain with us? You can share it in the sensitive data section if you don’t want to share it publicly.

    Looking forward to helping you.

    Thank you.

    RankBot

    2 years, 2 months ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)

The ticket ‘blocking subdomain from indexing using robots.txt’ is closed to new replies.