Hello,
Thank you for your query and we are so sorry about the trouble this must have caused.
The first thing to note is that the sitemap is just a crawling mechanism as the experts on Google forums mentioned and you should really focus on building your internal site structure and have some good content.
Also, please note that it is completely upto Google if they want to index or not index your website. So, if you haven’t willingly or accidentally made your posts noindex and Google still isn’t indexing your page – it means that either Google doesn’t want to index or hasn’t crawled your page yet because it hasn’t allocated that much crawl budget to your website.
Please check this website for more details.
Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.
Hi Miguel,
Please find the below URL log file, I have checked my site Crawl issues in my jetoctopus.com
I have found lot of URL 429 response code occurred, how to resolve the problem?
This is hosting server side issue and Development code issue.
Please help me,
https://docs.google.com/spreadsheets/d/1Zg2V_cWS7GywJ0fHjKEVhWAqKxCHg38X/edit?usp=sharing&ouid=113919484961542528268&rtpof=true&sd=true
Regards
Prbanjan
Hello,
Checking some of the URLs with 429 responses in the log, they seem to be loaded properly with the correct response:
Most of the URLs from your log are empty archives however, the URL from my screenshot is indeed not indexed on the Google SERP.
Kindly submit that URL in your GSC’s URL inspection tool and share a screenshot of it here.
Looking forward to helping you.
I found the settings in Google search console slow down the crawl speed.
But how much I need to set the second crawl rate? how to calculate this second?
https://prnt.sc/2pDNJSn7gDQR
RegardsPrbanjan
Hello,
This depends on how much crawling rate your hosting can handle. You may contact your hosting regarding this and ask them how much crawling budget Google should impose for your website.
As for the product page, although, there’s no referring page at the moment, the URL is already been discovered so it’s up to Google to decide if they want to index the said page. You may refer to this guide: https://rankmath.com/kb/discovered-currently-not-indexed/
Looking forward to helping you.
Still I am not receive good answer for questions, let me tell us Google crawl my site pages Yes or No,
I have keep on checking other search engine Bing and Yandex crawl my site day by day. but Google
only not crawl my website pages, taking very long time, if I was given index request by manually using
google search page inspection tool the web will come in google engine result page. otherwise auto
crawl not happening in my search console. please find attached file for your : https://prnt.sc/I_1kgSMtG-a2
Hello,
As we have mentioned earlier, the crawl rate depends on how your server can handle it. If you are really not sure, you can simply just let Google optimize the crawling rate for you by selecting the option “Let Google optimize for my site”.
Please try that instead and if it seems the issue is still there, please contact your hosting if there are any issues with your website that prevent Googlebot from accessing your pages.
At the moment, we don’t see settings on your website that can prevent Google from crawling it.
Looking forward to helping you with this one.
Thanks for your reply, I have checked the issue with our hosting provider, they said: nothing issue in server side, contact our SEO support.
For your reference, I have mentioned below URL Google search console crawl reports.
https://docs.google.com/spreadsheets/d/1BHgkgWuzM-uqDegrtvTLW1NzzFMegcca6JTxqhen2Zs/edit?usp=sharing
https://prnt.sc/ryUEnFhHRSBl
https://docs.google.com/spreadsheets/d/1HqV7N3GPFi8nk6K-TW-TlOmcoy-RUooC9SIQ8Moa5Iw/edit?usp=sharing
Please check the Log and let me know. what I do next.
Thanks in advance
Prabanjan
Hello,
In this case, you just need to depend on Google if they wish to crawl the URLs for you. As far as we have looked into your website, we do not see any issues that will hinder the crawl.
Just focus on creating good quality internal links and backlinks as well. This is one of the factors that will increase the change of getting crawled right away.
Don’t hesitate to get in touch with us if you have any other questions.
Thanks for your Email, I keep waiting for Google crawling very slow, and doing the same internal & external link-building.
or else I need change the Google crawl rate settings in Google search console?
Thanks for your support
Regards
Prabanjan
Hello,
Thank you for your patience.
We recommend changing the settings to the first one so Google will decide the crawl on your site.
The option to limit the crawl is for websites that are slowing down the server due to crawling.
Looking forward to helping you.
Thank for your support
Regarding this I have follow up my hosting provider and Rankmath support as well, but still not satisfied the answer for me because my site was 4 month old, but still, Google search console not crawl my site taking very long time, but bing,yandex search results it will coming soon. currently shown the SERP of Google results my web pages, I was submit the manual option: https://prnt.sc/382fZvSguG3I
Then only google SERP result coming. I will did my all website page google index using by this option is the best way? or I will wait for Automatic by Google crawl?
Please help me, long time struggling with the issue.
Please tell me the best way.
Thanks in advance
Regards
Prabanjan
Hello,
If Google is crawling and indexing the site after requesting the indexing manually, then you should do that for all the important URLs of your site. You can also use Instant Indexing for Google to submit the URLs to Google’s Indexing API. Here is a guide that can help: https://rankmath.com/blog/google-indexing-api/
Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.
Please reply to me, I am waiting for your support.
Hello,
Google estimates even more pages than the ones it shows as indexed as you can see here:
Also, looking at the screenshots we don’t see any crawling issue with your website.
The only pages that are not indexed is because Google has decided that their content is not relevant enough to be on the SERPs.
That’s what the error “Discovered – currently not indexed” usually means. The Googlebot is aware of the pages but has decided that they shouldn’t be crawled/indexed and most of the time that is due to the content not being up to standards.
On top of that, John Mueller from Google, already stated that the intention of Google is not to index 100% of every website. Most websites will have some pages that will never get indexed and that is fine.
Don’t hesitate to get in touch if you have any other questions.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
