.Google's John Mueller answered a concern regarding why Google marks web pages that are actually refused coming from creeping through robots.txt as well as why the it's safe to ignore the associated Explore Console reports regarding those crawls.Robot Traffic To Inquiry Criterion URLs.The person talking to the concern recorded that bots were actually developing links to non-existent question parameter Links (? q= xyz) to web pages with noindex meta tags that are additionally blocked out in robots.txt. What cued the concern is actually that Google is actually creeping the hyperlinks to those web pages, acquiring obstructed through robots.txt (without seeing a noindex robotics meta tag) then acquiring turned up in Google.com Browse Console as "Indexed, though shut out by robots.txt.".The individual asked the following inquiry:." But right here is actually the big question: why would Google.com index webpages when they can't even observe the content? What is actually the perk because?".Google.com's John Mueller validated that if they can not creep the web page they can't find the noindex meta tag. He also makes an appealing reference of the website: hunt operator, suggesting to neglect the results considering that the "common" individuals will not view those results.He wrote:." Yes, you're appropriate: if we can't crawl the webpage, we can not observe the noindex. That pointed out, if we can't crawl the web pages, then there is actually not a lot for our company to index. Therefore while you might view some of those pages along with a targeted website:- inquiry, the normal consumer won't see them, so I wouldn't fuss over it. Noindex is actually likewise fine (without robots.txt disallow), it merely suggests the URLs will certainly find yourself being crept (and end up in the Browse Console record for crawled/not listed-- neither of these standings trigger problems to the remainder of the website). The integral part is actually that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the constraints in using the Website: hunt evolved hunt operator for diagnostic reasons. Some of those factors is actually due to the fact that it is actually not hooked up to the regular hunt index, it is actually a distinct factor completely.Google.com's John Mueller discussed the web site hunt operator in 2021:." The quick response is that an internet site: inquiry is not meant to become comprehensive, nor made use of for diagnostics purposes.A web site question is actually a details type of hunt that limits the results to a specific site. It is actually basically simply words web site, a colon, and then the internet site's domain name.This concern restricts the end results to a specific site. It's not meant to be a complete selection of all the pages from that website.".2. Noindex tag without using a robots.txt is alright for these type of scenarios where a crawler is actually linking to non-existent webpages that are actually receiving found out by Googlebot.3. Links with the noindex tag will certainly produce a "crawled/not recorded" item in Browse Console and also those won't possess a negative impact on the rest of the site.Read through the concern and respond to on LinkedIn:.Why will Google mark webpages when they can not also view the content?Included Photo by Shutterstock/Krakenimages. com.