.Google's John Mueller addressed a question regarding why Google marks pages that are actually forbidden coming from creeping by robots.txt and why the it's safe to overlook the similar Browse Console records regarding those creeps.Crawler Web Traffic To Concern Parameter URLs.The individual talking to the inquiry documented that crawlers were actually developing links to non-existent question specification URLs (? q= xyz) to webpages along with noindex meta tags that are likewise obstructed in robots.txt. What prompted the inquiry is actually that Google is creeping the hyperlinks to those webpages, getting blocked out by robots.txt (without envisioning a noindex robotics meta tag) then obtaining reported in Google Browse Console as "Indexed, though obstructed through robots.txt.".The person asked the adhering to inquiry:." Yet here's the large inquiry: why would certainly Google mark web pages when they can not even find the material? What's the perk during that?".Google.com's John Mueller affirmed that if they can not creep the page they can not see the noindex meta tag. He also helps make an intriguing reference of the site: hunt driver, urging to overlook the end results considering that the "typical" customers will not find those outcomes.He wrote:." Yes, you're appropriate: if our experts can not creep the page, our experts can not view the noindex. That claimed, if our team can not crawl the webpages, after that there is actually certainly not a lot for our team to index. Thus while you could find several of those web pages along with a targeted web site:- inquiry, the normal consumer will not observe them, so I wouldn't bother it. Noindex is likewise fine (without robots.txt disallow), it simply implies the Links will definitely wind up being actually crawled (and also wind up in the Explore Console report for crawled/not listed-- neither of these conditions induce issues to the remainder of the website). The vital part is that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the constraints in using the Internet site: hunt progressed hunt operator for analysis reasons. One of those explanations is actually since it is actually not attached to the routine search mark, it is actually a separate point completely.Google.com's John Mueller commented on the site hunt operator in 2021:." The short response is that a website: concern is certainly not suggested to become complete, nor utilized for diagnostics purposes.A site question is a details kind of search that limits the end results to a certain site. It is actually generally just words web site, a bowel, and afterwards the site's domain name.This question confines the outcomes to a particular site. It is actually certainly not indicated to become a complete assortment of all the webpages coming from that website.".2. Noindex tag without using a robots.txt is actually great for these type of scenarios where a crawler is linking to non-existent pages that are actually receiving found by Googlebot.3. URLs along with the noindex tag will definitely create a "crawled/not listed" entry in Search Console and also those won't have a bad impact on the remainder of the website.Review the question and also respond to on LinkedIn:.Why would Google.com index webpages when they can not even see the information?Featured Picture by Shutterstock/Krakenimages. com.