A fake Googlebot is a bot that disguises itself with the identifier of Google’s real crawler.
It hopes websites will let it through unchecked because nobody wants to block the search engine.
Understanding Bots and Web Crawlers
Web crawlers are bots that open and analyse pages automatically.
Search engines use them to index the web;
other crawlers collect prices, content or addresses.
The Role of Googlebot
Googlebot collects pages for Google Search.
It respects robots.txt and comes from IP ranges that Google publishes and that can be verified by DNS lookup.
The Emergence of Fake Googlebots
Fake Googlebots copy content and prices, look for security holes, send spam through forms or overload servers.
Because many websites treat Googlebot generously, they bypass blocks and rate limits this way.
Identifying a Fake Googlebot
The user agent string can be faked at will and proves nothing.
Checking the IP address is reliable: the reverse lookup must end in googlebot.com or google.com, and the forward lookup of that name must return the same IP again.
Preventing Fake Googlebot Attacks
Exceptions for search engines should only apply to verified IP addresses.
Rate limiting, firewall rules and logging unusual access help as well.
The Role of CAPTCHA
Real search engines do not fill in forms.
Protection such as CDPCaptcha on contact, login and order forms stops fake crawlers there without affecting the indexing of pages.
Conclusion
A bot is not trustworthy just because it calls itself “Googlebot”.
Checking its origin via DNS and protecting forms lets the real search engine in and keeps fakes out.
