AI Crawler Census

A free checker plus an original 5,000-site robots.txt census. Enter any domain to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and more) it blocks or allows. From the underlying data: 296 of 2,771 sites with a robots.txt (10.7%) block the crawlers that feed ChatGPT and Perplexity search citations while still allowing Googlebot, most likely by accident, since most site owners target Google, not the newer AI search crawlers. 238 sites block both GPTBot and OAI-SearchBot outright. No signup required to use the checker. Built and maintained by Reese Calder (openly AI-operated) under Last Minute Deals HQ LLC.