Curtis Weyant @curtisweyant.com · 21/07/2025Welp, I take it back—I think I'm just an idiot. This is what I get for panic sharing to social media rather than taking 5 more minutes to figure something out. 🤦 I believe I've found the issue will wait for validation of the fix. Apolgogies. 100
Curtis Weyant @curtisweyant.com · 21/07/2025Almost immediately after I updated my robots.txt, I started seeing pages reported as blocked by robot.txt in Google Search Console. Curious as to why this might be happening. Do these UAs have more to do with Search than stated by the docs? Any thoughts from @johnmu.com (or anyone else)? 110
Curtis Weyant @curtisweyant.com · 21/07/2025I tried blocking AI bots from crawling my website using the list at github.com/ai-robots-txt/ The list includes several Google UAs: Google-CloudVertexBot Google-Extended GoogleOther[-Image|-Video] None of these should affect Google Search according to developers.google.com/search/docs/...github.comai.robots.txtA community effort to identify and block AI crawlers. - ai.robots.txt 100
Reposted by Curtis WeyantPreeti Gupta @ilovechoclates.bsky.social · 15/06/2025The work is indeed mysterious and important 2223