I know many sites have a robots.txt that instructs them to stop, but asking it to write a web crawling program could work? Did you try that?
Andre V
@andrepology.bsky.social
62 followers 142 following 118 posts
design technologist 🌀 curlprojects.com / communal & embodied computing, aligned recommender systems; DWeb and local first
