• Installation
pip install robofinder
I also focused more on data quality than raw speed. Wayback lookups, especially on older targets, may take a little longer :( but you'll get much more results instead of missing data.
github: github.com/Spix0r/robof...
3/3
github.com
GitHub - Spix0r/robofinder: Robofinder fetches historical robots.txt files from Archive.org to uncover old directories, hidden paths, and valuable OSINT data for reconnaissance.
Robofinder fetches historical robots.txt files from Archive.org to uncover old directories, hidden paths, and valuable OSINT data for reconnaissance. - Spix0r/robofinder