Add robots.txt to archive.softwareheritage.org to avoid crawlers
We don't want robots crawling archive.softwareheritage.org using API endpoints.
We need to add a robots.txt to that effect.
Migrated from T662 (view on Phabricator)
We don't want robots crawling archive.softwareheritage.org using API endpoints.
We need to add a robots.txt to that effect.
Migrated from T662 (view on Phabricator)