We want to be able to serve different robots.txt files depending on the fully qualified domain name.
We need to start with having both versions of robots.txt available in production, however the deploy currently overwrites the one that bans all bots.
Looks good on staging:
https://test.archiveofourown.org/robots.txt is still the private version.
http://insecure-test.archiveofourown.org/robots.txt is still the private version.
https://test.archiveofourown.org/robots.private.txt is the private version.
https://test.archiveofourown.org/robots.public.txt is the public version, same as https://archiveofourown.org/robots.txt.
Setting to pending attention on beta, things to check:
https://archiveofourown.org/robots.txt is (still) the public version.
https://archiveofourown.org/robots.public.txt is (still) the public version.
https://archiveofourown.org/robots.private.txt (new) is the private version.
http://insecure.archiveofourown.org/robots.txt is changed to the private version.
https://www.archiveofourown.org/robots.txt is changed to the private version.