Wayback Machine's Save Page Now may be processing code comments in robots.txt, blocking users from archiving pages

Description

Attempting to save a page on the Wayback Machine (e.g. by going to https://web.archive.org/save/https://archiveofourown.org/admin_posts/10026) currently results in a message saying that "Page cannot be displayed due to robots.txt"

Correspondence with the Wayback Machine maintainers has indicated that the current version of Save Page Now may be interpreting code comments as actual code:

It is our new experimental version of Save Page Now that will ignore:

  1. User-Agent: *

  2. Disallow: /

We want to take the comments out so archive.org is less confused.

Activity

Show:
Sarken
April 17, 2018, 2:48 PM

Staging has its own robots.txt, so setting this to PAOB

Done

Assignee

james_

Reporter

james_

Roadmap

Misc

Priority

Medium

Affects versions

Fix versions

Components

None

Difficulty

Medium

Milestone

Internal 0.9