Rerunning failed Resque tasks breaks assignments for gift exchanges hit by the stuck-in-a-matching-loop bug

Description

The problem

Occasionally, a gift exchange's matching will get stuck in a loop and a database admin will have to ensure that it completes. All the failed runs for this exchange end up in the Resque failures list.

When someone goes to the Failed page on Resque and presses the Retry Failed Jobs button (e.g. because they want to send out failed subscription emails), it retries the failed matching jobs. EDIT: This might only happen when rerunning the tasks using `bundle exec rake resque:run_failures`, not when using the Retry Failed Jobs button.

If the exchange has already set assignments, this effectively unsends them by setting the ChallengeAssignment's sent_at to nil. The assignments remain visible on the challenge's assignments page (https://test.archiveofourown.org/collections/COLLECTION_NAME/assignments), but they disappear from the participants' assignments pages (https://test.archiveofourown.org/users/USERNAME/assignments).

The means the challenge mods either have to purge assignments and rerun matching and recreate the assignments themselves or ask a database admin to fix things by setting the sent_at date. (For assignments that were not automatically generated, such as pinch-hits or write-in recipients or givers, request_signup_id and offer_signup_id information is actually lost from the ChallengeAssignment objects and can only be restored if the challenge mods have kept a record.)

The solution

Failed matching jobs should not be rerun, especially not if the challenge has successfully generated potential matches and/or sent assignments.

Testing

Unfortunately, we do not currently know how to reproduce the failed matching loops that cause this issue.

Activity

Show:
ticking instant
April 24, 2018, 11:22 PM

For testing purposes, you can pick a collection/challenge that has already sent out assignments, and run the following line in the console:

PotentialMatch.generate_in_background(Collection.find_by(name: "CollectionName").id)

Then verify that collection participants can still see their assignments.

It would also be good to try running this line, as well, and performing a similar check:

ChallengeAssignment.delayed_generate(Collection.find_by(name: "CollectionName").id)

(Depending on your logging level, you may also be able to check the SQL statements that each command generates to verify that there are no updates, inserts, or deletes in the PotentialMatch or ChallengeAssignment tables.)

Sarken
April 25, 2018, 4:09 PM

testy has an assignment on https://test.archiveofourown.org/users/testy/assignments for the Aardvark Gift Exchange (aardvark_gift_exchange).

2.3.0 :001 > PotentialMatch.generate_in_background(Collection.find_by(name: "aardvark_gift_exchange").id)
=> nil

Refreshed the page, could still see the assignment.

2.3.0 :003 > ChallengeAssignment.delayed_generate(Collection.find_by(name: "aardvark_gift_exchange").id)
=> nil

Refresh the page and could still see the assignment.

Looks good! Thank you x 10,000!

Lady Oscar
May 10, 2018, 1:57 AM

Looks good!

DeployedToBeta

Assignee

ticking instant

Reporter

Sarken

Roadmap

Challenges

Priority

High

Affects versions

Fix versions

Components

BackEnd
Utilities

Difficulty

Medium

Milestone

Internal 0.9