These steps will not always reproduce the bug, but here is the process:
Browse > Collections > New Collection
Fill in required information
Check “This collection is unrevealed”
Follow the “Post to Collection” link
Fill in required information (the collection info will already be filled in because you used the “Post to Collection” link)
Post Without Preview
Repeat previous 3 steps until you have your desired number of works
Return to the collection, e.g. by following the link in the work meta
Follow the “Collection Settings” link
Uncheck "This collection is unrevealed"
Most of the works are revealed, but not all of them.
All of the works should be revealed.
The CollectionItem for the work correctly has its unrevealed column updated to false. However, the corresponding Work does not get its in_unrevealed_collection column updated to false. It's not a case of the work belonging to multiple unrevealed collections, as you can see from the following example (identifying data removed):
2.3.0 :007 > work = Work.find(000)
=> #<Work id: 000, expected_number_of_chapters: 1, created_at: "2018-10-01 18:06:16", updated_at: "2018-10-19 01:59:30", major_version: 1, minor_version: 2, posted: true, language_id: 1, restricted: false, word_count: 0, hidden_by_admin: false, delta: false, revised_at: "2018-10-01 18:07:07", backdate: false, imported_from_url: nil, hit_count_old: 0, last_visitor_old: nil, complete: true, summary_sanitizer_version: 1, notes_sanitizer_version: 1, endnotes_sanitizer_version: 1, work_skin_id: nil, in_anon_collection: false, in_unrevealed_collection: true, anon_commenting_disabled: false, spam: false, spam_checked_at: "2018-10-01 18:10:38", moderated_commenting_enabled: false>
2.3.0 :008 > work.collection_items
=> #<ActiveRecord::Associations::CollectionProxy [#<CollectionItem id: 000, collection_id: 000, item_id: 000, item_type: "Work", user_approval_status: 1, collection_approval_status: 1, created_at: "2018-10-01 18:06:17", updated_at: "2018-10-01 22:40:14", anonymous: false, unrevealed: false>]>
Quite possibly a race condition and quite possibly related to AO3-5533.
CollectionItem.update_work is one of the few after_commit callbacks we have that tries to read data that was just written in the previous commit. Because it's an after_commit callback, not an after_save, the database queries to look up user_approved_collection_items (in update_anon_unrevealed, which is called by CollectionItem.update_work) occur outside of the transaction, which in theory would allow them them be routed to the read-only SQL servers, which have some chance of lagging behind the master SQL server by a transaction or two. (h/t ticking instant for the discovery and explanation.)
AO3-5532 can also cause some works in a collection to stay unrevealed, but the major difference is that involves a 500 error when updating the collection settings. This does not have an error.
This can probably happen with anonymous works, too, but we’re not sure.
Because this cannot be reproduced at will, it is unlikely the fix can be tested. All we can do is take several shots at posting to and then revealing an unrevealed collection to confirm the correct behavior still happens and watch for any new reports of this bug after the fix is deployed.