Wrangling: Removing a subtag from its metatag does not remove the subtag's works from the metatag's metatag

Description

Originally reported on Google Code with ID 4085
What archive revision are you testing on? (See the version label in
the footer, for example v0.8.13.8.)
otwarchive test-0.9.20.13

If appropriate, enter the URL of a page where the problem can be seen:
http://test.archiveofourown.org/tags/Highest%20Tag/works

What steps will reproduce the problem?
1. Log in as a tag wrangler
2. Post three separate works, each with a different fandom, e.g. Lowest Tag, Middle
Tag, Highest Tag
3. Make your fandom tags canonical and set them up so (1) Middle Tag is a subtag of
Highest Tag and (2) Lowest Tag is a subtag of Middle Tag
4. Remove Lowest Tag as a subtag of Middle Tag
5. Remove Middle Tag as a subtag of Highest Tag

What is the expected output? What do you see instead?
The works index for Highest Tag should only show the work tagged with Highest Tag.
Instead, it shows the work tagged with Highest Tag and the work tagged with Lowest
Tag. Sadly, it is not caching.

You can also see the problem if you check the works index for Highest Tag in between
steps 4 & 5.

Activity

Show:
Sarken
February 15, 2018, 11:43 PM

Hmm, okay – I'll take it back out of the release. It did used to be reproducible 100% of the time on staging, so something happened to make it better, at least. (Neither I nor another tester were able to reproduce it.)

ticking instant
February 16, 2018, 6:38 PM

Maybe there was an update that decreased the load on the workers? (Or a configuration change?)

The code for removing a meta-tagging through the interface first calls Tag.remove_association asynchronously, and then when the meta-tagging is deleted it triggers an asynchronous call to Tag.remove_meta_filters. So if you have two meta-taggings being deleted with sufficient delay between the deletions, and the Resque utilities workers are very responsive (i.e. very few things on the queue, or a lot of workers), these steps get executed in the following order:

1. In the interface, remove Lowest Tag as a subtag of Middle Tag.
2. "Middle Tag".remove_association("Lowest Tag")
3. "Lowest Tag".remove_meta_filters("Middle Tag")
4. In the interface, remove Middle Tag as a subtag of Highest Tag.
5. "Highest Tag".remove_association("Middle Tag")
6. "Middle Tag".remove_meta_filters("Highest Tag")

That order ensures that when step 3 is executed, "Highest Tag" is still a meta-tag of "Middle Tag." So when dissociating "Lowest Tag" from "Middle Tag," it will correctly dissociate "Lowest Tag" from "Highest Tag" as well (deleting the associated filter-taggings, and updating the appropriate Elasticsearch indexes).

However, if step 3 occurs after step 5 instead, the association between "Middle Tag" and "Highest Tag" will be gone when step 3 is executed, and "Lowest Tag" will never be dissociated from "Highest Tag." This leaves an incorrect, invisible meta-tagging in place, as well as all of the filter-taggings associated with that meta-tagging (and because the Elasticsearch filter_ids are built from the filter-taggings, Elasticsearch also gets it wrong).

I simulated this by suspending the workers while I made the changes in the interface, then re-enabling them. With only one worker, suspending the worker will cause it to fail every time; with more than one worker, there's a slim chance that the sequence of steps would work out, but it's still very likely to fail. In theory, suspending the workers should behave similarly to when all of the workers for the utilities queue are busy with other tasks.

I can also simulate it without suspending the workers by performing the two meta-tagging changes in a single edit (editing the Middle Tag and checking the boxes to remove both "Highest Tag" and "Lowest Tag"), which seems to cause it to fail pretty consistently. But I'm only running two workers, so I don't know if that would work with more.

teyla
April 26, 2020, 8:22 AM
  1. created three canonicals with metatag "Other Media" and one work each:
    This Tag is a Bottom http://test.archiveofourown.org/tags/This Tag is a Bottom
    This Tag is a Switch http://test.archiveofourown.org/tags/This Tag is a Switch
    This Tag is a Top http://test.archiveofourown.org/tags/This Tag is a Top

  2. created the following tag tree:
    This Tag is a Top
    is a metatag of
    This Tag is a Switch
    is a metatag of
    This Tag is a Bottom

  3. waited until all three works appeared on the This Tag is a Top filter page (~10 min): http://test.archiveofourown.org/tags/This Tag is a Top/works

  4. removed the following tag associations on the This Tag is a Switch edit page with a single submit action:
    http://test.archiveofourown.org/tags/This Tag is a Switch/edit
    * removed This Tag is a Bottom as subtag of This Tag is a Switch
    * removed This Tag is a Top as a metatag of This Tag is a Switch
    Note: I had to submit twice -- check the sub- and metatag to remove, submit, check them again and submit again. This is something that happens to me on the beta archive during regular wrangling all the time, so I'm assuming this is a known bug? I'll admit I've never checked.

  5. checked This Tag is a Top filter page immediately: all three works still visible

  6. waited ~5 min, then checked again: only 1 work visisble, tagged with "This Tag is a Top"

  7. recreated tag tree and waited for all three works to appear on This Tag is a Top filter page

  8. removed meta- and subtag from This is a Switch edit page again (again, see note above – I had to remove them twice for them to actually go away)

  9. checked This Tag is a Top filter page: all 3 works visible

  10. waited about 10 min and reloaded filter page: only 1 work visible, tagged with "This Tag is a Top"

This seems to work now; however, I don’t know how to suspend workers on staging, so someone with access to do that may want to re-test.

lydia-theda
May 8, 2020, 7:08 AM
  1. posted 3 works with one of three uncategorized fandoms:
    uncategorized metatag
    uncategorized tag
    uncategorized subtag

  2. canonized fandom tags and built tag tree from the top down (adding “tag” as a subtag via metatag’s edit page, then adding “subtag” as a subtag via tag’s edit page)

  3. 1 work appeared on subtag/works, 2 works appeared on tag/works, and 3 works appeared on metatag/works (as expected; did not time it but was within 5–10 minutes)

  4. went into tag/edit and removed both meta and subtag associations simultaneously. Both vanished immediately; did not have to refresh or re-remove from tag/edit page.

  5. metatag/works took about 10 minutes to stop showing the works from its previous subtags.

  6. rebuilt fandom tag tree, this time from the bottom up (adding “tag” as a metatag via subtag’s edit page, then adding “metatag” as a metatag via tag’s edit page)

  7. metatag/works took about 7 minutes to properly show all three works.

  8. went into tag/edit and removed subtag association to “subtag”; went into metatag/edit and removed subtag association to “tag”. Did not have to refresh to see a blank meta or subtag field.

  9. metatag/works took about 4 minutes to stop showing the two works from its previous subtags.

  10. rebuilt fandom tag tree, this time simultaneously (adding “metatag” as a metatag via tag’s edit page at the same time as adding “subtag” as a subtag via tag’s edit page)

  11. metatag/works took about 10 minutes to properly show all three works.

  12. went into subtag/edit and removed metatag association to “tag”; went into tag/edit and removed metatag association to “metatag”. Did not have to refresh to see a blank meta or subtag field.

  13. metatag/works took about 15 minutes to stop showing the two works from its previous subtags.

Since I seemed to have even fewer issues than teyla, would agree that manually forcing a suspension could be a potentially valuable third test.

redsummernight
May 16, 2020, 1:31 AM
Edited

Pending test steps:

  1. Log in as a tag wrangler

  2. Post three separate works, each with a different fandom, e.g. Lowest Tag, Middle Tag, Highest Tag

  3. Make your fandom tags canonical and set them up so (1) Middle Tag is a subtag of Highest Tag and (2) Lowest Tag is a subtag of Middle Tag

  4. Suspend the workers (bundle exec god stop workers)

  5. Remove Lowest Tag as a subtag of Middle Tag

  6. Remove Middle Tag as a subtag of Highest Tag

  7. Resume the workers (bundle exec god start workers)

ETA: built the tag tree. On the tag page for "High High High - Fandom", we could see:

  • Suspended the workers.

  • Edited "Mid Mid Mid" to remove subtag "Low Low Low".

  • Edited "High High High - Fandom" to remove subtag "Mid Mid Mid".

  • The "utilities" queue showed 20+ pending jobs, 2 of which are:

    • Tag ["perform_on_instance", 1221155, "update_inherited_meta_tags"]

    • Tag ["perform_on_instance", 1221158, "update_inherited_meta_tags"]

  • Resumed the workers. Waited for the queue to clear. Immediately checked the works page of "High High High - Fandom". All 3 works still appeared.

  • Checked the filters of 3 works, they looked correct:

Assignee

ticking instant

Reporter

Sarken

Roadmap

Tag Wrangling

Priority

Medium

Affects versions

None

Fix versions

Components

BackEnd

Difficulty

Medium

Required Access Level

Tag Wrangler

Milestone

Internal 0.9

Google Code Issue ID

4085
Configure