Enable Rack::Attack for rate limiting searches

Description

We'd like to be able to rate limit searches (including filtering). Luckily, we have Rack::Attack! So let's put that to work.

There will be two rate limit settings. One will be for bookmarks (unicorn_elastic_bookmarks), the other (unicorn_elastic) will be for everything else.

Testing instructions

The limits have been adjusted to allow for easier testing, so these numbers will of course not apply to real use.

  • Go to one of the URLs affected by the issue

  • Enter the Lockup password if necessary

  • In your browser's dev tools, go to the Network tab, locate the GET request for the page, and right click on that and choose "Copy as cURL". More info on Copy as cURL in a variety of browsers.

  • Paste the cURL command you just copied somewhere you can edit it, e.g. a text editor. It will look something like this:

  • We want to run this command 60 times and hit a different URL each time to avoid caching. We’re going to add a line at the top telling it how many times to run (for i in {1..60}; do) and then modify the URL we’re hitting (here the collection name is replaced with $i – if you’re using a tag’s or user’s works or bookmarks URL, you would replace the tag or username with $i) and then end the loop at the end with ; done

  • Now copy this into your command line and run it

The first output you see should list the response code as HTTP/2 200, indicating you’ve successfully reached the page, and then later ones should list HTTP/2 429, indicating you’ve been rate limited. If you try to go to any of the listed Elasticsearch page in your browser, you should get a “Retry later” message. If you try to go to a different page, e.g. the home page, you should be able to access it.

Activity

Show:
james_
December 16, 2019, 11:33 PM

I will need to apply a few changes to nginx brfore this can be tested.

 

I note that you have to do at most two times the rate requests as the requests have to happen in a single time period and it could be in two.

Lady Oscar
December 23, 2019, 2:34 AM

Besides checking to see if searching/filtering still works, is there any way in particular to test this?

Sarken
January 13, 2020, 2:29 AM

james_ 06:33
( Using test to see if I have headers set right ).

( For the rate limiting )

Ok the per unicorn set of rate limits look to be working, I have been rate limited on bookmarks on collections but not on the rest of the site.

Sarken
January 13, 2020, 9:21 PM

Working as expected on collection work pages! I got 429s and was able to access other, non-ES pages.

 

redsummernight
January 16, 2020, 2:11 AM
Edited

I won't post the cURL command here as it has the lockup password in the body. I use -i -s -o /dev/null -w "%{http_code}\n" to make the output easier to read (include protocol headers, be silent, download response body but don't output it, write out only the response status codes one per line).

Followed testing instructions, hammered https://test.archiveofourown.org/users/redsummernight/works (just that one URL, no variation) 120 times in less than a minute, got 429s towards the end. Then in a browser, I got "Retry later" plain text message on work search related pages, but I could still access bookmark search related pages normally, as well as the rest of the site. After 5-10 minutes, I could access work search related pages as usual.

Hammered https://test.archiveofourown.org/users/redsummernight/bookmarks (no variations) 60 times in about a minute, got 429s towards the end. Then in a browser, I got "Retry later" plain text message on bookmark search related pages, but I could still access work search related pages normally, as well as the rest of the site. After 5-10 minutes, I could access bookmark search related pages as usual.

Looks good!

We need to adjust the limits for beta before deploy.

DeployedToBeta

Assignee

james_

Reporter

Sarken

Roadmap

Search

Priority

Highest

Affects versions

Fix versions

Components

BackEnd

Difficulty

Medium

Milestone

Internal 0.9
Configure