Email specs fail when emails include characters that are not in US-ACCII, e.g. ç

Description

It all started when Sarken, in her infinite wisdom, changed an email spec to use Faker::Books.title when checking for a work title in the email (AO3-5815). This test failure happened:

 

So, naturally, it was time to use staging to check if a claim notification email for a work titled Françoise Sagan looked okay:

  1. Made a post on Dreamwidth that would import with that title: https://ao3testing.dreamwidth.org/4352.html

  2. Logged in as testy, who has archivist privileges

  3. Post > Import Work

  4. Filled in that URL, check the "Import for others ONLY with permission" checkbox and then filled in the name and email address associated with Sarken's staging account, which sent a claim notification email to that email address upon posting the work (note: I didn't check the option to post without previewing)

The HTML email looked normal, right down to the ç, but the plain text/source for the email had lots of =0D and =3D.

Bwuh? Turns out the Content-Transfer-Encoding on the plain text and HTML portions of the email (but not the email overall) are set to something different than usual:

Normally, when there is no ç, the Content-Transfer-Encoding on those parts is 7bit as well.

This may be because the Mail library sets the Content-Transfer-Encoding to 7bit automatically if the characters in the email are all inside the US ASCII set:

Mail assumes that if your text in the body is only us-ascii, that your transfer encoding is 7bit and it is text/plain. You can override this by explicitly declaring it. (Source: )

The ç means that condition isn't met. Ergo, different Content-Transfer-Encoding.

Not only does this mean plain text emails look like

=0D

Archive of Our Own=0D

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0D

=0D

=0D

Hello from the Archive of Our Own!=0D

=0D

You're receiving this e-mail because you had works in a fanworks archiv=

e that has been imported by Open Doors (http://opendoors.transformativewo=

rks.org) into the Archive of Our Own (AO3 - https://test.archiveofourown.=

org/). Because this e-mail address is connected to one registered on the =

imported archive, the associated fanworks (listed below) have been automa=

tically added to your AO3 account.=0D

=0D

You can read announcements about recent archive moves at AO3 News (http=

s://test.archiveofourown.org/admin_posts?tag=3D18), and find additional i=

nformation on Open Doors's FAQ page (http://opendoors.transformativeworks=

.org/faq) or tutorials page (http://opendoors.transformativeworks.org/tut=

orials). For any questions not answered in the FAQ, tutorials, or this e-=

mail, please contact AO3 Support at https://test.archiveofourown.org/supp=

ort.=0D

=0D

If this is a mistake and these are not your works, please don't delete =

them! Please contact Open Doors (http://transformativeworks.org/contact/o=

pen%20doors) and we will sort it out.=0D

=0D

Depending on the archive, your works may have been imported restricted =

to registered users only (to keep them out of Google searches). If this i=

s the case, the works will only be accessible by logged-in users unless y=

ou choose to make them fully visible. For help unlocking, orphaning, or d=

eleting your works, please contact Support.=0D

=0D

These works were written under the e-mail: [email removed for issue]=0D

-Fran=C3=A7oise Sagan https://test.archiveofourown.org/works/107088=

4 (No Fandom)=0D

=0D

To preserve rec lists and bookmarks, the imported archive's web address=

es may redirect to the imported copy of these works for a limited time (c=

heck the announcement post for your archive to be sure). If you've alread=

y uploaded a copy of these works and you did NOT use the import from URL =

feature, there will be two copies of the same work on the archive.=0D

=0D

But GoldenFalls reports that sometimes similar yuck loads in place of the HTML email in Mail on the iPhone (but it is possible to refresh and make it load correctly):

I don't see any ç in the email of mine which I know was broken looking, but it does have a lot of diacritic marks because the fic used canonicals for the Chinese characters, so that might have been what pushed it

Fixing notes

redsummernight 01:45
https://api.rubyonrails.org/classes/ActionMailer/Base.html#class-ActionMailer::Base-label-Default+Hash

class NotifierMailer < ApplicationMailer
default 'Content-Transfer-Encoding' => '7bit'
end

let's hope that does it

if we set it to base64

you may want to add a base mailer like Elz did in this PR: https://github.com/otwcode/otwarchive/pull/3397

More notes

Turns out this is only an issue with tests! We need to check the decoded text, not the encoded text.

Environment

None

Status

Assignee

Sarken

Reporter

Sarken

Roadmap

Internationalization
Misc

Priority

Lowest

Affects versions

Fix versions

Components

AutomatedTests

Difficulty

Medium

Required Access Level

None

Milestone

Internal 0.9
Configure