Stake and Ward leader emails are going to SPAM folder

[p.harrington]

Hello all,

I don't know of a better way to bring this issue to the attention of the appropriate people, but with the new Social Distancing guidelines in place our Stake and Ward leaders have been heavily using the website emailing tools.

The problem is that through a combination of old technology and how headers are formed for the emails, the messages from leaders are already 80% of the way towards being marked as SPAM.

I use Spamassassin to analyze email that comes into my server, that is combined with Thunderbird's internal junk filtering to produce a pretty good spam filter. I run my own mail server and my email address has been floating around the Internet for > 20 years, so I get a lot of spam.

I choose to run my own server because services like Gmail are even worse at false positives and place draconian restrictions on what can be sent to and from their email boxes.

With my reasoning out of the way, let me show you what the Spamassassin report for a recent Stake Presidency email looked like:

Code: Select all

X-Spam-Report: 
	* -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at
	*      https://www.dnswl.org/, no trust
	*      [216.49.179.112 listed in list.dnswl.org]
	* -0.0 SPF_PASS SPF: sender matches SPF record
	*  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
	*  0.0 HTML_MESSAGE BODY: HTML included in message
	* -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	*      author's domain
	* -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from
	*      envelope-from domain
	*  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
	*       valid
	* -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
	*  1.2 INVALID_MSGID Message-Id is not valid, according to RFC 2822
	*  1.5 PDS_FREEMAIL_REPLYTO_URISHRT Freemail replyto with URI
	*      shortener
	*  2.5 FREEMAIL_FORGED_REPLYTO Freemail in Reply-To, but not From
	*  1.5 PDS_TINYSUBJ_URISHRT Short subject with URL shortener

The body of the email read:

Code: Select all

Dear Brothers and Sisters of the Stake, 

    As we take time in our homes this weekend to celebrate the sacrifice and atonement of the Lord Jesus Christ, may we be inspired to more fully understand and appreciated the Savior and the true meaning of the Easter season. 
   The following Church video may help in this endeavor.

    We love and miss all of you, and hope to be able to meet together soon.

Sincerely, the Stake Presidenc

https://youtu.be/u0PmMX27LgE

I will not post the headers because I don't want to identify the person who sent the message.

There are 3 problems with how the Church website sends emails and these problems can be rectified, greatly reducing the penalty while sending messages:

1. The Message-Id header is not RFC compliant, it reads:

Code: Select all

Message-ID: <ESM_Email_Serivce.JavaMail.dc0df918-2c6c-4d28-af45-36d2c8d1b14f>

A proper RFC-2822 Message-ID header will look like this:

Code: Select all

Message-ID: <87DErSlXSuKxNKX44HtERg@mail.example.com>

The Message-ID is being generated by the JavaMail MUA that is sending the messages to membership, either the MUA (Mail User Agent) needs to be changed to a newer compliant Java Class, or the Message-ID needs to be explicitly set with a properly formatted header.

2. The Reply-To and the From header are not the same.

When a leader sends a message through the Church website the From header is modified to use <noreply@churchofjesuschrist.org> instead of their actual email address, yet the Reply-To header is set to the actual email address of the leader.

The RFC standard states that email replies should be sent based on this priority:

Reply-To
From
Return-Path

Error messages should be sent to the Return-Path, known as the envelope sender.

It serves no technical or practical purpose to set Reply-To to the leader's email address, but to put the Return-Path (envelope sender) in the From header of the message. This discontinuity causes Spamassassin to demerit the email message from the leader.

The fix is to set the From and Reply-To to contain the email address of the leader.

If a message is sent with no reply intended, then the From should contain <noreply@churchofjesuschrist.org>, the Return-Path should contain <noreply@churchofjesuschrist.org>, and the Reply-To should be omitted.

3. Attachments sent in broadcast emails are not broken up into lines of 80 characters each, they are sent as a single line.

Attachments are encoded in a text format which is called MIME encoding. It's a way of taking a file like a PDF, which contains non-printable characters (binary), and encoding it into characters which are technically printable (0-9, A-Z, etc).

The MIME encoding process takes an attachment as input and produces a long string of text as output, this long string of text should be broken up into lines that are shorter than 80 characters when embedded in an email message.

Because the MIME attachment is not broken up into lines, it is considered a characteristic commonly shared with SPAM and counts negatively towards the SPAM score.

The fix is to simply split the long MIME attachment into multiple lines with a CRLF (Carriage Return + Line Feed) delimiter between each line.

The reason why these 3 technical flaws are so critical is that it gives leader no leeway when drafting emails. The cited message simply included a short subject and a shortened YouTube URL, but because these are common traits shared with SPAM, they were enough to trigger the threshold and send the message to the junk folder.

Fixing these problems should take no more than a few hours at most and will help ensure messages from leaders make it to membership.

Thanks,

--Perry

[russellhltn]

It serves no technical or practical purpose to set Reply-To to the leader's email address, but to put the Return-Path (envelope sender) in the From header of the message. This discontinuity causes Spamassassin to demerit the email message from the leader.

The fix is to set the From and Reply-To to contain the email address of the leader.

I beg to differ. Many email servers will check the SPF record for the domain of the "From". If the actual server sending the email doesn't match the list of authorized servers for the "From" address, the email is discarded outright. It's not even put into the spam folder. AOL/Yahoo in particular is very strong about this. The church used to "forge" the "From" address and ran into problems.

To do what you suggest would require changing it so the email is actually sent from the leader's email. But that's likely to trigger other problems. Not the least that many users do not have their "MailTo:" feature setup to connect with their email system. And that's assuming the leader's email will cooperate in sending to a 100+ email addresses.

There has to be a way for a server to send "in behalf of". I thought it was by accurately stating the real sender in the "From" while using the "Reply to" to indicate where responses were to go.

[p.harrington]

My interpretation of SPF clearly states that the Envelope-from is evaluated only. DMARC is used with the Message-From to implement checking of the From.

I agree that the current technologies prevent leaders from effectively sending as their private address, however there is a mechanism which can mitigate this. Google uses Authenticated Received Chain to mitigate this issue.

In essence, a leader would send their email message to a special email address, that address would be a blind recipient list, the Church mail server would implement the ARC system, signing the forwarded message.

Some tricks would be needed to ensure the *modified* (anonymized) message was signed, because the entire message, sans ARC headers, is hashed.

This solves the problem of spoofing the original message and it solves the problem of forwarding the message, since the forwarded message would have ARC-Seal, ARC-Message-Signature, and ARC-Authentication-Results headers.

In effect, the Church mail server receives the leader's email, authenticates it via SPF, DKIM, and DMARC, signs their message assuring the next MTA that it is legitimate, then passes it on.

Alternatively, if they continue to do what they are doing, there is no way for the Church website to authentically send an email with a return address that does not belong to the Church. SPF will be valid (because the envelope sender is noreply@chuchofjesuschrist.org), but DMARC will not pass because the Message-From is different than the Envelope-From.

The most "right" way to solve this problem is to have a leader send an email to a unique email address @churchofjesuschrist.org, which is a blind mailing list to their membership.
The second most "right" way would be to have real email addresses for each leader that have a return address @churchofjesuschrist.org

The second solution requires relatively less effort, but still not negligible.
The first solution is the most standards compliant but needs to have proper security measures in place to prevent abuse. Either a unique address can be generated dynamically, or an obscure fixed address can be used with a whitelist of senders that can send to the list. This solution should also have some sanitizing steps to ensure no private or unintended data is forwarded.

The ARC method is technically the right solution, but it means more work. Creating "web mail" addresses for each leader can abstract their personal addresses and solve the problem too.

What I know is that the current solution is very problematic with SPAM detection. I personally have SPF, DKIM, DMARC, PGP signing, and TLS transports, but my emails get flagged as spam probably 40-50% of the time when emailing people.

--Perry

[jdlessley]

While this can be discussed ad infinitum in the forum it will get nothing done since those who are in a position to consider the topic discussed here most likely will never see the discussion. The best way to get this before those who can consider a proposal is to send feedback using the feedback link found in the footer of most churchofjesuschrist.org web pages. Since the topic is most likely the Send a Message app in Leader and Clerk Resources (LCR) then the link to use could be either the Do You Have Feedback on This Page? link at the bottom right of the LCR Send a Message page or the feedback link in the footer menu of that page.

[sbradshaw]

Thank you for all of your research – it's not uncommon to see reports here on the forum about LCR emails being filtered out, but it sounds like you understand the technical issues better than most of us, and I'm sure your expertise would be helpful to the developers. I think the feedback emails may have a character limit – if so, you could point the developers to refer to this thread.

[p.harrington]

I no longer have a calling that gives me access to the LCR, so I cannot submit feedback directly on those pages. I will try submitting feedback on the Access Denied page.

If indeed the people responsible are not reading this forum, then we just need to get them a link to this forum discussion. It doesn't matter who does that.

[russellhltn]

My interpretation of SPF clearly states that the Envelope-from is evaluated only. DMARC is used with the Message-From to implement checking of the From.

So, that would seem to suggest making the Envelope-from being from the church, but the Message-From being from the leader. The mis-match doesn't trigger anything? Which is used in the reply fuction?

I agree that the current technologies prevent leaders from effectively sending as their private address, however there is a mechanism which can mitigate this. Google uses Authenticated Received Chain to mitigate this issue.

In essence, a leader would send their email message to a special email address, that address would be a blind recipient list, the Church mail server would implement the ARC system, signing the forwarded message.

The issue there is that the current system is designed to allow the leader to customize the recipient list. That's currently being done to honor requests to not receive so many emails.

The second issue would be trying to get the leader to send a properly formatted message to make the system work. With the constant turnover, that's likely to become a full-time job for someone.

[davesudweeks]

I no longer have a calling that gives me access to the LCR, so I cannot submit feedback directly on those pages. I will try submitting feedback on the Access Denied page.

If indeed the people responsible are not reading this forum, then we just need to get them a link to this forum discussion. It doesn't matter who does that.

You can provide feedback from any page and indicate that it is in reference to the Send A Message tool on LCR. You should be able to provide a link to this forum topic in the feedback. The service missionaries who route the feedback will be able to get it to the right place. You may not get an individualized response, but I believe all feedback is read and considered.

[p.harrington]

My interpretation of SPF clearly states that the Envelope-from is evaluated only. DMARC is used with the Message-From to implement checking of the From.

So, that would seem to suggest making the Envelope-from being from the church, but the Message-From being from the leader. The mis-match doesn't trigger anything? Which is used in the reply fuction?

My server doesn't have DMARC checking enabled presently, so I don't have headers to examine. But DMARC should generate a warning due to the mismatch of sending host and From @domain.

The issue there is that the current system is designed to allow the leader to customize the recipient list. That's currently being done to honor requests to not receive so many emails.

The second issue would be trying to get the leader to send a properly formatted message to make the system work. With the constant turnover, that's likely to become a full-time job for someone.

I did email my bishop and let him know that attachments were causing his messages to go to SPAM, but the SPAM/notSPAM calculation is heavily weighted towards SPAM due to the issues I enumerated in my OP.

The mismatch of Reply-To/From count 2.5/5 points.

The invalid Message-ID counts 1.2/5 points.

So adding up, the technical issues I enumerated count 1.2+2.5=3.7/5 or 74% towards SPAM before a leader types ANYTHING into a message.

This means leaders are only allowed 1.3 points of leeway in drafting their messages.

Using a free email with a shortened URL counts 1.5 points, and a short subject with shortened URL is 1.5 points, bringing that to 3 points.

There is nothing improper or invalid about the message our Stake Presidency sent out, a subject of "Easter" and a shortened URL to YouTube is perfectly fine.

My point is simply that if we can fix the MIME attachment issue, the Reply-To/From mismatch, and Message-ID, while not perfect, it would give leaders MUCH more leeway in drafting their messages.

--Perry

[ambldsorg]

2. The Reply-To and the From header are not the same.

I don't think this is that critical. I'm not aware of any RFC, standard, or recommendation that Reply-to and From MUST be the same, and in fact, I think that would kind of defeat the purpose to have Reply-to the same as From in the first place. Reply-to is specified precisely in the case where the From address is not the correct place for a response to go. It would be senseless to have both Reply-to and From be the same address because an email client can already respond to the From header without any hints coming from a Reply-to. Some spam rating software might classify it as more suspicious, but that doesn't make it wrong to use.

I believe your other points are relevant and your analysis of the Message-Id and extremely disastrous violation of RFC 5322 and 5321 line lengths is extremely apropos [what better example of garbage in, garbage out can be presented?]. The LCR system still has this problem and I've seen a line that has over 11 million characters on it:

viewtopic.php?t=41439

I first noticed this problem in 2019 when I started getting corrupted attachments in emails that were sent from LCR. As it turns out, my email client was truncating the lines to the RFC specified length of 998 characters. This made the base64 decoding fail, and resulted in a garbage attachment. I have found others complaining of corrupted attachments sent via LCR in other threads. I have found others who have identified the problem correctly.

Here we are in 2024. I've opened tickets with Global Services Department, submitted feedback, and done just about everything available to the hoi polloi; not sure what else to do but come here and commiserate with others (as someone else in another thread said).

What can we do to help get this fixed?