Stake and Ward leader emails are going to SPAM folder

Some discussions just don't fit into a well defined box. Use this forum to discuss general topics and issues revolving around the Church and the technology offerings we use and share.
p.harrington
New Member
Posts: 6
Joined: Mon Mar 04, 2019 12:47 am

Stake and Ward leader emails are going to SPAM folder

Postby p.harrington » Sun Apr 12, 2020 7:48 pm

Hello all,

I don't know of a better way to bring this issue to the attention of the appropriate people, but with the new Social Distancing guidelines in place our Stake and Ward leaders have been heavily using the website emailing tools.

The problem is that through a combination of old technology and how headers are formed for the emails, the messages from leaders are already 80% of the way towards being marked as SPAM.

I use Spamassassin to analyze email that comes into my server, that is combined with Thunderbird's internal junk filtering to produce a pretty good spam filter. I run my own mail server and my email address has been floating around the Internet for > 20 years, so I get a lot of spam.

I choose to run my own server because services like Gmail are even worse at false positives and place draconian restrictions on what can be sent to and from their email boxes.

With my reasoning out of the way, let me show you what the Spamassassin report for a recent Stake Presidency email looked like:

Code: Select all

X-Spam-Report:
   * -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at
   *      https://www.dnswl.org/, no trust
   *      [216.49.179.112 listed in list.dnswl.org]
   * -0.0 SPF_PASS SPF: sender matches SPF record
   *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
   *  0.0 HTML_MESSAGE BODY: HTML included in message
   * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
   *      author's domain
   * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from
   *      envelope-from domain
   *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
   *       valid
   * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
   *  1.2 INVALID_MSGID Message-Id is not valid, according to RFC 2822
   *  1.5 PDS_FREEMAIL_REPLYTO_URISHRT Freemail replyto with URI
   *      shortener
   *  2.5 FREEMAIL_FORGED_REPLYTO Freemail in Reply-To, but not From
   *  1.5 PDS_TINYSUBJ_URISHRT Short subject with URL shortener


The body of the email read:

Code: Select all

Dear Brothers and Sisters of the Stake,

    As we take time in our homes this weekend to celebrate the sacrifice and atonement of the Lord Jesus Christ, may we be inspired to more fully understand and appreciated the Savior and the true meaning of the Easter season.
   The following Church video may help in this endeavor.

    We love and miss all of you, and hope to be able to meet together soon.

Sincerely, the Stake Presidenc

https://youtu.be/u0PmMX27LgE


I will not post the headers because I don't want to identify the person who sent the message.

There are 3 problems with how the Church website sends emails and these problems can be rectified, greatly reducing the penalty while sending messages:

1. The Message-Id header is not RFC compliant, it reads:

Code: Select all

Message-ID: <ESM_Email_Serivce.JavaMail.dc0df918-2c6c-4d28-af45-36d2c8d1b14f>

A proper RFC-2822 Message-ID header will look like this:

Code: Select all

Message-ID: <87DErSlXSuKxNKX44HtERg@mail.example.com>


The Message-ID is being generated by the JavaMail MUA that is sending the messages to membership, either the MUA (Mail User Agent) needs to be changed to a newer compliant Java Class, or the Message-ID needs to be explicitly set with a properly formatted header.

2. The Reply-To and the From header are not the same.

When a leader sends a message through the Church website the From header is modified to use <noreply@churchofjesuschrist.org> instead of their actual email address, yet the Reply-To header is set to the actual email address of the leader.

The RFC standard states that email replies should be sent based on this priority:

Reply-To
From
Return-Path

Error messages should be sent to the Return-Path, known as the envelope sender.

It serves no technical or practical purpose to set Reply-To to the leader's email address, but to put the Return-Path (envelope sender) in the From header of the message. This discontinuity causes Spamassassin to demerit the email message from the leader.

The fix is to set the From and Reply-To to contain the email address of the leader.

If a message is sent with no reply intended, then the From should contain <noreply@churchofjesuschrist.org>, the Return-Path should contain <noreply@churchofjesuschrist.org>, and the Reply-To should be omitted.

3. Attachments sent in broadcast emails are not broken up into lines of 80 characters each, they are sent as a single line.

Attachments are encoded in a text format which is called MIME encoding. It's a way of taking a file like a PDF, which contains non-printable characters (binary), and encoding it into characters which are technically printable (0-9, A-Z, etc).

The MIME encoding process takes an attachment as input and produces a long string of text as output, this long string of text should be broken up into lines that are shorter than 80 characters when embedded in an email message.

Because the MIME attachment is not broken up into lines, it is considered a characteristic commonly shared with SPAM and counts negatively towards the SPAM score.

The fix is to simply split the long MIME attachment into multiple lines with a CRLF (Carriage Return + Line Feed) delimiter between each line.

The reason why these 3 technical flaws are so critical is that it gives leader no leeway when drafting emails. The cited message simply included a short subject and a shortened YouTube URL, but because these are common traits shared with SPAM, they were enough to trigger the threshold and send the message to the junk folder.

Fixing these problems should take no more than a few hours at most and will help ensure messages from leaders make it to membership.

Thanks,

--Perry

russellhltn
Community Administrator
Posts: 30532
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Re: Stake and Ward leader emails are going to SPAM folder

Postby russellhltn » Sun Apr 12, 2020 10:26 pm

p.harrington wrote:It serves no technical or practical purpose to set Reply-To to the leader's email address, but to put the Return-Path (envelope sender) in the From header of the message. This discontinuity causes Spamassassin to demerit the email message from the leader.

The fix is to set the From and Reply-To to contain the email address of the leader.

I beg to differ. Many email servers will check the SPF record for the domain of the "From". If the actual server sending the email doesn't match the list of authorized servers for the "From" address, the email is discarded outright. It's not even put into the spam folder. AOL/Yahoo in particular is very strong about this. The church used to "forge" the "From" address and ran into problems.

To do what you suggest would require changing it so the email is actually sent from the leader's email. But that's likely to trigger other problems. Not the least that many users do not have their "MailTo:" feature setup to connect with their email system. And that's assuming the leader's email will cooperate in sending to a 100+ email addresses.

There has to be a way for a server to send "in behalf of". I thought it was by accurately stating the real sender in the "From" while using the "Reply to" to indicate where responses were to go.
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.

So we can better help you, please edit your Profile to include your general location.

p.harrington
New Member
Posts: 6
Joined: Mon Mar 04, 2019 12:47 am

Re: Stake and Ward leader emails are going to SPAM folder

Postby p.harrington » Wed Apr 15, 2020 12:20 am

My interpretation of SPF clearly states that the Envelope-from is evaluated only. DMARC is used with the Message-From to implement checking of the From.

I agree that the current technologies prevent leaders from effectively sending as their private address, however there is a mechanism which can mitigate this. Google uses Authenticated Received Chain to mitigate this issue.

In essence, a leader would send their email message to a special email address, that address would be a blind recipient list, the Church mail server would implement the ARC system, signing the forwarded message.

Some tricks would be needed to ensure the *modified* (anonymized) message was signed, because the entire message, sans ARC headers, is hashed.

This solves the problem of spoofing the original message and it solves the problem of forwarding the message, since the forwarded message would have ARC-Seal, ARC-Message-Signature, and ARC-Authentication-Results headers.

In effect, the Church mail server receives the leader's email, authenticates it via SPF, DKIM, and DMARC, signs their message assuring the next MTA that it is legitimate, then passes it on.

Alternatively, if they continue to do what they are doing, there is no way for the Church website to authentically send an email with a return address that does not belong to the Church. SPF will be valid (because the envelope sender is noreply@chuchofjesuschrist.org), but DMARC will not pass because the Message-From is different than the Envelope-From.

The most "right" way to solve this problem is to have a leader send an email to a unique email address @churchofjesuschrist.org, which is a blind mailing list to their membership.
The second most "right" way would be to have real email addresses for each leader that have a return address @churchofjesuschrist.org

The second solution requires relatively less effort, but still not negligible.
The first solution is the most standards compliant but needs to have proper security measures in place to prevent abuse. Either a unique address can be generated dynamically, or an obscure fixed address can be used with a whitelist of senders that can send to the list. This solution should also have some sanitizing steps to ensure no private or unintended data is forwarded.

The ARC method is technically the right solution, but it means more work. Creating "web mail" addresses for each leader can abstract their personal addresses and solve the problem too.

What I know is that the current solution is very problematic with SPAM detection. I personally have SPF, DKIM, DMARC, PGP signing, and TLS transports, but my emails get flagged as spam probably 40-50% of the time when emailing people.

--Perry

jdlessley
Community Moderators
Posts: 8468
Joined: Sun Mar 16, 2008 11:30 pm
Location: USA, TX

Re: Stake and Ward leader emails are going to SPAM folder

Postby jdlessley » Wed Apr 15, 2020 3:18 am

While this can be discussed ad infinitum in the forum it will get nothing done since those who are in a position to consider the topic discussed here most likely will never see the discussion. The best way to get this before those who can consider a proposal is to send feedback using the feedback link found in the footer of most churchofjesuschrist.org web pages. Since the topic is most likely the Send a Message app in Leader and Clerk Resources (LCR) then the link to use could be either the Do You Have Feedback on This Page? link at the bottom right of the LCR Send a Message page or the feedback link in the footer menu of that page.
JD Lessley
Have you tried finding your answer on the ChurchofJesusChrist.org Help Center?

User avatar
sbradshaw
Community Moderators
Posts: 5281
Joined: Mon Sep 26, 2011 8:42 pm
Location: Utah
Contact:

Re: Stake and Ward leader emails are going to SPAM folder

Postby sbradshaw » Wed Apr 15, 2020 9:25 am

Thank you for all of your research – it's not uncommon to see reports here on the forum about LCR emails being filtered out, but it sounds like you understand the technical issues better than most of us, and I'm sure your expertise would be helpful to the developers. I think the feedback emails may have a character limit – if so, you could point the developers to refer to this thread.
Samuel Bradshaw • If you desire to serve God, you are called to the work.

p.harrington
New Member
Posts: 6
Joined: Mon Mar 04, 2019 12:47 am

Re: Stake and Ward leader emails are going to SPAM folder

Postby p.harrington » Wed Apr 15, 2020 12:10 pm

I no longer have a calling that gives me access to the LCR, so I cannot submit feedback directly on those pages. I will try submitting feedback on the Access Denied page.

If indeed the people responsible are not reading this forum, then we just need to get them a link to this forum discussion. It doesn't matter who does that.

russellhltn
Community Administrator
Posts: 30532
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Re: Stake and Ward leader emails are going to SPAM folder

Postby russellhltn » Wed Apr 15, 2020 1:02 pm

p.harrington wrote:My interpretation of SPF clearly states that the Envelope-from is evaluated only. DMARC is used with the Message-From to implement checking of the From.

So, that would seem to suggest making the Envelope-from being from the church, but the Message-From being from the leader. The mis-match doesn't trigger anything? Which is used in the reply fuction?


p.harrington wrote:I agree that the current technologies prevent leaders from effectively sending as their private address, however there is a mechanism which can mitigate this. Google uses Authenticated Received Chain to mitigate this issue.

In essence, a leader would send their email message to a special email address, that address would be a blind recipient list, the Church mail server would implement the ARC system, signing the forwarded message.

The issue there is that the current system is designed to allow the leader to customize the recipient list. That's currently being done to honor requests to not receive so many emails.

The second issue would be trying to get the leader to send a properly formatted message to make the system work. With the constant turnover, that's likely to become a full-time job for someone.
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.

So we can better help you, please edit your Profile to include your general location.

davesudweeks
Senior Member
Posts: 2058
Joined: Sun May 09, 2010 8:16 pm
Location: Oklahoma, USA

Re: Stake and Ward leader emails are going to SPAM folder

Postby davesudweeks » Wed Apr 15, 2020 1:05 pm

p.harrington wrote:I no longer have a calling that gives me access to the LCR, so I cannot submit feedback directly on those pages. I will try submitting feedback on the Access Denied page.

If indeed the people responsible are not reading this forum, then we just need to get them a link to this forum discussion. It doesn't matter who does that.

You can provide feedback from any page and indicate that it is in reference to the Send A Message tool on LCR. You should be able to provide a link to this forum topic in the feedback. The service missionaries who route the feedback will be able to get it to the right place. You may not get an individualized response, but I believe all feedback is read and considered.

p.harrington
New Member
Posts: 6
Joined: Mon Mar 04, 2019 12:47 am

Re: Stake and Ward leader emails are going to SPAM folder

Postby p.harrington » Wed Apr 15, 2020 2:31 pm

russellhltn wrote:
p.harrington wrote:My interpretation of SPF clearly states that the Envelope-from is evaluated only. DMARC is used with the Message-From to implement checking of the From.

So, that would seem to suggest making the Envelope-from being from the church, but the Message-From being from the leader. The mis-match doesn't trigger anything? Which is used in the reply fuction?


My server doesn't have DMARC checking enabled presently, so I don't have headers to examine. But DMARC should generate a warning due to the mismatch of sending host and From @domain.

russellhltn wrote:The issue there is that the current system is designed to allow the leader to customize the recipient list. That's currently being done to honor requests to not receive so many emails.

The second issue would be trying to get the leader to send a properly formatted message to make the system work. With the constant turnover, that's likely to become a full-time job for someone.


I did email my bishop and let him know that attachments were causing his messages to go to SPAM, but the SPAM/notSPAM calculation is heavily weighted towards SPAM due to the issues I enumerated in my OP.

The mismatch of Reply-To/From count 2.5/5 points.

The invalid Message-ID counts 1.2/5 points.

So adding up, the technical issues I enumerated count 1.2+2.5=3.7/5 or 74% towards SPAM before a leader types ANYTHING into a message.

This means leaders are only allowed 1.3 points of leeway in drafting their messages.

Using a free email with a shortened URL counts 1.5 points, and a short subject with shortened URL is 1.5 points, bringing that to 3 points.

There is nothing improper or invalid about the message our Stake Presidency sent out, a subject of "Easter" and a shortened URL to YouTube is perfectly fine.

My point is simply that if we can fix the MIME attachment issue, the Reply-To/From mismatch, and Message-ID, while not perfect, it would give leaders MUCH more leeway in drafting their messages.

--Perry


Return to “General Discussions”

Who is online

Users browsing this forum: No registered users and 0 guests