Broadcast Email Is Unreliable

Share discussions around the Classic Local Unit Website (LUWS).
User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Wed Feb 11, 2009 9:36 am

dmaynes wrote:There is no indication of spam filtering (I'm not sure that I would see any).

If the issue is spam filtering, why would the filtering be applied for general broadcasts, but not individual e-mails?

You're right, you probably wouldn't see any indication. There are so many servers these emails probably go through before they get to the recipients. Filtering could be applied at any step along the way and in the case of spam filtering, it doesn't really get returned to sender. I'm sure there are some filtering implementations out there just drop the emails rather than delivering them to the user's Spam filter.

As for being applied to general broadcasts and not individual emails, I personally think the reason is obvious. General broadcasts are being delivered to any number between 5 and 500 email addresses at a time. Any email being delivered to anything over 20 or 25 users I'm sure qualifies it for running through spam detection logic.
dmaynes wrote:So the time on the e-mail server handling the general broadcasts is wrong.

This could be yet another reason for a spam filter to flag an email as a problem.
dmaynes wrote:I'm not getting the e-mail verification messages. I just think the server is dropping the request to send the broadcast, or some software packing/parsing bug is causing the request to not be created. That is the simplest explanation.

It's definitely a possibility. However, I disagree with it being the "simplest explanation". I think this problem would be the easiest to detect, but that is certainly not the same thing.

The potential problem you describe sounds like it would be a completely located only on the Church's servers, which would far easier to detect than any other proposed email issue that has been suggested thus far. I would think there would be some sort of logging in the Church's systems that would reveal your proposed problem almost instantly. This issue has obviously been dragging on for a long time, and that is why I would think the Church would have found the problem you proposed by now. The Church must have all kinds of logging and network monitoring set up to detect this kind of stuff.

So again, I second the notion that is likely an issue outside of the Church's servers. That is not to say that it is beyond the Church's control. There are many steps the Church could take to lessen the chance of these email broadcasts being flagged as spam. To summarize some of the more likely factors (and the ones the Church could actually do something about), the list would include the following in my opinion:
  • Sending a message to a large number of recipients
  • Sending from an account different from the sending domain
  • Discrepancy in time stamps (I imagine some filtering solutions would see an email with this problem as coming from a bogus email client or intentionally trying to be deceptive).

doughy
New Member
Posts: 21
Joined: Fri Oct 03, 2008 10:39 pm
Location: Salt Lake City, UT, USA

Postby doughy » Wed Feb 11, 2009 9:45 am

If this is a spam issue out of the Church's control, why does 1 out of 6 actually go through? Why would a spam filter mark the same exact email as spam one time, but 6 times later, let it through?

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Wed Feb 11, 2009 9:59 am

Doughy wrote:If this is a spam issue out of the Church's control, why does 1 out of 6 actually go through? Why would a spam filter mark the same exact email as spam one time, but 6 times later, let it through?

Excellent point...

The only somewhat reasonable explanation I can offer would be to quote Alan_Brown from a little earlier in the thread:
Alan_Brown wrote:Typically, the various potential violations are given ratings and weights and so a message ends up with a final score that determines if it is considered spam. Since there are so many factors involved in the rating, and they can even depend on other traffic through your ISP, it can seem somewhat random. It can even be the case that the exact same message gets filtered one day and sent through just fine on another.

The emails could be taking a different "route" on the Information Superhighway every time it's attempted to be delivered as well.

doughy
New Member
Posts: 21
Joined: Fri Oct 03, 2008 10:39 pm
Location: Salt Lake City, UT, USA

Postby doughy » Wed Feb 11, 2009 10:14 am

Has someone from the church's IT department given their mailing script a code review? I think that trying to diagnose the problem should at least start there.

doughy
New Member
Posts: 21
Joined: Fri Oct 03, 2008 10:39 pm
Location: Salt Lake City, UT, USA

Postby doughy » Wed Feb 11, 2009 10:35 am

I took a quick look at the headers for the individual emailing system, and the broadcast. Since the individual mailer seems to be working for everyone perfectly, it is interesting to see the differences in the headers.

Here is an individual email header:

Code: Select all

Delivered-To: doughywilson@gmail.com
Received: by 10.224.46.16 with SMTP id h16cs28827qaf;
        Wed, 11 Feb 2009 09:26:42 -0800 (PST)
Received: by 10.64.220.20 with SMTP id s20mr4697990qbg.47.1234373200842;
        Wed, 11 Feb 2009 09:26:40 -0800 (PST)
Return-Path: <DoughyWilson@gmail.com>
Received: from mail3.gmhwh.org (mail3.gmhwh.org [216.49.178.93])
        by mx.google.com with ESMTP id 28si304878qbw.23.2009.02.11.09.26.40;
        Wed, 11 Feb 2009 09:26:40 -0800 (PST)
Received-SPF: neutral (google.com: 216.49.178.93 is neither permitted nor denied by domain of DoughyWilson@gmail.com) client-ip=216.49.178.93;
Authentication-Results: mx.google.com; spf=neutral (google.com: 216.49.178.93 is neither permitted nor denied by domain of DoughyWilson@gmail.com) smtp.mail=DoughyWilson@gmail.com
Received: from chqpvua1283.ldschurch.org ([10.97.1.83])
   by mail6.gmhwh.org (8.14.1/8.14.1) with ESMTP id n1BHQd1L024321
   (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
   for <DoughyWilson@gmail.com>; Wed, 11 Feb 2009 10:26:40 -0700
Received: from chqpvuu4532 ([10.97.246.15])
   by chqpvua1283.ldschurch.org (8.14.1/8.14.1) with SMTP id n1BHQdTt019381
   for <DoughyWilson@gmail.com>; Wed, 11 Feb 2009 10:26:39 -0700
Message-Id: <200902111726.n1BHQdTt019381@chqpvua1283.ldschurch.org>
From: Joey Wilson <DoughyWilson@gmail.com>
To: <DoughyWilson@gmail.com>
Subject: Test
Date: Wed, 11 Feb 2009 10:26:39 MST
X-Proofpoint-Virus-Version-Internal: vendor=fsecure engine=1.12.7400:2.4.4,1.2.40,4.0.166 definitions=2009-02-11_02:2009-02-10,2009-02-11,2009-02-11 signatures=0
X-Proofpoint-Spam-Details-Internal-: safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.7400:2.4.4,1.2.40,4.0.166 definitions=2009-02-11_08:2009-02-10,2009-02-11,2009-02-11 signatures=0

Test message body here


Now, here is the header for a broadcast.

Code: Select all

Delivered-To: doughywilson@gmail.com
Received: by 10.224.46.16 with SMTP id h16cs28553qaf;
        Wed, 11 Feb 2009 09:23:38 -0800 (PST)
Received: by 10.64.241.18 with SMTP id o18mr1585470qbh.99.1234373018181;
        Wed, 11 Feb 2009 09:23:38 -0800 (PST)
Return-Path: <DoughyWilson@gmail.com>
Received: from mail2.gmhwh.org (mail2.gmhwh.org [216.49.178.92])
        by mx.google.com with ESMTP id p30si13593794qbp.2.2009.02.11.09.23.33;
        Wed, 11 Feb 2009 09:23:38 -0800 (PST)
Received-SPF: neutral (google.com: 216.49.178.92 is neither permitted nor denied by domain of DoughyWilson@gmail.com) client-ip=216.49.178.92;
Authentication-Results: mx.google.com; spf=neutral (google.com: 216.49.178.92 is neither permitted nor denied by domain of DoughyWilson@gmail.com) smtp.mail=DoughyWilson@gmail.com
Received: from chqpvua1283.ldschurch.org ([10.97.1.83])
   by mail5.gmhwh.org (8.14.1/8.14.1) with ESMTP id n1BHNS8h029131
   (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
   Wed, 11 Feb 2009 10:23:28 -0700
Received: from 10.98.40.114 ([10.97.246.15])
   by chqpvua1283.ldschurch.org (8.14.1/8.14.1) with SMTP id n1BHNRiP015719;
   Wed, 11 Feb 2009 10:23:27 -0700
Message-Id: <200902111723.n1BHNRiP015719@chqpvua1283.ldschurch.org>
[color=black]X-Mailer: TCL EMail Library 1.7[/color]
From: Joey Wilson <DoughyWilson@gmail.com>
Subject: Test
Date: Wed, 11 Feb 2009 10:23:27 -0600
X-Proofpoint-Virus-Version-Internal: vendor=fsecure engine=1.12.7400:2.4.4,1.2.40,4.0.166 definitions=2009-02-11_02:2009-02-10,2009-02-11,2009-02-11 signatures=0
X-Proofpoint-Spam-Details-Internal-: safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.7400:2.4.4,1.2.40,4.0.166 definitions=2009-02-11_08:2009-02-10,2009-02-11,2009-02-11 signatures=0

Test message body here


Notice a few differences. First, there is no "To:" in the header of the broadcast. Second, the broadcast has an "X-Mailer: TCL EMail Library 1.7", but the individual doesn't.

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Wed Feb 11, 2009 10:49 am

Doughy wrote:Notice a few differences. First, there is no "To:" in the header of the broadcast. Second, the broadcast has an "X-Mailer: TCL EMail Library 1.7", but the individual doesn't.

Also, the two "Received:" headers in the middle of the broadcast snippet don't have the statement that says "for <emailaddresshere@domain.com>", like the individual snippet does.

Both sets of headers have a "Received:" "from" <IP Address> "by" <some server> "with SMTP id" <crazy id string> and also a date time stamp. But the broadcast doesn't have a "for" statement (with an accompanying email address, that I assume is the sender of the broadcast or individual communication).

russellhltn
Community Administrator
Posts: 30311
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Postby russellhltn » Wed Feb 11, 2009 11:00 am

mkmurray wrote:There are so many servers these emails probably go through before they get to the recipients. Filtering could be applied at any step along the way and in the case of spam filtering, it doesn't really get returned to sender.


I disagree with that analysis. Ignoring complexities at the ends, there's only two mail servers - the sender and the receiver. The connection may be run though multiple routers, but emails do not "route" though the Internet like IP packets.

An individual sending a email will contact their ISP/mail vendor's outgoing mail server. That server in turn will contact the recipient's mail server based on the recipient's domain. If the mail was addressed to a .gmail recipient and a .hotmail recipient, then the sending server will connect to two different servers to send the message.

What is being described here is an intermittent problem where sending is either 100% or 0%. The only way you can have intermittent total failure is if the message never got sent. Aggressive spam filters on the recipients side can result in intermittent individual failures where only some people get the email, but can't account for total failure. I suppose a case could be made that all recipients subscribe to the same spam filtering software, but I have a hard time buying that.

In this case the church owns the sending mail server. The church's ISP only provides a connection, not mail service. So the only way I can see having a total failure is if the problem is within the church's network. Perhaps the problem is a spam filter - but it would be a spam filter on the sending mail server - owned by the church.
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.

So we can better help you, please edit your Profile to include your general location.

doughy
New Member
Posts: 21
Joined: Fri Oct 03, 2008 10:39 pm
Location: Salt Lake City, UT, USA

Postby doughy » Wed Feb 11, 2009 11:06 am

RussellHltn wrote:I disagree with that analysis. Ignoring complexities at the ends, there's only two mail servers - the sender and the receiver. The connection may be run though multiple routers, but emails do not "route" though the Internet like IP packets.

An individual sending a email will contact their ISP/mail vendor's outgoing mail server. That server in turn will contact the recipient's mail server based on the recipient's domain. If the mail was addressed to a .gmail recipient and a .hotmail recipient, then the sending server will connect to two different servers to send the message.

What is being described here is an intermittent problem where sending is either 100% or 0%. The only way you can have intermittent total failure is if the message never got sent. Aggressive spam filters on the recipients side can result in intermittent individual failures where only some people get the email, but can't account for total failure. I suppose a case could be made that all recipients subscribe to the same spam filtering software, but I have a hard time buying that.

In this case the church owns the sending mail server. The church's ISP only provides a connection, not mail service. So the only way I can see having a total failure is if the problem is within the church's network. Perhaps the problem is a spam filter - but it would be a spam filter on the sending mail server - owned by the church.


I agree 100% with Russell. I think this is a problem within the church's system. Can anyone confirm if someone from the church is actually investigating this problem?

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Wed Feb 11, 2009 11:13 am

RussellHltn wrote:I disagree with that analysis. Ignoring complexities at the ends, there's only two mail servers - the sender and the receiver. The connection may be run though multiple routers, but emails do not "route" though the Internet like IP packets.

Thank you for clearing up my misconceptions about email protocols.

dmaynes
Member
Posts: 233
Joined: Sat Nov 01, 2008 9:50 am
Location: Pleasant Grove, Utah

Postby dmaynes » Wed Feb 11, 2009 12:02 pm

mkmurray wrote:The potential problem you describe sounds like it would be a completely located only on the Church's servers, which would far easier to detect than any other proposed email issue that has been suggested thus far. I would think there would be some sort of logging in the Church's systems that would reveal your proposed problem almost instantly. This issue has obviously been dragging on for a long time, and that is why I would think the Church would have found the problem you proposed by now. The Church must have all kinds of logging and network monitoring set up to detect this kind of stuff.


I agree with your analysis, except if the software that is supposed to initiate the e-mail never initiates said e-mail, then it is very likely that the failure is not being logged. I believe this is what is happening.

I have not kept careful records, but to the best of my recollection whenever the "verification" e-mail is sent the main e-mail is also sent. I think a user reported earlier (Doughy?) some cases where the verification e-mail was sent but the main e-mail was never sent. If this is happening, it is not happening to the same degree as the error that we are discussing.


Return to “Classic Ward & Stake Sites (LUWS)”

Who is online

Users browsing this forum: No registered users and 1 guest