In T134886#2288880, @faidon wrote:

Yes, the full bounce would definitely be helpful. I could also check our logs for a delivery error (or a lack of thereof) in the Wikimedia-Gmail hand-off, which could tell us more. For that I'll need an address and approximate timestamp. It'd have to be for a relatively recent failure (we only keep short-term logs).

Thanks for offering to help. I've emailed you the details of all the bounces that people specifically complained about on Phabricator. Some of them are 6 days old, I don't know if that's recent enough, but there were also some that are only 2 days old. I also sent you the commands you can run to get a list of all gmail bounces in the past 24 hours (usually 10-20 per 24h period).

For what it's worth, I checked for delivery errors to Google's servers in general; there were a few 421s (temporary failure; rate limits or greylisting, usually) and ~1830 552s (permanent failures). The 552s had this for the delivery failure reason:

552-5.7.0 This message was blocked because its content presents a potential
552-5.7.0 security issue. Please visit
552-5.7.0 https://support.google.com/mail/answer/6590 to review our message
552 5.7.0 content and attachment content guidelines.

Note however that we use the same relays for most of our emails — thesecould just as easily be e.g. phishing sent to wikimedia.org aliases that expand into Gmail addresses. It's not necessarily indicative of wiki-mail failures.

Interesting. I wonder if there's something about the Special:Emailuser emails (or other types of emails for that matter) that we send that makes them look suspicious.

I checked the logs for all the emails Roan mailed me about (T134674, T134913, T134886 — 5 emails in total). In all of those cases, those were errors about various Yahoo domains, e.g.

550-5.7.1 Unauthenticated email from yahoo.es is not accepted due to domain's
550-5.7.1 DMARC policy. Please contact administrator of yahoo.es domain if this
550-5.7.1 was a legitimate mail. Please visit
550-5.7.1 https://support.google.com/mail/answer/2451690 to learn about DMARC
550 5.7.1 initiative.

The envelope From of those emails was wiki-<wiki>-<hash>@wikimedia.org, but I'm suspecting that the body's From was a Yahoo address (we're not logging that).

DMARC, an email authentication/antispam/antiphishing spec, introduced the concept of "aligned identifiers", which means that a provider may specify a policy where the envelope From must match the body's From and the connecting host for both should be allowed to mail from such a domain.

Yahoo is (in)famous for having a stringent DMARC policy on their domain, meaning that all emails with a body From @yahoo.com (and other ccTLDs) must have a corresponding @yahoo.com envelope From and can only be sent from a Yahoo mail exchange and signed with a Yahoo DKIM key, otherwise the receiving end should reject them. Gmail and various other providers enforce such declared DMARC policies (as they should), which results in those bounces. This Yahoo behavior also breaks traditional mailing lists, such as all of our mailing lists.

The DMARC FAQ addresses this but the TL;DR is that there is no easy solution for this. The only real solution would be for Wikimedia to set the body From to a @wikimedia.org (or similar, e.g. a subdomain) address and forward potential responses accordingly — or, in other words, always be on the path of the emails. That would, unfortunately, be a non-trivial project :(

In T134886#2291395, @faidon wrote:

[…]
Yahoo is (in)famous for having a stringent DMARC policy on their domain, meaning that all emails with a body From @yahoo.com (and other ccTLDs) must have a corresponding @yahoo.com envelope From and can only be sent from a Yahoo mail exchange and signed with a Yahoo DKIM key, otherwise the receiving end should reject them. Gmail and various other providers enforce such declared DMARC policies (as they should), which results in those bounces. This Yahoo behavior also breaks traditional mailing lists, such as all of our mailing lists.
[…]

I think I asked this once on another task, but it looks to me as if the problem occurs when users with a Yahoo mail address try to send mail via Wikipedia. In that case, we could add a check to MediaWiki that, if the user's mail address is covered by DMARC (or, if that test is too expensive, if the mail address ends in @yahoo.com, etc.), Special:EmailUser for that sending user is disabled and replaced with an informative message explaining why. This would not "fix" it, but avoid the black hole where mails just appear to vanish.

If I can add more data to the ticket:

1. I have received emails through WP since the original bounce (though I think we've moved past that part)
2. I can confirm that the original bounce was coming from a yahoo.de address
3. The last major incident I had with not receiving emails is documented at https://en.wikipedia.org/wiki/Wikipedia_talk:OUP#Oxford_Handbooks_Online In short, the same user (@Harrias) tried to email me multiple times and I never received it (even to this day—I imagine they gave up)

I may be suffering from this issue. I have been unable to send emails to other users on Commons for some months. When I send them, they do not receive the email nor do I get the copy email. But they do get a notification to say someone sent an email. Please can someone investigate why my emails don't work. Thanks

(IIUC) I think this is a duplicate of T66795: Email server's DMARC config prevents users from sending emails via Special:EmailUser, and so we should merge this task into that.
(and possibly this is also duplicated by T123068: Emails sent via EmailUser function not working for user on en.wp? The comments there are less clear.)

And then we should copy Faidon's excellent explanation above (T134886#2291395) into the description at T66795, as a TLDR.

And then file a new task, to work on the feature suggested at T66795 and also above by scfc (T134886#2291576). (and now written up at T137603: Block Special:EmailUser for people with a DMARC-problem address)

Is all of that correct?

Further to my earlier comment, I have managed to send one email to someone with their own personal email address (don't know who manages their domain for them) but failed for others on gmail or hotmail and I myself have a yahoo.co.uk address. So this is probably specific to the addresses. I didn't get a confirmation email, however.

What I think is unacceptable is that there appear to be problems with this for years, and yet nobody is notified or told there may be an issue with their emails. Just silence, and wondering if the recipient got it.

If the problem is spoofing the reply address, then simply use a no-reply address for that and include the user's email inside the message or header. We'll just have to copy/paste that to reply, but better than nothing. This is a serious issue, as the send-to-user may be the only way a user can privately contact an admin or 'crat on the site.

In T134886#2294703, @Colin wrote:

What I think is unacceptable is that there appear to be problems with this for years, and yet nobody is notified or told there may be an issue with their emails. Just silence, and wondering if the recipient got it.

But that's the reason the BounceHandler was developed and deployed to WMF wikis; to try and track these problems, and give some more insights. It's progress and visibility in the right direction.

It may be possible for us to improve some notifications; possibly telling both the sender that the email has bounced? I'm not sure if this is a privacy issue

I'm struggling to see what the reason for delay is. I get emails from wiki all the time, when people write to my talk page. So that works just fine. If certain domains of email address provider are rejecting your spoofed emails for wiki-user-email, or the email gets bounced, then just send a mail as you do for notifications, and include the address in the body. This is surely a few hours work?

My report:

I have tried to send emails via ru.wiki.

I use also the option to become the copy of my emails.The last 2-3 Email wasnt deliver to me as a copy. The users I sent my emails to havent reveived it too.

I wanted to send a test email to mylelf and got "Your registered email adress xxxxxx has been unsubscribed due to multiple message delivery failures. You can verify your email address again."

After it I have verified my email (again) and try to test it second time.
And again: "Your registered email adress xxxxxx has been unsubscribed due to multiple message delivery failures. You can verify your email address again."

I have verified it and Im not using the email service since this time.

In T134886#2311838, @AnnaMariaKoshka wrote:

I wanted to send a test email to mylelf and got "Your registered email adress xxxxxx has been unsubscribed due to multiple message delivery failures. You can verify your email address again."

After it I have verified my email (again) and try to test it second time.
And again: "Your registered email adress xxxxxx has been unsubscribed due to multiple message delivery failures. You can verify your email address again."

I think what would've happened is:

1. There were multiple > 5 bounces from your registered email id, and the bouncehandler extension decided to unsubscribe you (unconfrm your mail id)
2. You get the notification, verifies again.
3. Someone sends you a message in between, which again bounces - causing for the bouncehandler to unconfirm you again - the second time, it waits for only 1 more.
4. You tried sending an email again, and it cannot go through - as you are already unsubcribed.

Can someone take a look at the logs, and verify if things happened in the right order. Specially (3), which seems to be mysterious here.

In T134886#2311873, @01tonythomas wrote:

In T134886#2311838, @AnnaMariaKoshka wrote:

I wanted to send a test email to mylelf and got "Your registered email adress xxxxxx has been unsubscribed due to multiple message delivery failures. You can verify your email address again."

After it I have verified my email (again) and try to test it second time.
And again: "Your registered email adress xxxxxx has been unsubscribed due to multiple message delivery failures. You can verify your email address again."

I think what would've happened is:

1. There were multiple > 5 bounces from your registered email id, and the bouncehandler extension decided to unsubscribe you (unconfrm your mail id)
2. You get the notification, verifies again.
3. Someone sends you a message in between, which again bounces - causing for the bouncehandler to unconfirm you again - the second time, it waits for only 1 more.
4. You tried sending an email again, and it cannot go through - as you are already unsubcribed.

Can someone take a look at the logs, and verify if things happened in the right order. Specially (3), which seems to be mysterious here.

Your number 1.

Why can it happen?

I used email service from ru wiki and didnt got any problems earlier. The first time I have noticed, that I have not become the copy of my email was about 2 or max 3 days ago.

I just had the same problem on ptwiki. I received notification but no email. Asked somebody else to send a test email, and received both the notification and the email, so we figure that the problem might be in the bounced message having an email address within it and it bounced because of that.

Also see T58414: Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail

At the very least, would it be possible to send a bounce notification to the email sender, possibly with the option to resend the email? Otherwise the email is lost and both parties are out of luck. And if Yahoo isn't going to resolve their issue, shouldn't we warn applicable editors (who use email addresses that can bounce) about this bug?

In T134886#2368534, @czar wrote:

Also see T58414: Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail

At the very least, would it be possible to send a bounce notification to the email sender, possibly with the option to resend the email? Otherwise the email is lost and both parties are out of luck. And if Yahoo isn't going to resolve their issue, shouldn't we warn applicable editors (who use email addresses that can bounce) about this bug?

Okey. So here we have a problem. We are sending an email again to a bounced user. That has got higher chance of bouncing ? One thing which we can do is create an echo wiki only notification - which might be a solution!

Its not a yahho email problem

Also other email providers have problems - mail.ru, gmail

Above, someone suggests informing users their email failed and this is questioned "we are sending an email again to a bounced user". But I get emails just fine from other users and from the server (someone has written on my talk page, etc, etc). So in principle, you can send me emails just fine. The issue (for me at least) is that some recipients of my emails don't get them (but they get emails from other people via wiki just fine) and I don't get a copy email.

So notifying me that the email failed to send is possible. I can guess this from the lack of copy email, though others (newbies especially) may just wonder why their email wasn't responded to. I'm concerned that that this problem is not being investigated or the known issues dealt with. And if it concerns spoofing email addresses then I'm not particularly sympathetic as this sounds dodgy and not something I get from any other website.

In T134886#2291395, @faidon wrote:

The envelope From of those emails was wiki-<wiki>-<hash>@wikimedia.org, but I'm suspecting that the body's From was a Yahoo address (we're not logging that).

If From: is forged (=doesn't correspond to who the letter is actually from), that's more than legitimate a reason to bounce an e-mail.

There are additional fields to add some details on who se4nd and forwarded it like Sender,List, etc., yes. The problem is, Yahoo didn't authorize us to compose letters on behalf of its users - the fields don't solve that (DMARC does allow to forward letters as is). That's the conceptual reason behind the failure.

The DMARC FAQ addresses this but the TL;DR is that there is no easy solution for this. The only real solution would be for Wikimedia to set the body From to a @wikimedia.org (or similar, e.g. a subdomain) address and forward potential responses accordingly — or, in other words, always be on the path of the emails. That would, unfortunately, be a non-trivial project :(

This is the option 3.C in the aforementioned FAQ.

Since Mediawiki formats e-mail in such a way that users interact directly after the first transmission, the option 3.B looks like the way to go for us instead: "Replace From: address, set Reply-To: to message author"

@faidon, I failed to get a message through to Sealle@ruwiki about 2 days ago. Could you check this? I doubt either of our domains is yahoo - if it's not, there's more than one problematic domain.

Got a new feedback from IRC about it, with exactly the same behavior as described in the task description.
And an other user does not receive emails when he has checked the option "send me a copy".

Hello. Today an user (@Wladek92) sent me an email (I think that through Meta-Wiki Special:EmailUser, but can't really tell), or so Echo says; however I can't find it on any folder in my mailbox.

In T134886#2611796, @MarcoAurelio wrote:

Hello. Today an user (@Wladek92) sent me an email (I think that through Meta-Wiki Special:EmailUser, but can't really tell), or so Echo says; however I can't find it on any folder in my mailbox.

Hi all
Im user Wladek92 - Yes, mail was sent via Meta-Wiki Special:EmailUser but just to say my email is a yahoo email. / Christian/.

In T134886#2613891, @Wladek92 wrote:

Hi all
Im user Wladek92 - Yes, mail was sent via Meta-Wiki Special:EmailUser but just to say my email is a yahoo email. / Christian/.

I see that you've sent me another email, but I've not received it either (email notification via Echo again).

If it helps, this just happened when https://en.wikipedia.org/wiki/User:VG31 tried to send me (User:MusikAnimal) an email via Special:EmailUser (around 18:23 UTC, 20 October 2016). I am using Gmail. I tested Special:EmailUser using an alternate account and it worked, though.

Sometimes it works and sometimes it does. I got an email today.

This continues to be a problem. Email works about 50% of the time. I guess the question is should I simply turn off the email option until this is fixed?

In T134886#2798129, @Huji wrote:

We should also add the write WMF-related tag to this issue. My gut feeling is it is not a MediaWiki bug, but a WMF mail server issue.

Your gut feeling is incorrect. This is a MediaWiki bug.

It was an uneducated guess, so no surprise!

Since the end of September, I do not get any of the mailing list messages (I was subscribed t several of them, including wikimedia-l) but occasionally I get wiki-mail. Could it be related to this ticket?

In T134886#2821901, @Ymblanter wrote:

Since the end of September, I do not get any of the mailing list messages (I was subscribed t several of them, including wikimedia-l) but occasionally I get wiki-mail. Could it be related to this ticket?

I don't think so. Emails posted from wikis and emails posted from lists.wikimedia.org are two separate systems.

This bug is still active (at least for my account)

These same problem. In the 26.11.2016 User:ZUFAr‬ from ba-wp was send me e-mail (I see it in the notification list) -- but i not a received this (and no relevant items in spam-box). My e-mail is @gmail.com

As admin of wikimedia-ru mailing list I also see a lot of _working_ email-addresses were recently disabled due to the "delivery error." from @mail.ru servers (the largest e-mail provider in Russia)

Reason - "SMTP error from remote mail server after end of data: host mxs.mail.ru [94.100.180.150]: 550 5.7.1 This message was not accepted due to domain owner DMARC policy (RFC 7489)"

Maybe DMARC settings is broken or not configured correctly?

@Kaganer: See https://phabricator.wikimedia.org/T66795#2858497 - this task is closed as a duplicate.