Page MenuHomePhabricator

Help contact bot owners about the end of HTTP access to the API
Closed, ResolvedPublic

Description

Assist with communications to reduce unpleasant surprises related to T105794: Insecure POST traffic.

@BBlack is providing lists of frequent users of HTTP; CLs are contacting them. The most centralized discussion point is enwiki's WP:BOTN board.

Event Timeline

@WhatamIdoing - would you like recaps/updates of account lists here in this ticket?

If that's considered sensitive information, then definitely not.

Well that's been a question. The raw logs with IP addresses are sensitive. Username lists have been sent to mailing lists in the past, e.g. https://lists.wikimedia.org/pipermail/wikitech-l/2015-June/081931.html .

As discussed in email, now that we're past the first deadline date and we've been posting username lists on public wikis anyways, will place further updates here.

From log of past 24H of insecure API accesses from logstash, taken at 2016-06-16 14:40 UTC:

New usernames (not in previous notification lists):

KalanBot
Hubertl
Say8har
Туча

Old usernames (previously notified, still making insecure requests in this recent log, ordered by largest request count first):

EmausBot
MerlBot
RileyBot
Theo's_Little_Bot
HarrivBOT
FacebookBot
Acebot
Ananthanns
Kellergassen_Niederösterreich_2016
HAL
Ботчо
BracketBot
LaSabiduria
Gerd_Leibrock
LivingBot
Curly_Turkey
Der.Traeumer
Bj.schoenmakers
Sz-iwbot
EdinBot
DschwenBot
Nuno_Tavares
BeneBot*
DVdm
BeriBot
Seppl2013
SpaceFactsBot
Kautilya3
Faebot
CatWatchBot
BOTzilla
DanmicholoBot
Compteur_d'éditions_(bot)

I've posted notes for the newest four.

Fixed CatWatchBot and hopefully the remaining tasks for DanmicholoBot

New usernames in the past 24H:

Raboe001

(I figure no point repeating the already-notified list every day, but checking for new names every day will keep from missing some that slip through the cracks on the inbetween the bigger list days).

3 day log (over the weekend, basically since the last update above on the 17th):

New usernames over the past 3 days:

Wdwdbot
Reports_bot
Jtcurses
Miniapolis
Fiwiki-tools-bot

Old names (already notified, still accessing):

EmausBot
MerlBot
RileyBot
Theo's_Little_Bot
HarrivBOT
FacebookBot
Acebot
PastilleBot
BOTzilla
Ananthanns
HAL
Ботчо
LaSabiduria
BracketBot
EdinBot
LivingBot
Nevit
Hubertl
Curly_Turkey
BeneBot*
Gerd_Leibrock
DschwenBot
Sz-iwbot
DVdm
Kautilya3
Faebot
Say8har
Bj.schoenmakers
オランウータン
Der.Traeumer
Compteur_d'éditions_(bot)
Seppl2013
HydrizBot
ZsergheiBot
LaninBot
Nuno_Tavares

Brandon/Sherry asked me to contact user Paulis for his bot Fkraus because he speaks German. I mailed him in German about it.

Since the last update (past ~4 days):

New usernames:

Electron_Bot
Pahles
KSFT
Amalthea_(bot)
Qsx753698
AlphamaBot

Previously-notified usernames:

MerlBot
RileyBot
Theo's_Little_Bot
EmausBot
HarrivBOT
FacebookBot
Ananthanns
Acebot
HAL
Ботчо
Bottine
EdinBot
BracketBot
LaSabiduria
Curly_Turkey
BOTzilla
ClemRutter
LivingBot
Nevit
DVdm
Sz-iwbot
BeneBot*
Der.Traeumer
Berthold_Werner
Faebot
Gerd_Leibrock
Kautilya3
PastilleBot
Rainbot
オランウータン
Compteur_d'éditions_(bot)
Bj.schoenmakers
SpaceFactsBot
DschwenBot
LaninBot

Hi. I'm really surprised that my bot (LivingBot) is still failing this. Is there any way to get more diagnostic info?

Hi. I'm really surprised that my bot (LivingBot) is still failing this. Is there any way to get more diagnostic info?

@Jarry1250 - The insecure accesses with account LivingBot have the User-Agent string Peachy MediaWiki Bot API Version 2.0 (alpha 8). Shortly before this post, there was a burst of 5 hits in the same minute starting at 2016-06-25T08:13:05.000Z (so a little over 2h before your post above), and then another isolated hit at 2016-06-25T10:18:33.000Z (about 2-3 minutes before your post above). The accesses are coming from Labs IP addresses, and are hitting enwiki. That's about all I can tell from logstash data.

@Jarry1250 - The insecure accesses with account LivingBot have the User-Agent string Peachy MediaWiki Bot API Version 2.0 (alpha 8). Shortly before this post, there was a burst of 5 hits in the same minute starting at 2016-06-25T08:13:05.000Z (so a little over 2h before your post above), and then another isolated hit at 2016-06-25T10:18:33.000Z (about 2-3 minutes before your post above). The accesses are coming from Labs IP addresses, and are hitting enwiki. That's about all I can tell from logstash data.

Great, thanks. Looks like it's a problem with the version of the framework I'm running. I'll look into it.

@Ladsgroup The issue with "Fkraus" bot by Paulis that we talked about at Wikimania is part of this ticket.

Okay, hopefully LivingBot is fixed now... let's see.

I've contact all of the new names in the list (Electron_Bot, Pahles, KSFT, Amalthea_(bot), Qsx753698, and AlphamaBot).

@Johan, if you are planning to run an announcement about this in Tech News soon, then may I recommend a link to whatever the most current list is, so that bot owners and script users will know which ones are still at risk?

Electron Bot is supposed to be fixed now.

Talked with Paulis, the owner of FKraus bot and xqt, pywikibot-Framework author. Paulis has reinstalled python and bot and runs pywikibot 2.0rc4. He says there are still a few errors but none that should be an issue for this bug anymore.

New lists from just the past 24H (shortly before this post):

New usernames:

Poudou99

Previously notified, still insecurely accessing:

EmausBot
MerlBot
RileyBot
Theo's_Little_Bot
HarrivBOT
FacebookBot
Acebot
Ananthanns
HAL
Ботчо
BracketBot
EdinBot
Curly_Turkey
DVdm
LaSabiduria
Faebot
Sz-iwbot
BeneBot*
BOTzilla
Der.Traeumer
Bj.schoenmakers
Compteur_d'éditions_(bot)

I've left a message for the new user today.

Also, the list finally seems to be getting smaller. @Fae ran into some problems with converting Noaabot, and I believe that a couple of other people are working on changes now, too.

For the future, it may be useful to use some kind of confirmation system (maybe like the one Commons uses for sysops who risk losing their status. They basically have to sign on a page to prove they have understood what's being asked of them, confirm they still want to be sysops etc.) In case privacy is a concern this can be done on a "more private" venue than a wiki page.

For the suggestion by Elitre, we could possibly use the "L" system here in Phabricator. Because that is what it does, let's people sign pages with custom content. Compare L2, L3, L4 etc..

If the interested parties can see a list of everyone who's already signed, then that's a possibility (although I can't promise that everyone will want to create a Phab account just to do that.)

That's a potentially useful system, and I'm glad to know that it exists.

I'm not sure that it's necessary for this particular project, though; whether they know what's asked of them or not, the API will be changing on the stated date.

It's for us: it's so that we know at a glance who's been contacted, who has acknowledged there's action required on their side, and who's still struggling with what and may use a hand.

The cutoff date is coming up tomorrow!

One more list update, from the past 48H:

New usernames not seen before:

HWY_Shield_Bot
Galaxies00
H2Bot

Previously-notified:

MerlBot
RileyBot
EmausBot
Theo's_Little_Bot
W2Bot
HarrivBOT
Ananthanns
Acebot
HAL
Ботчо
EdinBot
BracketBot
Nevit
Gerd_Leibrock
Curly_Turkey
BeneBot*
LaSabiduria
Sz-iwbot
BOTzilla
Der.Traeumer
Reports_bot
Faebot
HydrizBot
Compteur_d'éditions_(bot)
Bj.schoenmakers
DVdm
LaninBot
Kautilya3

I've contacted the newest three. I'm going to post general messages to all the WP:BOTN pages and a few VPTs as well. Brandon, you're likely to get pinged in every one of those messages.

@Whatamidoing-WMF Thanks! I'm still getting caught up a bit from being on vacation....

The original plan (and still the current publicly-announced plan!) was to cut off all insecure access tomorrow. My current thinking is it's probably prudent to give one more week of grace time just for our internal labs networks (but not the outside world), since I haven't been here during the final week before the cutoff to help push things along. That exception would include the notable Merlbot case. I'm going to do a little more digging on the data first and upload the intended technical changes (for deployment tomorrow) sometime in the next couple of hours, before communicating any of that more-clearly and/or over on WP:BOTN.

What should I do to make my bot (AryanBot) is not broken?

Quoting a linked list message:

Either of the above solutions may be tested immediately, you'll know it
works because you stop seeing the warning.

Change 298336 had a related patch set uploaded (by BBlack):
Insecure POST: 20% fail for labs, 100% for external

https://gerrit.wikimedia.org/r/298336

The patch link above is pretty self-descriptive, and I'm planning to deploy that tomorrow. Will update WP:BOTN with a link to this as well.

Change 298336 merged by BBlack:
Insecure POST: 20% fail for labs, 100% for external

https://gerrit.wikimedia.org/r/298336

Change 299532 had a related patch set uploaded (by BBlack):
insecure post: 100% failure, loophole closed

https://gerrit.wikimedia.org/r/299532

@Racso is the owner of BOTzilla. Is that bot still using insecure HTTP?

Hello. I'm the owner of BOTzilla. As far as I know, the bot is currently inactive. Is it actually doing something with insecure HTTP? Thanks!

The final patch to block insecure HTTP is going out a few hours from now, so if the bot is inactive you don't really have to worry about it from this perspective.

Change 299532 merged by BBlack:
insecure post: 100% failure, loophole closed

https://gerrit.wikimedia.org/r/299532