Change 864819 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.40.0-wmf.13] Branch commit for wmf/1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864819

Change 864819 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.13] Branch commit for wmf/1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864819

Change 864888 had a related patch set uploaded (by TrainBranchBot; author: MediaWiki PreSync):

[operations/mediawiki-config@master] testwikis wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864888

Change 864888 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864888

Mentioned in SAL (#wikimedia-operations) [2022-12-06T04:02:28Z] <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-06T12:14:54Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-06T12:21:26Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-06T12:27:18Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 (duration: 05m 52s)

Not sure if T323542: Forbid new modern code from not targeting mobile should have been a train blocker, but if it was, it no longer is: the revert is merged into wmf.13.

FYI https://gerrit.wikimedia.org/r/865128 will need to be merged before the train can continue.

I take care of it.

Change 865154 had a related patch set uploaded (by TrainBranchBot; author: Chad Horohoe):

[operations/mediawiki-config@master] group0 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865154

Change 865154 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865154

Mentioned in SAL (#wikimedia-operations) [2022-12-06T19:19:35Z] <demon@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-07T13:34:45Z] <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-07T13:42:30Z] <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 (duration: 07m 45s)

Change 865733 had a related patch set uploaded (by TrainBranchBot; author: Chad Horohoe):

[operations/mediawiki-config@master] group1 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865733

Change 865733 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865733

Mentioned in SAL (#wikimedia-operations) [2022-12-07T20:13:49Z] <demon@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-07T20:20:53Z] <demon@deploy1002> Synchronized php: group1 wikis to 1.40.0-wmf.13 refs T320518 (duration: 07m 03s)

Hi, around the time this rolled to group 1, T324711: UnexpectedValueException: Parsoid does not support content model proofread-index started — ~4k exceptions an hour

I can be one of T320534: Put Parsoid output into the ParserCache on every edit or T320529: Configure VE backend to use Parsoid directly, instead of calling RESTbase or T322672: Make ParsoidHandler::wt2html write to parser cache instead of wmf.13

Change 866498 had a related patch set uploaded (by TrainBranchBot; author: Chad Horohoe):

[operations/mediawiki-config@master] group2 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866498

Change 866498 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866498

Mentioned in SAL (#wikimedia-operations) [2022-12-08T20:12:26Z] <demon@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.13 refs T320518

We're seeing a 3x increase in throttled logs since this rolled out. I suspect it's from Module "{module}" not loadable on target "{target}". (logstash) but can't be sure because the throttler is fully engaged on these messages since 2022-12-07@20:00Z.

To protect the Logstash cluster, we start dropping logs when the cluster sees 100 duplicate logs per second matching the template <type><channel><level><normalized_message>.

@Jdlrobson I think it's due to the module target changes.

Is it because a module is marked to target mobile but something in the tree is not? If that's the case, I think I can write a script to go through all of RL modules dependency tree and find the ones that can be easily converted without causing issues.

Didn't "mediawiki.action.view.metadata" switch? https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2022.12.08?id=SnDA84QBIFwrQt9tiW67 the only dependency is mediawiki.action.view.filepage which got fixed in the same patch

Ah, it seems the patches didn't make it to branch cut. Backporting.

Change 866469 had a related patch set uploaded (by Ladsgroup; author: Jdlrobson):

[mediawiki/core@wmf/1.40.0-wmf.13] File pages: Add mobile targets to modules that are silently being removed

https://gerrit.wikimedia.org/r/866469

Change 866469 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.13] File pages: Add mobile targets to modules that are silently being removed

https://gerrit.wikimedia.org/r/866469

Mentioned in SAL (#wikimedia-operations) [2022-12-08T23:02:10Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:866469|File pages: Add mobile targets to modules that are silently being removed (T324723 T320518)]]

Mentioned in SAL (#wikimedia-operations) [2022-12-08T23:03:58Z] <ladsgroup@deploy1002> ladsgroup and ladsgroup: Backport for [[gerrit:866469|File pages: Add mobile targets to modules that are silently being removed (T324723 T320518)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-12-08T23:11:23Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:866469|File pages: Add mobile targets to modules that are silently being removed (T324723 T320518)]] (duration: 09m 12s)

It's in a much better state now:
https://grafana.wikimedia.org/d/000000561/logstash?from=1670529261667&orgId=1&to=1670542372306&viewPanel=45

It'll be cleaned up more in the future. If it's still causing issues, ping us to start sampling.

In T320518#8455688, @Ladsgroup wrote:

It's in a much better state now:
https://grafana.wikimedia.org/d/000000561/logstash?from=1670529261667&orgId=1&to=1670542372306&viewPanel=45

The drop rate has settled to a 2x increase since the deployment. Thanks @Ladsgroup!

Are these logs revealing a deeper issue that is being worked on? The logging pipeline is not in immediate danger, but we would prefer to avoid normalizing the deviance.

Hey @colewhite I need a week's worth of data to address a deeper issue. I'm going to turn this logging off next week in time for the holidays. Feel free to +2 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/864722 as I wasn't sure if I can self merge that :-)

The train was rolled back because of T324801#8455874.

The blocker had a patch which needs to be backported. I can run the train at 13:00 UTC to move forward.

I'm backporting it asap

Change 866586 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] all wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866586

Change 866586 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866586

Mentioned in SAL (#wikimedia-operations) [2022-12-09T13:13:16Z] <hashar@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.13 refs T320518

In T320518#8455841, @Jdlrobson wrote:

Hey @colewhite I need a week's worth of data to address a deeper issue. I'm going to turn this logging off next week in time for the holidays.

Sounds like a plan. Thanks!

Mentioned in SAL (#wikimedia-operations) [2022-12-09T17:03:31Z] <claime> eventgate-analytics bumped to 30 replicas to absorb increased load - T320518