Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

service.version: 1.41.0-wmf.27
trace.id: 147bc53a-3478-4166-80db-919af19d86ba
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522

error.stack_trace

from /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MemcachedBagOStuff.php(109)
#0 /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MemcachedBagOStuff.php(120): MemcachedBagOStuff->validateKeyEncoding(string)
#1 /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MemcachedPeclBagOStuff.php(182): MemcachedBagOStuff->validateKeyAndPrependRoute(string)
#2 /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): MemcachedPeclBagOStuff->doGet(string, integer)
#3 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/CentralAuthUtilityService.php(96): MediumSpecificBagOStuff->get(string)
#4 [internal function]: MediaWiki\Extension\CentralAuth\CentralAuthUtilityService::MediaWiki\Extension\CentralAuth\{closure}()
#5 /srv/mediawiki/php-1.41.0-wmf.27/vendor/wikimedia/wait-condition-loop/src/WaitConditionLoop.php(96): call_user_func(Closure)
#6 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/CentralAuthUtilityService.php(107): Wikimedia\WaitConditionLoop->invoke()
#7 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/Special/SpecialCentralLogin.php(241): MediaWiki\Extension\CentralAuth\CentralAuthUtilityService->getKeyValueUponExistence(MemcachedPeclBagOStuff, string)
#8 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/Special/SpecialCentralLogin.php(91): MediaWiki\Extension\CentralAuth\Special\SpecialCentralLogin->doLoginComplete(string)
#9 /srv/mediawiki/php-1.41.0-wmf.27/includes/specialpage/SpecialPage.php(721): MediaWiki\Extension\CentralAuth\Special\SpecialCentralLogin->execute(string)
#10 /srv/mediawiki/php-1.41.0-wmf.27/includes/specialpage/SpecialPageFactory.php(1622): MediaWiki\SpecialPage\SpecialPage->run(string)
#11 /srv/mediawiki/php-1.41.0-wmf.27/includes/MediaWiki.php(354): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#12 /srv/mediawiki/php-1.41.0-wmf.27/includes/MediaWiki.php(953): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.41.0-wmf.27/includes/MediaWiki.php(601): MediaWiki->main()
#14 /srv/mediawiki/php-1.41.0-wmf.27/index.php(50): MediaWiki->run()
#15 /srv/mediawiki/php-1.41.0-wmf.27/index.php(46): wfIndexMain()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}

Impact

Notes

Probably triggered by tempering with URLs. Reproducible with GET request.

Details

Request URL: https://www.mediawiki.org/wiki/Special:CentralLogin/complete?token=*

Subject	Repo	Branch	Lines +/-
Drop "$wgCentralAuthDatabase:$keygroup:$components..." format (p.3)	mediawiki/extensions/CentralAuth	master	+43 -145
Try reading from newKey first before falling back to oldKey (p.2)	mediawiki/extensions/CentralAuth	master	+38 -36
Fix centralauthtoken key schema migration	mediawiki/extensions/CentralAuth	master	+1 -4
Begin migrating key formats for cache key lookups or writes (p.1)	mediawiki/extensions/CentralAuth	master	+181 -48
Fix centralauthtoken key schema migration	mediawiki/extensions/CentralAuth	wmf/1.42.0-wmf.4	+1 -4

Customize query in gerrit

Related Objects

Mentioned In: T350723: GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`
Mentioned Here: T350723: GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`

Event Timeline

Ammarpad created this task.Sep 23 2023, 7:09 AM

Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptSep 23 2023, 7:09 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Ammarpad updated the task description. (Show Details)Sep 23 2023, 7:17 AM

Krinkle moved this task from Untriaged to Sep 2023 on the Wikimedia-production-error board.Sep 25 2023, 1:17 PM

Bmueller moved this task from Inbox, needs triage to Current Sprint on the MediaWiki-Platform-Team board.Sep 25 2023, 1:31 PM

Bmueller assigned this task to DAlangi_WMF.Sep 25 2023, 1:36 PM

Change 960643 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Escape/normalize cache key before retreiving data from cache backend

https://gerrit.wikimedia.org/r/960643

gerritbot added a project: Patch-For-Review.Sep 25 2023, 3:44 PM

DAlangi_WMF changed the task status from Open to In Progress.Sep 28 2023, 4:21 PM

larissagaulia moved this task from Current Sprint to Blocked/waiting on the MediaWiki-Platform-Team board.Oct 9 2023, 1:41 PM

Change 961794 had a related patch set uploaded (by Krinkle; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Drop support for: "$wgCentralAuthDatabase:$keygroup:$components..." key

https://gerrit.wikimedia.org/r/961794

DAlangi_WMF moved this task from Blocked/waiting to Current Sprint on the MediaWiki-Platform-Team board.Oct 23 2023, 8:46 AM

Change 967880 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Try reading from newKey first before falling back to oldKey (p.2)

https://gerrit.wikimedia.org/r/967880

Change 961794 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Drop "$wgCentralAuthDatabase:$keygroup:$components..." format (p.3)

https://gerrit.wikimedia.org/r/961794

Change 960643 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Begin migrating key formats for cache key lookups or writes (p.1)

https://gerrit.wikimedia.org/r/960643

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Nov 6 2023, 6:00 AM

We should probably test the changes (all three patches) after they are deployed by the train. I think a test run on Tuesday (ie. after new code got deployed to group0) would look something like this:

log in on some group 0 wiki (let's say akwiki)
check that shared cookies work by checking in on a wiki on the same second-level domain (e.g. howiki)
check that central login works and edge login works by checking that you are logged in at another group 0 wiki that is edge login enabled (e.g. mediawikiwiki) and a group 1 wiki that's edge login enabled (e.g. metawiki)
check that central autologin works by logging in on some group 0 wiki that is not edge login enabled (e.g. wikimania2016wiki) and some group 1 wiki that's not edge login enabled (e.g. outreachwiki)
get a token using action=centralauthtoken in Special:ApiSandbox, use it to authenticate an action API request and a REST API request. (Note the token is only valid for 10 sec.) The REST API doesn't have any endpoint where the authenticated and the unathenticated response would differ, but an incorrect token will result in an error, so it's still testable.

A test on Wednesday would look like this:

repeat the steps above (to test new code <-> new code communication)
repeat the steps above again, but using group 2 wikis instead of group 0 wikis (to test new code <-> old code communcation). There is no group2 that isn't edge login enabled so that part can be skipped.

There is no need to test on Thursday, the previous two tests should already cover all the possible code pathes with all the possible old code / new code combinations.

(Note that these tests require third-party cookie blocking to be disabled in the browser (default for some browsers, but not for all). There is a guide on how the various authentication flows work, and debugging instructions (which are not very user-friendly and shouldn't be needed, but just in case).)

Tgr mentioned this in T350723: GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`.Nov 7 2023, 8:07 PM

Change 972464 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972464

Change 972464 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972464

Change 972262 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@wmf/1.42.0-wmf.4] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972262

Change 972262 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.42.0-wmf.4] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972262

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:45:03Z] <tgr@deploy2002> Started scap: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:46:23Z] <tgr@deploy2002> tgr: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:58:20Z] <tgr@deploy2002> Finished scap: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]] (duration: 13m 17s)

Aboodhassanio subscribed.Nov 7 2023, 10:30 PM

Change 967880 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Try reading from newKey first before falling back to oldKey (p.2)

https://gerrit.wikimedia.org/r/967880

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.5; 2023-11-14); removed MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Nov 13 2023, 1:00 AM

Change 961794 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Drop "$wgCentralAuthDatabase:$keygroup:$components..." format (p.3)

https://gerrit.wikimedia.org/r/961794

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.7; 2023-11-28); removed MW-1.42-notes (1.42.0-wmf.5; 2023-11-14).Nov 19 2023, 7:00 AM

larissagaulia moved this task from Current Sprint to Blocked/waiting on the MediaWiki-Platform-Team board.Nov 20 2023, 3:41 PM

(update): The last patch on the chain of patches is riding the train this week. At this point, the migration is already complete and we're dropping support for the old key format. No incidents so far and things are working as expected.

By tomorrow, the aforementioned patch will hit group 2 wikis (by end of day) and if everywhere is still silent, I'll go ahead and resolve this ticket. Also, logstash looks clear but I'll like to hear from @Ammarpad for further confirmation ☺️ on logstash from their side.

In T347223#9352116, @DAlangi_WMF wrote:

(update): The last patch on the chain of patches is riding the train this week. At this point, the migration is already complete and we're dropping support for the old key format. No incidents so far and things are working as expected.

By tomorrow, the aforementioned patch will hit group 2 wikis (by end of day) and if everywhere is still silent, I'll go ahead and resolve this ticket. Also, logstash looks clear but I'll like to hear from @Ammarpad for further confirmation ☺️ on logstash from their side.

This will happen next week with wmf.7; there's no train this week.

This will happen next week with wmf.7; there's no train this week.

Thanks for the indication @Jdforrester-WMF. I missed that entirely. Next week then!

All clear, this can be resolved now. Thanks for the collaboration @Tgr, @Krinkle and @Jdforrester-WMF.

Most especially, thanks @Ammarpad for reporting.

Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522Closed, ResolvedPublicPRODUCTION ERRORActions

Description

Error

Impact

Notes

Details

Related Objects

Event Timeline

Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522
Closed, ResolvedPublicPRODUCTION ERROR
Actions