Also at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/964599 when trying to cherry pick to wmf.30. The same patch passed fine yesterday on master https://gerrit.wikimedia.org/r/c/mediawiki/core/+/962724

Not sure if this is (almost) the same as T313775: MinervaNeue "Wikitext Editor.Closing editor (overlay button)" test is flaky?

In T348539#9260801, @Michael wrote:

Not sure if this is (almost) the same as T313775: MinervaNeue "Wikitext Editor.Closing editor (overlay button)" test is flaky?

Underlying issue is the same any way.

This is happening at greater frequency now, adding @zeljkofilipin who will know better

Having this affect a good chunk of test / gate-and-submit runs for changes on the ReportIncident extension.

There's also recent repeated failure in Math repo because of this. It seems no one is interested in fixing it. I think it should be disabled like the other test in the suite T313775 which was disabled with even lower failure rate.

Occurring on https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium-docker/73752/console

From the associated assets it looks like VisualEditor is not loading here (or is loading slowly) and back button is disabled here. Might be worth getting the editing team to investigate this separately. Selenium seems to be doing what it should do here.

IN terms of the test it should be easy to modify it for a short term fix to either:

1. check for the slow button in addition to the editor (which it's currently waiting on)
2. Increase the waitUntil condition time.

Change 970454 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@master] Account for load basic button

https://gerrit.wikimedia.org/r/970454

Change 970454 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Account for load basic button

https://gerrit.wikimedia.org/r/970454

Change 970759 had a related patch set uploaded (by Reedy; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@REL1_41] Account for load basic button

https://gerrit.wikimedia.org/r/970759

Change 970760 had a related patch set uploaded (by Reedy; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@REL1_40] Account for load basic button

https://gerrit.wikimedia.org/r/970760

Seems it's still happening https://gerrit.wikimedia.org/r/c/mediawiki/core/+/971331

In T348539#9305439, @Jdlrobson wrote:

According to the video this still has something to do with a slow connection. We can skip this test for the time being but it's a little concerning that our browser tests are hitting this and I'd be curious if the editing team knows why "slow connection" is showing so quickly in this simulation. It suggests some kind of bug is occurring in the loading of the editor.

Wikitext-Editor-Closing-editor-(browser-button)-2023-11-03T13-25-20-128Z.mp4349 KBDownload

It is not showing quickly, the recording is just weird.

Time	Screenshot	Notes
0.633 seconds		You can see that it started one of the 'Wikitext Editor' tests (it's on a page titled like 'NewPage <date>').
1.468 seconds		For some reason, another browser window opens!
3.803 seconds		You can see that the new window is running one of the 'Manage Watchlist' tests (it's on a page titled like 'I am on the "Selenium mobile unwatched test <date>').
4.671 seconds		The other test finishes, and we see the browser window running our test again, which has displayed the slow connection message in the meantime. This is expected, as this message should appear after 3 seconds.

Is it normal that it's running multiple browser tests at once? That doesn't seem like it would be very reliable.

In T348539#9292289, @Jdlrobson wrote:

Occurring on https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium-docker/73752/console

From the associated assets it looks like VisualEditor is not loading here (or is loading slowly) and back button is disabled here. Might be worth getting the editing team to investigate this separately. Selenium seems to be doing what it should do here.

Wikitext-Editor-Closing-editor-(browser-button)-2023-10-30T11-25-12-081Z.mp4205 KBDownload

IN terms of the test it should be easy to modify it for a short term fix to either:

1. check for the slow button in addition to the editor (which it's currently waiting on)
2. Increase the waitUntil condition time.

I disagree with the analysis here. The editor is not loading slowly; rather, the editor loading is cancelled (by pressing the browser back button), but the "splash screen" remains open. This seems like the bug causing this problem, and the patch would not have fixed it.

You can match up this video closely to the test steps: https://gerrit.wikimedia.org/g/mediawiki/skins/MinervaNeue/+/6af218d9ee47676e319d6737008273f34c2681a4/tests/selenium/specs/editor_wikitext_nosave.js#13

Time	Screenshot	Test step	Notes
0.667 seconds		iAmOnAPageThatDoesNotExist(); iClickTheEditButton();
0.800 seconds			Note the ?action=edit in the browser address bar – we navigated to another page. Maybe the editor should load without navigating to another page? That would speed up the test, at least
1.234 seconds			URL has been reformatted and presumably the ?action=edit replaced by #/editor/0 – we can't see that because the page title is too long, but we probably reached this line: https://gerrit.wikimedia.org/g/mediawiki/extensions/MobileFrontend/+/50e08f5c68208402711a6ef99b637a526696fad3/src/mobile.init/editor.js#456
1.701 seconds		iSeeTheWikitextEditorOverlay(); iClickTheBrowserBackButton();
2.002 seconds		iDoNotSeeAnOverlay();	We have navigated back (as evindenced by the browser "forward" button being active), and the overlay background has disappeared, but not the rest of it – thus this step never passes and we wait until timeout

I would recommend using shorter page titles in the tests, so that it's easier to see what happens.

I would also recommend moving the test code from MinervaNeue to MobileFrontend, since that's where the editor is implemented. Keeping them in separate repos makes this difficult to maintain.

tl;dr: The recordings show that browser back button does not reliably close the editor loading screen if the editor hasn't loaded yet. That may be a bug worth investigating. If it's not practical to investigate, we should instead change the iSeeTheWikitextEditorOverlay(); step to wait for the editor to load fully, not only for the loading screen – the browser back button handling may be more reliable in this case.

Thanks for looking at this @matmarex. If there is nothing wrong here with the editor workflow, to be honest I would just suggest removing the test altogether as it doesn't seem useful in current form. If we want to retain a test to check back button behaviour, then the language overlay is a better bet, since that's a synchronous overlay and doesn't require a server roundtrip.

Regarding the multiple test running at same time - that may be an optimization made previously by RelEng that's worth re-evaluating. I think in general use of Selenium at WMF should be reassessed given these tests seem to cause more inconvenience then being helpful. I've heard good things about Cypress for example.

Still happening, e.g. in this build for a WikibaseQualityConstraints change.

Change 977779 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/skins/MinervaNeue@master] Disable another flakey test

https://gerrit.wikimedia.org/r/977779

Change 977779 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Disable another flakey test

https://gerrit.wikimedia.org/r/977779

Change 970759 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@REL1_41] Account for load basic button

https://gerrit.wikimedia.org/r/970759

Change 970760 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@REL1_40] Account for load basic button

https://gerrit.wikimedia.org/r/970760