Page MenuHomePhabricator

Selenium test "Search.Clicking on a watchstar toggles the watchstar" is flaky
Open, Needs TriagePublic

Description

Failure on https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/5355/console
The failure was for a gate-and-submit patch set of the cldr extension, the test definition is in MinervaNeue skin.

 Running: Chrome (v90.0.4430.212) on linux
 Session ID: 5c2d8780-17d0-4954-be5a-8c7cc8c89f3e

 » /tests/selenium/specs/search_loggedin.js
 Search
    ✖ Clicking on a watchstar toggles the watchstar

 1 failing (9.6s)

 1) Search Clicking on a watchstar toggles the watchstar
 Expected values to be strictly equal:
01:06:31 
01:06:31 false !== true
01:06:31       
01:06:31       actual expected
01:06:31       
01:06:31       falsetrue
01:06:31       
 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

 false !== true

     at iAmLoggedIn (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:63:9)
     at async iAmLoggedIntoTheMobileWebsite (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:68:2)
     at async Context.<anonymous> (/workspace/src/skins/MinervaNeue/tests/selenium/specs/search_loggedin.js:22:3)

Video at https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/5355/artifact/log/Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-04-12T23-06-15-187Z.mp4
Looks like wrong skin is loaded on the test

Event Timeline

Here are a couple videos captured for a Parsoid updated in mediawiki/vendor https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/927241


Extracted from https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/7767/console

The MediaWiki error log is empty

21:21:08 [0-4] RUNNING in chrome - /tests/selenium/specs/search_loggedin.js
21:21:08 [0-5] RUNNING in chrome - /tests/selenium/specs/user_page.js
21:21:09 [0-5] PASSED in chrome - /tests/selenium/specs/user_page.js
21:21:09 [0-2] Nearby item will only appear in main menu if $wgMFNearby is configured
21:21:10 [0-2] PASSED in chrome - /tests/selenium/specs/mainmenu_loggedin.js
21:21:10 [0-6] RUNNING in chrome - /tests/selenium/specs/watchstar.js
21:21:12 [0-4] AssertionError in "Search.Clicking on a watchstar toggles the watchstar"
21:21:12 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
21:21:12 
21:21:12 false !== true
21:21:12 
21:21:12     at iAmLoggedIn (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:63:9)
21:21:12     at async iAmLoggedIntoTheMobileWebsite (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:68:2)
21:21:12     at async Context.<anonymous> (/workspace/src/skins/MinervaNeue/tests/selenium/specs/search_loggedin.js:22:3)
21:21:13 [0-4] RETRYING in chrome - /tests/selenium/specs/search_loggedin.js
21:21:13 [0-4] RUNNING in chrome - /tests/selenium/specs/search_loggedin.js
21:21:17 [0-1] PASSED in chrome - /tests/selenium/specs/editor_wikitext_nosave.js
21:21:26 [0-4] AssertionError in "Search.Clicking on a watchstar toggles the watchstar"
21:21:26 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
21:21:26 
21:21:26 false !== true
21:21:26 
21:21:26     at iAmLoggedIn (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:63:9)
21:21:26     at async iAmLoggedIntoTheMobileWebsite (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:68:2)
21:21:26     at async Context.<anonymous> (/workspace/src/skins/MinervaNeue/tests/selenium/specs/search_loggedin.js:22:3)
21:21:26 [0-4] FAILED in chrome - /tests/selenium/specs/search_loggedin.js (1 retries)
21:21:27 [0-6] PASSED in chrome - /tests/selenium/specs/watchstar.js
21:21:27

The test is retried and fails a second time (hence the two videos):

21:21:13 [0-4] RETRYING in chrome - /tests/selenium/specs/search_loggedin.js

There are some rate limitation reached, but they don't seem related.

...
2023-06-05 21:11:14 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting linkpurge rate for 127.0.0.1
2023-06-05 21:13:40 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting growthexperimentsuserimpacthandler rate for NewUser-0.42622400257586124-Iñtërnâtiônà lizætiøn
2023-06-05 21:13:43 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting stashbasehtml rate for NewUser-0.42622400257586124-Iñtërnâtiônà lizætiøn
2023-06-05 21:13:54 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting growthexperimentsuserimpacthandler rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:13:57 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting stashbasehtml rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:14:05 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting edit rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:14:05 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting linkpurge rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:17:15 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting edit rate for 127.0.0.1
...

So the user is supposed to be logged in and on the mobile website for this test, but in these videos, they are logged out and on desktop website.

There are currently several issues open about logging in on mobile not working, see e.g. T312042 and the other reports linked there. This is probably the root cause, and not the test itself.

Change 942458 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@master] Disable flakey test

https://gerrit.wikimedia.org/r/942458

Change 942458 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Disable flakey test

https://gerrit.wikimedia.org/r/942458

So the user is supposed to be logged in and on the mobile website for this test, but in these videos, they are logged out and on desktop website.

There are currently several issues open about logging in on mobile not working, see e.g. T312042 and the other reports linked there. This is probably the root cause, and not the test itself.

That is resolved now. Let's try to restore it, and see if it's still flaky.

Change 975889 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/skins/MinervaNeue@master] Revert "Disable flakey test"

https://gerrit.wikimedia.org/r/975889

Change 975889 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Revert "Disable flakey test"

https://gerrit.wikimedia.org/r/975889

matmarex claimed this task.

Ugh, it is still flaky, it just failed on https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/975350 (https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php80-docker/2248/console).

It looks like it's just using the desktop mode?


Change 977621 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/skins/MinervaNeue@master] Revert "Revert "Disable flakey test""

https://gerrit.wikimedia.org/r/977621

Change 977621 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Revert "Revert "Disable flakey test""

https://gerrit.wikimedia.org/r/977621