Page MenuHomePhabricator
Create Task
Maniphest T147359

Cannot access the database: No working replica DB server: Unknown error
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Rollout of wmf.21 to testwiki caused 503s with the error message, Cannot access the database: No working replica DB server: Unknown error

{
  "args": [],
  "type": "->",
  "class": "LoadBalancer",
  "function": "reportConnectionError",
  "line": 565,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php"
},
{
  "args": [
    "boolean",
    "array",
    "boolean"
  ],
  "type": "->",
  "class": "LoadBalancer",
  "function": "getConnection",
  "line": 3073,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/GlobalFunctions.php"
},
{
  "args": [
    "integer"
  ],
  "function": "wfGetDB",
  "line": 739,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/user/User.php"
},
{
  "args": [
    "string"
  ],
  "type": "::",
  "class": "User",
  "function": "idFromName",
  "line": 1839,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/extensions/CentralAuth/includes/CentralAuthUser.php"
},
{
  "args": [],
  "type": "->",
  "class": "CentralAuthUser",
  "function": "canAuthenticate",
  "line": 1888,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/extensions/CentralAuth/includes/CentralAuthUser.php"
},
{
  "args": [
    "string"
  ],
  "type": "->",
  "class": "CentralAuthUser",
  "function": "authenticateWithToken",
  "line": 179,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php"
},
{
  "args": [
    "WebRequest"
  ],
  "type": "->",
  "class": "CentralAuthSessionProvider",
  "function": "provideSessionInfo",
  "line": 486,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/session/SessionManager.php"
},
{
  "args": [
    "WebRequest"
  ],
  "type": "->",
  "class": "MediaWiki\\Session\\SessionManager",
  "function": "getSessionInfoForRequest",
  "line": 189,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/session/SessionManager.php"
},
{
  "args": [
    "WebRequest"
  ],
  "type": "->",
  "class": "MediaWiki\\Session\\SessionManager",
  "function": "getSessionForRequest",
  "line": 735,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/WebRequest.php"
},
{
  "args": [],
  "type": "->",
  "class": "WebRequest",
  "function": "getSession",
  "line": 128,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/session/SessionManager.php"
},
{
  "args": [],
  "type": "::",
  "class": "MediaWiki\\Session\\SessionManager",
  "function": "getGlobalSession",
  "line": 759,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/Setup.php"
},
{
  "args": [
    "string"
  ],
  "function": "include",
  "line": 137,
  "file": "/srv/mediawiki/php-1.28.0-wmf.21/includes/WebStart.php"
},
{
  "args": [
    "string"
  ],
  "function": "include",
  "line": 3,
  "file": "/srv/mediawiki/w/favicon.php"
}

Details

SubjectRepoBranchLines +/-
Make LoadMonitor use $serverIndexes in the cache keymediawiki/coremaster+24 -41
Make LoadMonitor use $serverIndexes in the cache keymediawiki/corewmf/1.28.0-wmf.21+24 -41
Add version to LoadMonitor::getCacheKey()mediawiki/coremaster+3 -0
Add version to LoadMonitor::getCacheKey()mediawiki/corewmf/1.28.0-wmf.21+3 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

thcipriani created this task.Oct 4 2016, 8:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2016, 8:49 PM
thcipriani updated the task description. (Show Details)Oct 4 2016, 8:49 PM
thcipriani added a parent task: T145220: MW-1.28.0-wmf.21 deployment blockers.
greg triaged this task as Unbreak Now! priority.Oct 4 2016, 8:51 PM
Restricted Application added subscribers: Jay8g, Luke081515, TerraCodes. · View Herald TranscriptOct 4 2016, 8:51 PM
gerritbot subscribed.Oct 4 2016, 9:08 PM

Change 314173 had a related patch set uploaded (by Aaron Schulz):
Add version to LoadMonitor::getCacheKey()

https://gerrit.wikimedia.org/r/314173

gerritbot added a project: Patch-For-Review.Oct 4 2016, 9:08 PM
gerritbot added a comment.Oct 4 2016, 9:10 PM

Change 314176 had a related patch set uploaded (by Thcipriani):
Add version to LoadMonitor::getCacheKey()

https://gerrit.wikimedia.org/r/314176

gerritbot added a comment.Oct 4 2016, 9:21 PM

Change 314173 merged by jenkins-bot:
Add version to LoadMonitor::getCacheKey()

https://gerrit.wikimedia.org/r/314173

gerritbot added a comment.Oct 4 2016, 9:31 PM

Change 314176 merged by jenkins-bot:
Add version to LoadMonitor::getCacheKey()

https://gerrit.wikimedia.org/r/314176

Stashbot subscribed.Oct 4 2016, 9:34 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-04T21:34:46Z] <thcipriani@tin> Synchronized php-1.28.0-wmf.21/includes/libs/rdbms/loadmonitor/LoadMonitor.php: [[gerrit:314176|Add version to LoadMonitor::getCacheKey() (T147359)]] (duration: 00m 53s)

thcipriani closed this task as Resolved.Oct 4 2016, 9:57 PM
thcipriani assigned this task to aaron.

1.28.0-wmf.21 is now on group0 wikis and I'm no longer seeing this error. Marking as resolved.

ReleaseTaggerBot added projects: MW-1.28-release (WMF-deploy-2016-10-11_(1.28.0-wmf.22)), MW-1.28-release (WMF-deploy-2016-10-04_(1.28.0-wmf.21)), MW-1.28-release-notes.Oct 4 2016, 10:00 PM
thcipriani reopened this task as Open.Oct 5 2016, 5:40 PM

Spoke too soon. Although the rate at which this error is occurring has fallen considerably, this error is still in the logs.

gerritbot added a comment.Oct 5 2016, 7:59 PM

Change 314340 had a related patch set uploaded (by Aaron Schulz):
Make LoadMonitor use $serverIndexes in the cache key

https://gerrit.wikimedia.org/r/314340

aaron merged a task: T147422: log spammed with: Notice: Undefined index: 1 in /srv/mediawiki/php-1.28.0-wmf.21/includes/libs/rdbms/loadmonitor/LoadMonitor.php on line 67.Oct 5 2016, 8:01 PM
aaron added a subscriber: Matanya.
gerritbot added a comment.Oct 5 2016, 8:53 PM

Change 314414 had a related patch set uploaded (by Thcipriani):
Make LoadMonitor use $serverIndexes in the cache key

https://gerrit.wikimedia.org/r/314414

gerritbot added a comment.Oct 5 2016, 8:54 PM

Change 314340 merged by jenkins-bot:
Make LoadMonitor use $serverIndexes in the cache key

https://gerrit.wikimedia.org/r/314340

gerritbot added a comment.Oct 5 2016, 9:01 PM

Change 314414 merged by jenkins-bot:
Make LoadMonitor use $serverIndexes in the cache key

https://gerrit.wikimedia.org/r/314414

Stashbot added a comment.Oct 5 2016, 9:08 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-05T21:08:15Z] <thcipriani@tin> Synchronized php-1.28.0-wmf.21/maintenance/lag.php: [[gerrit:314414|Make LoadMonitor use $serverIndexes in the cache key (T147359)]] PART I (duration: 00m 50s)

Stashbot added a comment.Oct 5 2016, 9:09 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-05T21:09:20Z] <thcipriani@tin> Synchronized php-1.28.0-wmf.21/includes/libs/rdbms: [[gerrit:314414|Make LoadMonitor use $serverIndexes in the cache key (T147359)]] PART II (duration: 00m 55s)

thcipriani closed this task as Resolved.Oct 6 2016, 10:49 PM

Seems to be fully gone from the logs afaict

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits