Page MenuHomePhabricator
Create Task
Maniphest T370962

Southward Datacenter Switchover (September 2024)
Open, Needs TriagePublic
Actions

Description

This is the umbrella task for the September 2024 Southward Datacenter Switchover (eqiad to codfw).

Please file tasks for related work as subtasks.

Important Dates:

Note: The above times are all 15:00 UTC (i.e., 1h later than the 14:00 UTC target of the prior switchover).

Previous switchover (March 2024): T357547

Details

SubjectRepoBranchLines +/-
hieradata: update deployment_server to deploy2002operations/puppetproduction+2 -2
debug.json: order codfw (primary) DC backends firstoperations/mediawiki-configmaster+4 -4
wmnet: update deployment CNAME record to deploy2002operations/dnsmaster+1 -1
geo-maps: update map default to list codfw firstoperations/dnsmaster+1 -1
wmnet: update CNAME record for maintenance host to codfwoperations/dnsmaster+1 -1
wmnet: update CNAME records for DB masters to codfwoperations/dnsmaster+22 -22
Customize query in gerrit

Related Objects
Search...

Event Timeline

Scott_French created this task.Jul 24 2024, 9:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 24 2024, 9:07 PM
Scott_French updated the task description. (Show Details)Jul 24 2024, 9:13 PM
Scott_French updated the task description. (Show Details)Jul 24 2024, 9:30 PM
Scott_French added a project: Datacenter-Switchover.Jul 24 2024, 10:21 PM
kamila subscribed.Jul 25 2024, 12:10 PM
akosiaris subscribed.Jul 26 2024, 8:39 AM
• Marostegui subscribed.Jul 30 2024, 2:10 PM
Scott_French added a subtask: T369921: Support warmup for local caches in mw-on-k8s.Jul 30 2024, 4:26 PM
Scott_French closed subtask T369921: Support warmup for local caches in mw-on-k8s as Resolved.Aug 1 2024, 11:53 PM
Scott_French added a subtask: T372649: Audit / update switchover-related cookbooks.Aug 16 2024, 2:56 PM
Scott_French added a subtask: T372849: Determine switchover changes for migration of video scaling to k8s.Aug 20 2024, 1:46 AM
jasmine_ subscribed.Aug 20 2024, 2:27 AM
Clement_Goubert mentioned this in T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.Aug 20 2024, 11:22 AM
Clement_Goubert mentioned this in T332015: Migrate poolcounter hosts to bookworm.Mon, Sep 9, 10:57 AM
Dreamy_Jazz subscribed.Tue, Sep 10, 9:26 PM
Quoting from T357547

While stopping all maintenance scripts (01-stop-maintenance), we found a user triggered script which we fiercely killed manually, and continued the process

Not sure exactly which script this was, but it might have been https://wikitech.wikimedia.org/wiki/MediaModeration. If it was, then this will be running like last time and so would need to be manually killed.

Scott_French closed subtask T372649: Audit / update switchover-related cookbooks as Resolved.Fri, Sep 13, 1:29 AM
gerritbot added a comment.Wed, Sep 18, 7:59 PM

Change #1073894 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: update deployment_server to deploy2002

https://gerrit.wikimedia.org/r/1073894

gerritbot added a project: Patch-For-Review.Wed, Sep 18, 7:59 PM

Change #1073895 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/mediawiki-config@master] debug.json: order codfw (primary) DC backends first

https://gerrit.wikimedia.org/r/1073895

gerritbot added a comment.Wed, Sep 18, 8:02 PM

Change #1073897 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] wmnet: update CNAME records for DB masters to codfw

https://gerrit.wikimedia.org/r/1073897

gerritbot added a comment.Wed, Sep 18, 8:02 PM

Change #1073898 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] wmnet: update CNAME record for maintenance host to codfw

https://gerrit.wikimedia.org/r/1073898

gerritbot added a comment.Wed, Sep 18, 8:02 PM

Change #1073899 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] geo-maps: update map default to list codfw first

https://gerrit.wikimedia.org/r/1073899

gerritbot added a comment.Wed, Sep 18, 8:02 PM

Change #1073900 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] wmnet: update deployment CNAME record to deploy2002

https://gerrit.wikimedia.org/r/1073900

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits