Page MenuHomePhabricator
Create Task
Maniphest T272388

Restart m5 master (db1128)
Closed, ResolvedPublic
Actions

Description

In order to enable report_host flag (T266483) on m5 master (db1128), we need to restart its MySQL.
This host currently has the following active databases:

labsdbaccounts
labswiki
striker
test_labsdbaccounts
testreduce
testreduce_vd

When: Thursday 28th January at 09:00AM UTC

Impact: The above databases (including wikitech) will be unavailable for a few minutes (no reads/no writes)
This is just a daemon restart, so it shouldn't take too long, maybe a couple of minutes.

@Andrew @aborrero @Bstorm @ssastry would you be ok with this date?

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 19 2021, 3:37 PM
Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptJan 19 2021, 3:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Jan 19 2021, 3:37 PM
Marostegui moved this task from Triage to Ready on the DBA board.
Marostegui updated the task description. (Show Details)
Marostegui added a parent task: T266483: Enable report_host for mariadb.
LSobanski subscribed.Jan 19 2021, 3:42 PM
aborrero moved this task from Inbox to Watching on the cloud-services-team (Kanban) board.Jan 19 2021, 3:43 PM

I think this works for us, thanks for the heads up.

Do you think the downtime should be announced to stakeholders? Wikitech being down seems like something some folks may want to know in advance.

Marostegui added a comment.Jan 19 2021, 3:46 PM
In T272388#6758254, @aborrero wrote:

I think this works for us, thanks for the heads up.

Do you think the downtime should be announced to stakeholders? Wikitech being down seems like something some folks may want to know in advance.

Ideally sending an email to wikitech-l + User-notice tag should be enough, but I wanted to wait for the date/time confirmation, let's see if this time also works for @ssastry

ssastry added a comment.Jan 19 2021, 3:53 PM

Works for me. Tangentially, we are currently in process of possibly stopping all use of testreduce database for our tests and it is possible we might get it done by then as well.

ssastry added a comment.Jan 19 2021, 3:53 PM

Works for me. Tangentially, we are currently in process of possibly stopping all use of testreduce database for our tests.

Marostegui added a comment.Jan 19 2021, 3:54 PM
In T272388#6758291, @ssastry wrote:

Works for me. Tangentially, we are currently in process of possibly stopping all use of testreduce database for our tests and it is possible we might get it done by then as well.

Oh sweet - in that case, please create a ticket so we can drop those databases :)

• jcrespo subscribed.EditedJan 19 2021, 3:54 PM

One small detail- I am unsure if labswiki use the proxy / has its failover service configured, due to it being handled by mediawiki, so for wikitech it may be a hard down (not only a read-only status). Please Marostegui confirm (I may have outdated info).

Correction, you already mentioned that on the body, ignore.

Marostegui added a comment.Jan 19 2021, 3:55 PM

@jcrespo there is no active proxy for m5 - as I stated on the task there will be no reads and no writes.

Marostegui added a project: User-notice.Jan 19 2021, 4:00 PM
Marostegui updated the task description. (Show Details)
Johan moved this task from To Triage to In current Tech/News draft on the User-notice board.Jan 20 2021, 5:21 AM
Johan subscribed.Jan 20 2021, 5:24 AM

Tech News might normally be a bit overkill for Wikitech being down for a couple of minutes (I'd recommend wikitech-l and a few short posts to a couple of technical wiki pages, in that case), but since there's already an item about Commons, we can just merge the two into one item and have this one tag along. It'll go out on Monday 25 January.

Andrew added a comment.Jan 20 2021, 5:33 AM

@Marostegui that timing sounds fine to me, especially if someone other than me (@Johan?) announces the downtime in advance.

Marostegui added a comment.Jan 20 2021, 5:42 AM
In T272388#6760598, @Andrew wrote:

@Marostegui that timing sounds fine to me, especially if someone other than me (@Johan?) announces the downtime in advance.

I sent the email to wikitech-l and ops@ yesterday.
Regarding the banner, I will leave that up to @Johan to decide.

Thanks

Johan added a comment.Jan 20 2021, 5:51 AM

To be very clear we're talking about the same thing, by "Wikitech" we're just referring to wikitech.wikimedia.org/wiki/ here and nothing else right?

Marostegui added a comment.Jan 20 2021, 5:51 AM

Yes, that's it

Johan added a comment.Jan 20 2021, 5:58 AM

I think the mailing lists and Tech News can be considered having announced it in advance. Were it a wiki seeing a lot of traffic, or we were expecting more than a couple of minutes of downtime, that might change things, but I think this should be enough for now.

Andrew added a comment.Jan 20 2021, 2:34 PM
In T272388#6760633, @Johan wrote:

I think the mailing lists and Tech News can be considered having announced it in advance. Were it a wiki seeing a lot of traffic, or we were expecting more than a couple of minutes of downtime, that might change things, but I think this should be enough for now.

agreed!

Marostegui added a comment.Jan 21 2021, 9:44 AM

Added to the deployments calendar

Marostegui added a parent task: T272568: Add m* and es4/es5 sections to Orchestrator.Jan 26 2021, 9:29 AM
Marostegui moved this task from Ready to In progress on the DBA board.Jan 27 2021, 6:36 AM
Marostegui added a comment.EditedJan 27 2021, 6:38 AM

Procedure:

Pre restart

  • Silence m5 hosts
  • buffer pool dump + disablement in advance to make the restart faster

Restart

  • !log m5 master restart, wikitech will be unavailable - T272388
  • set db1128 in RO
  • db1128: restart mysql
  • verify report_host is enabled
  • verify read_only is OFF
  • Once mysql is back: reload haproxy on dbproxy1021 (non active) and dbproxy1017 (non active)
  • check everything is ok
  • close this task
Johan moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Jan 28 2021, 4:27 AM
Marostegui added a comment.Jan 28 2021, 8:50 AM

Pre-restart steps are done

Marostegui closed this task as Resolved.Jan 28 2021, 9:04 AM

This was done.
Downtime start: 09:00:23
Downtime stop: 09:00:51
Total: 28 seconds

+--------------------+
| @@report_host      |
+--------------------+
| db1128.eqiad.wmnet |
+--------------------+
1 row in set (0.001 sec)
Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:54 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 13 2022, 2:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits