As we are getting closer and closer to a fully functional MW-on-K8s image, we can start discussing our testing in production and rolling out options.
(Task description will be updated as we are figuring out our next steps)

Background

History
When we migrated to PHP7, users were served by PHP7 based on the existence of a cookie, set by mediawiki, and the X-Seven header, set at the traffic layer. On the application layer, Apache would route a request to the relevant backend, HHVM or php-fpm.

Having a cookie allowed us to first let beta users in, and in turn progressively increase the amount of anonymous user traffic served via PHP7. We then continued by converting API servers to php7_only servers, and finally converted all jobrunners. Additionally, we split the outer caching layer into PHP7 rendered pages and HHVM rendered pages (vary-slotting). Note that we didn't though do this for parsercache too, At the time, we were only using Varnish, so all this logic was written in VCL, with some additional apache config.

Now
This migration is slightly different this time:

• Caching layer consists of Varnish and ATS (VCL and LUA)
• Decision of where to route an incoming request will be taken at the caching layer
• We have 4 mediawiki clusters: api, app, jobrunners, and parsoid
• we are older

Proposed Plans

After a brief discussion with Traffic and Performance-Team, we have:

Proposal #1: URL routing

Given that app and api servers share the same configuration, and assuming that initially we will have the same discovery URL, eg mw-k8s-rw.discovery.wmnet,
we can start by routing a some low traffic URLs to Kubernetes, for example https://en.wikipedia.org/wiki/Barack_Obama. When we are more comfortable, we can start migrating some small wikis, and eventually, migrate them all.

Prons

• No complex and dangerous VCL and LUA changes
• Edge cache will not be polluted since we will always have the k8s rendered article
• Easy edge cache invalidation (single pages or entire wikis)

Cons

• Less control over traffic served
• Won't be able to create a beta feature
• Longer rolling out strategy
• Slightly complex rollbacks (traffic layer change + edge cache invalidation)

Beta users

In parsercache we have the ability to specify a key prefix and TTL for specific objects. Additionally, logged-in users bypass caches in our traffic layer, that being said, we could possibly have beta users:

• A user has a special cookie indicating they are part of the k8s beta
• When a server is storing in parcercache (and in turn, in memcached), uses a key prefix and a shorter TTL (cache slotting/Vary)
• Beta users can always compare a page by simply opening it as anonymous
• Beta users are more likely to report problems.
• We can run this for as long as we want

Prons

• No edge and parser cache pollution (pages rendered by k8s mixed with pages render by baremetal servers)
• User reports

Cons

• Browsing will be considerably slower for beta users, we could consider fiddling with the TTL a bit

Rollout Example

1. X-Wikimedia-Debug
2. Beta users/parsercache slotting
3. Low traffic urls
4. Low traffic wikis from group0
5. Some group1 wikis
6. Parsoid (?)
7. All wikis except enwiki
8. enwiki (Fin)

Note: Running jobs, timers, and standalone scripts are going to be approached differently

Proposal #2: Use a k8s cookie

Users with the cookie will be routed to k8s, and will have their own cache in the traffic layer (varnish+ATS). This is similar to how we rolled out PHP7, the difference is that previously, routing and cookie setting took place within the application layer, while now we have to do this in the traffic layer.

Prons

• We have previous experience in rolling out
• Beta users
• Better control over amount of traffic served
• Easier to roll back (?)

Cons

• Complex VCL and LUA changes for edge cache slotting (not enough test coverage there)
• Edge cache invalidation issue (ie how do we invalidate only k8s rendered cache from the edge?)
• Where will we calculate if an anonymous user should get the k8s cookie or not?
• Traffic would like to avoid this solution

Proposal #3: Per cluster rollout (winner)

We can create kubernetes services to server some (initially internal) traffic, and then do a per cluster migration. For instance, we could create api-internal-r{w,o}.discovery.wment service, and then start moving services to start using it.
This has been used at the beginning for all migrations, and will continue to be used for T333120: Migrate internal traffic to k8s

Proposal #4: Percentage-based global traffic redirect (followup to winner)

See T336038: Add traffic sampling support to mw-on-k8s.lua ATS script
A LUA script was added to ATS. It supports:

• Sending any percentage of traffic for a domain to mw-on-k8s (including 0 and 100%)
• Sending any percentage of global traffic to mw-on-k8s

This approach will be used going forward, with the current thresholds described in Roll out phase 2

After discussions, serviceops has decided to mix and match ideas from the above proposals.

Roll out

Roll out phase 1: Start serving a small portion of content from specific wikis

• Serve test2.wikipedia.org from k8s
• T292707 Wikitech is ideal to dogfood mw-on-k8s, there are challenges though that we need to over come T292707
• Migrate the rest of group0, except testwiki, starting with mediawiki.org T337490: Migrate group0 to Kubernetes
• Coordinate a temporary switch of testwiki to mw-on-k8s with testers T337489: Run QTE test suite on testwiki on kubernetes
• T340549: Migrate group1 to Kubernetes , except commons, votewiki, and wikidata Abandoned approach, from then on we go for the more statistically significant percentage based approach.

Roll out phase 2: Migrate global traffic by increments

• T341078: Direct 0.5% of all traffic to mw-on-k8s
• T341463: Direct 1% of all traffic to mw-on-k8s
• T341780: Direct 5% of all traffic to mw-on-k8s
• T346422: Move 10% of mediawiki external requests to mw on k8s
• T348122: Move 25% of mediawiki external requests to mw on k8s
• T355532: Move 40% of mediawiki external requests to mw on k8s
• T357507: Move 50% of mediawiki external requests to mw on k8s
• T357508: Move 60% of mediawiki external requests to mw on k8s
• T360763: Move 70% of mediawiki external requests to mw on k8s
• T362323: Move 100% of external traffic to Kubernetes

Roll out phase 3: Cleanup, scripts, and stragglers, oh my!

• Legacy clusters cleanup T367949: Spin down api_appserver and appserver clusters
• Hardware repurpose T351074: Move servers from the appserver/api cluster to kubernetes and T354791: Reclaim jobrunner hardware for k8s
• Maintenance scripts T341560: Migrate mwmaint server functionality to mw-on-k8s
• Periodic scripts T341555: Allow running periodic jobs for mw on k8s
• mwdebug T276994: Provide an mwdebug functionality on kubernetes
• Dumps T352650: Migrate current-generation dumps to run from our containerized images
• Wikitech T292707: ☂ Migrate Wikitech to Kubernetes
• Videoscaling T355292: Port videoscaling to kubernetes