Koschei SOP

Koschei consists of frontend and backend.

Frontend is a web application written in Python using Flask framework. It is ran under Apache httpd with mod_wsgi as a WSGi application. Frontend displays information to users and allows editing package groups and changing priorities.

Backend consists of a couple of loosely-coupled microservices, including:

Koschei deployment is managed by an Ansible playbook:

The above playbook is idempotent, which means that running it has no effect when everything is already configured as expected.

Koschei is fully-containerized. It is deployed on OpenShift.

Koschei is stateless. It doesn’t use any persistent storage. All non-volatile information is stored in PostgreSQL database, which is not part of Koschei, but an external service that Koschei depends on.

There is one common container image for different Koschei workloads — frontend and backend containers are all ran from the same image.

Koschei images are built by upstream on Quay.io. Upstream implements continuous delivery of container images to Quay.io registry. Code pushed to fedora-prod or fedora-stage git branches in upstream GitHub repository are automatically built as container images and pushed to Quay.io registry with appropriate tags.

Pristine upstream Koschei images are then imported into internal OpenShift registry — Fedora OpenShift does not build any Koschei container images by itself. Image import into OpenShift is always done manually by a Koschei sysadmin, usually by running a manual Ansible playbook. This way we ensure that developers who can push code to GitHub repository don’t have any control over Fedora infrastructure deployment process.

Upstream images don’t contain any Fedora-specific configuration. Such configuration is mounted into containers as read-only volumes backed by Kubernetes Secrets.

Frontend is ran as Kubernetes Deployment with multiple replicas for high availability. Frontend supports rolling update, which allows it to be updated with no user-visible downtime.

Each of backend services has its own Kubernetes Deployment with a single replica. Because backend downtime is not user-visible, rolling updates are not used by backend.

In addition to frontend and backend, there is also admin Deployment, which runs a container that does nothing but waits for sysadmin to rsh into it for running manual admin commands.

Besides the forementioned Kubernetes Deployments, some ad-hoc tasks are ran as Kubernetes Jobs, either created on a time schedule from CronJobs or created by running manual Ansible playbooks by Koschei sysadmins.

Upgrading Koschei to a new upstream version is done by running one of manual Ansible playbooks:

The first rolling update playbook should be used when given update is known not to change database schema. In this case new upstream image is simply imported into internal OpenShift registry and all Deployments are restarted. OpenShift takes care of doing rolling update of frontend, so that no downtime is experienced by users. Backend Pods are also recreated with the new image.

The second full update playbook is used when given update changes database schema. This playbook pauses all Deployments and terminates all Pods. Users experience frontend downtime. When everything is stopped, the playbook creates Kubernetes Jobs to run database migrations and perform other maintenance tasks. Once the Jobs are done, new Deployments are rolled.

Certain Koschei operation tasks are done with the koschei-admin CLI tool. The container where the tool is available can be accessed with:

…​ oc project koschei oc rsh deploy/admin …​

For stopping builds from being scheduled, scaling down the scheduler Deployment to zero replicas is enough. For planned Koji outages, it’s recommended to stop the scheduler service. It is not necessary, as Koschei can recover from Koji errors and network errors automatically, but when Koji builders are stopped, it may cause unexpected build failures that would be reported to users. Other backend services can be left running as they automatically restart themselves on Koji and network errors.

Koschei is by default limited to 30 concurrently running builds. This limit can be changed in the configuration under koji_config.max_builds key. There’s also Koji load monitoring, that prevents builds from being scheduled when Koji load is higher that certain threshold. That should prevent scheduling builds during mass rebuilds, so it’s not necessary to stop scheduling during those.

Koschei can display announcement in web UI. This is mostly useful to inform users about outages or other problems.

To set announcement, run:

or:

To clear announcement, run:

Packages can be added to one or more group.

To add new group named mynewgroup, run:

To add new group named mynewgroup and populate it with some packages, run:

Some packages are more or less important and can have higher or lower priority. Any user can change manual priority, which is reset after package is rebuilt. Admins can additionally set static priority, which is not affected by package rebuilds.

To set static priority of package foo to value 100, run:

After branching occurs and Koji build targets have been created, Koschei should be updated to reflect the new state. There is a special admin command for this purpose, which takes care of copying the configuration and also last builds from the history.

To branch the collection from Fedora 27 to Fedora 28, use the following:

Then you can optionally verify that the collection configuration is correct by visiting https://koschei.fedoraproject.org/collections and examining the configuration of the newly branched collection.

When a Fedora release goes EOL its collection must be removed from Koschei. Otherwise Koschei will flood Koji with scratch build attempts that fail. To do this, from the admin pod, run:

with '39' changed to the number of the release going EOL.

To turn mygroup group created by user someuser into a global group thegroup, run: