A PRACTICAL GUIDE

TO WORKING WITH
KUBERNETES
SYNOPSIS
The IT world is having a love affair with Kubernetes. Seemingly everyone
everywhere is talking about Kubernetes as an essential tool for orchestrating
containerized applications.

There are several reasons for Kubernetes’s popularity. One is the strong
open source community that undergirds it. Another is the fact that it’s a
community-developed platform, rather than one closely associated with a single
commercial company (as most other container orchestrators are). The fact that
Kubernetes is free and easily customizable helps, too.

Yet, amidst all of the excitement about Kubernetes, it can be easy to overlook
a crucial reality. While Kubernetes is very useful as a container orchestration
tool, it doesn’t perform magic. To use Kubernetes effectively, organizations
need to carefully plan how they will implement and optimize their Kubernetes
deployment. They need to assess which Kubernetes features are the most
important for their needs, and determine how to configure those features to
maximum effect. They must address challenges such as securing Kubernetes,
and ensuring that Kubernetes deployments remain flexible and scalable as
the applications Kubernetes manages evolve. And they need to integrate
Kubernetes effectively into their DevOps-based workflows.

With those challenges in mind, Alcide has prepared this eBook to provide
organizations with a practical guide to working with Kubernetes. The eBook
explains how to get Kubernetes up and running. It highlights Kubernetes’s most
significant features and offers tips on optimizing them. And it explains how to
use Kubernetes to advance your organization’s DevOps culture.

The pages below don’t go into exhaustive detail about the technical details
of Kubernetes; for that information, the Kubernetes documentation is a great
resource. But we do hope that readers of this eBook will find helpful and
practical tips for making the most out of Kubernetes, and see why Kubernetes —
when used effectively — is such a powerful tool.

2 www.alcide.io • [email protected]
GETTING STARTED WITH
KUBERNETES
If you’re planning your Kubernetes deployment or assessing whether
Kubernetes is a good fit for your needs, an obvious question you need to answer
is what the process of setting up Kubernetes actually looks like. Here’s a primer
on the steps you’ll follow in order to start your Kubernetes journey.

PICK YOUR PATH TO

KUBERNETES
Getting started with Kubernetes presents
three paths you can take. The first is to
use a managed offering through cloud
providers like AKS by Microsoft Azure. The
second is to deploy your own servers, and
install one of many enterprise-focused
distributions, like OpenShift by Red Hat. The
third option is to build it from scratch. This
is the most complicated and work-intensive
choice. Kubernetes the Hard Way by Kelsey
Hightower is an often-used guide for those
who are willing to put in the extra effort.

3 www.alcide.io • [email protected]
DAY 1
GET KUBERNETES
RUNNING IN YOUR
ENVIRONMENT
Using the forerunner of hosted solutions,
GKE on GCP, is as simple as one command
line:

$ gcloud container clusters

create [CLUSTER_NAME]

On the distribution and do-it-yourself paths,

there are a few more steps — starting with
at least three master nodes to have high
availability because of how etcd works.
Then you’ll need to add compute nodes
and any other support servers required for
supplementary infrastructure, like software-
defined storage, and more enhanced load
balancing options.

Now that you are up and running with

Kubernetes, you have some more
decisions to make before it is ready for use.
These decisions at their highest level are
based on topics like how to build and store
containers and what to use for networking,
like the default — wide open networking
— or a more advanced networking option.
(There are a couple of CNI options
available if you are interested.)

4 www.alcide.io • [email protected]
DAY 2
NOW WE NEED TO MAINTAIN THIS
KUBERNETES MONSTER
A full Kubernetes multi-node cluster The best approach right now is to use a
deployment with live applications is not a part- project like that offered by Alcide, which
time endeavour for the average operational understands both Kubernetes and the
team. Not only is it typically new to the cloud it is running on, with a single,
environment, it is work in addition to any easy-to-use interface.
existing application deployments. Over time,
For monitoring and alerting, the CNCF
other application deployment models can be
has a top-level project that co-exists with
retired or migrated to Kubernetes, but there is
Kubernetes called Prometheus, which can
a transition period where making operations
handle most environments’ basic needs.
as simple as possible is key.
If more in-depth diagnostics are needed
Most Kubernetes deployments end up using from deployed applications, Jaeger is also
a networking layer that supports network supported by the CNCF.
policies, as they are the most flexible and
easiest to use for developers to manage
their own network security. Managing these
security policies only becomes more difficult
over time as the number of policies configured
grows.

This — along with having to ensure the

underpinning infrastructure’s network is also
properly secured — is daunting, especially
when using a public cloud like AWS and its
security groups.

5 www.alcide.io • [email protected]
DAY 3
DON’T FORGET
KUBERNETES
SECURITY
As with any modern network-connected
infrastructure, security is a big part of the
Kubernetes landscape. This goes beyond the
initial concerns people have when security is
mentioned (often residing around edge security
like firewalls). Network security policies play a
big part in what used to be exclusively in the
domain of edge security.

Another common concern around security is

secrets management. Secrets can be anything
from passwords to private certificates to API
keys. Kubernetes has a native way to manage
secrets so they can be referenced from other
points in the configuration, and it provides a
solid foundation layer. If additional layers of
security are required, then consider products
like HashiCorp Vault or CyberArk Conjur (aka
Application Access Manager).

The biggest areas that are overlooked for

security are security visibility, misconfiguration
and enforcing all the policies which are required
to keep the actual platform running, and not
just support the different applications.

For Kubernetes, this covers areas like Pod

Security Policies, which define the criteria a
pod needs to meet to even run. RBAC (AuthZ)
handles what access different accounts
have to cluster-wide and namespace-specific
resources, and authentication (AuthN) services,
which define what methods are used to
externally validate any users in a Kubernetes
cluster.

6 www.alcide.io • [email protected]
 UNDERSTANDING AND
OPTIMIZING KEBERNETES
KEY FEATURES
Kubernetes is a complex tool. Although at a high level, it
does one thing (container orchestration), its functionality
breaks down into several distinct categories.

In this section, we outline what the main features of

Kubernetes are. We also (and this is critical for actually
using Kubernetes effectively) explain how to approach
each of these features in a way that will optimize your
Kubernetes experience.

7 www.alcide.io • [email protected]
UNDERSTANDING
AND OPTIMIZING
KUBERNETES’S KEY
FEATURES

AUTOMATION & One of the most common reasons for using Kubernetes is to
automate tasks that would be difficult to perform manually.
SELF-SERVICE
Out of the box, Kubernetes automates many complicated
procedures.

But taking advantage of these automation features is not

enough to get the most out of Kubernetes. In order to
maximize your team’s ability to take advantage of Kubernetes
automation, you should implement a self-service model
that lets anyone on the DevOps team leverage Kubernetes’s
automation features (provided they have the proper
authorization to do so, of course).

Ideally, you want to have a self-service environment that

lets everyone trigger a deployment, from infrastructure
components to application or service deployments. For
example, you could have a dashboard from a CI/CD page
that you can trigger when a pipeline procedure is completed.
Kubernetes makes it easy to spin up CI/CD environments that
enable this functionality. That way, you can be confident that
you can deliver new features or fix bugs more easily, and you’ll
give more ownership to engineering teams.

CREATING A Kubernetes is a very powerful platform, but it can be used

and abused in many ways if not configured properly. To have
BETTER SECURITY
the best possible Kubernetes security, you need to evaluate
MODEL all the available security controls, without sacrificing usability
and availability. That includes setting up access controls,
configuring firewalls and ports, adding security scans,
throttling and load balancing, and setting up VPCs and pod
security policies. Ideally, you want security to be built-in and
integrated as part of the build pipeline so that incidents are
reported immediately.

But that alone is not enough. A better security model has

multiple layers of security that allow segregated controls and
automation without sacrificing the user experience. Luckily,
with Alcide’s Kubernetes Security Module, you can enable
advanced security controls to protect against vulnerabilities
and attack vectors.

8 www.alcide.io • [email protected]
UNDERSTANDING
AND OPTIMIZING
KUBERNETES’S KEY
FEATURES

RELIABILITY Utilizing microservices and distributed systems in the

cloud automatically means that you expect more failures
and downtime. You need to acknowledge that no system
is infallible. It’s also critical to embrace failure and learn
how to overcome it. What better way than to use system
resiliency patterns like Chaos Engineering, where scenarios
are introduced to randomly shut down modules or services
in production sites. By testing the behavior of systems under
varying loads, you can verify the functionality, scalability,
performance and reliability of architecture in real time.

The benefits of this approach are many. It enables systems to

be resilient to failure and lets teams apply changes confidently,
with minimal disruptions — so when a real failure occurs, the
operating staff will be better prepared to handle it in advance.

AUDITING Inspection of Kubernetes operations, logs, or configuration

changes should be baked into every environment because of
the need to track actions that occur in the environment and
to act in case of inconsistencies. Auditing is also required for
compliance.

Kubernetes has first-class support for auditing, with

configurable backends, policies, and log collectors that aim to
give administrators a complete picture of current operational
information.

9 www.alcide.io • [email protected]
UNDERSTANDING
AND OPTIMIZING
KUBERNETES’S KEY
FEATURES

REPRODUCIBILITY You should ideally maintain development, QA, validation

and production environments that are as consistent with
one another as possible, so that you can reproduce existing
scenarios — every time. Since there are moving parts with
containers, it’s very easy to create drift between environments,
leading to infamous “It Works On My Machine” moments.

This is why it's important to invest proactively in maintaining

reproducibility (sometimes also called parity) across
environments so that the engineering team can have higher
confidence that a decision made about one part of the
Kubernetes environment will result in predictable actions in
other parts.

The best way to achieve parity is to export the same

configuration for every Kubernetes environment. For example,
if using the kubectl tool, you can export a single parameterized
definition for a deployment (service, pod, secret):

kubectl get service name -o yaml --export

If using kube-aws you can use the export flag to create

CloudFormation stack files to apply them on demand:

kube-aws up --export

If using Openshift Origin, you can leverage the oc-export tool

to export all deployment resources so they can be applied
elsewhere. They can also be used to create boilerplate
templates that can accept parameters. For example, exporting
all services as a YAML file, we can run:

oc export service -o yaml

A best practice for the exported stack files is to commit them

with a version control system so they are reviewable and
verifiable as part of the CI/CD pipeline.

10 www.alcide.io • [email protected]
 KUBERNETES SECURITY AND
PERFORMANCE CONSIDERATIONS
Security and performance are complex topics within the realm of
Kubernetes. Kubernetes is not a security tool per se. Nor does it optimize
performance on its own. In order to address these needs, developers and IT
staff must understand which security and performance features Kubernetes
offers, and how to work with them.

Below, we highlight these features, with a focus on those introduced in the

most recent release of Kubernetes, version 1.14.

11 www.alcide.io • [email protected]
KUBERNETES
SECURITY AND
PERFORMANCE
CONSIDERATIONS

SECURITY: Kubernetes role-based access control gives you the ability

to set permissions for specific sets of users over cluster
USING ROLE-
resources. This includes how users can interact with your
BASED ACCESS cluster or a cluster namespace. The Kubernetes API gives
CONTROL (RBAC) administrators control to dynamically change policies and
groups. Prior to 1.14, cluster role discovery was included and
permissioned by default within Kubernetes pods, leading to
potential security holes that led people to avoid RBAC.

With Kubernetes 1.14, however, API discovery endpoints

are inaccessible to unauthenticated users by default,
greatly improving data privacy and cluster security. The
specific change was to remove the system:unauthenticated
subjects from the system:discovery and system:basic-user
ClusterRoleBindings. A new best practice is to use RBAC
by default, knowing that users and resources won’t be able
to perform sensitive operations without having the explicit
permission to do so.

SECURITY: Most Linux users and administrators are aware of process

ID (PID) limitations, especially with container-based
POD ISOLATION
services that quickly spin new processes up and down. PID
exhaustion is a situation where Kubernetes pods can no
longer spin up new tasks. For pods that induce exhaustion
for whatever reason, the risk is that it can impact other
workloads running on the node.

Kubernetes v1.14 now allows administrators to place

limits on new PID allocations, providing workload isolation
via these limits and new PID reservation capabilities.
As an example from a recent Kubernetes 1.14 blog, if a
Linux machine supports 32,768 PIDs and 100 pods, an
administrator can implement a budget of 300 PIDs per pod
to prevent PID exhaustion. This feature helps to increase
overall system stability, and avoids denial-of-service attacks
on container-based applications through PID exhaustion.

12 www.alcide.io • [email protected]
KUBERNETES
SECURITY AND
PERFORMANCE
CONSIDERATIONS

Kubernetes now officially supports adding Windows servers

AUTOMATION: as worker nodes and scheduling Windows containers,
UNIFY WINDOWS enabling Windows applications (new and legacy) to leverage
the Kubernetes platform. If you have existing investments
AND LINUX in both Windows-based and Linux-based applications, or
ORCHESTRATION are looking to expand with Windows-specific workloads,
you no longer need separate orchestrators to manage the
different workloads. In the past, this often led to operational
inefficiencies across deployments. Now that Kubernetes
1.14 officially supports this, you can safely deploy Windows
workloads knowing they will only be migrated to pods
running on Windows-based servers. The metrics and quotas
for Windows-based pods closely match those for Linux
containers, increasing the consistency.

Enabling this leads to additional advantages, including

operational efficiency (as mentioned), scalability across
clusters comprised of varying platform implementations, and
sharing of container knowledge across development teams,
regardless of platform or language chosen.

ORCHESTRATION: The implementation of Kubernetes pod priority and

preemption enables the Kubernetes scheduler to schedule
USING POD more important Pods and associated workloads first. For
PRIORITY example, when a cluster becomes constrained in some
way, it will remove less important pods (specified with
lower pod priority relative to other pods) to create room
for more important (higher priority) pods. To guard against
malicious activity, where an unethical individual creates a
demanding pod with highest priority to effectively starve all
other pods in a cluster, version 1.14 extends ResourceQuota
to include priority as a “stable” feature. (The feature was
first introduced in beta form in Kubernetes 1.12.) With this
functionality, an admin can specify ResourceQuota for users
at specific priority levels, preventing them from creating
pods with high priorities.

As a Kubernetes 1.14 best practice, one can use pod priority

to help the scheduler know which pods can be shut down
and which pods need to be maintained when resources
become constrained, or activity spikes unexpectedly. Be sure
to extend PriorityClass in your pod templates, as existing
pods will not have this feature turned on by default.

13 www.alcide.io • [email protected]
KUBERNETES
SECURITY AND
PERFORMANCE
CONSIDERATIONS

PERFORMANCE: Although this one mainly falls under the category

of performance, it also applies to orchestration and
LOCAL
automation.
PERSISTENT
Kubernetes workloads tend to use different types of block
VOLUMES
and file storage to persist data. While most use cases
involve remote storage (e.g. cloud-based), they don’t
always provide the level of consistent performance needed
for many applications. With the Local Persistent Volume
plugin, officially released as stable as of v1.14, Kubernetes
workloads can now use local storage (typically with higher
performance) using the same volume APIs as in the past.

The difference is that, compared with the similarly featured

Kubernetes HostPath Volume, the Kubernetes scheduler
understands which node a Local Persistent Volume belongs
to. This means, unlike HostPath Volumes, that the scheduler
will ensure workloads that rely on local storage stick to
nodes that have that storage mounted and available. Since
it can be referenced only via a Persistent Volume Claim,
using Persistent Local Volumes adds an additional layer of
security, as you have more control over access to them.

In terms of best practices, remember that using Persistent

Local Storage limits the nodes the Kubernetes scheduler can
run dependant workloads on. This is where the orchestration
advantages of this new feature become important for
stability. However, for these same reasons, Local Persistent
Volumes should only be considered for workloads that
require the highest performance, or that handle data
replication and backup at the application layer. This helps
make the applications resilient to node or data failures,
remaining available despite the lack of guarantees at the
individual disk level.

Examples of candidate workloads include software defined

storage systems and replicated databases. Other types of
applications should continue to use highly available cloud-
based storage.

14 www.alcide.io • [email protected]
 KUBERNETES AND
DEVOPS
A final essential requirement for making the most
of Kubernetes is to integrate Kubernetes effectively
into your DevOps workflow and culture. On this topic,
there is not a specific set of technical procedures to
follow; instead, the focus is on understanding which
values a DevOps culture prioritizes, and identifying
ways in which Kubernetes can reinforce them.

15 www.alcide.io • [email protected]
KUBERNETES AND
DEVOPS

REDUCING One key goal of any DevOps practice is to implement tools

and processes that reduce the complexity developers
COMPLEXITY
and IT Ops teams have to contend with when delivering
applications.

In theory, containers naturally help to reduce complexity.

Containers can be black-box tested, deployed consistently
across environments (e.g. development, QA, and production),
and provide a convenient way to enable cloud and
microservice-based architectures and deployments.

However, if developers simply throw containers over the wall,

it’s left up to operations teams to deploy and manage them,
and figure out how to scale them to meet user demand. This
increases rather than decreases the complexity required to
work with containerized applications, undercutting the main
reason for adopting containers in the first place.

Fortunately, Kubernetes helps to resolve this pain point

by allowing developers and operations to work together
to automate the deployment and runtime orchestration of
containers across nodes in your datacenter or cloud. Going
further, Kubernetes can be used to orchestrate the process
of upgrading containers in production so that users aren’t
impacted.

16 www.alcide.io • [email protected]
KUBERNETES AND
DEVOPS

CONTINUOUS In a DevOps world, continuous is an important and

overloaded word. For example, tools are available to
DEVELOPMENT,
automatically and continuously perform software builds and
INTEGRATION, deployments based on developer actions, called triggers.
AND DELIVERY One trigger may be the simple act of checking code into
a repository. The automation that continuous integration
(CI) tools provides removes much of the manual effort (as
well as bottlenecks and human error) from an otherwise
mundane set of tasks. With the rise of containers, this
automation needs to be extended to support a CI process.

Instead of moving code across different servers, or VMs

running on cloud infrastructure, the same code can be
automatically deployed across pods within a Kubernetes
cluster. Kubernetes expands your DevOps culture within a
container-based architecture by:

y Increasing collaboration between developers and

operations staff

y Managing rolling container updates across Kubernetes

nodes

y Orchestrating deployments regardless of platform (e.g.

for Windows or Linux workloads)

y Scaling your containers across pods based on need

predictably

y Increasing productivity by deploying software to

containers for QA immediately

y Optimizing costs by efficiently utilizing server resources

across a Kubernetes cluster (e.g. not using more cloud
compute or bandwidth other than what’s required to
meet user demand)

17 www.alcide.io • [email protected]
KUBERNETES AND
DEVOPS

TEST-DRIVEN A test-driven approach to development (TDD) requires

application owners to define tests alongside requirements
DEVELOPMENT
before code is even written. The theory is: If you can’t
describe how to test it, how can you be sure you’re
specifying the requirement thoroughly, or even correctly?
Additionally, thinking about how to test (or simply use) a
software feature and its code often leads to more well-
thought out software design, and fewer bugs in production.

The trick is to make TDD part of your deployment process

(and part of your entire DevOps practice) by integrating it
into your tools. Breaking down your application into very
testable stand-alone services increases confidence that
the system won’t break when the services are individually
deployed. Kubernetes supports this by taking your container-
based microservice architecture and deploying container
updates without disturbing other parts of your application.

In effect, Kubernetes extends the TDD mentality to your

operations teams. This moves testing beyond individual
feature, unit, and system tests, and enables a DevOps
culture of end-to-end testing that more closely resembles
the system-level experience your own users will get when
they use your software in production.

KUBERNETES Good DevOps tools adapt to your process as well as

development tools. Kubernetes includes an API to help you
INCREASES
integrate it into your CI/CD DevOps practice. And since the
SECURITY goal of DevOps is not only to improve developer efficiency
but also to better serve your users’ needs, security is often
at the top of the list when driving the DevOps culture. To help
implement security into your practice, Kubernetes allows
you to control who can access container clusters, manage
and restrict updates to container code and environment
variables, limit resource usage on specific pods, control the
privileges that container-based workloads run with, restrict
network access across containers, and much more.

18 www.alcide.io • [email protected]
KUBERNETES AND
DEVOPS

INTRODUCING Alcide provides continuous K8s security from dev-to-

production supporting DevSecOps by auditing their
ALCIDE
Kubernetes clusters, nodes and pods configuration to make
KUBERNETES sure their cluster is tuned and runs according to security
ADVISOR best practices and internal guidelines.

For this purpose we developed Alcide Advisor which

provides a real-time summary of your cluster’s compliance
and security status by looking at the following:

y Kubernetes vulnerability scanning

y Hunting misplaced secrets, or excessive secret access

y Workload hardening from Pod Security to network

policies

y Istio security configuration and best practices

y Ingress controllers for security best practices

y Kubernetes API server access privileges

y Kubernetes operators security best practices

The Alcide Advisor produces a report that contains:

y A summary of your cluster’s compliance and security

status

y A detailed list of identified compliance and security

issues, grouped by category. Each identified issue is
assigned a category, a severity to reflect its importance
and criticality, and recommendation for a quick
remediation.

y In addition, you can create a baseline profile based on

specific cluster scan results and compare it to other
clusters.

The result: A single-pane view for all your K8s-related

issues: audit, compliance, topology, network, policies and
threats.

19 www.alcide.io • [email protected]
CONCLUSION
Kubernetes is eating the world, with good reason. Kubernetes offers a powerful
set of features for deploying and managing containerized applications.

However, as this eBook has explained, improving IT operations with the help
of Kubernetes is not as simple as flipping a switch to turn on Kubernetes
and calling it a day. Instead, organizations must carefully plan how to set up
and manage their Kubernetes deployments. They must also choose how to
configure the long list of features that Kubernetes offers, and they must take
extra steps to optimize Kubernetes security and performance.

It’s only through careful planning and implementation that Kubernetes can
achieve its promise of making the delivery of containerized applications faster
and more efficient — and, by extension, advance the goals of organizations that
have embraced DevOps.

HOW CAN WE HELP

Alcide Advisor closes the loop between people who code, deliver and secure
apps. The Alcide Advisor is also available for a 30-day free trial.

GET IT HERE

20 www.alcide.io • [email protected]