1) Node Selector:: Let Us Start With A Simple Example
1) Node Selector:: Let Us Start With A Simple Example
1) Node Selector:: Let Us Start With A Simple Example
Limitations:
We used a single label and selector to achieve our goal here.
But what if our requirement is much more complex.
For example, we would like to say something like place the pod on a large or medium node
or something like place the pod on any nodes that are not small.
You cannot achieve this using Node selectors for this node affinity and anti affinity features
were introduced.
2) Node Affinity:
The primary purpose of node affinity feature is to ensure that pods are hosted on particular
node. Affinity feature provides us with advanced capabilities to limit pod placement on
specific nodes with great power comes great complexity.
Available:
Type 1: requiredDuringSchedulingIgnoredDuringExecution
This type will be used in cases where the placement of the pod is crucial, If a matching
node does not exist the pod will not be scheduled.
Ex: If there is no label on Node, and POD tries to placed on this node with some label then
it will not be scheduled. Because node mandating the type of node.
Type 2: preferredDuringSchedulingIgnoredDuringExecution
But let's say the pod placement is less important than running the workload itself.
In that case you could set it to preferred and in cases where a matching node is not found.
The scheduler will simply ignore node affinity rules and place the card on any available
note. This is a way of telling the scheduler hey try your best to place the pod on matching
node but if you really cannot find one just plays it anywhere.
As you can see the two types of node affinity available today has this value set too ignored
which means pods will continue to run and any changes in node affinity will not impact them
once they are scheduled.
Planned:
Type 3: requiredDuringSchedulingRequiredDuringExecution
A pod running on the large node will be evicted or terminated if the label large
is removed from the node.
3) Taints and Toleration and Node Affinity:
In example we want to place green pod in green Node and so on..
To achieve this we are taking example of above mentioned methods.
2) Node Affinity:
In this other pod my end up in landing in Red Node, so individually this method is also not
usable.
3) Combination of both:
In this case combination of both may solve the issue.
Ex: It first try to schedule on 1st Node but resources not available and scheduled on Node 2.
If there is no sufficient resources available on any of the nodes, Kubernetes holds back
scheduling the POD, and you will see the POD in a pending state.
If you look at the events, you will see the reason – insufficient cpu. Let us now focus on the
resource, resource requirements for each POD.
5) Daemon set:
Daemon set are like replica sets, as in it helps you deploy multiple instances of pod. But it
runs one copy of your pod on each node in your cluster.
Whenever a new node is added to the cluster a replica of the pod is automatically added to
that node and when a node is removed the pod is automatically removed.
The demon set ensures that one copy of the pod is always present in all nodes in the
cluster.
6) Static Pod:
PODs that are created by the kubelet on its own without the intervention from the API server
or rest of the kuberentes cluster components are known as Static PODs.
You can configure the kubelet to read the pod definition files from a directory on the server
designated to store information about pods.
The pods definition files in this directory the Kubelet periodically checks this directory for
files reads these files and creates pods on the host.
Not only does it create the pod it can ensure that the pod stays alive. If the application
crashes, the kubelet attempts to restart it.
If you make a change to any of the file within this directory, the kubelet recreates the pod for
those changes to take effect.
If you remove a file from this directory the part is deleted automatically.
You cannot create replicasets or deployments or services by placing a definition file in the
designated directory.
So what is that designated folder and how do you configure it.
It could be any directory on the host. And the location of that directory is passed in to the
kubelet as a option while running the service.
The option is named pod manifest path and here it is set to /etc/Kubernetes/manifests.
There is also another way to configure this
Instead of specifying the option directly in the kubelet.service file, you could provide a path
to another config file using the config option, and define the directory path as staticPodPath
in that file.
Clusters setup by the kubeadmin tool uses this approach. If you are inspecting an existing
cluster, you should inspect this option of the kubelet to identify the path to the directory.
Since static pods are not dependent on the Kubernetes control plane, you can use static
pods to deploy the control plane components itself as pods on a node. Start by installing
kubelet on all the master nodes.
Then create pod definition files that uses Docker images of the various control plane
components such as the api server, controller, etcd etc.
Place the definition files in the designated manifests folder. And kubelet takes care of
deploying the control plane components themselves as PODs on the cluster.
This way you don't have to download the binaries configure services or worry about so the
service is crashing.
If any of these services were to crash since it's a static pod it will automatically be restarted
by the kubelet.
Difference betn Static Pods and DaemonSets
7) Multiple Scheduler:
You can write your own kubernetes scheduler program, package it and deploy it as the
default scheduler or as an additional scheduler in the kubernetes cluster.
That way all of the other applications can go through the default scheduler, however one
specific application can use your custom scheduler.
Your kubernetes cluster can have multiple schedulers at the same time.
As per configuration file shown below, left side is default file whereas the right side is custom one.
Adding 2 parameters,
Scheduler name : New custom scheduler name
In case you do have multiple masters, you can pass in an additional parameter to set a lock
object name. This is to differentiate the new custom scheduler from the default during the leader
election process
You may see in kube-system, two scheduler defined, one is default and other one is customized.
We need a solution that will monitor these metrics store them and provide analytics around
this data as of this recording, Kubernetes does not come with a full featured built-in
monitoring solution.
Heapster is now Deprecated and a slimmed down version was formed known as the Metrics
Server.
Metric server is only an in memory monitoring solution and does not store the metrics on
the desk and as a result you cannot see historical performance data.
Kubernetes runs an agent on each node known as the kubelet, which is responsible
for receiving instructions from the kubernetes API master server and running PODs
on the nodes.
The kubelet also contains a subcomponent known as as cAdvisor or Container
Advisor. cAdvisor is responsible for retrieving performance metrics from pods, and
exposing them through the kubelet API to make the metrics available for the Metrics
Server.
This provides the CPU and Memory consumption of each of the nodes.Use the kubectl top
pod command to view performance metrics of pods in kubernetes.
This helps us keep track of the changes made to our deployment and enables us to roll
back to a previous version of deployment if necessary.
There are 2 strategies for deployments:
A) Re-Create Strategy
Say for example you have five replicas of your web application instance deployed.
One way to upgrade these to a newer version is to destroy all of these and then create
newer versions of application instances meaning first destroy the five running instances and
then deploy five new instances of the new application version.
The problem with this as you can imagine is that during the period after the older versions
are down and before any newer version is up the application is down and inaccessible to
users this strategy is known as the Recreate strategy,
Rollback:
After deployment if any issue observe you may rollback to previous version by using rollout
undo command.
Use command to check the Strategy used to schedule a deployment, Re-Create or Rolling
update:
Kubectl describe deployment < >
2) Configuration Environment Variable:
ConfigMaps:
When you have a lot of pod definition files it will become difficult to manage the environment
data stored within the query files.
We can take this information out of the pod definition file and manage it centrally using
Configuration Maps. ConfigMaps are used to pass configuration data in the form of key
value pairs in Kubernetes.
When it pod is created inject the config map into the pod, so the key value pairs that are
available as environment variables for the application hosted inside the container in the pod.
There are 2 ways to create configMap,
Imperative Way: Without using a ConfigMap definition file. The parameter will be specified
in ConfigMap command itself.
You may also specify the parameter in file and you mention the path of file, like 3 rd and 4th
option.
Declarative Way: In this we will use the create command with definition file.
Use the from file option to specify a path to the file that contains the required data.
The data from this file is read and stored under the name of the file let us now look at the
declarative approach.
View ConfigMaps:
Inject in Pod:
Specify the name of the configmap we created earlier.
This is how we inject a specific configmap from the ones we created before. Creating the
pod definition file now creates a web application with a blue background. What we just saw was
using configMaps to inject environment variables.
Secrets:
To separate the username and password used in pods, can be configure in secrets. This will provide
encoding of the normal username and password.
Steps involve for execution is same as ConfigMap 1. Create the Secret
2. Inject in pod.
Again While creating secret 2 ways, Imperative and Declarative.
Imperative:
Declarative:
Secrets in Pod:
Scale Applications
We have already discussed about scaling applications in the Deployments
and Rolling updates and Rollback sections.
MultiContainer Pod:
Which enables us to develop and deploy a set of independent small and reusable code.
This architecture can then help us scale up down as well as modify each service as
required as opposed to modifying the entire application, however at times you may need
two services to work together such as a web server and a logging service.
If we need one Log Agent per Web Server then there is possibility we may accomadate
them in one single POD. One Pod can have multiple container.
These containers share the same lifecycle which means they are created together and
destroyed together. They share the same network space which means they can refer to
each other as local host and they have access to the same storage volumes.
There are 3 different types of Multi Container design: The details are not included in CKA
exam.
InitContainers:
That is a task that will be run only one time when the pod is first created Or a
process that waits for an external service or database to be up before the
actual application starts. That's where initContainers comes in.
1. apiVersion: v1
2. kind: Pod
3. metadata:
4. name: myapp-pod
5. labels:
6. app: myapp
7. spec:
8. containers:
9. - name: myapp-container
10. image: busybox:1.28
11. command: ['sh', '-c', 'echo The app is running! && sleep 3600']
12. initContainers:
13. - name: init-myservice
14. image: busybox
15. command: ['sh', '-c', 'git clone <some-repository-that-will-be-used-by-
application> ; done;']
When a POD is first created the initContainer is run, and the process in the
initContainer must run to a completion before the real container hosting the
application starts.
You can configure multiple such initContainers as well, like how we did for multi-pod
containers. In that case each init container is run one at a time in sequential order.
If any of the initContainers fail to complete, Kubernetes restarts the Pod repeatedly
until the Init Container succeeds.
1. apiVersion: v1
2. kind: Pod
3. metadata:
4. name: myapp-pod
5. labels:
6. app: myapp
7. spec:
8. containers:
9. - name: myapp-container
10. image: busybox:1.28
11. command: ['sh', '-c', 'echo The app is running! && sleep 3600']
12. initContainers:
13. - name: init-myservice
14. image: busybox:1.28
15. command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice;
sleep 2; done;']
16. - name: init-mydb
17. image: busybox:1.28
18. command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2;
done;']
When a POD is first created the initContainer is run, and the process in the
initContainer must run to a completion before the real container hosting the
application starts.
You can configure multiple such initContainers as well, like how we did for
multi-pod containers. In that case each init container is run one at a time in
sequential order.
It helps in ensuring enough replicas of the application are running at all times.
1) OS Upgrades:
In normal scenario, if any node goes down, application on respective Pods also goes down.
If same pods are available on other node, Application will be running.
But if no other node carries similar Pod then application will be down.
However, if the node was down for more than 5 minutes, then the pods are terminated from that
node.
If the PODs where part of a replicaset then they are recreated on other nodes.
The time it waits for a pod to come back online is known as the pod eviction timeout and is
set on the controller manager with a default value of five minutes.
So whenever a node goes offline, the master node waits for upto 5 minutes before
considering the node dead.
When the node comes back on line after the pod eviction timeout it comes up blank without
any pods scheduled on it.
Kubectl cordon node-2 : - Marks the node un-schedulable, so It simply makes sure that new
pods are not scheduled on that node.
2) Kubernetes Releases:
While upgrade, you cannot jump from 1.10 to 1.13 directly, we need to first upgrade from 1.10 to
1.11 then 1.12 and so-on.
Upgrade Process:
If you have 3 worker nodes and 1 Master node.
We will upgrade first master node, drain the node and shift the pods to worker node. Getting down
master node will not impact on other worker nodes and respective application.
Here in example, Master node upgraded to V1.11 from V1.10.
Strategy-1:
Upgrade all of them at once but then your ports are down and users are no longer able to
access the applications.
Once the upgrade is complete the nodes are back up new paths are scheduled and users
can resume access.
Strategy-2:
Upgrade the first node where the workloads move to the second and third node and users
are so from there.
Once the first node is upgraded and backed up with an update the second node where the
workloads move to the first and third nodes and finally the third node where the workloads
are shared between the first two until we have all nodes upgraded to a newer version.
Strategy 3:-
Add new node to the cluster nodes with newer software version.
This is especially convenient if you're on a cloud environment where you can easily
provision new nodes and decommission old ones nodes with the newer software version
can be added to the cluster.
Kubeadm-upgrade:
We use kubeadm to upgrade the component of cluster.
With mentioned command, you may see current version of component and available
version. As mentioned, kubelet we need to upgrade manually. Commands to upgrade
mentioned below
As per standard process, you need to drain the node first by drain command.
Then login to respective node with ssh prompt, execute mentioned command.
1) Kubectl drain node-1
Drain the node
Normally we do definition with Imperative and Declarative way, if we use delclarative method all
the way, then we may copy the Resource file to GitHub as repository.
But many times we use imperative way, which may loose the backup data
Better way to take backup by kube-api server,one of the commands that can be used in a
backup script is to get all pods, deployments and services in all namespaces using the
kubectl utility’s get all command and extract the output in a YAML format. Then save that
file. And that’s just for a few resource groups.
The ETCD cluster stores information about the state of our cluster. So information about the
cluster itself. The nodes and every other resource as created within the cluster are stored
here. So instead of backing up resource as before, you may chose to backup the ETCD
server itself.
As we have seen the ETCD cluster is hosted on the master nodes. While configuring ETCD
we specified a location where all the data would be stored.
The data directory. That is the directory that can be configured to be backup by your backup
tool.
A) Security Primitives:
Kube-api server is first point of contact to Kubernetes system, we need to apply security in
Kube-api for authentication and authorization.
Authentication: who can access the host ?
Authorization: What can user do ?
All communication with the cluster, between the various components such as the ETCD
cluster, kube controller manager, scheduler, api server, as well as those running on the
worker nodes such as the kubelet and and kubeproxy is secured using TLS Encryption.
By default all PODs can access all other PODs within the cluster.
You can restrict access between them using Network Policies.
B) Authentication:
There are 2 types of user who access the nodes
1) Humans (Admin or developer)
2) Boats (Third party server )
All user access is managed by the API server. Weather you are accessing the cluster
through kubectl tool or the API directly, all of these requests go through the kube api server.
The kube-api server authenticates the requests before processing it.
Different way of Auth Mechanism:
Static Password File :- Here password saved in one of file and reference given in kub-api
Static Token File: Here instead of password, token mentioned in file.
Certificate: You may authenticate by TLS certificate
Identity Service : Third party auth protocol LDAP.
If you setup your cluster using the kubeadm tool, then you must modify the kube-apiserver
POD definition file. Kube-adm tool will automatically restart the kube-api server once you
update the file.
While authentication, need to use command like mentioned
“user:password”
C) TLS Certificate:
A certificate is used to guarantee trust between two parties during a transaction.
For example, when a user tries to access a web server, TLS certificates ensure that the
communication between the user and the server is encrypted and the server is who it says it
is.
There are 2 types of Encryption,
Symmetric Encryption
Asymmetric Encryption : In this private and public key concept will be there.
Client Side:
Ssh-keygen command: To generate the private and public key.
The complete process for TLS certificate and authentication bent Client and Server:
1) The server uses a pair of keys to secure HTTPS traffic.
2) For this the server first sends a certificate signing request to a CA.
3) The CA uses its private key to sign the CSR.
4) Remember all users have a copy of the CAs public key.The signed certificate is then
sent back to the server the server configure is the web application withthe signed
certificate.
5) Whenever a user accesses the web application the server first sends the certificate
with its public key.
6) The user or rather the user's browser reads the certificate and uses the CA's public
key to validate and retrieve the servers.
7) Public key it then generates a symmetric key that it wishes to use going forward for
all communication. The symmetric key is encrypted using the server as public key
and sent back to the server
8) The server uses its private key to decrypt the message and retrieve the symmetric
key.
9) The symmetric key is used for communication going forward so the administrator
generates a key pair for securing SSH.
10) The web server generates a key pair for securing the web site with HTTPS, the
Certificate Authority generates its own set of key pair to sign certificates.
11) The end user though only generates a single symmetric key.
12) Once he establishes trust with the Web site he uses his username and password to
authenticate the Web server with the servers key pairs.
13) The client was able to validate that the server is who they say they are but the server
does not for sure know if the client is who they say they are.
14) It could be a hacker impersonating a user by somehow gaining access to his
credentials not over the network for sure.
15) The server can request a certificate from the client and so the client must generate a
pair of keys and a signed certificate from a valid CA the client then sends the
certificate to the server for it to verify that the client is who they say they are.
The process of generating distributing and maintaining digital certificates is known as public
key infrastructure or PKI.
Public key file always have *.crt or *.pem format, whereas private key will have *.key or
*.key.pem format.
Every component in Kubernetes communicate with other component with secure way.
So every component has its own private key and public key.
Kube-api communicate with ETCD and Kubelet, in this case kube-api becomes client and
ETCD and kubelet becomes Server, so it can have multiple public and private key
respectively. One for other componenent, one for ETCD and one for kubelet.
o Since this is for the CA itself, it is self-signed by the CA using its own private
key that it generated
opensslx509 -req -in ca.csr-signkeyca.key-out ca.crt
Certificate for Admin User:
In this we need to use ca.key and ca.crt of CA certificate for generation of Admin user certificate.
This will be different for each node and will have different name accordingly.
Every node will have standard name “system:node:node01” and so on. Similarly certificate
will also mention the group of certificate “System:nodes”
View Certificate existing cluster:
There is two type you have install the cluster by Hardway or by automated using Kubeadm.
The parameter we need to check in certificate are-
Certificate path:
CN Name:
ALT Name:
Organization:
Issuer:
Expiration:
Make a table to troubleshoot the issue.
In mentioned path file you will find the certificate name, use openssl command to decode
and view details of those certificate.
Inspect the Service log to troubleshoot.
View Logs in kubernets:
Kubectl logs etcd-master
If kubectl is down and service unavailable, need to check in docker.
docker ps-a
docker logs 87fc
Section 7: Storage
A) Volumes:
When the container created, it processes the data. When container got deleted the data
also get deleted.
In Kubernetes, we attach volume to the container, so the data will be stored in Volume and
even though the container got deleted data will be persistent.
In mentioned yml file of pod creation, we have mentioned args, which will give a random
number.
Volumes: Storage definition and its directory path in node /data, it will also be stored on
database
volumeMounts: Where as volumeMounts will be /opt in node, which is not persistent.
Here we are using volume as aws Elastic Block Store.
B) Persistent Volume:
As we know we may use volume from external storage, then we may create definition file for
Persistent Volume.
C) Persistent Volume Claims:
An Administrator creates a set of Persistent Volumes and a user creates Persistent Volume
Claims to use to storage.
Once the Persistent Volume Claims are created, Kubernetes binds the Persistent Volumes
to Claims based on the request and properties set on the volume.
Every Persistent Volume Claims is bound to single Persistent volume during the binding
process Kubernetes tries to find a persistent volume that has sufficient capacity as
requested by the claim and any other request properties such as access modes volume
modes storage class etc.
When the claim is created, kubernetes looks at the volume created previously. The access
Modes match.
The capacity requested is 500 Megabytes but the volume is configured with 1 GB of
storage. Since there are no other volumes available.
The persistent volume claim is bound to persistent volume when we run to get volumes
command again.
Section 8 : Networking
A) Switching
B) DNS
C) Network Namespaces:
# Create network namespaces
ipnetnsadd red
ipnetnsadd blue
# Create vethpairs
iplink add veth-red type vethpeer name veth-blue
# Create Add vethto respective namespaces
iplink set veth-red netnsred
iplink set veth-blue netnsblue
# Set IP Addresses
ip-n red addradd 192.168.1.1 dev veth-red
ip-n blue addradd 192.168.1.2 dev veth-blue
# Check IP Addresses
ip-n red addr
ip-n blue addr
# Bring up interfaces
ip-n red link set veth-red up
ip-n blue link set veth-blue up
# Bring Loopback devices up
ip-n red link set lo up
ip-n blue link set lo up
# Add default gateway
ipnetnsexec red iproute add default via 192.168.1.1 dev veth-red
ipnetnsexec blue iproute add default via 192.168.1.2 dev
ipnetnsdel red
ipnetnsdel blue
iplink del v-net-0
iptables -t nat-D POSTROUTING 1
#
ipnetnsadd red
ipnetnsadd blue
iplink add veth-red type vethpeer name veth-red-br
iplink add veth-blue type vethpeer name veth-blue-br
iplink set veth-red netnsred
iplink set veth-blue netnsblue
ip-n red addradd 192.168.15.2/24 dev veth-red
ip-n blue addradd 192.168.15.3/24 dev veth-blue
brctladdbrv-net-0
iplink set dev v-net-0 up
iplink set veth-red-brup
iplink set