Study notes on Kubernetes and Helm

• Kubernetes probes:  Liveness and Readiness
• Liveness probe: container level, if hung/crash, kubelet kill container, then based on restart policy may restart container locally, or restart on other nodes due to lack of resources or due to Kubernetes QoS policies;
• Readiness probe: service level, if service interrupted, Pod IP will be removed from endpoints, thus malfunctioning node taken off load balancer while healthy/ready nodes remain in service
• Three types handlers for both liveness and readiness probes:
• ExecAction:    execute command, exit code 0 success, others fail; better for non-tcp and non-http services;
• TCPSocketAction:  test connectivity of ports on container IP; if can connect, success; only test connectivity, if service is not functioning but port is still listening, it may not cache the error condition;
• HTTPGetAction: HTTP GET requests, code 200 < > 400 is success;  better for http service health check, need provide health check path;
• Three test results:
• Success
• Failure
• Unknown
• Recommend to have both liveness and readiness probes,  can be co-existed in the same containers; timeout for readiness probe should be less than that of liveness probes; First get the malfunctioning service node off load balancer; if can't self-healing/recover,  liveness probe kicks in and restart container based on restart policy (restart container or start a new pod from other nodes)
• Challendges introduced by Kubernetes framework: 
• manage/update configuration files in k8s;
• deploy k8s apps with large amount of configuration files;
• share and re-use k8s configuration files /settings;
• parameterized templates supporting different running environment;
• manage app lifecycles: release, rollback, diff, history etc;
• service orchestration/control specific steps in a release cycle;
• verification after release/deployment;
• Helm introduced to solve above challenges;  like apt-get/yum to linux as Helm to Kubernetes; Built by Deis, acquired by Microsoft; Help package Kubernetes resources (such as deployments, services, ingress etc) into a chart;  in turn, charts can be stored and shared in chart repository; Official chart repositories maintened by Kubernetes Charts, but Helm allow us to build private chart repositories; It has two components:  helm client (CLI) and Tiller Server; both implemented in Go, using gRPC for Helm client/server communications; Tiller server using kubernetes client library to communicate with Kubernetes using REST + JSON format; 
• Helm has three important concepts:
• chart:  include all information needed to deploy a service instance on Kubernetes;
• config: include all configuration/settings in an application release;
• release:  A running instance of a chart and its associated config/settings;
• Helm can do following:
• create a new chart, 
• package chart in to tgz format; 
• upload/download chart to chart repository; 
• deploy and delete chart into kubernete cluster; 
• manage release lifecycles of charts installed by Helm;
• Helm Client CLI can do following:
• Develop chart locally;
• Manage chart repository;
• Communicate/Interact with Tiller server;
• Release chart;
• Qurey release status;
• Upgrade or delete release 
• Tiller Server is a server deployed on kubernetes cluster,it doesn't have any DB backend, instead it uses Kubernetes ConfigMaps to store information;  it interacts with Helm client and Kubernetes API server; it is responsible for:
• Listen on requests from Helm client;
• Construct a release using chart and its configuration/settings;
• Install chart onto Kubernetes cluster and follow along later releases deployed;
• Interacte with Kubernetes to upgrade or delete chart;
• Helm Client manages charts, Tiller Server manages releases
• Chart repository can store index.yml and packaged chart files on HTTP server. Any HTTP server that can provide YAML and tar files can be turned into chart repository, such as GCS (Google Cloud Storage), Amazon S3, Github Pages, or your own web server;
• An example Helm release process:
• helm fetch stable/mysql --version 0.2.6 --untar    (retrieve mysql 0.2.6 package and untar)
• helm lint mysql   ( checking failures in chart)
• helm create mychart   (create mychart directory, including mychart/{charts, Chart.yaml, templates/{deployment.yaml,_helpers.tpl,ingress.yaml,NOTES.txt,service.yaml}, values.yaml}) ,  2 directories, 7 files
• Chart.yaml:  metadata of chart,  includes: name, description, release version etc;
• values.yaml:  variables stored in template files;
• templates:   all template files;
• charts/:   storage path for chart
• NOTES.txt:  information for post-deployment uses, such as how to use chart, default values etc;
• Install chart has following methods:
• helm install stable/mariadb   (specify chart name)
• helm install ./nginx-2.1.0.tgz   (specify package file)
• helm install ./nginx  (specify directory contains packages)
• helm install https://example.com/charts/nginx-2.1.0.tgz (specify chart URL)
• Set vlaues in configuration has following methods:
• helm install -f myvalues.yaml ./redis   (Specify a config file)
• helm install --set name=prod ./redis (Specify key=value format)
• helm install -n mysql -f mysql/values.yaml --set resources.requests.memory=512Mi mysql     (deploy a release)
• helm status mysql   (check release status)
• helm list -a  (check release, tag "-a" for all releases, deployed/failed-deployment/deleting/deleted-releases )
• helm upgrade mysql -f mysql/values.yaml --set reources.requests.memory=1024Mi mysql  (upgrade a deployed release)
• helm get --revision 1 mysql   ( you would see memory was set to 512Mi)
• helm get --revision 2 mysql   ( you would see memory was set to 1024Mi)
• helm rollback --debug mysql 1    ( it would rollback and memory would be set to 512Mi)
• helm delete mysql   ( even after deleted, the release history would still be preserved and can be recovered )
• helm rollback --debug mysql 2  ( a deleted release got recovered, because information are stored in Kubernetes ConfigMap, when delete/upgrade Tiller, the application config/setting/data are preserved )
• helm list -a   (see all releases)
• helm hist  (check a specific release)
• helm get --reviesion (get configuration information for a specific release , even the release was deleted, information won't lost and can still be rollback)
• Helm configuratrion management function,  Helm Chart is using Go template language,  variable values are stored in values.yaml, its values can be replaced using  --values YAML_FILE_PATH or --set key1=value1, key2=value2 during chart deployment; {{...}} for the template varibles, it can also be set in following ways:
• PIPELINES:  like Linux/Unix pipeline,  drink: {{.Values.favorite.drink|repeat 10 |quote }}; food: {{ .Values.favorite.food|upper|quote }} 
• Named template,  defined in _helpers.tpl,  in the format of  ({{/* ... */}}) , then we can use it using include such as:  {{- include "my_labels" }} to insert into configMap metadata or {{ include $mytemplate }};
• SubChart:    a chart depends on other chart, the one depend on is subchart, 
• subchart don't need depend on parent chart, it is independant and has its own values and templates; 
• subchart can't access parent chart values, but parent chart can overwirte subchart values
• subchart or parent chart can both access global values in helm;
• Helm Hook, (helm install), (helm upgrade), (helm rollback), (helm delete), all can have hooks It has following types:
• pre-install
• post-install
• pre-delete:
• post-delete
• pre-upgrade
• post-upgrade
• pre-rollback
• post-rollback
• Helm hook has weights, can be negative or positive values, will be loaded according to the weight but not garenteed, if weight is the same, it will be in a-z alphabetic order, may be changed in future releases;  Helm can have two strategies for deleting hook resource:  
• hook-succeeded
• hook-failed 
• Control service start sequence, two strategies:
• Init Container:  pod/rc/deployment yams files only describe container start sequence, not service start sequence, but Kubernetes Init Container can do this; command: ['sh', '-c', "curl --connect-timeout 3 --max-time 5 --retry 10 --retry-delay 5 --retry-max-time 60 serviceB:portB/pathB/"]
• Helm Hook:  pre-install hook, put above command in to  pre-install-job container as pre-install hook;  Since Helm Hook is blocking operation, it will block before helm install, thus control the service start sequence;
• QoS in Kubernetes, two categories:
• requests: 0~node-allocatable;   0 <= requests <=node-allocatable
• limits:  requests~unlimited;  requests <= limits <= Infinity
• If CPU usage > limits,  pod won't be killed but will be limited; if no limits set, pod could use all cpu 
• If MEM usage > limits, pod container processes will be killed by OOM, if killed by OOM, it will try to restart container or create new pod on local node or other node;
• QoS class: 
• Guaranteed: all containers in a pod must have the same limits, if one container has requests, all other containers need have it set to the same value as limits, thus this pod is having a Guaranteed QoS class; (by default, request = limit if request is not defined)
• Burstable: if at least one of containers in a pod has different requests and limits, this pod has burstable QoS class;
• Best-Effort: if all resources in pod do not have request and limits, this pod has Best-Effort QoS class;
• CPU resource can be squeezed:  if pod is exceeding limits, process CPU will be limited but won't be killed;  MEM resource can't be squeezed: if a node is short on memory, process will be kernel killed by OOM;
• Best-Effort pods -> Burstable pods -> Guaranteed pods  (highest  QoS class);  
• Best-Effort pods:  if MEM is used up, these pods will be killed first;
• Burstable pods:  if MEM is used up, and there is no other Best-Effort containers, these pods will be killed;
• Guaranteed pods:  if MEM is used up, and there is no other Burstable and/or Best-Effort container, these pods will be killed;
• If a pod processes exceed the limits but node still has resources, system will try restart container on that node or create new pod on local node or other nodes;
• If have enough resources, using Guaranteed QoS pods to get stability;
• Use different QoS class for different services to gain better resource utilization while maintain service quality;
• Tiller service is in the form of a pod; using cluster IP,  helm client using kubectl proxy to connect; 
• Kubernetes service discovery has 2 methods:
• Environment  variables:  may not be visible  if the pod defines the variable is started later; 
• Kubernetes DNS: recommended
• Can  use  HAProxy/Nginx to replace kibe-proxy for better performance