My Kubernetes Book

Kubectl tips

Mon, 01 Feb 2021 00:00:00 +0000

This page will gather some tips to use kubectl in a more effective way.

Watch resources status change

kubectl have a useful feature to monitor the changes in the status of a resource. For example, when you deploy a pod, there are multiple status before Running (See Pod Lifecycle documentation).

Using the --watch/-w option for a get command, enable to watch continuously the status changes of resources:

# Will provide a new line for each status change in the pods list
kubectl get <resource-type> --watch  # long format
kubectl get <resource-type> -w       # short format

More and more details

The following commands will provide you more and more details about a specific resource.

# get resource items names only
kubectl get <resource-type> -o name
# list resource items
kubectl get <resource-type>
# display labels
kubectl get <resource-type> --show-labels
# list resource items with more details
kubectl get <resource-type> -o wide
# get a specific resource item
kubectl get <resource-type> <resource-name>
# get a specific resource item with more details
kubectl describe <resource-type> <resource-name>
# yaml export
kubectl get <resource-type> <resource-name> -o yaml
# json export
kubectl get <resource-type> <resource-name> -o json

Troubelshooting

You can use kubectl to troubleshoot your clusters, deployments or other resources.

# Monitor pod metrics
kubectl top pod <pod-name>
# Monitor node metrics
kubectl top node <node-name>

# Look at pod's logs
kubectl logs -f <pod-name>
# Look at a specific pod's container logs
kubectl logs -f <pod-name> -c <container>

# Mark node as unschedulable
kubectl cordon <node-name>
# Empty node in preparation for maintenance
kubectl drain <node-name>

# Cluster info
kubectl cluster-info
# Dump cluster state
kubectl cluster-info dump --output-directory=/tmp/dump.txt

# Config
kubectl config view

Pod interractions

Interact with your pods with the following commands:

# Run an interactive pod
kubectl run -i --tty busybox --image=busybox -- sh
# Attach to an existing pod
kubectl attach <pod-name> -i
# or
kubectl exec --stdin --tty <pod-name> -- /bin/sh
# Exec a single command in a pod (single container pod)
kubectl exec <pod-name> -- ls /
# Exec a single command in a pod (multi container pod)
kubectl exec <pod-name> -c <container> -- ls /

# Forward port 6000 from the pod to local machine port 5000
kubectl port-forward <pod-name> 5000:6000
# Forward port 6000 from the pod to machine port 5000
kubectl port-forward --address 0.0.0.0 <pod-name> 5000:6000

All !

Use following commands to list (all) resources from (all) namespace(s):

# List resource-type items from all namespaces
kubectl get <resource-type> -A
# List all resources in current context
kubectl get all
# List all resources from all namespaces
kubectl get all -A

All nodes pre-requisites

Sun, 24 Jan 2021 00:00:00 +0000

Network configuration

# Network setup
vi /etc/netplan/50-cloud-init.yaml
netplan apply

Software pre-requisites

# Upgrade packages
sudo apt update
sudo apt upgrade -y

# Install software pre-requisites
sudo apt-get install python3-pip gcc nfs-common -y

NTP time

sudo apt-get install ntp -y
# check configured servers in:
sudo vi /etc/ntp.conf
# restart services if needed:
sudo /etc/init.d/ntp restart
# check ntp status:
ntpq -p

Basics tools

Sun, 24 Jan 2021 00:00:00 +0000

Generic pre-requisites

We will need pip and gcc for some tools, so lets install them if needed:

sudo apt-get install python3-pip gcc -y

Install `kubectl` CLI tool

In order to test and manage the k3s cluster, it could be useful to have a kubectl cli install in the master nodes:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

# Test it
kubectl version --client

`kubectl` auto-complete

Use the following commands to add auto-completion to your bash shell for the kubectl command (or it’s k alias)

echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -F __start_kubectl k' >>~/.bashrc

Helm

Easy peasy:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Traefik as ingress controler

Sun, 24 Jan 2021 00:00:00 +0000

As mentionned on Traefik website: Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience.

It could be used as a powerfull Ingress controler to handle incoming requests on the cluster and forward them to the appropriate service.

Installation

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install traefik traefik/traefik

Test it

Get the whoami.yaml file form the bellow attachement files, then:

Related files

whoami.yaml (0 kb)

# Deploy it
kubectl apply -f ./whoami.yaml

# Run a test command
curl -i http://k3s.vlab.lcl -H "Host: mydomain.com"

# Output:
HTTP/1.1 200 OK
Content-Length: 406
Content-Type: text/plain; charset=utf-8
Date: Fri, 22 Jan 2021 14:59:46 GMT

Hostname: whoami-app-7564dd9695-dtzrh
IP: 127.0.0.1
IP: ::1
IP: 10.42.2.6
IP: fe80::f099:c3ff:fe3e:7071
RemoteAddr: 10.42.1.6:46964
GET / HTTP/1.1
Host: mydomain.com
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.42.0.8
X-Forwarded-Host: mydomain.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-79fdc967bf-5nkxt
X-Real-Ip: 10.42.0.8

Here we simulate a DNS record mydomain.com by using the Host HTTP header. It provides a easy way to make a test but of course, for a real service, a DNS record is mandatory.

Here is a quick demo of the load-balancing made on the whoami service with 4 replicas:

Expose the dashboard

It is possible to expose both Traefik UI dashboard and API through Traefik itself.

Get the dashboard.yaml file from below attachements. Before applying, ensure that the content fits your needs, especially the host specified as a matching criteria.

Related files

dashboard.yaml (0 kb)

Create a secret for basic-authentication

# To get `htpasswd` tool
sudo apt install apache2-utils -y

# Create an auth file with data for user "admin"
htpasswd -c ./users admin # you will be prompted for password and confirmation

# If needed, add other user(s) to the file with **almost** the same command:
htpasswd ./users otheradmin

# Deploy the file as a new secret
kubectl create secret generic admin-auth --from-file users --namespace=default

Deploy the dashboard IngressRoute

# Deploy it
kubectl apply -f ./dashboard.yaml

# Test it:
curl -I -X GET http://lb.vlab.lcl/dashboard/ | head -n1
# Output: HTTP/1.1 401 Unauthorized

curl -Is -X GET http://lb.vlab.lcl/dashboard/ -u 'admin:*****' | head -n 1
# Output: HTTP/1.1 200 OK

Now, you probably can reach: http://lb.vlab.lcl/dashboard/# from your favorite web brower.

Arkade

Sun, 24 Jan 2021 00:00:00 +0000

Arkade is a great open-source Kubernetes “market-place” project initiated by @alexellis.

The arkade CLI provides a simple method to download your favourite devops CLIs and install helm charts, with a single command.

# Note: you can also run without `sudo` and move the binary yourself
curl -sLS https://dl.get-arkade.dev | sudo sh

# Test
arkade --help

Install a CLI tool

# List available CLI tools
arkade get

# Example:
arkade get yq

Install a K8S app

# List available apps
arkade install / --help

# Example: kubernetes-dashboard
arkade install kubernetes-dashboard

Initial master node

Sun, 24 Jan 2021 00:00:00 +0000

The following section will explain how to deploy the initial master node of the k3s cluster.

Install `kubectl` CLI tool

In order to test and manage the k3s cluster, it could be useful to have a kubectl cli install in the master nodes:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

# Test it
kubectl version --client

Auto-complete

Use the following commands to add auto-completion to your bash shell for the kubectl command (or it’s k alias)

echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -F __start_kubectl k' >>~/.bashrc

Deploy

# enable read for other users
export K3S_KUBECONFIG_MODE=0644

# Deploy a new k3s master node with embeded etcd
curl -sfL https://get.k3s.io | sh -s - --disable=traefik --cluster-init

Configuration file

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

You can test the connection to the cluster by listing the deployed pods and the list of nodes in the cluster (only one will be listed by now):

kubectl get pods --all-namespaces
kubectl get nodes

#Output:
NAME          STATUS   ROLES                       AGE     VERSION
k3s-mstr-01   Ready    control-plane,etcd,master   2m32s   v1.20.2+k3s1

Retrieve the cluster token

This token will be used to join the other nodes to the current k3s deployment:

sudo cat /var/lib/rancher/k3s/server/node-token

Monitoring with Prometheus

Sun, 24 Jan 2021 00:00:00 +0000

The kube-prometheus project collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

git clone https://github.com/prometheus-operator/kube-prometheus.git /tmp/kube-prometheus
cd /tmp/kube-prometheus
# create the namespace and CRDs
kubectl create -f manifests/setup
# wait for them to be availble
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
# create the remaining resources
kubectl create -f manifests/

Port-forward method to access dashboard

# Prometheus on https://127.0.0.1:9090
kubectl --namespace monitoring port-forward --address 0.0.0.0 svc/prometheus-k8s 9090

# Grafana with user:password admin:admin on https://127.0.0.1:3000
kubectl --namespace monitoring port-forward --address 0.0.0.0 svc/grafana 3000

# Alert Manager on http://127.0.0.1:9093
kubectl --namespace monitoring port-forward --address 0.0.0.0 svc/alertmanager-main 9093

Ingress access to dashboards

The following section will explain how you can expose the 3 previously created dashboards through a simple ingress configuration with a basic authentication.

DNS records

Create DNS records for 3 new hosts with a CNAME type and target on the endpoint of your Kubernetes cluster. For example:

alert.vlab.lcl
prometheus.vlab.lcl
grafana.vlab.lcl

Authentication

# To get `htpasswd` tool
sudo apt install apache2-utils -y

# Create an auth file with data for user "admin"
htpasswd -c ./users admin # you will be prompted for password and confirmation

# If needed, add other user(s) to the file with **almost** the same command:
htpasswd ./users otheradmin

# Create a secret named `admin-auth`
kubectl create secret generic admin-auth --from-file users --namespace=monitoring

Deploy an ingress based on Traefik

If needed, please apply content from the Traefik section to prepare Ingress controller.

Download the kube-prometheus-ingress-traefik.yaml attached file, customized with your DNS records and apply it:

Related files

kube-prometheus-ingress-traefik.yaml (1 kb)

kubectl apply -f kube-prometheus-ingress-traefik.yaml

Test access from ingress

Following commands provide a quick way to test the behavior of deployed ingress:

# Show ingress status
kubectl describe ingress grafana-gateway-ingress --namespace=monitoring

# Test your access without authentication
curl http://grafana.vlab.lcl -Is | head -n 1
# Outpout: HTTP/1.1 401 Unauthorized

# Test your access with authentication
curl http://grafana.vlab.lcl -u 'admin:*****' -Is | head -n 1
# Outpout: HTTP/1.1 200 OK

Worker node(s)

Sun, 24 Jan 2021 00:00:00 +0000

The following section will explain how to join additional worker nodes to your k3s deployment.

First of all, you will need to get the cluster token, as explained in the previous section and the master node fqdn.

Deploy k3s to join an existing cluster

curl -sfL https://get.k3s.io | \
    K3S_URL="https://k3s-mstr-01.vlab.lcl:6443" \
    K3S_TOKEN="K10d1...fe537::server:26a92...e34f9" \
    sh -

Check

From the master node or any client device with access to the Kubernetes cluster, ensure that the node is now added to the cluster:

kubectl get nodes

Additional master nodes

Sun, 24 Jan 2021 00:00:00 +0000

The following section will explain how to join additional master nodes to your k3s deployment.

To run K3s in this mode, you must have an odd number of server nodes.

First of all, you will need to get the cluster token, as explained in the previous section and the master node fqdn.

Deploy k3s to join an existing cluster


curl -sfL https://get.k3s.io | \
    K3S_URL="https://k3s-mstr-01.vlab.lcl:6443" \
    K3S_TOKEN="K10d1...fe537::server:26a92...e34f9" \
    INSTALL_K3S_EXEC="server" \
    sh - --no-deploy=traefik

Check

From the master node or any client device with access to the Kubernetes cluster, ensure that the node is now added to the cluster:

kubectl get nodes

# Output:
NAME          STATUS   ROLES                       AGE     VERSION
k3s-mstr-01   Ready    control-plane,etcd,master   4m25s   v1.20.2+k3s1
k3s-mstr-02   Ready    control-plane,etcd,master   3m33s   v1.20.2+k3s1
k3s-mstr-03   Ready    control-plane,etcd,master   2m27s   v1.20.2+k3s1

k9s

Sun, 24 Jan 2021 00:00:00 +0000

K9s is a terminal based UI to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your deployed applications in the wild.

K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.

CLI installation

curl -L https://github.com/derailed/k9s/releases/download/v0.24.2/k9s_Linux_x86_64.tar.gz | tar -xz
chmod +x k9s && sudo mv ./k9s /usr/local/bin/k9s
k9s version

Skin

Apply a pre-defined skin to the output:

mkdir -p ~/.k9s
curl -L https://raw.githubusercontent.com/derailed/k9s/master/skins/one_dark.yml > ~/.k9s/skin.yml
# or
curl -L https://raw.githubusercontent.com/derailed/k9s/master/skins/solarized_dark.yml > ~/.k9s/skin.yml

kubens & kubectx

Sun, 24 Jan 2021 00:00:00 +0000

Thanks to @ahmetb: https://github.com/ahmetb/kubectx

kubectxis a utility to manage and switch between kubectl contexts.
kubensis a utility to switch between Kubernetes namespaces.

sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx
sudo ln -s /opt/kubectx/kubens /usr/local/bin/kubens

Then you can easly switch both context and namespace by:

kubectx # list contexts
kubectx <name> # switch context

kubens # list namespace
kubens <name> # switch namespace

jsonnet

Sun, 24 Jan 2021 00:00:00 +0000

Jsonnet is a data templating language for app and tool developers

Generate config data
Side-effect free
Organize, simplify, unify
Manage sprawling config

# Update or install golang support
sudo apt install golang

# Expand PATH with `go get` binary folder
echo 'PATH="~/go/bin/:"$PATH' >> ~/.bashrc
. ~/.bashrc

# Install with go
go get github.com/google/go-jsonnet/cmd/jsonnet

# Test it
jsonnet --version

gojsontoyaml

jsonnet provides json files but it is more easy to use yaml ones. gojsontoyaml provide a simple way to convert the jsonnet result files to yaml ones.

# Install with go
go get github.com/brancz/gojsontoyaml

# Test if
echo "{'installed': true}" | gojsontoyaml

Octant

Sun, 24 Jan 2021 00:00:00 +0000

Octant is a tool for developers to understand how applications run on a Kubernetes cluster.

It aims to be part of the developer’s toolkit for gaining insight and approaching complexity found in Kubernetes. Octant offers a combination of introspective tooling, cluster navigation, and object management along with a plugin system to further extend its capabilities.

Installation

curl -L https://github.com/vmware-tanzu/octant/releases/download/v0.16.3/octant_0.16.3_Linux-64bit.deb > /tmp/octant.deb
dpkg -i /tmp/octant.deb

# To run and access octant from http://localhost:7777
octant

# To remotly run and access octant:
OCTANT_LISTENER_ADDR=X.X.X.X:7777 OCTANT_ACCEPTED_HOSTS=Y.Y.Y.Y octant --disable-open-browser

Argo cli

Sun, 24 Jan 2021 00:00:00 +0000

If you need to use Argo from this client machine.

curl -sL https://github.com/argoproj/argo/releases/download/latest/argo-linux-amd64.gz | gunzip > argo
chmod +x argo && sudo mv ./argo /usr/local/bin/argo
argo version

Configuration

# Argo namespace
export ARGO_NAMESPACE=argo

# Start port forwarding
kubectl -n $ARGO_NAMESPACE port-forward deployment/argo-server 2746:2746 &
# keep it running in bg; user the following command to get the job number:
jobs

# if needed: get it back in foreground:
fg 1

OpenFaaS CLI

Sun, 24 Jan 2021 00:00:00 +0000

If you need to use OpenFaaS from the client machine.

One-liner:

# openfaas-cli
curl -sSL https://cli.openfaas.com | sudo sh

Configuration

# Set namespace
export OS_NS="openfaas"

# Get password
export OF_PASS=$(echo $(kubectl -n $OS_NS get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode))
echo $OF_PASS

# Get URI
echo "export OPENFAAS_URL=http://"$(kubectl -n $OS_NS describe pods $(kubectl -n $OS_NS get pods | grep "gateway-" | awk '{print $1}') | grep "^Node:" | awk -F "/" '{print $2}')":31112"

echo $OF_PASS | faas-cli login --password-stdin

My Kubernetes Book

Kubectl tips

Watch resources status change

More and more details

Troubelshooting

Pod interractions

All !

All nodes pre-requisites

Network configuration

Software pre-requisites

NTP time

Basics tools

Generic pre-requisites

Install kubectl CLI tool

kubectl auto-complete

Helm

Traefik as ingress controler

Installation

Test it

Expose the dashboard

Create a secret for basic-authentication

Deploy the dashboard IngressRoute

Arkade

Install a CLI tool

Install a K8S app

Initial master node

Install kubectl CLI tool

Auto-complete

Deploy

Configuration file

Retrieve the cluster token

Monitoring with Prometheus

Port-forward method to access dashboard

Ingress access to dashboards

DNS records

Authentication

Deploy an ingress based on Traefik

Test access from ingress

Worker node(s)

Deploy k3s to join an existing cluster

Check

Additional master nodes

Deploy k3s to join an existing cluster

Check

k9s

CLI installation

Skin

kubens & kubectx

jsonnet

gojsontoyaml

Octant

Installation

Argo cli

Configuration

OpenFaaS CLI

Configuration

Login

Install `kubectl` CLI tool

`kubectl` auto-complete

Install `kubectl` CLI tool