Vaskata84 - Overview

📋 Table of Contents

This guide provides a comprehensive walkthrough for deploying a production-ready Kubernetes cluster on bare metal infrastructure. Perfect for DevOps engineers, system administrators, and anyone looking to build their own Kubernetes environment from scratch.

🌟 What You'll Learn

🔧 Complete bare metal Kubernetes setup
🐳 Container runtime (containerd) configuration
🌐 CNI plugin deployment (Calico AND Flannel options)
🐋 Docker support with cri-dockerd
🔐 Security best practices
📊 Cluster management basics

✅ Prerequisites

Before starting, ensure you have:

🖥️ Ubuntu/Debian-based system (tested on Ubuntu 20.04+)
💾 Minimum 2GB RAM per node (4GB+ recommended)
🔌 2 CPU cores per node minimum
🌐 Network connectivity between all nodes
👤 Root or sudo access
📡 Static IP addresses for all nodes

🏗️ Architecture

┌─────────────────────────────────────────────────────┐
│              Kubernetes Cluster Architecture        │
├─────────────────────────────────────────────────────┤
│                                                     │
│  ┌──────────────────┐      ┌──────────────────┐     │
│  │   Master Node    │      │   Worker Nodes   │     │
│  │  (k8s-control)   │◄────►│                  │     │
│  │                  │      │  - Node 1        │     │
│  │  - API Server    │      │  - Node 2        │     │
│  │  - Scheduler     │      │  - Node N        │     │
│  │  - Controller    │      │                  │     │
│  └──────────────────┘      └──────────────────┘     │
│           │                                         │
│           ▼                                         │
│  ┌──────────────────────────────────────────┐       │
│  │  Container Runtime: containerd v1.6.16   │       │
│  └──────────────────────────────────────────┘       │
│           │                                         │
│           ▼                                         │
│  ┌──────────────────────────────────────────┐       │
│  │  CNI Plugin (Choose one):                │       │
│  │  • Calico v3.25                          │       │
│  │  • Flannel (Recommended)                 │       │
│  └──────────────────────────────────────────┘       │
│                                                     │
└─────────────────────────────────────────────────────┘

🚀 Installation Steps

Step 1: Disable Swap

Kubernetes requires swap to be disabled for optimal performance.

Why? Kubernetes scheduler needs to manage memory allocation precisely. Swap can interfere with this process.

Step 2: Update /etc/fstab

Prevent swap from being re-enabled after system reboot.

sudo sed -i '/\sswap\s/s/^/#/' /etc/fstab

Explanation: This comments out any swap entries in /etc/fstab

Step 3: Update /etc/hosts

Configure hostname resolution for your cluster nodes.

sudo sed -i '/<IP_address>/d; $a\<IP_address>\t<hostname>' /etc/hosts

Example:

# For master node
sudo sed -i '/192.168.1.100/d; $a\192.168.1.100\tk8s-master' /etc/hosts

# For worker nodes
sudo sed -i '/192.168.1.101/d; $a\192.168.1.101\tk8s-worker1' /etc/hosts
sudo sed -i '/192.168.1.102/d; $a\192.168.1.102\tk8s-worker2' /etc/hosts

Step 4: Enable Bridged Traffic in IPTABLES

Enable necessary kernel modules and network settings for Kubernetes networking.

echo -e "overlay\nbr_netfilter" | sudo tee /etc/modules-load.d/containerd.conf >/dev/null && cat /etc/modules-load.d/containerd.conf

sudo modprobe overlay
sudo modprobe br_netfilter

echo -e "net.bridge.bridge-nf-call-iptables  = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward                 = 1" | sudo tee /etc/sysctl.d/k8s.conf >/dev/null && cat /etc/sysctl.d/k8s.conf

Verify the configuration:

# Check if modules are loaded
lsmod | grep br_netfilter
lsmod | grep overlay

# Apply sysctl parameters
sudo sysctl --system

Step 5: Install Container Runtime (containerd)

Install containerd as the container runtime for Kubernetes.

wget https://github.com/containerd/containerd/releases/download/v1.6.16/containerd-1.6.16-linux-amd64.tar.gz -P /tmp/

tar Cxzvf /usr/local /tmp/containerd-1.6.16-linux-amd64.tar.gz

wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service -P /etc/systemd/system/

systemctl daemon-reload
systemctl enable --now containerd

Verify installation:

systemctl status containerd
containerd --version

Step 6: Installing runc for (containerd)

runc is the container runtime that containerd uses to run containers.

wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64 -P /tmp/

install -m 755 /tmp/runc.amd64 /usr/local/sbin/runc

Verify installation:

Step 7: Installing CNI Plugin

Container Network Interface (CNI) plugins for network configuration.

wget https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz -P /tmp/

mkdir -p /opt/cni/bin

tar Cxzvf /opt/cni/bin /tmp/cni-plugins-linux-amd64-v1.2.0.tgz

Verify installation:

Step 8: Configure containerd

Generate and customize containerd configuration.

mkdir -p /etc/containerd

containerd config default | tee /etc/containerd/config.toml

⚠️ IMPORTANT: Manually edit /etc/containerd/config.toml and change SystemdCgroup to true

# Edit the config file
sudo nano /etc/containerd/config.toml

# Find this line (around line 125):
#   SystemdCgroup = false
# Change it to:
#   SystemdCgroup = true

Restart containerd:

systemctl restart containerd

Step 9: Adding Kubernetes Repo to Install kubectl, kubeadm, kubelet

Install Kubernetes components from official repository.

apt-get update

apt-get install -y apt-transport-https ca-certificates curl

curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

apt-get update

sudo -s

apt-get install -y kubelet=1.26.1-00 kubeadm=1.26.1-00 kubectl=1.26.1-00

apt-mark hold kubelet kubeadm kubectl

Verify installation:

kubectl version --client
kubeadm version
kubelet --version

Creating Control Panel on Master Node Only

Initialize the Kubernetes cluster on the master node.

Option 1: Command Line Initialization

kubeadm init --pod-network-cidr 10.10.0.0/16 --kubernetes-version 1.26.1 --node-name k8s-control

Note: Change 10.10.0.0/16 to your network CIDR if needed

Option 2: Configuration File

kubeadm init --config=kubeadm-config.yaml --upload-certs

⚠️ IMPORTANT: Save the kubeadm join command from the output! You'll need it for worker nodes.

Configure kubectl access:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verify cluster:

kubectl cluster-info
kubectl get nodes

Adding Calico 3.25 CNI or Flannel

Choose one CNI plugin for your cluster networking.

🟢 Option A: Calico CNI (Recommended)

# Install Tigera Operator
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml

# Download and customize resources
wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml

# Edit the CIDR for pods if it's custom
vi custom-resources.yaml

# Apply configuration
kubectl apply -f custom-resources.yaml

Watch Calico pods deployment:

watch kubectl get pods -n calico-system

🔵 Option B: Flannel CNI

# Apply Flannel manifest
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Alternative deployment: Check Flannel Manual Deployment

Restart services if needed:

sudo systemctl restart containerd
systemctl daemon-reload

Install cri-dockerd (Optional)

If you need Docker support alongside containerd.

Check Docker Status

Install cri-dockerd on Debian-based Systems

# Update and install prerequisites
sudo apt update
sudo apt install git wget curl

# Get latest version
VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d '"' -f 4|sed 's/v//g')
echo $VER

For Intel 64-bit CPU

# Download cri-dockerd
wget https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd-${VER}.amd64.tgz

# Extract and install
tar xvf cri-dockerd-${VER}.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/

# Verify installation
cri-dockerd --version

Configure systemd Services

# Download service files
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket

# Move to systemd directory
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/

# Update service path
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

# Reload systemd
sudo systemctl daemon-reload

# Enable services
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket
sudo systemctl enable cri-docker.service
sudo systemctl enable cri-docker.socket

# Start services
sudo systemctl start cri-docker.service
sudo systemctl start cri-docker.socket

Check service status:

journalctl -u cri-docker.service
sudo systemctl status cri-docker.service

Join the Cluster with the Token

Add worker nodes to your Kubernetes cluster.

On each worker node:

# Use the join command from kubeadm init output
sudo kubeadm join <MASTER_IP>:6443 \
  --token <YOUR_TOKEN> \
  --discovery-token-ca-cert-hash sha256:<YOUR_HASH>

If you lost the join command, generate a new one on master:

kubeadm token create --print-join-command

✅ Verification

Get Running Nodes

Expected output:

NAME          STATUS   ROLES           AGE   VERSION
k8s-control   Ready    control-plane   15m   v1.26.1
k8s-worker1   Ready    <none>          10m   v1.26.1
k8s-worker2   Ready    <none>          10m   v1.26.1

Additional Verification Commands

# Check all pods across all namespaces
kubectl get pods -A

# Check node details
kubectl get nodes -o wide

# Check cluster info
kubectl cluster-info

# Check component status
kubectl get componentstatuses

# Check namespaces
kubectl get namespaces

# Verify CNI plugin (for Calico)
kubectl get pods -n calico-system

# Verify CNI plugin (for Flannel)
kubectl get pods -n kube-flannel

🔧 Troubleshooting

Common Issues and Solutions

⚠️ Nodes showing NotReady status

# Check kubelet status
sudo systemctl status kubelet

# View kubelet logs
sudo journalctl -u kubelet -n 100 --no-pager

# Check CNI installation
kubectl get pods -n kube-system

# Restart kubelet
sudo systemctl restart kubelet

⚠️ containerd not starting

# Check containerd status
sudo systemctl status containerd

# View containerd logs
sudo journalctl -u containerd -n 100 --no-pager

# Verify configuration
sudo containerd config dump

# Restart containerd
sudo systemctl restart containerd

⚠️ Pod network not working

# Verify network modules
lsmod | grep br_netfilter
lsmod | grep overlay

# Check sysctl settings
sysctl net.bridge.bridge-nf-call-iptables
sysctl net.bridge.bridge-nf-call-ip6tables
sysctl net.ipv4.ip_forward

# Reload network configuration
sudo sysctl --system

⚠️ CoreDNS pods pending

# Check CoreDNS status
kubectl get pods -n kube-system | grep coredns

# Describe CoreDNS pod
kubectl describe pod -n kube-system <coredns-pod-name>

# Restart CoreDNS
kubectl rollout restart deployment coredns -n kube-system

⚠️ Worker node join fails

# On worker node - reset kubeadm
sudo kubeadm reset

# Clean up
sudo rm -rf /etc/cni/net.d
sudo rm -rf $HOME/.kube/config

# Generate new token on master
kubeadm token create --print-join-command

# Try joining again

📚 Additional Resources

📖 Official Documentation

Kubernetes: https://kubernetes.io/docs/
containerd: https://containerd.io/docs/
Calico: https://docs.tigera.io/calico/latest/about/
Flannel: https://github.com/flannel-io/flannel#deploying-flannel-manually

🛠️ Useful Commands Cheat Sheet

# Cluster Management
kubectl cluster-info                    # Display cluster info
kubectl get nodes                       # List all nodes
kubectl get pods -A                     # List all pods in all namespaces
kubectl get services -A                 # List all services

# Pod Management
kubectl get pods -n <namespace>         # List pods in namespace
kubectl describe pod <pod-name>         # Detailed pod info
kubectl logs <pod-name>                 # View pod logs
kubectl exec -it <pod-name> -- /bin/bash # Access pod shell

# Node Management
kubectl describe node <node-name>       # Detailed node info
kubectl cordon <node-name>              # Mark node unschedulable
kubectl uncordon <node-name>            # Mark node schedulable
kubectl drain <node-name>               # Drain node for maintenance

# Troubleshooting
kubectl get events -A                   # View cluster events
journalctl -u kubelet -f                # Follow kubelet logs
journalctl -u containerd -f             # Follow containerd logs
kubectl top nodes                       # Node resource usage
kubectl top pods -A                     # Pod resource usage