注意： 這份筆記是 2020 年的安裝記錄。Kubernetes 1.24 之後已移除 dockershim，Docker 不再作為 CRI 直接支援。現行安裝請考慮改用 containerd 或 CRI-O。
環境
- Ubuntu 18.04 / 20.04 LTS
- Docker（Container Runtime）
- Kubernetes 版本：1.17.x ~ 1.19.x
Step.1 安裝 Docker
CRI（Container Runtime Interface）除了 Docker，還有 CRI-O、containerd、rkt、Kata Containers 等選擇。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| apt update
apt install -y docker.io
# 設定 Docker daemon：cgroup driver 使用 systemd（Kubernetes 要求）
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl enable docker.service
systemctl daemon-reload
systemctl restart docker
|
確認版本：
1
2
| docker version
# Client: 19.03.8 / Server Engine: 19.03.8
|
Step.2 安裝 kubeadm、kubelet、kubectl
1
2
3
4
5
6
7
8
9
10
11
| apt update && apt install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | tee -a /etc/apt/sources.list.d/kubernetes.list
apt update
apt install -y kubelet kubeadm kubectl bash-completion
# 鎖定套件，避免自動更新
apt-mark hold kubelet kubeadm kubectl
|
啟用 kubectl shell 自動補全：
1
| kubectl completion bash > /etc/bash_completion.d/kubectl
|
確認版本：
1
2
3
4
5
| kubeadm version
# kubeadm version: v1.19.2
kubectl version --client
# Client Version: v1.19.2
|
若還未設定連線叢集，kubectl version 會出現 localhost:8080 was refused 錯誤，可先忽略。
Step.3 預先下載 Image（可選）
初始化 Master Node 時需要 pull 多個 image，可提前下載：
1
2
3
4
| kubeadm config images pull
# [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.19.2
# [config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.19.2
# ...
|
Step.4 初始化 Control Plane（Master Node）
先關閉 SWAP（Kubernetes 強制要求）：
執行初始化：
1
| kubeadm init --pod-network-cidr=10.244.0.0/16
|
順利的話約 2-3 分鐘完成，輸出末尾會出現：
1
2
3
4
5
6
7
8
9
10
11
12
| Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.1.20:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
|
設定 kubectl config：
1
2
3
| mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
Step.5 佈署 CNI（Cilium）
初始化完成後 Master Node 狀態會是 NotReady，原因是 CNI 尚未安裝：
1
2
3
| kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# k8s-master NotReady master 26m v1.19.2
|
安裝 Cilium CNI：
1
| kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/install/kubernetes/quick-install.yaml
|
等待 pods 就緒：
1
| kubectl get pods -n kube-system --watch
|
Cilium 啟動後，coredns 也會一起帶起，Master Node 變為 Ready。
Step.6 加入 Worker Node
在 Master Node：取得 join 指令
1
2
| kubeadm token create --print-join-command
# kubeadm join 172.16.1.20:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
|
在 Worker Node：執行前置安裝
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| swapoff -a
apt update
apt install -y docker.io
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {"max-size": "100m"},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl enable docker.service
systemctl daemon-reload
systemctl restart docker
apt update && apt install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | tee -a /etc/apt/sources.list.d/kubernetes.list
apt update
apt install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
|
設定 hostname 並重啟：
1
2
3
| hostnamectl set-hostname k8s-node-1
echo "127.0.0.1\t$(hostname)" | tee -a /etc/hosts
reboot
|
加入叢集：
1
2
| kubeadm join 172.16.1.20:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
|
驗證
回到 Master Node 確認 node 已加入：
1
2
3
4
5
| kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# k8s-master Ready master 4h30m v1.19.2
# k8s-node-1 Ready <none> 4m54s v1.19.2
# k8s-node-2 Ready <none> 4m23s v1.19.2
|
Step.7 安裝 Kubernetes Dashboard
1
| kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml
|
建立 admin-user ServiceAccount：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
EOF
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
|
取得登入 token：
1
2
| kubectl -n kubernetes-dashboard describe secret \
$(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
|
開啟 proxy：
瀏覽器連到：http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
若透過 SSH 連到 Master Node，建立 SSH tunnel：ssh username@<master-ip> -NfL 8001:localhost:8001
Step.8 安裝 Rook（Ceph 儲存）
⚠️ 以下記錄未完成：OSD 無法建立，僅作參考。
安裝 Rook Operator
1
2
3
4
5
6
7
8
9
10
| # 安裝 common.yaml（處理 RBAC 版本相容問題）
curl https://raw.githubusercontent.com/rook/rook/release-1.4/cluster/examples/kubernetes/ceph/common.yaml \
| sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' \
| kubectl apply -f -
# 安裝 operator
kubectl apply -f https://raw.githubusercontent.com/rook/rook/release-1.4/cluster/examples/kubernetes/ceph/operator.yaml
# 等待 operator 就緒後，安裝 Ceph cluster
kubectl apply -f https://github.com/rook/rook/raw/release-1.4/cluster/examples/kubernetes/ceph/cluster.yaml
|
1
2
3
4
5
6
7
| kubectl apply -f https://github.com/rook/rook/raw/release-1.4/cluster/examples/kubernetes/ceph/toolbox.yaml
kubectl -n rook-ceph exec -it \
$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') \
-- bash
# 常用指令：ceph status / ceph osd status / ceph df / rados df
|
若遇到 FailedScheduling
1
| kubectl taint nodes --all node-role.kubernetes.io/master-
|
故障排除
Swap 未關閉
錯誤訊息:
1
| [ERROR Swap]: running with swap on is not supported. Please disable swap
|
解法 (臨時性):
開機後需再次執行，或編輯 /etc/fstab 永久停用。
Docker service 未啟用
錯誤訊息:
1
| [WARNING Service-Docker]: docker service is not enabled
|
解法:
1
| systemctl enable docker.service
|
References
