K8s 學習筆記 - 維護與常見問題


延續 K8s 學習筆記 - kubeadm 手動安裝 的整理,本文整理 K8s Cluster 安裝或者維護過程遇到的問題。


管理

使用 Rancher 管理 K8s Cluster

Rancher 是很棒的 K8s 管理平台,在團隊裡,建議使用它作為主要的管理入口平台,他本身跑在 K3s 上,是一個微型的 K8s 應用程式。底下是快速跑起一個 Docker 的紀錄:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
TS=$(date +"%Y%m%d")
APP_NAME="rancher"
#APP_VER="latest"
#APP_VER="v2.3.8" ## 穩定使用一年多
APP_VER="v2.5.2-rc5" ## 改動很大,多了 Cluster Explorer

APP_CONTAINER_NAME="${APP_NAME}-${APP_VER}-${TS}"

## see: https://hub.docker.com/r/rancher/rancher/tags?page=1
## see: https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/

docker run -d \
--name ${APP_CONTAINER_NAME} \
-p 1080:80 \
-p 1443:443 \
--restart=unless-stopped \
--privileged \ # v2.5 之後要加
rancher/rancher:${APP_VER}

docker logs -f ${APP_CONTAINER_NAME}

要注意的是,Rancher 本身需要不少資源,建議至少 4core, 8G 以上的環境。

維護

刪除 Worker Node

  • 了解如何刪除 worker node
  • 了解刪除過程中會發生的事情

打算刪除 k8s14-worker02-u1604 這個 worker node,先檢查現況

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 1. 確認現在的 node
~$ k get no
NAME STATUS ROLES AGE VERSION
k8s14-master01-u1604 Ready master 10d v1.14.0
k8s14-worker02-u1604 Ready <none> 4m31s v1.14.0
k8s14-worker03-u1604 Ready <none> 32m v1.14.0
k8s14-worker04-u1604 Ready <none> 24m v1.14.0

## 檢查有哪些 pod 跑在 worker02
~$ k get po -o wide | grep worker02
kube-proxy-zx6fn 1/1 Running 0 3m53s 192.168.2.16 k8s14-worker02-u1604
weave-net-fkb5w 2/2 Running 1 3m53s 192.168.2.16 k8s14-worker02-u1604

## 2. 到 worker02 裡面,檢查 docker ps
root@k8s14-worker02-u1604:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
239c836beb6a 047e0878ff14 "/tini -- /usr/local…" 3 minutes ago Up 3 minutes k8s_rook-ceph-agent_rook-ceph-agent-bsgqk_rook-ceph_71d9fc74-605e-11e9-a972-92cde7b04430_1
fe68d8f88ed1 1f394ae9e226 "/home/weave/launch.…" 3 minutes ago Up 3 minutes k8s_weave_weave-net-fkb5w_kube-system_6bd4a2fa-605e-11e9-a972-92cde7b04430_1
ae00e498728f 789b7f496034 "/usr/bin/weave-npc" 4 minutes ago Up 4 minutes k8s_weave-npc_weave-net-fkb5w_kube-system_6bd4a2fa-605e-11e9-a972-92cde7b04430_0
... 略 ...

開始執行刪除 worker node,主要步驟如下:

  1. Drain Node
  2. Delete Node
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
# 1. Drain Node, 同時刪除一些資料
~$ kubectl drain k8s14-worker02-u1604 --delete-local-data --force --ignore-daemonsets
node/k8s14-worker02-u1604 cordoned
WARNING: Ignoring DaemonSet-managed pods: kube-proxy-zx6fn, weave-net-fkb5w, rook-ceph-agent-bsgqk, rook-discover-p5ptq, weave-scope-agent-wsmd2
pod/rook-ceph-mon-b-775ff945c5-rzzt6 evicted

## 1-1. 確認狀態
~$ kubectl get no
NAME STATUS ROLES AGE VERSION
k8s14-master01-u1604 Ready master 10d v1.14.0
k8s14-worker02-u1604 Ready,SchedulingDisabled <none> 8m36s v1.14.0
k8s14-worker03-u1604 Ready <none> 36m v1.14.0
k8s14-worker04-u1604 Ready <none> 28m v1.14.0

## 1-2. Begin of Checking: 沒啥變化
### 檢查跑在 worker02 上的 pod
~$ k get po -o wide | grep worker02
kube-proxy-zx6fn 1/1 Running 0 8m56s 192.168.2.16 k8s14-worker02-u1604
weave-net-fkb5w 2/2 Running 1 8m56s 192.168.2.16 k8s14-worker02-u1604

### 到 worker02 裡面,檢查 docker ps
root@k8s14-worker02-u1604:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
239c836beb6a 047e0878ff14 "/tini -- /usr/local…" 3 minutes ago Up 3 minutes k8s_rook-ceph-agent_rook-ceph-agent-bsgqk_rook-ceph_71d9fc74-605e-11e9-a972-92cde7b04430_1
fe68d8f88ed1 1f394ae9e226 "/home/weave/launch.…" 3 minutes ago Up 3 minutes k8s_weave_weave-net-fkb5w_kube-system_6bd4a2fa-605e-11e9-a972-92cde7b04430_1
ae00e498728f 789b7f496034 "/usr/bin/weave-npc" 4 minutes ago Up 4 minutes k8s_weave-npc_weave-net-fkb5w_kube-system_6bd4a2fa-605e-11e9-a972-92cde7b04430_0
... 略 ...
## End of Checking: 沒啥變化


# 2. 刪除 worker01 node
~$ kubectl delete node k8s14-worker02-u1604
node "k8s14-worker02-u1604" deleted

## 2-1. 檢查 node
~$ k get no
NAME STATUS ROLES AGE VERSION
k8s14-master01-u1604 Ready master 10d v1.14.0
k8s14-worker03-u1604 Ready <none> 42m v1.14.0
k8s14-worker04-u1604 Ready <none> 33m v1.14.0


## 2-2. 到 worker02 裡執行 kubeadm reset 清除資料, 如下:
root@k8s14-worker02-u1604:~# kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0416 23:57:25.138327 18457 reset.go:234] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your systems IPVS tables.


### (optional) 到 worker02 機器裡面看看 docker ps, container 很快就刪完了,沒有 container 再跑了
root@k8s14-worker02-u1604:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bf7310c8d3c6 d9ece03f45e7 "/home/weave/entrypo…" 9 minutes ago Up 9 minutes k8s_scope-agent_weave-scope-agent-wsmd2_weave_6bd48236-605e-11e9-a972-92cde7b04430_0
79d7b777a525 k8s.gcr.io/pause:3.1 "/pause" 9 minutes ago Up 9 minutes k8s_POD_weave-scope-agent-wsmd2_weave_6bd48236-605e-11e9-a972-92cde7b04430_0

root@k8s14-worker02-u1604:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@k8s14-worker02-u1604:~#

### (optional) docker images 還在
root@k8s14-worker02-u1604:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rook/ceph master 047e0878ff14 3 days ago 698MB
wordpress latest 837092bc87de 5 days ago 421MB
istio/proxyv2 1.1.2 c7fb421f087e 12 days ago 378MB
... 略 ...

Troubleshooting

Q: CoreDNS 無法啟動

我的機器開機後,CoreDNS pod 狀態一直是 Completed

1
2
3
4
5
6
7
8
9
10
11
12
13
14
~$ k get po
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-f4kcl 0/1 Completed 3 10d
coredns-fb8b8dccf-n5tj6 0/1 Completed 1 23h
etcd-k8s14-master01-u1604 1/1 Running 5 10d
kube-apiserver-k8s14-master01-u1604 1/1 Running 5 10d
kube-controller-manager-k8s14-master01-u1604 1/1 Running 5 10d
kube-proxy-2bjb8 1/1 Running 5 10d
kube-proxy-fcm2v 1/1 Running 8 10d
kube-proxy-hn44m 1/1 Running 6 10d
kube-scheduler-k8s14-master01-u1604 1/1 Running 5 10d
weave-net-gbjvt 2/2 Running 5 22h
weave-net-rjj5p 2/2 Running 7 22h
weave-net-rns74 2/2 Running 5 22h

這個問題發生在 worker node 故障了,後來解決方法是 join new node, drain old node 就解掉了。

Q: 使用 Rancher 出現 Controller Manager / Scheduler Error

把 K8s Cluster 放到 Rancher 管理後,出現 Controller Manager / Scheduler 異常的狀況,如下圖:

這問題在 v1.16, v1.18, v1.19 都會出現,v1.14 則沒遇到。

主要的原因是 Rancher 無法正確取得 controller-manager, scheduler 的狀態,透過 kubectl get cs 或者 kubectl get componentstatuses 取得狀態,有問題狀態如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
## v1.16.4
## 這看不出狀況,是因為記錄當下的 kubectl 版本比 cluster 還新,最後面補充說明。
~$ k get cs
NAME AGE
scheduler <unknown>
controller-manager <unknown>
etcd-0 <unknown>

## v1.19.3
~$ k get cs
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {"health":"true"}

原因是因為 controller-manager / scheduler 的 port 指定問題,他們的設定在 Master Node 位置如下:

  • /etc/kubernetes/manifests/kube-scheduler.yaml
  • /etc/kubernetes/manifests/kube-controller-manager.yaml

兩個的設定都指定 --port 0,底下是 kube-controller-manager.yaml,只要把 --port=0 註解即可:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --cluster-cidr=10.244.0.0/16
- --cluster-name=kubernetes
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
- --controllers=*,bootstrapsigner,tokencleaner
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --leader-elect=true
- --node-cidr-mask-size=24
# - --port=0 # <-- 註解此行
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --use-service-account-credentials=true
image: k8s.gcr.io/kube-controller-manager:v1.18.6
imagePullPolicy: IfNotPresent

kube-scheduler 文件針對 --port 的說明如下:

–port int Default: 10251

DEPRECATED: the port on which to serve HTTP insecurely without authentication and authorization. If 0, don't serve plain HTTP at all. See --secure-port instead.

kube-controller-manager 則沒有找到對應的說明。

將這兩個的 --port 都註解掉之後,執行 systemctl restart kubelet 或者重新啟動 cluster,執行 kubectl get cs 就會看到正常狀態,同時 Rancher 看到的狀態也會正常。

1
2
3
4
5
~$ k get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}

註:如果 kubectl 的版本比 cluster 還新,就會出現 無法解析 Response Body 的問題,除錯過程可以增加 -v=9 看看實際的 payload,判斷這兩個服務的狀況。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
~$ k get cs -v=9
... 略 ....
I1108 16:54:23.472713 10668 round_trippers.go:449] Response Headers:
I1108 16:54:23.472717 10668 round_trippers.go:452] Cache-Control: no-cache, private
I1108 16:54:23.472720 10668 round_trippers.go:452] Content-Type: application/json
I1108 16:54:23.472723 10668 round_trippers.go:452] Content-Length: 661
I1108 16:54:23.472726 10668 round_trippers.go:452] Date: Sun, 08 Nov 2020 08:54:23 GMT
I1108 16:54:23.472752 10668 request.go:1068] Response Body: {"kind":"ComponentStatusList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/componentstatuses"},"items":[{"metadata":{"name":"controller-manager","selfLink":"/api/v1/componentstatuses/controller-manager","creationTimestamp":null},"conditions":[{"type":"Healthy","status":"True","message":"ok"}]},{"metadata":{"name":"scheduler","selfLink":"/api/v1/componentstatuses/scheduler","creationTimestamp":null},"conditions":[{"type":"Healthy","status":"True","message":"ok"}]},{"metadata":{"name":"etcd-0","selfLink":"/api/v1/componentstatuses/etcd-0","creationTimestamp":null},"conditions":[{"type":"Healthy","status":"True","message":"{\"health\":\"true\"}"}]}]}

... 略 ....

NAME AGE
scheduler <unknown>
controller-manager <unknown>
etcd-0 <unknown>

只要確認 Healthy 狀態是 OK 即可。


延伸閱讀

站內文章:K8s 相關

VPC - 網路規劃

參考資料




Comments