在 Macbook Pro M1 (Apple Silicon) 安裝 K3s 的筆記。
摘要 K3d vs K3s K3d 其實是 K3s in Docker 的縮寫，也就是透過 docker 把 K8s 的角色跑起來，包含 api, kube-api, kube-proxy, coredns … 等。兩者的使用對象有所差異，整理如下：
開發者：適合用 K3d 快速建立 cluster，測試自己的應用程式。
如果要在 Apple Silicon 的環境使用 k8s，也可以用 k3d 就可以。
K8s admin：適合用 K3s 設計與規劃 K8s cluster，模擬各種 cluster 的需求。
K3s 的架構與原理如下圖：
Source: https://docs.k3s.io/architecture
類似於 dapr 的 hosted mode 與 cluster mode.
Multipasss multipass 是 ubuntu 的公司出的，針對 ubuntu 在三大作業系統上可以順利運行 ubuntu vm 的工具，本身提供的 ubuntu image 也是輕量化過的。
安裝 multipass 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 ❯ brew install --cask multipass Running `brew update --auto-update`... ==> Auto-updated Homebrew! Updated 1 tap (homebrew/core). ==> New Formulae aptos cargo-about ecoji grayskull kwctl podsync twm xorg-server aws-sam-cli cargo-deny fgbio jbang mpfrcx quartz-wm video-compare xrdb bossa cascadia flagd jreleaser ocm ruby@3.1 xinit bzip3 cdebug gokey kubent openvino simdutf xmodmap You have 3 outdated formulae installed. You can upgrade them with brew upgrade or list them with brew outdated. ==> Tapping homebrew/cask Cloning into '/opt/homebrew/Library/Taps/homebrew/homebrew-cask' ... remote: Enumerating objects: 702522, done . remote: Counting objects: 100% (57/57), done . remote: Compressing objects: 100% (41/41), done . Receiving objects: 39% (273984/702522), 98.74 MiB | 3.09 MiB/s ==> Installing Cask multipass ==> Running installer for multipass; your password may be necessary. Package installers may write to any location; options such as `--appdir` are ignored. Password: installer: Package name is multipass installer: Installing at base path / installer: The install was successful. 🍺 multipass was successfully installed!
基本的使用參數：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ❯ multipass version multipass 1.10.1+mac multipassd 1.10.1+mac ❯ multipass help Usage: multipass [options] <command > Create, control and connect to Ubuntu instances. This is a command line utility for multipass, a service that manages Ubuntu instances. ❯ multipass help launch Usage: multipass launch [options] [[<remote:>]<image> | <url>] Create and start a new instance. ❯ multipass find Image Aliases Version Description snapcraft:core18 18.04 20201111 Snapcraft builder for Core 18 snapcraft:core20 20.04 20210921 Snapcraft builder for Core 20 snapcraft:core22 22.04 20220426 Snapcraft builder for Core 22 18.04 bionic 20230112 Ubuntu 18.04 LTS 20.04 focal 20230111 Ubuntu 20.04 LTS 22.04 jammy,lts 20230107 Ubuntu 22.04 LTS anbox-cloud-appliance latest Anbox Cloud Appliance charm-dev latest A development and testing environment for charmers docker latest A Docker environment with Portainer and related tools jellyfin latest Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media. minikube latest minikube is local Kubernetes
啟動虛擬機 透過 multipass 啟動一個虛擬機，指定虛擬機名稱、記憶體、Disk、CPU 數量、網路模式
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 ❯ multipass launch --name k3s-main \ --mem 4G --disk 20G \ --cpus 2 Launched: k3s-main ❯ multipass info k3s-main Name: k3s-main State: Running IPv4: 192.168.64.3 Release: Ubuntu 22.04.1 LTS Image hash : 3100a27357a0 (Ubuntu 22.04 LTS) Load: 0.88 0.37 0.14 Disk usage: 2.4G out of 19.2G Memory usage: 749.0M out of 3.8G Mounts: -- ❯ mkdir -p ~/Temp/K8s ❯ multipass mount ~/Temp/K8s k3s:~/k8s Enabling support for mounting \ ❯ multipass delete k3s-main ❯ multipass purge
進去虛擬機
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ❯ multipass shell k3s-main ubuntu@k3s-main:~$ df -h Filesystem Size Used Avail Use% Mounted on tmpfs 393M 864K 392M 1% /run /dev/vda1 20G 1.9G 18G 10% / tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/vda15 105M 5.3M 100M 5% /boot/efi tmpfs 393M 4.0K 393M 1% /run/user/1000 ubuntu@k3s-main:~$ cpu-info Packages: 0: Intel Core i9-9900K 1: Intel Core i9-9900K Microarchitectures: 2x Sky Lake Cores: 0: 1 processor (0), Intel Sky Lake 1: 1 processor (1), Intel Sky Lake Logical processors (System ID): 0 (0): APIC ID 0x00000000 1 (1): APIC ID 0x00000001
整理幾個 multipass 運作的資訊：
multipass 程序主要工作在 /Library/Application Support/com.canonical.multipass
multipass 預設使用 qemu 為 Hypervisor，虛擬磁碟為 COW2 格式，網路為 Bridge 模式
虛擬機的預設路徑為：/var/root/Library/Application Support/multipassd/ (需要 root 權限才能存取)
multipass 預設透過 luanchd 跟著作業系統啟動
配置檔位置： /Library/LaunchDaemons/com.canonical.multipassd.plist
安裝 K3s 直接安裝 如果在 macOS 直接安裝 k3s 會出現以下訊息：
1 2 ❯ curl -sfL https://get.k3s.io | sh - [ERROR] Can not find systemd or openrc to use as a process supervisor for k3s
不管是 Intel or Apple Silicon 都是。
在 VM 裡面安裝 K3s 透過 multipass 裝起好一台 VM 之後，進去裡面再安裝 K3s 就可以了。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 root@k3s:~ [INFO] Finding release for channel stable [INFO] Using v1.25.5+k3s2 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.5+k3s2/sha256sum-arm64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.5+k3s2/k3s-arm64 [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env : Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service. [INFO] systemd: Starting k3se ubuntu@k3s-main:~$ sudo kubectl get po,no --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/local-path-provisioner-79f67d76f8-w946b 1/1 Running 0 8m15s kube-system pod/coredns-597584b69b-9mx2w 1/1 Running 0 8m15s kube-system pod/metrics-server-5f9f776df5-p826z 1/1 Running 0 8m15s kube-system pod/helm-install-traefik-crd-bq27h 0/1 Completed 0 8m15s kube-system pod/helm-install-traefik-b6vb4 0/1 Completed 1 8m15s kube-system pod/svclb-traefik-09095d96-v9fpk 2/2 Running 0 7m45s kube-system pod/traefik-66c46d954f-r48tq 1/1 Running 0 7m45s NAMESPACE NAME STATUS ROLES AGE VERSION node/k3s-main Ready control-plane,master 8m31s v1.25.5+k3s2 ubuntu@k3s-main:~$ sudo kubectl api-resources NAME SHORTNAMES APIVERSION NAMESPACED KIND bindings v1 true Binding componentstatuses cs v1 false ComponentStatus configmaps cm v1 true ConfigMap endpoints ep v1 true Endpoints events ev v1 true Event ... 略 ... ubuntu@k3s-main:~$ sudo kubectl get rs --all-namespaces NAMESPACE NAME DESIRED CURRENT READY AGE kube-system local-path-provisioner-79f67d76f8 1 1 1 10m kube-system coredns-597584b69b 1 1 1 10m kube-system metrics-server-5f9f776df5 1 1 1 10m kube-system traefik-66c46d954f 1 1 1 10m ❯ multipass exec k3s-main sudo cat /var/lib/rancher/k3s/server/node-token K1082f1ce868a513c55999a7eba68ab888dc2035baf2323131466364f65bf5f0220::server:27b105f74351819669c0481da7c87baa cat /etc/rancher/k3s/k3s.yaml
新增 Worker Node 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 multipass launch --name k3s-worker1 --mem 2G --disk 20G multipass launch --name k3s-worker2 --mem 2G --disk 20G multipass shell k3s-worker1 multipass shell k3s-worker2 curl -sfL https://get.k3s.io | K3S_URL=https://192.168.64.3:6443 K3S_TOKEN="K102a46f234cf614bd49a50092389d8e03fed00618846936782fbdae904c32ee9d5::server:5fc3190151a2de0126199cc6fbd94acf" sh - ubuntu@k3s-main:~$ sudo kubectl get no NAME STATUS ROLES AGE VERSION k3s-main Ready control-plane,master 65m v1.25.5+k3s2 k3s-worker1 Ready <none> 75s v1.25.5+k3s2 k3s-worker2 Ready <none> 2s v1.25.5+k3s2
移除 Worker Node 1 2 3 4 5 kubectl drain k3s-worker --delete-emptydir-data --force --ignore-daemonsets root@k3s:~ node "k3s-worker" deleted
測試 底下都在 k3s-master node 裡面執行。
建立測試網頁: index.html，內容如下：
1 2 3 4 5 6 <html > <head > <title > Hello World!</title > </head > <body > Hello World!</body > </html >
把這個 HTML 設定成 configmap:
1 ~$ kubectl create configmap hello-world --from-file index.html
建立 K8s YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello-world annotations: kubernetes.io/ingress.class: "traefik" spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: hello-world port: number: 80 --- apiVersion: v1 kind: Service metadata: name: hello-world spec: ports: - port: 80 protocol: TCP selector: app: hello-world --- apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-nginx spec: selector: matchLabels: app: hello-world replicas: 3 template: metadata: labels: app: hello-world spec: containers: - name: nginx image: nginx ports: - containerPort: 80 volumeMounts: - name: hello-world-volume mountPath: /usr/share/nginx/html volumes: - name: hello-world-volume configMap: name: hello-world
執行這段 YAML
1 2 3 4 5 6 ~$ kubectl apply -f hello-world.yml ~$ kubectl get po NAME READY STATUS RESTARTS AGE hello-world-nginx-5876d86d96-k2mm6 1/1 Running 0 4m25s hello-world-nginx-5876d86d96-krsrh 1/1 Running 0 4m25s hello-world-nginx-5876d86d96-wq9hf 1/1 Running 0 4m25s
測試：
1 2 3 4 5 6 7 ~$ curl localhost:80 <html> <head > <title>Hello World!</title> </head> <body>Hello World!</body> </html>
Q & A Q: 可以把 master 和 worker node 裝在同一台 VM? 不可以，Port 會出現衝突，要裝在同一台 VM，就用 K3d
如果已經執行了，記得跑 k3s-agent-uninstall.sh 或者 k3s-uninstall.sh 刪掉造成衝突的一方，然後重新啟動 service 即可:
1 2 systemctl start k3s.service systemctl status k3s.service
