系统环境：

Ubuntu 22.04 LTS
containerd 1.6.20
k8s-mater：192.168.60.160
k8s-node1：192.168.60.161
k8s-node2：192.168.60.162

主机基础优化（所有节点）：

cat /etc/sysctl.conf
net.ipv4.ip_forward=1
vm.max_map_count=262144
kernel.pid_max=4194303
fs.file-max=1000000
net.ipv4.tcp_max_tw_buckets=6000
net.netfilter.nf_conntrack_max=2097152
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0

net.ipv4.ip_forward=1: 启用IP转发功能，用于将接收到的网络数据包转发到另一个网络接口，有助于实现路由、防火墙等网络功能。

vm.max_map_count=262144：设置进程可以拥有的内存区域数量上限，通常应该大于或等于 Elasticsearch 和 Solr 需要的最低数量。

kernel.pid_max=4194303：设置进程 ID 的上限，即最大 PID。这个值越大，允许的进程数也就越多。

fs.file-max=1000000：设置系统可以同时打开的文件句柄数上限，这个值的大小决定了系统能够同时运行的应用程序数量。

net.ipv4.tcp_max_tw_buckets=6000：设置本地核心分配的 TIME_WAIT sockets 的最大数量，这决定了可以生成的TCP连接数量。

net.netfilter.nf_conntrack_max=2097152：设置Netfilter连接跟踪状态表大小的上限，跟踪表缓存容量的大小设置会影响到连接跟踪的引擎,上限防止服务器中规模较大的回话跟踪出现阻塞网络流量情况而设置。

net.bridge.bridge-nf-call-ip6tables = 1：如果启用了 net.ipv4.ip_forward 参数，需要将该参数设置为1。这样可以确保内核能够在 Linux 虚拟网桥桥接 IPv6 数据流时自动应用 iptables 规则。

net.bridge.bridge-nf-call-iptables = 1：开启系统 IPVS 时，如果启用了 net.ipv4.ip_forward，还需将该参数设置为1，以确保内核在 Linux 虚拟网桥桥接 IPv4 数据流时自动应用 iptables 规则。

vm.swappiness=0：控制系统进行分页交换（swap）的倾向，值为0表示操作系统只在极端情况下才会进行交换（尽可能少）。

内核模块开机挂载（所有节点）:

cat <<EOF >/etc/modules-load.d/modules.conf
ip_vs
ip_vs_lc
ip_vs_lblc
ip_vs_lblcr
ip_vs_rr
ip_vs_wrr
ip_vs_sh
ip_vs_dh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
ip_tables
ip_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
xt_set
br_netfilter
nf_conntrack
overlay
EOF

关闭swap（所有节点）：

临时关闭：

sudo swapoff -a

永久关闭swap：

编辑/etc/fstab文件
root@k8s-master:~# cat  /etc/fstab
# /etc/fstab: static file system information.
:#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-B3JMt5tYZ6wr0jJ9KKyrqy9Wrkh0YnhH55zotzNSKjdq7mjFQ739NvDv5HIV5UP8 / ext4 defaults 0 1
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/39a04650-f093-4cde-ae62-1a6c36fa2b63 /boot ext4 defaults 0 1
#/swap.img    none    swap    sw    0    0

注释掉所有swap分区的行或者删除swap分区的行
重启系统

sudo reboot

重启后验证内核模块与内存参数：

lsmod | grep br_netfilter
br_netfilter 32768 0
bridge 307200 1 br_netfilter

sysctl -a | grep bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1

安装containerd（所有节点）：

wget https://shackles.cn/Software/runtime-docker20.10.19-containerd1.6.20-binary-install.tar.gz
tar xvf runtime-docker20.10.19-containerd1.6.20-binary-install.tar.gz
bash runtime-install.sh containerd

更换软件仓库镜像：

cat <<EOF >/etc/apt/sources.list
# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse

deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
EOF

各节点安装kubeadm、 kubectl、 kubelet：

apt-get update && apt-get install -y apt-transport-https -y
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update && apt-cache madison kubeadm
apt-get install -y kubeadm=1.27.2-00 kubectl=1.27.2-00 kubelet=1.27.2-00

下载kubenetes镜像:

kubeadm config images list --kubernetes-version v1.27.2

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controllermanager:v1.27.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.7-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1

#master节点所需image
cat <<EOF >/root/images-master-down.sh
#!/bin/bash
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.2
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controllermanager:v1.27.2
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.2
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.7-0
EOF

#node节点所需image
cat <<EOF >/root/images-node-down.sh
#!/bin/bash
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1
EOF

使用配置文件进行初始化master：

kubeadm config print init-defaults > kubeadm-init.yaml
vim kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s       #加入集群需要的token有效时常默认24小时
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.60.160      # master主机的IP地址也是API Server的监听地址
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock #可ls此文件查看是否存在
  imagePullPolicy: IfNotPresent
  name: k8s-master                      #当前主机名
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki        #证书路径
clusterName: kubernetes
#controlPlaneEndpoint: 192.168.60.160:6443  #公有云中负载均衡器的内网地址
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers    #修改为国内image仓库
kind: ClusterConfiguration
kubernetesVersion: 1.27.2                   #和当前版本对应
networking:
  dnsDomain: cluster.local
  podSubnet: 10.200.0.0/16                  #Pod的IP段
  serviceSubnet: 10.100.0.0/16              #SVC的IP段
scheduler: {}

--- #指定kubelet使⽤systemd
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

--- #指定KubeProxy使⽤ipvs
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

初始化：

kubeadm init --config kubeadm-init.yaml

初始化成功后会打印如下信息：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.60.160:6443 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:4e645a1f0d5d377f79d87139232e35e5972ebe86dbca1df808a54e818be3ec58

根据提示执行指令：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

这三条命令的作用是将 Kubernetes 集群的 admin.conf 配置文件拷贝到当前用户的 $HOME/.kube/config 文件中，并修改其拥有者为当前用户，以便可以通过 kubectl 命令管理 Kubernetes 集群。

添加Node节点：

kubeadm join 192.168.60.160:6443 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:4e645a1f0d5d377f79d87139232e35e5972ebe86dbca1df808a54e818be3ec58

检查node节点是否加入成功：

kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   NotReady    control-plane   20h   v1.27.2
k8s-node1    NotReady    <none>          16h   v1.27.2
k8s-node2    NotReady    <none>          16h   v1.27.2
如上图所示nodes又两台已经加入成功，STATUS为NotReady是因为没有配置CNI。

部署⽹络组件flannel：

cat <<EOF > kube-flannel.yml
apiVersion: v1
kind: Namespace
metadata:
labels:
    k8s-app: flannel
    pod-security.kubernetes.io/enforce: privileged
name: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
    k8s-app: flannel
name: flannel
namespace: kube-flannel
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
    k8s-app: flannel
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- networking.k8s.io
resources:
- clustercidrs
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
    k8s-app: flannel
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-flannel
---
apiVersion: v1
data:
cni-conf.json: |
    {
    "name": "cbr0",
    "cniVersion": "0.3.1",
    "plugins": [
        {
        "type": "flannel",
        "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
        }
        },
        {
        "type": "portmap",
        "capabilities": {
            "portMappings": true
        }
        }
    ]
    }
net-conf.json: |
    {
    "Network": "10.200.0.0/16",
    "Backend": {
        "Type": "vxlan"
    }
    }
kind: ConfigMap
metadata:
labels:
    app: flannel
    k8s-app: flannel
    tier: node
name: kube-flannel-cfg
namespace: kube-flannel
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
    app: flannel
    k8s-app: flannel
    tier: node
name: kube-flannel-ds
namespace: kube-flannel
spec:
selector:
    matchLabels:
    app: flannel
    k8s-app: flannel
template:
    metadata:
    labels:
        app: flannel
        k8s-app: flannel
        tier: node
    spec:
    affinity:
        nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
            - key: kubernetes.io/os
                operator: In
                values:
                - linux
    containers:
    - args:
        - --ip-masq
        - --kube-subnet-mgr
        command:
        - /opt/bin/flanneld
        env:
        - name: POD_NAME
        valueFrom:
            fieldRef:
            fieldPath: metadata.name
        - name: POD_NAMESPACE
        valueFrom:
            fieldRef:
            fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
        value: "5000"
        image: docker.io/flannel/flannel:v0.22.0
        name: kube-flannel
        resources:
        requests:
            cpu: 100m
            memory: 50Mi
        securityContext:
        capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
        privileged: false
        volumeMounts:
        - mountPath: /run/flannel
        name: run
        - mountPath: /etc/kube-flannel/
        name: flannel-cfg
        - mountPath: /run/xtables.lock
        name: xtables-lock
    hostNetwork: true
    initContainers:
    - args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        command:
        - cp
        image: docker.io/flannel/flannel-cni-plugin:v1.1.2
        name: install-cni-plugin
        volumeMounts:
        - mountPath: /opt/cni/bin
        name: cni-plugin
    - args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        command:
        - cp
        image: docker.io/flannel/flannel:v0.22.0
        name: install-cni
        volumeMounts:
        - mountPath: /etc/cni/net.d
        name: cni
        - mountPath: /etc/kube-flannel/
        name: flannel-cfg
    priorityClassName: system-node-critical
    serviceAccountName: flannel
    tolerations:
    - effect: NoSchedule
        operator: Exists
    volumes:
    - hostPath:
        path: /run/flannel
        name: run
    - hostPath:
        path: /opt/cni/bin
        name: cni-plugin
    - hostPath:
        path: /etc/cni/net.d
        name: cni
    - configMap:
        name: kube-flannel-cfg
        name: flannel-cfg
    - hostPath:
        path: /run/xtables.lock
        type: FileOrCreate
        name: xtables-lock
EOF
配置中"Network": "10.200.0.0/16",要和初始化时指定的podSubnet对应。

kubectl apply -f kube-flannel.yml

namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

验证flannel是否部署成功：

root@k8s-master:~# kubectl get pods -n kube-flannel
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-57hbz   1/1     Running   0          15h
kube-flannel-ds-hg48m   1/1     Running   0          15h
kube-flannel-ds-w5dsj   1/1     Running   0          15h

验证node节点状态：

kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   20h   v1.27.2
k8s-node1    Ready    <none>          16h   v1.27.2
k8s-node2    Ready    <none>          16h   v1.27.2

后期添加master节点：

kubeadm init phase upload-certs --upload-certs
I0414 16:21:59.283672 14673 version.go:256] remote version is much newer: v1.27.0;
falling back to: stable-1.26
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system"
Namespace
[upload-certs] Using certificate key:
e4226754e61e5cbcec412db6693597e2489ccf62187137357bd40a50b6c9b313

root@k8s-master2:~# kubeadm join 192.168.60.160:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash
sha256:5e36ce948f5f2ef7af33d61664d6572a4818239c4ec593d4fb0ee66eda17dfea \
--control-plane --certificate-key
e4226754e61e5cbcec412db6693597e2489ccf62187137357bd40a50b6c9b313
最后修改：2024 年 05 月 15 日
© 允许规范转载
如果觉得我的文章对你有用，请随意赞赏