|
kubeadm 搭建 K8s本篇主要记录一下 使用 kubeadm 搭建 k8s 详细过程 ，环境使用 VirtualBox 构建的3台虚拟机
1.环境准备- 操作系统：Centos7 （CentOS Linux release 7.9.2009）
- Master 主节点： 1台 虚拟机
- Node 计算节点： 2台 虚拟机
- K8s version：v1.23.5 （选的较新的版本）
- Docker version：20.10.14
虚拟机创建我就不详细展开了
2.虚拟机网络配置(保证能访问外网)- 虚拟机网卡 选择的是 1.网络地址转换NAT （主要用来访问外网的） 2.仅主机网络 （主要用来宿主机访问虚拟机的）
- 把对应的 ip 设置如下
- kube-master 192.168.56.20
- kube-node-1 192.168.56.21
- kube-node-2 192.168.56.22
2.1 3台虚拟机 静态ip配置cat /etc/sysconfig/network-scripts/ifcfg-enp0s3 TYPE="Ethernet"PROXY_METHOD="none"BROWSER_ONLY="no"BOOTPROTO="none" #关闭DEFROUTE="yes"IPV4_FAILURE_FATAL="no"IPV6INIT="yes"IPV6_AUTOCONF="yes"IPV6_DEFROUTE="yes"IPV6_FAILURE_FATAL="no"IPV6_ADDR_GEN_MODE="stable-privacy"NAME="enp0s3"UUID="35eac2d4-234d-4859-848e-ed2a30a64a81"DEVICE="enp0s3"#添加部分 IPADDR="10.0.2.20"PREFIX="24"GATEWAY="10.0.2.2"DNS1="10.0.2.3"DNS2="114.114.114.114"ONBOOT="yes" cat /etc/sysconfig/network-scripts/ifcfg-enp0s8 TYPE=EthernetPROXY_METHOD=noneBROWSER_ONLY=noBOOTPROTO=static #修改为 static 静态ipDEFROUTE=yesIPV4_FAILURE_FATAL=noIPV6INIT=yesIPV6_AUTOCONF=yesIPV6_DEFROUTE=yesIPV6_FAILURE_FATAL=noIPV6_ADDR_GEN_MODE=stable-privacyNAME=enp0s8UUID=d06bac6d-294f-4bfe-bf08-5cb65c0a431fDEVICE=enp0s8#新增部分IPADDR=192.168.56.20PREFIX=24ONBOOT=yes
2.2 虚拟机hosts 配置在 3台虚拟机 /etc/hosts 中配置 如下 192.168.56.20 kube-master192.168.56.21 kube-node-1192.168.56.22 kube-node-2 
3.k8s 基础环境设置
3.1 yum 换源(可选)yum -y install wgetcd /etc/yum.repos.dmv CentOS-Base.repo CentOS-Base.repo.bakwget -O CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repowget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repoyum clean allyum makecache #重新生成缓存
3.2 关闭 selinuxsetenforce 0 #临时关闭sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config #永久关闭
3.3 关闭swap分区或禁用swap文件swapoff -a# 注释掉关于swap分区的行yes | cp /etc/fstab /etc/fstab_bakcat /etc/fstab_bak |grep -v swap > /etc/fstab
3.4 允许iptables 检查桥接流量cat <<EOF | sudo tee /etc/modules-load.d/k8s.confbr_netfilterEOFcat <<EOF | sudo tee /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOFsudo sysctl --system
3.5 关闭防火墙systemctl stop firewalldsystemctl disable firewalld
3.6 /proc/sys/net/ipv4/ip_forwardecho 1 > /proc/sys/net/ipv4/ip_forward
4.kubectl、kubeadm、kubelet的安装注意 先换源 https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.64be1b11iddYWH 阿里换源文档
4.1 添加Kubernetes的alibaba yum源cat <<EOF > /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgEOFyum install -y kubelet kubeadm kubectl
4.2 安装 kubelet kubeadm kubectlyum install -y kubelet kubeadm kubectl systemctl enable kubelet && systemctl start kubelet 由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y --nogpgcheck kubelet kubeadm kubectl 安装
4.3 启动kubelet 服务systemctl enable kubelet && systemctl start kubelet 此时执行systemctl status kubelet查看服务状态，此时肯定有错误（因为我们还未进行初始化操作），错误可使用journalctl -xe查看错误信息。
5.Docker 安装和配置Docker centos 安装文档 地址 https://docs.docker.com/engine/install/centos/
5.1 Docker 安装sudo yum install -y yum-utilssudo yum-config-manager / --add-repo / https://download.docker.com/linux/centos/docker-ce.reposudo yum install docker-ce docker-ce-cli containerd.io # 也可以安装其他版本 yum list docker-ce --showduplicates | sort -r # sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io#启动dockersudo systemctl enable docker && systemctl start docker
5.2 Docker 镜像源更换sudo tee /etc/docker/daemon.json <<-'EOF'{ "registry-mirrors": ["https://9xxxx1clo.mirror.aliyuncs.com"] #请更换自己的 阿里云镜像地址}EOF
5.3 Docker 配置cgroup-driver由于 cgroup-driver ，默认docker 是 cgroupfs ，而k8s 默认是 systemd，所以需要设置 docker 的 cgroup 为 systemd 推荐使用方式二进行修改
#查看cgroup-driver docker info | grep -i cgroup# 方式一 追加 --exec-opt native.cgroupdriver=systemd 参数 sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service systemctl daemon-reload # 重新加载服务 systemctl enable docker # 启用docker服务(开机自起) systemctl restart docker # 启动docker服务# 方式二 修改docker配置文件 vim /etc/docker/daemon.json{ "exec-opts": ["native.cgroupdriver=systemd"]}
5.4 拉取k8s所需镜像可以通过 kubeadm config images list 查看所需的镜像
kubeadm config images list k8s.gcr.io/kube-apiserver:v1.23.5k8s.gcr.io/kube-controller-manager:v1.23.5k8s.gcr.io/kube-scheduler:v1.23.5k8s.gcr.io/kube-proxy:v1.23.5k8s.gcr.io/pause:3.6k8s.gcr.io/etcd:3.5.1-0k8s.gcr.io/coredns/coredns:v1.8.6 如果拉取不到镜像 可以通过 去 docker hub 上查找对应的 镜像 
docker pull k8simage/kube-apiserver:v1.23.5 docker tag k8simage/kube-apiserver:v1.23.5 k8s.gcr.io/kube-apiserver:v1.23.5 #重新tag改成上面所需的镜像即可， 其他的都按照这样来 
Tips: 完成上面操作后 你可以通过 docker save 导出镜像,以后再用的时候 可以通过 docker image load < 进行离线加载镜像
#导出镜像 ，批量导出所有镜像docker save $(docker images --format '{{.Repository}}:{{.Tag}}') -o k8s.images.tar #导入镜像 ，注意 是 < 不是 > 。。 > 后会导致tar 无法使用需要重新上传docker image load < ...tar/zip
6.kube-master 节点 初始化我这里选的是 flannel 网络插件，先来看看 flannel 的配置文件
6.1 查看flannel 网络范围flannel.yml https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml 不同的 网络插件 网络范围不同 
6.2 kubeadm init 初始化kubeadm init --pod-network-cidr=10.244.0.0/16 / #网卡插件的 网络范围 --apiserver-advertise-address=192.168.56.20 #master 的 ip 地址 等一会就会有如下结果 
可以把 kubeadm join 命令记下来 便于后面使用 保存输出中的kubeadm join部分内容，用于添加node节点，或者使用kubeadm token list 和kubeadm token create --print-join-command查看 执行后续步骤，可以看到提示 mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config
6.3 网络插件flannel 配置前面提到了flannel 把对应的flannel.yml文件下载到服务器中，可以使用wget 可以看到 在没配置 flannel 的时候 coredns 一直是 pending 状态 
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml #可能会下载不到，文章最后我会提供内容，直接copy过去即可#Tips: 可能需要指定一下网卡 ，在kube-flannel.yml中 - /opt/bin/flanneld args: 后面添加#- --iface=enp0s8 # 需要指定网卡，否则可能k8s会导致使用错误的网卡 kubectl apply -f kube-flannel.yml #应用该文件 
kubectl get pods --all-namespaces -o wide #查看是否安装成功，它会在 master 和 node 节点上都会自动部署 
最后通过 kubectl get nodes -o wide 查看 节点
7.kube-node 节点初始化
7.1 查看token可以通过该命令进行查看 token kubeadm token create --print-join-command#kubeadm join kube-master:6443 --token ii5xyw.mywlktar1e8f60cu --discovery-token-ca-cert-hash sha256:358f92418178342f79aa7b049e6c18a423f0fb88186189142459016fe6efb8e7
7.2 node节点上 执行 kubeadmin joinkubeadm join kube-master:6443 --token ii5xyw.mywlktar1e8f60cu --discovery-token-ca-cert-hash sha256:358f92418178342f79aa7b049e6c18a423f0fb88186189142459016fe6efb8e7
7.3 在master 上查看结果watch kubectl get nodes -o wide #可以通过watch 实时监听 加入情况 
至此使用 kubeadm 已经成功搭建k8s 集群
总结本篇主要讲解了 如何使用 kubeadm 进行搭建 k8s 集群 ，搭建的是 k8s 较新版本 v1.23.5 版本，使用kubeadm 搭建相对来说还是比较简单的 。
k8s官网文档https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
阿里 k8s 更换镜像文档https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.64be1b11iddYWH
Docker hub 地址https://hub.docker.com/r/k8simage/
Docker 安装地址https://docs.docker.com/engine/install/centos/
flannel.yml 文件---apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/defaultspec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny'---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelrules:- apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged']- apiGroups: - "" resources: - pods verbs: - get- apiGroups: - "" resources: - nodes verbs: - list - watch- apiGroups: - "" resources: - nodes/status verbs: - patch---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flanneldata: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } }---apiVersion: apps/v1kind: DaemonSetmetadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannelspec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni-plugin #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 command: - cp args: - -f - /flannel - /opt/cni/bin/flannel volumeMounts: - name: cni-plugin mountPath: /opt/cni/bin - name: install-cni #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) image: rancher/mirrored-flannelcni-flannel:v0.17.0 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) image: rancher/mirrored-flannelcni-flannel:v0.17.0 command: - /opt/bin/flanneld args: #- --iface=enp0s8 # 需要指定网卡，否则可能会导致问题 - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: EVENT_QUEUE_DEPTH value: "5000" volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ - name: xtables-lock mountPath: /run/xtables.lock volumes: - name: run hostPath: path: /run/flannel - name: cni-plugin hostPath: path: /opt/cni/bin - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg - name: xtables-lock hostPath: path: /run/xtables.lock type: FileOrCreate
下载地址:
docker上快速搭建gitlab、gitlab-runer及实现CI/CD功能
SpringBoot打包docker镜像发布的详细步骤 |
