一. 系统软件环境

软件	版本
操作系统	CentOS Linux release 7.8.2003 (Core)
Docker	docker-20.10.6-ce
Kubernetes	1.20.6
ETCD	3.4.15

节点组件

角色	IP	组件
k8s-master1	3.1.101.49	kube-apiserver, kube-controller-manager, kube-scheduler, docker, kubelet, kube-proxy,etcd,nginx,keepalived
k8s-master2	3.1.101.50	kube-apiserver, kube-controller-manager, kube-scheduler, docker, kubelet, kube-proxy,etcd,nginx,keepalived
k8s-node1	3.1.101.51	docker, kubelet, kube-proxy, etcd
k8s-node2	3.1.101.52	docker, kubelet, kube-proxy, etcd
k8s-node1	3.1.101.53	docker, kubelet, kube-proxy, etcd
VIP	3.1.101.45

master1和master2为基于Ningx+Keepalived的高可用

二. 基础环境配置

所有NODE节点

2.1 创建目录

## 创建目录结构
mkdir -pv /opt/etcd/{bin,cfg,ssl,logs}
mkdir -pv /opt/k8s/{bin,cfg,ssl,logs,yaml}
mkdir -pv /opt/cni/{bin,cfg,yaml}
mkdir -pv /etc/cni/

2.2 hosts配置

cat >> /etc/hosts << EOF
3.1.101.49 k8s-master1
3.1.101.50 k8s-master2
3.1.101.51 k8s-node1
3.1.101.52 k8s-node2
3.1.101.53 k8s-node3
3.1.101.49 etcd-1
3.1.101.50 etcd-2
3.1.101.51 etcd-3
3.1.101.52 etcd-4
3.1.101.53 etcd-5
EOF

2.3 主机名修改

在对应节点分别执行

hostnamectl set-hostname k8s-master1
hostnamectl set-hostname k8s-master2
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2
hostnamectl set-hostname k8s-node3

2.4 其他系统设置

## 启用IPVS模式相关配置
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
## 生效
sysctl --system

## 关闭缓存,配置/etc/fstab,永久关闭
# 临时关闭:
swapoff -a

## 关闭NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager

## 时间同步
/usr/bin/cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
ntpdate time.windows.com

##配置环境变量(根据节点情况,一般配置master节点即可)
echo 'export PATH=$PATH:/opt/k8s/bin/' >> /etc/profile
echo 'export PATH=$PATH:/opt/etcd/bin/' >> /etc/profile
source /etc/profile

## 为了便捷操作,在k8s-master1上创建免密登录其他节点
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-node1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-node2

三. 安装cfssl证书工具

master节点

## 创建自签证书目录
mkdir -pv /data/TLS/{etcd,k8s}

## 下载地址
https://github.com/cloudflare/cfssl/releases/download

## 移动到/usr/bin目录下
mv cfssl_1.5.0_linux_amd64 /usr/bin/cfssl
mv cfssl-certinfo_1.5.0_linux_amd64 /usr/bin/cfssl-certinfo
mv cfssljson_1.5.0_linux_amd64 /usr/bin/cfssljson

## 添加可执行权限
chmod +x /usr/bin/cfssl*

## 生成配置模版命令
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json

四. 部署ETCD集群

节点名称	IP
etcd-1	3.1.101.49
etcd-2	3.1.101.50
etcd-3	3.1.101.51
etcd-4	3.1.101.52
etcd-5	3.1.101.53

4.1 自签TLS证书

ETCD-1操作,然后同步到其他节点

自签证书颁发机构（CA）

1	cd /data/TLS/etcd/

自签CA

cd /data/TLS/etcd/
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "etcd": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成证书

1
2
3

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

使用自签CA签发Etcd HTTPS证书

创建证书申请文件(hosts中要包含所有etcd节点ip,也可以多写几个预留)

cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "3.1.101.49",
    "3.1.101.50",
    "3.1.101.51",
    "3.1.101.52",
    "3.1.101.53",
    "3.1.101.54",
    "3.1.101.55"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server

ls
ca-config.json  ca-key.pem  server-csr.json
ca.csr          ca.pem      server-key.pem
ca-csr.json     server.csr  server.pem

同步证书

cp /data/TLS/etcd/*.pem /opt/etcd/ssl/
ls /opt/etcd/ssl/
ca-key.pem  ca.pem  server-key.pem  server.pem
rsync -av /data/TLS/etcd/*.pem etcd-2:/opt/etcd/ssl/
rsync -av /data/TLS/etcd/*.pem etcd-3:/opt/etcd/ssl/
rsync -av /data/TLS/etcd/*.pem etcd-4:/opt/etcd/ssl/
rsync -av /data/TLS/etcd/*.pem etcd-5:/opt/etcd/ssl/

4.2 ETCD安装

下载地址

https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz

解压部署

ETCD-1操作,然后同步到其他节点

tar -zxf etcd-v3.4.15-linux-amd64.tar.gz
mv etcd-v3.4.15-linux-amd64/etcd* /opt/etcd/bin/

rsync -av /opt/etcd/bin/* etcd-1:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-2:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-3:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-4:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-5:/opt/etcd/bin/

4.3 创建ETCD配置文件

ETCD各节点配置基本相同, 注意修改如下配置, 修改成本机etcd-name或者IP

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://3.1.101.49:2380"
ETCD_LISTEN_CLIENT_URLS="https://3.1.101.49:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://3.1.101.49:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://3.1.101.49:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://3.1.101.49:2380,etcd-2=https://3.1.101.50:2380,etcd-3=https://3.1.101.51:2380,etcd-4=https://3.1.101.52:2380,etcd-5=https://3.1.101.53:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

4.4 创建ETCD启动文件

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

4.5 启动ETCD

## 重载启动配置文件
systemctl daemon-reload
## 启动etcd
systemctl restart etcd
## 加入开机自启动
systemctl enable etcd

4.6 验证ETCD状态

1
2
3

/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379" endpoint health

/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379" member list

五. 二进制部署DOCKER

5.1 下载地址

https://download.docker.com/linux/static/stable/x86_64/

1 2	tar zxf docker-20.10.6.tgz mv docker/* /usr/bin/

编辑docker配置文件

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://gsm39obv.mirror.aliyuncs.com"]
}
EOF

5.2 创建systemd启动文件

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
EOF

5.3 启动docker

1
2
3

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

六. kubenetes部署

二进制文件部署

下载地址

https://dl.k8s.io/v1.20.6/kubernetes-server-linux-amd64.tar.gz

解压

1 2	tar zxvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin

master节点

1	rsync -av kubectl kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy /opt/k8s/bin/

node节点

1
2
3

rsync -av kubelet kube-proxy root@k8s-node1:/opt/k8s/bin/
rsync -av kubelet kube-proxy root@k8s-node2:/opt/k8s/bin/
rsync -av kubelet kube-proxy root@k8s-node3:/opt/k8s/bin/

七. Master节点部署

7.1 部署kube-apiserver

生成kube-apiserver证书

自签证书颁发机构（CA）

cd /data/TLS/k8s/

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成CA证书

1	cfssl gencert -initca ca-csr.json \| cfssljson -bare ca

使用自签CA签发kube-apiserver HTTPS证书

## 创建证书申请文件：

cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "3.1.101.48",
      "3.1.101.49",
      "3.1.101.50",
      "3.1.101.51",
      "3.1.101.52",
      "3.1.101.53",
      "3.1.101.45",
      "3.1.101.46",
      "3.1.101.57",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

注：上述文件hosts字段中IP为所有Master/LB/VIP IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。

生成证书

1	cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json \| cfssljson -bare server

同步证书

# master 节点
cp /data/TLS/k8s/ca*pem /opt/k8s/ssl/
cp /data/TLS/k8s/server*pem /opt/k8s/ssl/

# 同步至node节点
scp /data/TLS/k8s/ca.pem root@k8s-node1:/opt/k8s/ssl
scp /data/TLS/k8s/ca.pem root@k8s-node2:/opt/k8s/ssl
scp /data/TLS/k8s/ca.pem root@k8s-node3:/opt/k8s/ssl

创建conf配置文件

cat > /opt/k8s/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--feature-gates=RemoveSelfLink=false \\
--v=2 \\
--log-dir=/opt/k8s/logs \\
--etcd-servers=https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379 \\
--bind-address=3.1.101.49 \\
--secure-port=6443 \\
--advertise-address=3.1.101.49 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/k8s/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/k8s/ssl/server.pem \\
--kubelet-client-key=/opt/k8s/ssl/server-key.pem \\
--tls-cert-file=/opt/k8s/ssl/server.pem  \\
--tls-private-key-file=/opt/k8s/ssl/server-key.pem \\
--client-ca-file=/opt/k8s/ssl/ca.pem \\
--service-account-key-file=/opt/k8s/ssl/ca-key.pem \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--service-account-signing-key-file=/opt/k8s/ssl/server-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/k8s/ssl/ca.pem \\
--proxy-client-cert-file=/opt/k8s/ssl/server.pem \\
--proxy-client-key-file=/opt/k8s/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/k8s/logs/k8s-audit.log"
EOF

创建TLS机制所需TOKEN

TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

1 2	## 创建kube-apiserver.conf中所需的token.csv echo "`head -c 16 /dev/urandom \| od -An -t x \| tr -d ' '`,kubelet-bootstrap,10001,"system:node-bootstrapper"" > /opt/k8s/cfg/token.csv

创建systemd启动文件

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/k8s/cfg/kube-apiserver.conf
ExecStart=/opt/k8s/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动apiserver

1
2
3

systemctl daemon-reload
systemctl restart kube-apiserver
systemctl enable kube-apiserver

7.2 部署kube-controller-manager

创建conf配置文件

cat > /opt/k8s/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/k8s/logs \\
--leader-elect=true \\
--kubeconfig=/opt/k8s/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/k8s/ssl/ca.pem \\
--cluster-signing-key-file=/opt/k8s/ssl/ca-key.pem  \\
--root-ca-file=/opt/k8s/ssl/ca.pem \\
--service-account-private-key-file=/opt/k8s/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

–kubeconfig：连接apiserver配置文件

–leader-elect：当该组件启动多个时，自动选举（HA）

–cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

生成kubeconfig配置文件

生成证书

kube-controller-manager.kubeconfig

cd /data/TLS/k8s

# 创建证书请求文件
cat > kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing", 
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

生成kubeconfig文件(在/data/TLS/k8s下执行)

KUBE_CONFIG="/opt/k8s/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://3.1.101.49:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-controller-manager \
  --client-certificate=./kube-controller-manager.pem \
  --client-key=./kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建systemd启动文件

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service

[Service]
EnvironmentFile=/opt/k8s/cfg/kube-controller-manager.conf
ExecStart=/opt/k8s/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-controller-manager

1
2
3

systemctl daemon-reload
systemctl restart kube-controller-manager
systemctl enable kube-controller-manager

7.3 部署kube-scheduler

创建conf配置文件

cat > /opt/k8s/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/k8s/logs \\
--leader-elect \\
--kubeconfig=/opt/k8s/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF

–kubeconfig：连接apiserver配置文件

–leader-elect：当该组件启动多个时，自动选举（HA）

生成kubeconfig配置文件

kube-scheduler.kubeconfig

cd /data/TLS/k8s

# 创建证书请求文件
cat > kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

生成kubeconfig文件(在/data/TLS/k8s下执行)

KUBE_CONFIG="/opt/k8s/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://3.1.101.49:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-scheduler \
  --client-certificate=./kube-scheduler.pem \
  --client-key=./kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-scheduler \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建systemd启动文件

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service

[Service]
EnvironmentFile=/opt/k8s/cfg/kube-scheduler.conf
ExecStart=/opt/k8s/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-scheduler

1
2
3

systemctl daemon-reload
systemctl restart kube-scheduler
systemctl enable kube-scheduler

7.4 查看集群状态

生成kubectl连接集群的证书

cd /data/TLS/k8s

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成kubeconfig配置文件

mkdir -pv /root/.kube
KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://3.1.101.49:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials cluster-admin \
  --client-certificate=./admin.pem \
  --client-key=./admin-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

查看集群状态

kubectl  get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-3               Healthy   {"health":"true"}
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-2               Healthy   {"health":"true"}
etcd-4               Healthy   {"health":"true"}

7.5 置到k8s-master2

master1节点

同步k8s-master1配置到k8s-master2

## 同步k8s目录
rsync -av /opt/k8s/* root@k8s-master2:/opt/k8s/
## 同步连接集群所需的kube认证目录
rsync -av /root/.kube root@k8s-master2:/root
## 同步master组件启动文件
rsync -av /usr/lib/systemd/system/kube-apiserver.service root@k8s-master2:/usr/lib/systemd/system/kube-apiserver.service
rsync -av /usr/lib/systemd/system/kube-controller-manager.service root@k8s-master2:/usr/lib/systemd/system/kube-controller-manager.service
rsync -av /usr/lib/systemd/system/kube-scheduler.service root@k8s-master2:/usr/lib/systemd/system/kube-scheduler.service

master节点

修改对应组件配置中以及kube中的IP为master2本机IP
启动组件

systemctl daemon-reload
systemctl restart kube-apiserver
systemctl enable kube-apiserver

systemctl daemon-reload
systemctl restart kube-controller-manager
systemctl enable kube-controller-manager

systemctl daemon-reload
systemctl restart kube-scheduler
systemctl enable kube-scheduler

查看集群状态

kubectl  get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-2               Healthy   {"health":"true"}
etcd-3               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-4               Healthy   {"health":"true"}
etcd-0               Healthy   {"health":"true"}

八. NODE节点部署

master1上执行

master也需要部署node节点相应组件: kubelet和kube-proxy

8.1 部署kubelet

创建conf配置文件

cat > /opt/k8s/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/k8s/logs \\
--hostname-override=k8s-master1 \\
--network-plugin=cni \\
--kubeconfig=/opt/k8s/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/k8s/cfg/bootstrap.kubeconfig \\
--config=/opt/k8s/cfg/kubelet-config.yml \\
--cert-dir=/opt/k8s/ssl \\
--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.1"
EOF

–hostname-override：显示名称，为节点hostname, 集群中唯一

–network-plugin：启用CNI

–kubeconfig：空路径，会自动生成，后面用于连接apiserver

–bootstrap-kubeconfig：首次启动向apiserver申请证书

–config：配置参数文件

–cert-dir：kubelet证书生成目录

–pod-infra-container-image：管理Pod网络容器的镜像

创建yml参数配置文件

kubelet-config.yml文件内容

cat > /opt/k8s/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/k8s/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

创建bootstrap.kubeconfig配置文件

kubelet初次加入集群引导kubeconfig文件

master节点操作

KUBE_CONFIG="/opt/k8s/cfg/bootstrap.kubeconfig"
KUBE_APISERVER="https://3.1.101.49:6443"
TOKEN=`cat /opt/k8s/cfg/token.csv|awk -F',' '{print $1}'` # 与token.csv里保持一致

# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建systemd启动文件

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/opt/k8s/cfg/kubelet.conf
ExecStart=/opt/k8s/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动kubelet

1
2
3

systemctl daemon-reload
systemctl restart kubelet
systemctl enable kubelet

同步kubelet配置到其余节点

同步kubelet.conf, kubelet-config.yml, bootstrap.kubeconfig, kubelet.service到所有节点, 修改对应masterIP

## 同步kubelet配置
rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-master2:/opt/k8s/cfg/
rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-node1:/opt/k8s/cfg/
rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-node2:/opt/k8s/cfg/
rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-node3:/opt/k8s/cfg/

## 同步启动文件
rsync -av /usr/lib/systemd/system/kubelet.service root@k8s-master2:/usr/lib/systemd/system/kubelet.service
rsync -av /usr/lib/systemd/system/kubelet.service root@k8s-node1:/usr/lib/systemd/system/kubelet.service
rsync -av /usr/lib/systemd/system/kubelet.service root@k8s-node2:/usr/lib/systemd/system/kubelet.service
rsync -av /usr/lib/systemd/system/kubelet.service root@k8s-node3:/usr/lib/systemd/system/kubelet.service

## 其余节点启动kubelet
systemctl daemon-reload
systemctl restart kubelet
systemctl enable kubelet

连接集群暂时使用master1节点IP, 后面集群全部正常后, 再做master1和master2的高可用, 将连接集群IP改为VIP

kubelet-bootstrap授权

到这里, 启动kubelet时候会报错

failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "kubelet-bootstrap" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

这是因为kubelet-bootstrap没有权限申请证书,在master上查看证书申请列表也是空的

1 2	kubectl get csr No resources found in default namespace.

这时候需要在master上操作,授权kubelet-bootstrap用户允许请求证书

1
2
3

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

或者执行yaml文件,效果相同

cat > /opt/k8s/yaml/kubelet-bootstrap-rbac.yaml << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-bootstrap 
subjects:
- kind: User
  name: kubelet-bootstrap 
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:node-bootstrapper
  apiGroup: rbac.authorization.k8s.io
EOF

kubectl apply -f kubelet-bootstrap-rbac.yaml

重新启动kubelet,然后在master上查看证书申请

1
2
3

kubectl get csr
NAME 					AGE   				SIGNERNAME 				REQUESTOR 				CONDITION
node-csr-dqVIp0rPbtw3PNeY25Z0V27I2wxANX8R29yjdXT9Q34   36s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

批准kubelet证书申请并加入集群

1	kubectl certificate approve node-csr-dqVIp0rPbtw3PNeY25Z0V27I2wxANX8R29yjdXT9Q34

再次查看证书申请

1
2
3

kubectl get csr
NAME 						AGE    					SIGNERNAME 				REQUESTOR           CONDITION
node-csr-dqVIp0rPbtw3PNeY25Z0V27I2wxANX8R29yjdXT9Q34   2m9s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued

查看节点状态

# kubectl get node
NAME          STATUS     ROLES    AGE   VERSION
k8s-master1   NotReady   <none>   11m   v1.20.6
k8s-master2   NotReady   <none>   12m   v1.20.6
k8s-node1     NotReady   <none>   3s    v1.20.6
k8s-node2     NotReady   <none>   25s   v1.20.6
k8s-node3     NotReady   <none>   14s   v1.20.6

注：由于CNI网络插件还没有部署，节点会没有准备就绪 NotReady

8.2 部署kube-proxy

创建conf配置文件

cat > /opt/k8s/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/k8s/logs \\
--config=/opt/k8s/cfg/kube-proxy-config.yml"
EOF

创建yml参数配置文件–IPVS模式

安装ipvs ipset

1	yum -y install ipvsadm ipset conntrack-tools

配置系统加载模块

cat > /etc/modules-load.d/ipvs.conf <<EOF 
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
ipip
EOF

## 配置开机启动
systemctl restart systemd-modules-load
systemctl enable systemd-modules-load

查看生效模块

1	lsmod \|grep ip_vs

修改kube-proxy-config.yml

cat > /opt/k8s/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
iptables:
  masqueradeAll: true
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
ipvs:
  masqueradeAll: true
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: "rr"
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
mode: "ipvs"
clientConnection:
  kubeconfig: /opt/k8s/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master1
clusterCIDR: 10.0.0.0/24
EOF

注意:

修改hostnameOverride为节点hostname

clusterCIDR: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT

clusterCIDR: 10.0.0.0/24这个是集群service段,和kube-apiserver.conf还有kube-controller-manager.conf中--service-cluster-ip-range=10.0.0.0/24参数保持一致

生成kube-proxy.kubeconfig文件

master节点操作

生成kube-proxy证书:

cd /data/TLS/k8s

# 创建证书请求文件
cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

生成kube-proxy.kubeconfig文件

KUBE_CONFIG="/opt/k8s/cfg/kube-proxy.kubeconfig"
KUBE_APISERVER="https://3.1.101.49:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建systemd启动文件

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=docker.service

[Service]
EnvironmentFile=/opt/k8s/cfg/kube-proxy.conf
ExecStart=/opt/k8s/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动kube-proxy

1
2
3

systemctl daemon-reload
systemctl restart kube-proxy
systemctl enable kube-proxy

验证IPVS模式

1 2	## 验证 ipvsadm -l

同步kube-proxy配置到其余节点

同步kube-proxy.conf, kube-proxy-config.yml, kube-proxy.kubeconfig, kube-proxy.service到所有节点, 修改对应masterIP

## 同步kube-proxy配置
rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-master2:/opt/k8s/cfg/
rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-node1:/opt/k8s/cfg/
rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-node2:/opt/k8s/cfg/
rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-node3:/opt/k8s/cfg/

## 同步启动文件
rsync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-master2:/usr/lib/systemd/system/kube-proxy.service
rsync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-node1:/usr/lib/systemd/system/kube-proxy.service
rsync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-node2:/usr/lib/systemd/system/kube-proxy.service
rsync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-node3:/usr/lib/systemd/system/kube-proxy.service

## 其余节点启动kubelet
systemctl daemon-reload
systemctl restart kube-proxy
systemctl enable kube-proxy

九. 授权apiserver访问kubelet

如果不进行授权, 将无法管理容器

cat > /opt/k8s/yaml/apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f apiserver-to-kubelet-rbac.yaml

十. 部署相关插件

为master节点打污点, master节点不运行pod

1 2	kubectl taint nodes k8s-master1 node-role.kubernetes.io/master=:NoSchedule kubectl taint nodes k8s-master2 node-role.kubernetes.io/master=:NoSchedule

master节点操作

10.1 部署cni网络-Calico

下载地址

https://docs.projectcalico.org/getting-started/kubernetes/installation/config-options

1	curl -k https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico-etcd.yaml

配置Secret

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # The keys below should be uncommented and the values populated with the base64
  # encoded contents of each file that would be associated with the TLS data.
  # Example command for encoding a file contents: cat <file> | base64 -w 0
  etcd-key: <server-key.pem转换内容>
  etcd-cert: <server.pem转换内容>
  etcd-ca: <ca.pem转换内容>

1 2	转换命令: cat <file> \| base64 -w 0

配置ConfigMap

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379"
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: "/calico-secrets/etcd-ca"
  etcd_cert: "/calico-secrets/etcd-cert"
  etcd_key: "/calico-secrets/etcd-key"
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"

etcd_endpoints: ETCD地址

修改Pod CIDR

查找关键字CALICO_IPV4POOL_CIDR; Pod CIDR要与控制器配置文件kube-controller-manager.conf中配置的对应,10.244.0.0/16

# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"

配置calico工作模式

默认IPIP模式,如果关闭后,模式就变为BGP模式

1
2
3

# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Always"

指定网卡

修改DaemonSet控制器下的containers.env
加入
# Specify interface
- name: IP_AUTODETECTION_METHOD
  value: "interface=ens192"

ens.*根据实际环境修改

部署calico网络

1	kubectl apply -f calico-etcd.yaml

部署calico管理工具

下载

1 2	wget -O /usr/bin/calicoctl https://github.com/projectcalico/calicoctl/releases/download/v3.18.1/calicoctl chmod +x /usr/bin/calicoctl

calicoctl配置文件

mkdir -pv /etc/calico/
cat > /etc/calico/calicoctl.cfg << EOF
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "etcdv3"
  etcdEndpoints: "https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379"
  etcdKeyFile: "/opt/etcd/ssl/server-key.pem"
  etcdCertFile: "/opt/etcd/ssl/server.pem"
  etcdCACertFile: "/opt/etcd/ssl/ca.pem"
EOF

calicoctl常用命令

1
2
3

calicoctl node status			//查看当前网络状态,不需要指定配置文件
calicoctl get nodes -o wide			//查看节点,需要指定配置文件
calicoctl get ippool -o wide	//查看 IPAM的IP地址池

10.2 部署Dashboard

下载yaml文件

1	curl https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml -o kubernetes-dashboard.yaml

替换镜像地址(二选一)

更换docker官方镜像更换为阿里云镜像地址(下载更快)

1	sed -i 's#kubernetesui#registry.cn-hangzhou.aliyuncs.com\/google_containers#g' kubernetes-dashboard.yaml

者将两个image地址更换为docker镜像仓库最新版本地址

1 2	kubernetesui/dashboard:v2.3.1 kubernetesui/metrics-scraper:v1.0.7

配置dashboard-service

默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部(kubernetes-dashboard部分), 如下：

cat >> kubernetes-dashboard.yaml << EOF
---
# ------------------- dashboard-service ------------------- #
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard
EOF

配置dashboard-admin帐号

cat >> kubernetes-dashboard.yaml << EOF
---
# ------------------- dashboard-admin ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
EOF

如果不使用上面方法,也可以使用命令创建帐号并授权

// 先创建一个帐号
kubectl create serviceaccount dashboard-admin-01 -n kubernetes-dashboard

// 给账号授权角色
kubectl create clusterrolebinding dashboard-admin-01 --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin-01

// 获取角色帐号TOKEN令牌
kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | awk '/dashboard-admin-01/{print $1}')

相关查询命令:
kubectl -n kubernetes-dashboard get/describe serviceaccount/clusterrolebinding/secret dashboard-admin-01
serviceaccount:	创建帐号
clusterrolebinding:	绑定角色
secret:	token相关

部署kubernetes-dashboard

## 部署
kubectl apply -f kubernetes-dashboard.yaml

## 查看部署状态
kubectl get all -n kubernetes-dashboard -o wide

## 获取令牌
kubectl describe secrets -n kubernetes-dashboard dashboard-admin

## 访问
https://NODE_IP:30001

10.3 部署coredns

下载yaml配置文件

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/coredns.yaml.base

下载coredns.yaml.base,修改后保存为coredns.yaml

修改yaml配置文件

70行左右   kubernetes cluster.local {  	-->大写部分修改成自己的域  一般为 cluster.local
135行左右	image: coredns/coredns:1.8.0 	-->image部分墙外的需要修改，coredns/coredns:1.8.0
140行左右	memory: 170Mi 		 			-->修改成自己适合的值，我这里修改为 170Mi
200行左右	clusterIP: 10.0.0.2				--> clusterIP 修改成kubelet-config.yml中设置的clusterDNS地址

PS: 结合官方模版修改,比如内存,image镜像地址,版本号等
https://github.com/coredns/deployment/blob/master/kubernetes/coredns.yaml.sed

部署coredns

## 部署
kubectl apply -f coredns.yaml

## 验证
kubectl get pod -n kube-system

## 测试
kubectl run busybox --image=busybox --command -- ping www.baidu.com
kubectl exec -it pod/busybox -- /bin/sh -il
或者直接
kubectl run -it --image=busybox:1.28.4 --rm test /bin/sh
------------------------------------------------------------------------------
执行nslookup:
# nslookup kubernetes
结果为如下,证明coredns生效
Server:    10.0.0.2
Address 1: 10.0.0.2

Name:      kubernetes
Address 1: 10.0.0.1
------------------------------------------------------------------------------
执行ping命令
ping www.baidu.com
PING www.baidu.com (220.181.38.149): 56 data bytes
64 bytes from 220.181.38.149: seq=0 ttl=51 time=20.448 ms
64 bytes from 220.181.38.149: seq=1 ttl=51 time=22.957 ms

10.4 部署Ingress

下载yaml配置文件

https://github.com/kubernetes/ingress-nginx/blob/nginx-0.30.0/deploy/static/mandatory.yaml

内容保存为ingress-nginx.yaml

修改ingress-nginx.yaml配置文件

使用Deployment控制器, 配置replicas为2; 或者直接使用DaemonSet控制器
修改rbac.authorization.k8s.io/v1beta1为rbac.authorization.k8s.io/v1(多处)
加入default-http-backend(在Deployment.nginx-ingress-controller之前)

还需要再ingress-nginx.yaml中加入参数

–default-backend-service=$(POD_NAMESPACE)/default-http-backend

位置如下:

containers:
  - name: nginx-ingress-controller
    image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
    args:
      - /nginx-ingress-controller
      - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      - --configmap=$(POD_NAMESPACE)/nginx-configuration
      - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      - --publish-service=$(POD_NAMESPACE)/ingress-nginx
      - --annotations-prefix=nginx.ingress.kubernetes.io

default-http-backend的yaml配置

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
       app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image:  mirrorgooglecontainers/defaultbackend-amd64:1.5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  # namespace: ingress-nginx
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---

配置为hostNetwork模式

需要在Ingress Controller的yaml配置文件中指定使用主机网络hostNetwork: true位置位于spec.tmplate.spec下

不需要配置Service,通过kubectl get ingress获取ingress信息时候,ADDRESS地址会为空

可以将控制器设置为DaemonSet,就可以将域名解析到任意节点进行访问了

部署ingress Controller节点端口(80,443)不能被占用

部署ingress-nginx

修改后nginx-ingress-controller部分配置如下

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      hostNetwork: true
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io

使用kubectl部署

1	kubectl apply -f ingress-nginx.yaml

查看部署状态

1	kubectl get all -n ingress-nginx

将ingress部署到指定节点

给节点打标签

1 2	kubectl label nodes k8s-node2 type=ingress kubectl label nodes k8s-node3 type=ingress

修改yaml文件, 添加pod选择器

spec:
  hostNetwork: true
  # wait up to five minutes for the drain of connections
  terminationGracePeriodSeconds: 300
  serviceAccountName: nginx-ingress-serviceaccount
  nodeSelector:
    kubernetes.io/os: linux
    type: "ingress"
  containers:
    - name: nginx-ingress-controller
      image: devops/ingress/nginx-ingress-controller:0.30.0

更新配置,查看pod所在节点

1 2	kubectl apply -f ingress-nginx.yaml kubectl get all -n ingress-nginx -o wide

验证ingress-nginx

创建deployment和service

## 创建一个nginx deployment
kubectl create deployment nginx-deploy --image=nginx

## 创建发布nginx deployment的service
kubectl expose deployment/nginx-deploy --port=80 --type=NodePort --target-port=80 --name=nginx-svc

配置nginx-ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress-test
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: test.ingress.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: nginx-svc
            port:
              number: 80

配置hosts,域名访问,如果成功,说明ingress已生效

十一. 配置master高可用

NGINX四层负载配置

nginx.conf

与http平级

stream {
    log_format proxy '$remote_addr - [$time_local] $status "$upstream_bytes_sent" "$upstream_bytes_received" - "$upstream_addr" "$upstream_connect_time"';

    access_log  /var/log/nginx/upstream-access.log  proxy;
    open_log_file_cache off;

    include /etc/nginx/upstream/*.conf;
}

k8s.conf

在/etc/nginx/upstream/目录下

upstream k8s-server {
   hash $remote_addr consistent;
   server 3.1.101.49:6443;
   server 3.1.101.50:6443;
}

server {
   listen 6443;
   proxy_pass k8s-server;
   access_log  /var/log/nginx/k8s-ha.log  proxy;
}

重启NGINX生效配置

Keepalived配置

Master

global_defs {
   notification_email {
     ha.localgit.com
   }
   notification_email_from ha.localhost.com
   router_id LB-M
}

vrrp_script check_k8s {
    script "</dev/tcp/127.0.0.1/6443"      #修改为自己需要监听的端口,理论上可以监听远程端口
    interval 2                             #检查脚本的频率,单位（秒）
    weight -30                             #端口检查失败,优先级减少30,weight的绝对值要大于两台priority的差值
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192                                    # 修改为实际网卡名
    virtual_router_id 51                                        # VRRP 路由 ID实例，每个实例是唯一的
    priority 100                                        # 优先级，备服务器设置 90
    advert_int 1                                                # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    # VIP
    virtual_ipaddress {
        3.1.101.45/24
    }
    track_script {
        check_k8s
    }
}

backup

global_defs {
   notification_email {
     ha.localgit.com
   }
   notification_email_from ha.localhost.com
   router_id LB-S
}

vrrp_script check_k8s {
    script "</dev/tcp/127.0.0.1/6443"      #修改为自己需要监听的端口,理论上可以监听远程端口
    interval 2                             #检查脚本的频率,单位（秒）
    weight -30                             #端口检查失败,优先级减少30,weight的绝对值要大于两台priority的差值
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens192                                    # 修改为实际网卡名
    virtual_router_id 51                                # VRRP 路由 ID实例，每个实例是唯一的
    priority 90                                         # 优先级，备服务器设置 90
    advert_int 1                                        # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    # VIP
    virtual_ipaddress {
        3.1.101.45/24
    }
    track_script {
        check_k8s
    }
}

启动keepalived

1	/usr/bin/keepalived -f /etc/keepalived/keepalived.conf

修改k8s配置文件

将配置中master节点IP修改为VIP

master节点

/root/.kube/config

/opt/k8s/cfg/bootstrap.kubeconfig

/opt/k8s/cfg/kube-controller-manager.kubeconfig

/opt/k8s/cfg/kube-scheduler.kubeconfig

node节点

/opt/k8s/cfg/bootstrap.kubeconfig

生效配置

## master节点,重启kube-controller-manager和kube-scheduler
systemctl restart kube-controller-manager
systemctl restart kube-scheduler
rm -f /opt/k8s/ssl/kubelet-client-*.pem
rm -f /opt/k8s/cfg/kubelet.kubeconfig
systemctl restart kubelet

## node节点,删除自申请成证书以及kubeconfig文件,重启kubelet重新申请(需要master节点重新批准)
rm -f /opt/k8s/ssl/kubelet-client-*.pem
rm -f /opt/k8s/cfg/kubelet.kubeconfig
systemctl restart kubelet

## 验证
kubectl get cs
kubectl get node

到此,k8s 1.20二进制版本高可用部署完成