编程技术分享
搜索Ctrl K

外观

etcd集群部署 ​

1. 概述 ​

这是一个序列总结文档。

1.1 VirtualBox虚拟机信息记录 ​

学习etcd时，使用以下几个虚拟机:

序号虚拟机主机名IPCPU内存说明
1ansible-masteransible192.168.56.1202核4GAnsible控制节点
2ansible-node1etcd-node1192.168.56.1212核2GAnsible工作节点1
3ansible-node2etcd-node2192.168.56.1222核2GAnsible工作节点2
4ansible-node3etcd-node3192.168.56.1232核2GAnsible工作节点3

后面会编写使用ansible部署etcd集群的剧本。

操作系统说明：

sh
[root@etcd-node1 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[root@etcd-node1 ~]# hostname -I
192.168.56.121 10.0.3.15
[root@etcd-node1 ~]#

2. 三服务器部署etcd集群 ​

参考官方文档How to Set Up a Demo etcd Cluster

  • ectd端口中是 2379 用于客户端连接，提供HTTP API服务，供客户端交互。而 2380 用于伙伴通讯。

由于是在三个服务器上部署etcd集群，我们可以直接使用默认端口，我们先规划一下：

序号节点名称节点运行目录IP通讯端口客户端端口
1node1/srv/etcd/node192.168.56.12123802379
2node2/srv/etcd/node192.168.56.12123802379
3node2/srv/etcd/node192.168.56.12123802379

参考单节点部署etcd服务，配置不同的启动脚本。

查看第1个启动脚本：

sh
[root@etcd-node1 etcd]# cat node/start.sh
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=192.168.56.121
HOST_2=192.168.56.122
HOST_3=192.168.56.123
PEER_PORT_1=2380
PEER_PORT_2=2380
PEER_PORT_3=2380
API_PORT_1=2379
API_PORT_2=2379
API_PORT_3=2379
START_PATH_1=/srv/etcd/node
START_PATH_2=/srv/etcd/node
START_PATH_3=/srv/etcd/node
CLUSTER=${NAME_1}=http://${HOST_1}:${PEER_PORT_1},${NAME_2}=http://${HOST_2}:${PEER_PORT_2},${NAME_3}=http://${HOST_3}:${PEER_PORT_3}
echo "CLUSTER:${CLUSTER}"

# 节点1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
cd ${START_PATH_1}
nohup etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:${PEER_PORT_1} --listen-peer-urls http://${THIS_IP}:${PEER_PORT_1} \
    --advertise-client-urls http://${THIS_IP}:${API_PORT_1} --listen-client-urls http://${THIS_IP}:${API_PORT_1} \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

[root@etcd-node1 etcd]#

查看第2个启动脚本：

sh
[root@etcd-node2 etcd]# cat node/start.sh
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=192.168.56.121
HOST_2=192.168.56.122
HOST_3=192.168.56.123
PEER_PORT_1=2380
PEER_PORT_2=2380
PEER_PORT_3=2380
API_PORT_1=2379
API_PORT_2=2379
API_PORT_3=2379
START_PATH_1=/srv/etcd/node
START_PATH_2=/srv/etcd/node
START_PATH_3=/srv/etcd/node
CLUSTER=${NAME_1}=http://${HOST_1}:${PEER_PORT_1},${NAME_2}=http://${HOST_2}:${PEER_PORT_2},${NAME_3}=http://${HOST_3}:${PEER_PORT_3}
echo "CLUSTER:${CLUSTER}"

# 节点2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
cd ${START_PATH_2}
nohup etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:${PEER_PORT_2} --listen-peer-urls http://${THIS_IP}:${PEER_PORT_2} \
    --advertise-client-urls http://${THIS_IP}:${API_PORT_2} --listen-client-urls http://${THIS_IP}:${API_PORT_2} \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

[root@etcd-node2 etcd]#

查看第3个启动脚本：

sh
[root@etcd-node3 etcd]# cat node/start.sh
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=192.168.56.121
HOST_2=192.168.56.122
HOST_3=192.168.56.123
PEER_PORT_1=2380
PEER_PORT_2=2380
PEER_PORT_3=2380
API_PORT_1=2379
API_PORT_2=2379
API_PORT_3=2379
START_PATH_1=/srv/etcd/node
START_PATH_2=/srv/etcd/node
START_PATH_3=/srv/etcd/node
CLUSTER=${NAME_1}=http://${HOST_1}:${PEER_PORT_1},${NAME_2}=http://${HOST_2}:${PEER_PORT_2},${NAME_3}=http://${HOST_3}:${PEER_PORT_3}
echo "CLUSTER:${CLUSTER}"

# 节点3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
cd ${START_PATH_3}
nohup etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:${PEER_PORT_3} --listen-peer-urls http://${THIS_IP}:${PEER_PORT_3} \
    --advertise-client-urls http://${THIS_IP}:${API_PORT_3} --listen-client-urls http://${THIS_IP}:${API_PORT_3} \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

[root@etcd-node3 etcd]#

三个节点分别执行启动脚本：

sh
[root@etcd-node1 ~]# cd /srv/etcd/node && ./start.sh 
CLUSTER:node1=http://192.168.56.121:23801,node2=http://192.168.56.121:23802,node3=http://192.168.56.121:23803
/root
[root@etcd-node1 ~]# nohup: appending output to ‘nohup.out’


[root@etcd-node2 ~]# cd /srv/etcd/node && ./start.sh 
CLUSTER:node1=http://192.168.56.121:23801,node2=http://192.168.56.121:23802,node3=http://192.168.56.121:23803
/root
[root@etcd-node1 ~]# nohup: appending output to ‘nohup.out’


[root@etcd-node3 ~]# cd /srv/etcd/node && ./start.sh 
CLUSTER:node1=http://192.168.56.121:23801,node2=http://192.168.56.121:23802,node3=http://192.168.56.121:23803
/root
[root@etcd-node1 ~]# nohup: appending output to ‘nohup.out’

查看etcd进程和端口监听：

sh
# 第1个节点
[root@etcd-node1 ~]# ps -ef|grep -v grep|grep etcd
root      1440     1  0 22:55 pts/0    00:00:03 etcd --data-dir=data.etcd --name node1 --initial-advertise-peer-urls http://192.168.56.121:2380 --listen-peer-urls http://192.168.56.121:2380 --advertise-client-urls http://192.168.56.121:2379 --listen-client-urls http://192.168.56.121:2379 --initial-cluster node1=http://192.168.56.121:2380,node2=http://192.168.56.122:2380,node3=http://192.168.56.123:2380 --initial-cluster-state new --initial-cluster-token token-01
[root@etcd-node1 ~]# netstat -tunlp|grep etcd
tcp        0      0 192.168.56.121:2379     0.0.0.0:*               LISTEN      1440/etcd
tcp        0      0 192.168.56.121:2380     0.0.0.0:*               LISTEN      1440/etcd
[root@etcd-node1 ~]#


# 第2个节点
[root@etcd-node2 ~]# ps -ef|grep -v grep|grep etcd
root      1458     1  0 22:57 pts/0    00:00:01 etcd --data-dir=data.etcd --name node2 --initial-advertise-peer-urls http://192.168.56.122:2380 --listen-peer-urls http://192.168.56.122:2380 --advertise-client-urls http://192.168.56.122:2379 --listen-client-urls http://192.168.56.122:2379 --initial-cluster node1=http://192.168.56.121:2380,node2=http://192.168.56.122:2380,node3=http://192.168.56.123:2380 --initial-cluster-state new --initial-cluster-token token-01
[root@etcd-node2 ~]# netstat -tunlp|grep etcd
tcp        0      0 192.168.56.122:2379     0.0.0.0:*               LISTEN      1458/etcd
tcp        0      0 192.168.56.122:2380     0.0.0.0:*               LISTEN      1458/etcd


# 第3个节点
[root@etcd-node3 ~]# ps -ef|grep -v grep|grep etcd
root      1459     1  0 22:57 pts/0    00:00:01 etcd --data-dir=data.etcd --name node3 --initial-advertise-peer-urls http://192.168.56.123:2380 --listen-peer-urls http://192.168.56.123:2380 --advertise-client-urls http://192.168.56.123:2379 --listen-client-urls http://192.168.56.123:2379 --initial-cluster node1=http://192.168.56.121:2380,node2=http://192.168.56.122:2380,node3=http://192.168.56.123:2380 --initial-cluster-state new --initial-cluster-token token-01
[root@etcd-node3 ~]# netstat -tunlp|grep etcd
tcp        0      0 192.168.56.123:2379     0.0.0.0:*               LISTEN      1459/etcd
tcp        0      0 192.168.56.123:2380     0.0.0.0:*               LISTEN      1459/etcd
[root@etcd-node3 ~]#

3. 查看集群信息 ​

将以下信息写入到~/.bashrc配置文件中：

sh
export ETCDCTL_API=3
export HOST_1=192.168.56.121
export HOST_2=192.168.56.122
export HOST_3=192.168.56.123
export API_PORT_1=2379
export API_PORT_2=2379
export API_PORT_3=2379
export ENDPOINTS=${HOST_1}:${API_PORT_1},${HOST_2}:${API_PORT_2},${HOST_3}:${API_PORT_3}

然后使用source ~/.bashrc使配置生效。

3.1 查看集群成员列表 ​

生效后，查看集群成员列表信息：

sh
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS member list
b8d58f2bdbad1d62, started, node1, http://192.168.56.121:2380, http://192.168.56.121:2379, false
c5a01234fd04a6a4, started, node3, http://192.168.56.123:2380, http://192.168.56.123:2379, false
fd3f25a01705c082, started, node2, http://192.168.56.122:2380, http://192.168.56.122:2379, false
[root@etcd-node1 ~]#


[root@etcd-node2 ~]# etcdctl --endpoints=$ENDPOINTS member list
b8d58f2bdbad1d62, started, node1, http://192.168.56.121:2380, http://192.168.56.121:2379, false
c5a01234fd04a6a4, started, node3, http://192.168.56.123:2380, http://192.168.56.123:2379, false
fd3f25a01705c082, started, node2, http://192.168.56.122:2380, http://192.168.56.122:2379, false
[root@etcd-node2 ~]#


[root@etcd-node3 ~]# etcdctl --endpoints=$ENDPOINTS member list
b8d58f2bdbad1d62, started, node1, http://192.168.56.121:2380, http://192.168.56.121:2379, false
c5a01234fd04a6a4, started, node3, http://192.168.56.123:2380, http://192.168.56.123:2379, false
fd3f25a01705c082, started, node2, http://192.168.56.122:2380, http://192.168.56.122:2379, false
[root@etcd-node3 ~]#

3.2 查看集群状态信息 ​

sh
etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint status

在三个节点查看到信息应该是一样的，类似如下所示：

sh
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint status
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.56.121:2379 | b8d58f2bdbad1d62 |  3.5.18 |   20 kB |      true |      false |         4 |         16 |                 16 |        |
| 192.168.56.122:2379 | fd3f25a01705c082 |  3.5.18 |   20 kB |     false |      false |         4 |         16 |                 16 |        |
| 192.168.56.123:2379 | c5a01234fd04a6a4 |  3.5.18 |   20 kB |     false |      false |         4 |         16 |                 16 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@etcd-node1 ~]#

Snipaste_2025-03-03_23-00-35.png

可以看到，节点node1是Leader。

3.3 查看集群健康状态 ​

sh
etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint health

在三个节点查看到信息应该是一样的，类似如下所示：

sh
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint health
+---------------------+--------+------------+-------+
|      ENDPOINT       | HEALTH |    TOOK    | ERROR |
+---------------------+--------+------------+-------+
| 192.168.56.121:2379 |   true | 2.240929ms |       |
| 192.168.56.122:2379 |   true | 2.163503ms |       |
| 192.168.56.123:2379 |   true | 9.061671ms |       |
+---------------------+--------+------------+-------+
[root@etcd-node1 ~]#

可以看到，三个节点健康都是true正常的。

3.4 集群的数据读写 ​

在节点1上写入数据：

sh
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS put greeting "Hello, etcd"
OK
[root@etcd-node1 ~]#

然后在三个节点上获取数据：

sh
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS get greeting
greeting
Hello, etcd
[root@etcd-node1 ~]#


[root@etcd-node2 ~]# etcdctl --endpoints=$ENDPOINTS get greeting
greeting
Hello, etcd
[root@etcd-node2 ~]#



[root@etcd-node3 ~]# etcdctl --endpoints=$ENDPOINTS get greeting
greeting
Hello, etcd
[root@etcd-node3 ~]#

三个节点都正常获取到数据了。

Gitalking ...

本首页参考 https://notes.fe-mm.com/ 配置而成

Copyright © 2019-2024 阿梅的博客 京ICP备19051502号