etcd 集群的备份和还原

准备工作

安装 etcdctl

方法1

apt install etcd-client=3.2.17+dfsg-1

方法2

export RELEASE="3.2.17"

test -d /tmp/etcd && mkdir -p /tmp/etcd && cd /tmp/etcd
wget https://github.com/etcd-io/etcd/releases/download/v${RELEASE}/etcd-v${RELEASE}-linux-amd64.tar.gz
tar -zxvf etcd-v${RELEASE}-linux-amd64.tar.gz
cd etcd-v${RELEASE}-linux-amd64
cp etcdctl /usr/local/bin

etcdctl --version

方法3

使用 docker cp 从 etcd 容器中拷贝。

备份

etcd 的备份有两种方式，选择其一即可。

方式一：使用 etcdctl snapshot 命令（推荐）

在任何一个 member 节点执行：

ETCDCTL_API=3 etcdctl snapshot save snapshot.db

方式二：拷贝 member/snap/db 文件

cp /var/lib/etcd/member/snap/db snapshot.db

如果使用此方法，etcdctl snapshot restore 时需要设置 --skip-hash-check=true

还原

方式一：单节点还原成功后，再将其他节点加入集群

根据 snapshot.db 生成新的 data dir：

# restore.sh

rm /var/lib/etcd -rf 
 
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --name k8s-etcd-host1 \
  --data-dir /var/lib/etcd \
  --initial-cluster k8s-etcd-host1=http://host1:2380 \
  --initial-cluster-token k8s-etcd \
  --initial-advertise-peer-urls http://host1:2380 \
  --skip-hash-check=false

启动单实例：

spec:
  containers:
  - command:
    - etcd
    - --name=k8s-etcd-host1
    - --initial-advertise-peer-urls=http://host1:2380
    - --listen-peer-urls=http://host1:2380
    - --listen-client-urls=http://0.0.0.0:2379
    - --advertise-client-urls=http://host1:2379
    - --data-dir=/var/lib/etcd
    - --initial-cluster-token=k8s-etcd
    - --initial-cluster=k8s-etcd-host1=http://host1:2380
    - --initial-cluster-state=existing

将其他节点依次加入集群（先执行 add 命令再启动实例），add 命令如下：

etcdctl member add k8s-etcd-host2 http://host2:2380

启动实例：

spec:
  containers:
  - command:
    - etcd
    - --name=k8s-etcd-host2
    - --initial-advertise-peer-urls=http://host2:2380
    - --listen-peer-urls=http://host2:2380
    - --listen-client-urls=http://0.0.0.0:2379
    - --advertise-client-urls=http://host2:2379
    - --data-dir=/var/lib/etcd
    - --initial-cluster-token=k8s-etcd
    - --initial-cluster=k8s-etcd-host1=http://host1:2380,k8s-etcd-host2=http://host2:2380
    - --initial-cluster-state=existing

其他实例操作方法类似。

方式二：同时还原多节点集群

将 snapshot.db 文件拷贝至所有 etcd 节点，根据 snapshot.db 生成 data dir：

ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --name k8s-etcd-host1 \
  --data-dir /var/lib/etcd \
  --initial-cluster k8s-etcd-host1=http://host1:2380,k8s-etcd-host2=http://host2:2380,k8s-etcd-host3=http://host3:2380 \
  --initial-cluster-token k8s-etcd \
  --initial-advertise-peer-urls http://host1:2380 \
  --skip-hash-check=false
 
 
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --name k8s-etcd-host2 \
  --data-dir /var/lib/etcd \
  --initial-cluster k8s-etcd-host1=http://host1:2380,k8s-etcd-host2=http://host2:2380,k8s-etcd-host3=http://host3:2380 \
  --initial-cluster-token k8s-etcd \
  --initial-advertise-peer-urls http://host2:2380 \
  --skip-hash-check=false
 
 
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --name k8s-etcd-host3 \
  --data-dir /var/lib/etcd \
  --initial-cluster k8s-etcd-host1=http://host1:2380,k8s-etcd-host2=http://host2:2380,k8s-etcd-host3=http://host3:2380 \
  --initial-cluster-token k8s-etcd \
  --initial-advertise-peer-urls http://host3:2380 \
  --skip-hash-check=false

还原后启动所有 etcd 实例 。启动参数如下，其他类似：

spec:
  containers:
  - command:
    - etcd
    - --name=k8s-etcd-host1
    - --initial-advertise-peer-urls=http://host1:2380
    - --listen-peer-urls=http://host1:2380
    - --listen-client-urls=http://0.0.0.0:2379
    - --advertise-client-urls=http://host1:2379
    - --data-dir=/var/lib/etcd
    - --initial-cluster-token=k8s-etcd
    - --initial-cluster=k8s-etcd-host1=http://host1:2380,k8s-etcd-host2=http://host2:2380,k8s-etcd-host3=http://host3:2380
    - --initial-cluster-state=existing

注意

启动 etcd 之前最好停掉 kube-apiserver

参考

• https://coreos.com/etcd/docs/3.1.12/op-guide/recovery.html
• https://coreos.com/etcd/docs/latest/op-guide/runtime-configuration.html