Velero

Velero简介

  • Velero 是vmware开源的一个云原生的灾难恢复和迁移工具，它本身也是开源的,采用Go语言编写，可以安全的备份、恢复和迁移Kubernetes集群资源数据，https://velero.io/
  • Velero 是西班牙语意思是帆船，非常符合Kubernetes社区的命名风格，Velero的开发公司Heptio，已被VMware收购。
  • Velero 支持标准的K8S集群，既可以是私有云平台也可以是公有云，除了灾备之外它还能做资源移转，支持把容器应用从一个集群
    迁移到另一个集群
  • Velero 的工作方式就是把kubernetes中的数据备份到对象存储以实现高可用和持久化，默认的备份保存时间为720小时，并在需要
    的时候进行下载和恢复。

Velero与etcd快照备份的区别：

  • etcd 快照是全局完成备份(类似于MySQL全部备份),即使需要恢复一个资源对象(类似于只恢复MySQL的一个库),但是也需要做全局恢复到备份的状态(类似于MySQL的全库恢复),即会影响其它namespace中pod运行服务(类似于会影响MySQL其它数据库的数据)。
  • Velero可以有针对性的备份,比如按照namespace单独备份、只备份单独的资源对象等,在恢复的时候可以根据备份只恢复单独的namespace或资源对象，而不影响其它namespace中pod运行服务。
  • velero支持ceph、oss等对象存储，etcd 快照是一个为本地文件。
  • velero支持任务计划实现周期备份，但etcd 快照也可以基于cronjob实现。
  • velero支持对AWS EBS创建快照及还原
    https://www.qloudx.com/velero-for-kubernetes-backup-restore-stateful-workloads-with-aws-ebs-snapshots/
    https://github.com/vmware-tanzu/velero-plugin-for-aws #Elastic Block Store（弹性块存储）

Velero整体架构:

Velero整体架构图

备份流程:

Velero备份流程图

  • Velero 客户端调用Kubernetes API Server创建Backup任务。
  • Backup 控制器基于watch 机制通过API Server获取到备份任务。
  • Backup 控制器开始执行备份动作，其会通过请求API Server获取需要备份的数据。
  • Backup 控制器将获取到的数据备份到指定的对象存储server端。
velero backup create myserver-ns-backup-${DATE} --include-namespaces myserver --kubeconfig=/root/.kube/config --namespace velero-system

部署环境

Velero部署环境图

部署minio：

1、下载minio image：

docker pull minio/minio:RELEASE.2022-04-12T06-55-35Z

2、创建数据目录：

mkdir -p /data/minio

3、创建minio容器

docker run --name minio \
-p 9000:9000 \
-p 9999:9999 \
-d --restart=always \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=12345678" \
-v /data/minio/data:/data \
minio/minio:RELEASE.2022-04-12T06-55-35Z server /data \
--console-address '0.0.0.0:9999'

注意：如果不指定用户名和密码，则默认用户名与密码为 minioadmin/minioadmin,可以通过环境变量自定义；
如上所示 用户名：admin；密码：12345678

3.1、webminio登录界面

登陆地址：

http://10.2.0.11:9999/

webminio登录界面

3.2、创建bucket：

Buckets=>Create Bucket=>输入Bucket Name => Create Bucket

实验环境创建的Bucket Name是velerodata

部署Velero Server:

https://github.com/vmware-tanzu/velero #版本兼容性
https://velero.io/docs/v1.8/contributions/minio/# #官方安装文档

wget https://github.com/vmware-tanzu/velero/releases/download/v1.12.0/velero-v1.12.0-linux-amd64.tar.gz
tar xvf velero-v1.12.0-linux-amd64.tar.gz && cp velero-v1.12.0-linux-amd64/velero  /usr/local/bin/

velero  --help
Usage:
velero [command]

Available Commands:
backup            Work with backups
backup-location   Work with backup storage locations
bug               Report a Velero bug
client            Velero client related commands
completion        Generate completion script
create            Create velero resources
debug             Generate debug bundle
delete            Delete velero resources
describe          Describe velero resources
get               Get velero resources
help              Help about any command
install           Install Velero
plugin            Work with plugins
repo              Work with repositories
restore           Work with restores
schedule          Work with schedules
snapshot-location Work with snapshot locations
uninstall         Uninstall Velero
version           Print the velero version and associated image
............

创建Velero用于访问存储桶的凭据文件：

cat ./velero-auth.txt
aws_access_key_id = admin
aws_secret_access_key = 1245678

注意：凭据文件中的变量名不能改变

启动服务器和本地存储服务。在Velero namespace中运行：

kubectl apply -f velero-v1.12.0-linux-amd64/examples/minio/00-minio-deployment.yaml

namespace/velero created
deployment.apps/minio created
service/minio created
job.batch/minio-setup created

安装velero：

velero --kubeconfig  /root/.kube/config \
    install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.5.5 \
    --bucket velerodata  \
    --secret-file ./velero-auth.txt \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.2.0.11:9000

CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client
CustomResourceDefinition/backuprepositories.velero.io: already exists, proceeding
CustomResourceDefinition/backuprepositories.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: already exists, proceeding
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: already exists, proceeding
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: already exists, proceeding
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: already exists, proceeding
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: already exists, proceeding
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: already exists, proceeding
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: already exists, proceeding
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: already exists, proceeding
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: already exists, proceeding
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
CustomResourceDefinition/datadownloads.velero.io: attempting to create resource
CustomResourceDefinition/datadownloads.velero.io: attempting to create resource client
CustomResourceDefinition/datadownloads.velero.io: already exists, proceeding
CustomResourceDefinition/datadownloads.velero.io: created
CustomResourceDefinition/datauploads.velero.io: attempting to create resource
CustomResourceDefinition/datauploads.velero.io: attempting to create resource client
CustomResourceDefinition/datauploads.velero.io: already exists, proceeding
CustomResourceDefinition/datauploads.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: already exists, proceeding
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

根据最后一句提示执行指令查看状态

Velero已安装！⛵ 使用“kubectl logs deployment/velero-n velero”查看状态

参数解释：

--kubeconfig：指定Kubernetes集群的kubeconfig文件。
--provider：指定Velero使用的存储提供商。
--plugins：指定Velero使用的插件。
--bucket：指定Velero使用的存储桶名称。
--secret-file：指定Velero用于访问存储桶的凭据文件。
--use-volume-snapshots：是否使用卷快照进行备份。
--backup-location-config：指定Velero备份和还原的存储位置配置。
s3ForcePathStyle="true"：强制使用S3的存储方式；S3=对象存储
s3Url :minio的API端口

备份default名称空间所有资源：

当前default资源

root@k8s-master-1:~# kubectl get pod
NAME                                      READY   STATUS    RESTARTS        AGE
mysql-0                                   2/2     Running   2 (2d19h ago)   22d
mysql-1                                   2/2     Running   2 (2d19h ago)   22d
mysql-2                                   2/2     Running   2 (2d19h ago)   22d
nginx-deployment-79b5755fc5-qrz4w         1/1     Running   2 (2d19h ago)   28d
redis-0                                   1/1     Running   1 (2d19h ago)   23d
redis-1                                   1/1     Running   1 (2d19h ago)   23d
redis-2                                   1/1     Running   1 (2d19h ago)   23d
redis-3                                   1/1     Running   1 (2d19h ago)   23d
redis-4                                   1/1     Running   1 (2d19h ago)   23d
redis-5                                   1/1     Running   1 (2d19h ago)   23d
tomcat-app1-deployment-7bd6894c5f-t5bx8   1/1     Running   2 (2d19h ago)   29d
zookeeper1-544f7fbfd4-d65mx               1/1     Running   1 (2d19h ago)   24d
zookeeper2-66b9896c7-x5669                1/1     Running   1 (2d19h ago)   24d
zookeeper3-84f4f9589d-5rbd6               1/1     Running   1 (2d19h ago)   24d
root@k8s-master-1:~# kubectl get deploy
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment         1/1     1            1           28d
tomcat-app1-deployment   1/1     1            1           29d
zookeeper1               1/1     1            1           24d
zookeeper2               1/1     1            1           24d
zookeeper3               1/1     1            1           24d

DATE=`date +%Y%m%d%H%M%S`

velero backup create default-backup-${DATE} \
--include-cluster-resources=true \
--include-namespaces default \
--kubeconfig=/root/.kube/config \
--namespace velero-system

Backup request "default-backup-20230924155439" submitted successfully.
Run `velero backup describe default-backup-20230924155439` or `velero backup logs default-backup-20230924155439` for more details

参数解释：

velero backup create：创建一个备份。
default-backup-{DATE}：备份的名称。${DATE} 将被替换为当前日期。
--include-cluster-resources=true：包含所有集群资源。
--include-namespaces default：备份 default 命名空间的所有资源。
--kubeconfig=/root/.kube/config：指定 Kubeconfig 文件的路径。
--namespace velero-system：在 velero-system 命名空间中创建备份。

注意：以上命令用于创建一个包含所有集群资源和 default 命名空间所有资源的备份。

验证备份：

root@k8s-master-1:~# velero backup describe default-backup-20230924155439 --kubeconfig=/root/.kube/config --namespace velero
Name:         default-backup-20230924155439
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.27.2
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=27

Phase:  Completed


Namespaces:
  Included:  default
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  included

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          auto
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-09-24 15:54:59 +0800 CST
Completed:  2023-09-24 15:55:04 +0800 CST

Expiration:  2023-10-24 15:54:59 +0800 CST

Total items to be backed up:  358
Items backed up:              358

Velero-Native Snapshots: <none included>

验证minio备份数据：

验证minio备份数据

删除resources并验证数据恢复：

root@k8s-master-1:~# kubectl delete deploy tomcat-app1-deployment

验证删除是否成功：

root@k8s-master-1:~# kubectl get deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1/1     1            1           28d
zookeeper1         1/1     1            1           24d
zookeeper2         1/1     1            1           24d
zookeeper3         1/1     1            1           24d
root@k8s-master-1:~# kubectl get pod
NAME                                      READY   STATUS        RESTARTS        AGE
mysql-0                                   2/2     Running       2 (2d19h ago)   22d
mysql-1                                   2/2     Running       2 (2d19h ago)   22d
mysql-2                                   2/2     Running       2 (2d19h ago)   22d
nginx-deployment-79b5755fc5-qrz4w         1/1     Running       2 (2d19h ago)   28d
redis-0                                   1/1     Running       1 (2d19h ago)   23d
redis-1                                   1/1     Running       1 (2d19h ago)   23d
redis-2                                   1/1     Running       1 (2d19h ago)   23d
redis-3                                   1/1     Running       1 (2d19h ago)   23d
redis-4                                   1/1     Running       1 (2d19h ago)   23d
redis-5                                   1/1     Running       1 (2d19h ago)   23d
zookeeper1-544f7fbfd4-d65mx               1/1     Running       1 (2d19h ago)   24d
zookeeper2-66b9896c7-x5669                1/1     Running       1 (2d19h ago)   24d
zookeeper3-84f4f9589d-5rbd6               1/1     Running       1 (2d19h ago)   24d

恢复tomcat-app1-deployment:

velero restore create --from-backup  default-backup-20230924155439 --wait --kubeconfig=/root/.kube/config --namespace velero

Restore request "default-backup-20230924155439-20230925105933" submitted successfully.  #已提交还原请求
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background. #正在等待恢复完成，可以安全地按ctrl-c停止等待-您的恢复将在后台继续
....................................
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe default-backup-20230924155439-20230925105933` and `velero restore logs default-backup-20230924155439-20230925105933`. 
#还原已完成，状态为：已完成。您可以使用命令“velero restore describe default-backup-2023092415539-202309225105933”和“velelo restore logs default-backup 20230924155439-2023092505933”检查更多信息。

参数解释：

velero restore create：创建一个 Restore 请求。
--from-backup default-backup-20230924155439：指定要恢复的备份的名称。
--wait：等待 Restore 请求完成。
--kubeconfig=/root/.kube/config：指定用于访问 Kubernetes 集群的 kubeconfig 文件的路径。
--namespace velero：指定创建 Restore 请求的 namespace。

验证恢复deploy：

root@k8s-master-1:~# kubectl get deploy
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment         1/1     1            1           28d
tomcat-app1-deployment   1/1     1            1           3s
zookeeper1               1/1     1            1           24d
zookeeper2               1/1     1            1           24d
zookeeper3               1/1     1            1           24d

查看tomcat-app1-deployment控制器所在行的AGE列时间为3s，意思是在3秒之前恢复的

备份指定namespace中的pod或特定资源：

kubectl run net-test1 --image=centos:7.9.2009 sleep 10000000000 -n qwx
pod/net-test1 created
kubectl run net-test2 --image=centos:7.9.2009 sleep 10000000000 -n qwx
pod/net-test2 created
root@k8s-master-1:~# kubectl get pod -n  qwx
NAME        READY   STATUS    RESTARTS   AGE
net-test1   1/1     Running   0          2s
net-test2   1/1     Running   0          3s

备份Pod资源

root@k8s-master-1:~# velero backup create pod-backup-20230925 --include-cluster-resources=true --ordered-resources='pods=qwx/net-test1,qwx/net-test2' --namespace velero --include-namespaces=qwx

Backup request "pod-backup-20230925" submitted successfully.
Run `velero backup describe pod-backup-20230925` or `velero backup logs pod-backup-20230925` for more details.

删除Pod资源

root@k8s-master-1:~# kubectl delete pod -n qwx net-test1 net-test2
pod "net-test1" deleted
pod "net-test2" deleted

验证是否删除成功：

root@k8s-master-1:~# kubectl get pod -n  qwx
No resources found in qwx namespace.

恢复Pod资源：

root@k8s-master-1:~# velero restore create --from-backup pod-backup-20230925 --wait \
--kubeconfig=/root/.kube/config \
--namespace velero

Restore request "pod-backup-20230925-20230925122657" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
...........................
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe pod-backup-20230925-20230925122657` and `velero restore logs pod-backup-20230925-20230925122657`.

验证Pod资源

root@k8s-master-1:~# kubectl get pod -n  qwx
NAME        READY   STATUS    RESTARTS   AGE
net-test1   1/1     Running   0          23s
net-test2   1/1     Running   0          23s

指定资源类型--ordered-resources

举例备份namespace为qwx中的deploy的控制器：
--ordered-resources 'deployments=qwx/deploy1,qwx/deploy2'
举例备份namespace为qwx中的PVC：
--ordered-resources 'persistentvolumeclaims=qwx/pvc4,qwx/pvc8'

批量备份不同的namespace:

#!/bin/bash
NS_NAME=`kubectl get ns | awk '{if (NR>2){print}}' | awk '{print $1}'`
DATE=`date +%Y%m%d%H%M%S`
cd /data/velero/
for i in $NS_NAME;do
velero backup create ${i}-ns-backup-${DATE} \
--include-cluster-resources=true \
--include-namespaces ${i} \
--kubeconfig=/root/.kube/config \   
--namespace velero-system
done
最后修改：2023 年 09 月 25 日
© 禁止转载
如果觉得我的文章对你有用，请随意赞赏