Disaster Recovery and Data Prote

In this session, we will provide some strategies on how to protect the critical data. We will discuss how to use OpenSDS, an open source Software Defined Storage project under Linux Foundation, to provision persistent volumes for Kubernetes using the CSI plugin, how to use a policy engine to periodically and asynchronously create snapshots as point-in-time protection for the Kubernetes persistent volumes, and how the array-based and host-based replication feature in OpenSDS can help protect the data residing on the persistent volumes in the case of a disaster.

1.Disaster Recovery and Data Protection for Kubernetes Persistent Volumes Xing Yang, Principal Architect, Huawei

2.Agenda • Kubernetes Persistent Volumes and CSI • Why OpenSDS for Kubernetes and CSI • OpenSDS Overview • Provision and Manage Persistent Volumes • Disaster Recovery for Persistent Volumes • Data Protection for Persistent Volumes • OpenSDS Roadmap for Aruba and Bali Release • OpenSDS Community • Demo

3.Kubernetes Persistent Volumes • A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator. • A PV can be provisioned statically or dynamically. • A PersistentVolumeClaim (PVC) is a request for storage by a user through a StorageClass. • A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to different quality-of-service levels (or ”profiles”) in other storage systems. • A StorageClass needs to specify a provisioner for dynamic provisioning.

4.Container Storage Interface (CSI) CSI is an industry standard defined to enable storage vendors to develop a plugin once and have it work across a number of container orchestration systems. Source: https://github.com/kubernetes/community/blob/master/contributors/ design-proposals/storage/container-storage-interface.md

5.What Happens When Disaster Strikes Pod Pod Container Container Container Container Container Container Container Container PVC PVC PV PV

6.Why OpenSDS for Kubernetes and CSI • Storage functionalities in Kubernetes and CSI are still evolving. • OpenSDS can provide additional storage functionalities such as data protection and disaster recovery. • Provide unified control for traditional cloud and cloud native environment. Pod Pod Container Container Container Container Container Container Container Container PVC PVC OpenSDS PV PV PV PV ✗ ✓

7.OpenSDS Overview - Core Projects OpenSDS is an Open Source Project under the Linux Foundation

8.OpenSDS Overview - Project Framework

9.OpenSDS Overview - Architecture

10.Provision and Manage Persistent Volumes using OpenSDS

11.Profiles: Mapping OpenSDS Profile to K8S StorageClass

12.Profiles: Policy Driven SPDM • OpenSDS profile is based on Swordfish specification. • The SNIA Swordfish™ specification helps to provide a unified approach for the management of storage and servers in hyperscale and cloud infrastructure environments, supported by multiple storage vendors. • An extension of the DMTF (Distributed Management Task Force) Redfish specification. • Redfish is designed by the DMTF’s Scalable Platforms Management Forum (SPMF) to create and publish an open industry standard specification and schema for management of scalable platform hardware. It is a RESTful interface over HTTPS in JSON format based on OData v4.

13.Profiles: Definitions Data proection profile properties: - DataProrectionLoS Replication profile properties: - RecoveryGeographicObject - DataProrectionLoS - RecoveryPointObjective - RecoveryGeographicObjective - RecoveryTimeObjective - RecoveryPointObjective - ReplicaTypes - RecoveryTimeObjective Provisioning profile properties: - ConsistencyEnabled - ReplicaType - DataStorageLoS - ReplicaInfos - RecoveryTimeObjective - ReplicaUpdateMode - ProvisioningPolicy - ConsistencyEnabled - IsSpaceEfficient - ReplicationPeriod - IOConnectivityLoS - ReplicationBandwidth - AccessProtocol - HostBasedReplication - MaxIOPs - DiskDrain (how to handle the - MaxBWs Profile ordering of dependent write - Provisioning profile properties requests) - Replication profile properties - Snapshot policies - ReadBalancing - Custom properties - ResyncRate Snapshot profile properties: - Fencing (avoid split-brain) - Schedule - Date Migration profile properties: - Time - Schedule - Occurrence - Date (daily/weekly/monthly) - Time Custom profile property examples: - Retention - Occurrence - DiskType - By number of snapshots - Latency - Rules - By duration to retain a - Deduplication - Define what to migrate snapshot - PreConditions - Compression - …… - Specify in which condition to trigger a rule

14.Profiles: Mapping Profiles to Capabilities Profile - Provisioning profile properties - Replication profile properties - Snapshot profile properties - Custom properties Capabilities Storage backend 1 Storage backend 1 Capabilities Storage backend 2 Storage backend 2 Selector Capabilities Storage backend 3 Storage backend 3

15.Provision: StorageClass with Profile Parameter

16.Provision: Running OpenSDS CSI Plugin • Create OpenSDS CSI plugin pods: kubectl create -f csi/server/deploy/kubernetes • Three pods can be found by kubectl get pod: Attacher Plugin Pod CSI OpenSDS Attacher CSI Plugin Node Plugin Pod Driver OpenSDS Registrar CSI Plugin Provisioner Plugin Pod CSI OpenSDS Provisioner CSI Plugin

17.Provision: Using OpenSDS Volume • Create nginx application kubectl create -f csi/server/examples/kubernetes/nginx.yaml • An OpenSDS volume is mounted at /var/lib/www/html. docker exec -it <nginx container id> /bin/bash

18.Disaster Recovery: Replication Profile • RecoveryTimeObjective • RecoveryPointObjective • RecoveryGeographicObjective • ReplicaType – Mirror • ReplicationUpdateMode – Sync, Async, Active, Adaptive • ConsistencyEnabled • ReplicationPeriod • ReplicationBandwidth

19.Disaster Recovery: Replication Example ReplicationPVC.yaml apiVersion: v1 ReplicationSC.yaml kind: PersistentVolumeClaim apiVersion: storage.k8s.io/v1 metadata: kind: StorageClass name: opensds-csi-replication-pvc metadata: spec: name: opensds-csi-replication-sc accessModes: provisioner: csi-opensdsplugin - ReadWriteOnce parameters: resources: profile: replication-profile requests: enableReplication: “true” storage: 10Gi storageClassName: opensds-csi-replication-sc

20.Disaster Recovery: Array-based Replication • Create source and target volume • Create replication on the arrays

21.Disaster Recovery: Host-based Replication Source: http://docs.linbit.com/docs/users-guide-9.0/ • Create source and target volume • Attach volumes • Create replication on the hosts

22.Disaster Recovery: Replication Functions Create Replication: osdsctl replication create <primary volume id> <secondary volume id> [flags] Flags: -d, --description string the description of created replication -h, --help help for create -n, --name string the name of created replication -p, --primary_driver_data string the primary replication driver data of created replication -m, --replication_mode string the replication mode of created replication, value can be sync/async -t, --replication_period int the replication period of created replication, the value must be greater than 0 -s, --secondary_driver_data string the secondary replication driver data of created replication Enable Replication: Disable Replication: osdsctl replication enable <replication id> osdsctl replication disable <replication id> Failover Replication: osdsctl replication failover <replication id> [flags] Flags: -a, --allow_attached_volume whether allow attached volume when failing over replication -h, --help help for failover -s, --secondary_backend_id string the secondary backend id of failover replication

23.Data Protection: Snapshot Profile • Schedule – Date – Time – Occurrence (daily/weekly/monthly) • Retention – By number of snapshots – By duration to retain a snapshot • Topology – Where to upload snapshot 23

24.Data Protection: Periodic Snapshotting • Take snapshots periodically based on snapshot profile • Upload snapshots to object store on-premise or in the cloud

25.Data Protection: Restore • Create volume from snapshot • Restore volume from backup • Provision PV and bind with PVC

26.OpenSDS Roadmap v0.14

27.Governance Technical Steering Committee End-User Advisory Committee

28.OpenSDS Community

29.Demo • Array-based replication: Failover storage provisioned by OpenSDS CSI plugin