使用 Rook 在 Kubernetes 之上自动部署 Ceph 集群——Dennis Chen，Arm

下载 1

ccone

发布于

2304

人观看

#信息技术

Rook 是由 CNCF 主持、用以编制带有云原生基元的分布式存储系统（如 Ceph）的开源项目。在本介绍中，Dennis 将首先介绍 Rook 的整体设计思想及其工作原理，然后针对 AArch64 服务器上的实际工作负荷，探讨如何借助 Rook 自动部署 Ceph 集群以及如何利用 Rook，包括容量提供和附件。最后将讨论本项目的一些新特性（如 CSI 支持）、挑战和问题。

展开查看详情

2 .Agenda • Overview of Design • Operator PoD • Agent and Discover PoD • Ceph cluster deployment with Rook • Flex Volume Driver -- rookflex • Ceph RBD volume operations • Status update on AArch64 2 © 2017 Arm Limited

3 .Overview of Design kubectl Kubernetes Rook Operator API kubelet Rook Agent Flexvolume Flexvolume plugin Driver etcd UDS Node 0 § Rook cluster is on top of k8s cluster § 3 key elements of Rook: Operator, Agent Agent Agent Agent and Discover Discover Discover Discover § Now Rook is based on out-of-tree . .. Flexvolume machenism § Rook doesn’t change the data plane of the Ceph cluster Node 1 Node 2 Node n 3 © 2017 Arm Limited

4 .Operator PoD • A simple container functions at cluster level bootstrap and monitor the storage cluster • `kubectl create –f operator.yaml` to create the Rook Operator PoD. • Rook Operator PoD is created through k8s’ Deployment obj with `replicas: 1` (only one instance running in the cluster), it’s mechanism of k8s not Rook’s. • The arguments to startup the operator pod container- args: ["ceph", "operator"], triggering the `/usr/local/bin/rook ceph operator` command line inside container. • So RunE = startOperator() is invoked to bootstrap the `Agent` and `Discover` PoDs in form of DaemonSet. 4 © 2017 Arm Limited

5 .Operator PoD • startOperator() workflow Create startOperator() Create Agent DS Create Discover DS ProvisionController & ClusterController Watch the cluster’s resource changes Agent Agent Agent • onAdd() Discover Discover . . . Discover • onUpdate() • onDelete() node 0 node 1 node n Cluster *Operator includes the provisioning component: ProvisionController 5 © 2017 Arm Limited

6 .Agent and Discover PoDs • A mini operator that functions at the node level • Agent PoD startups with args: “ceph agent” -- `/usr/local/bin/rook ceph agent` command line inside container, will call RunE=startAgent() • Install the Rook Flexvolume driver Node Agent PoD kubelet to the volume-plugin-dir directory on every Flexvolume server Flexvolume plugin node RPC RPC rook flexvolume • Perform storage operations on behalf Driver UDS of the Flexvolume driver, such as attaching/detaching, mount/unmount via UDS • Discover PoD is used to discover the available devices on the node periodically 6 © 2017 Arm Limited

7 .Ceph cluster deployment with Rook • Rook clusterController is watching the resources changes within the cluster, so: • `kubectl create –f cluster.yaml` will trigger the Ceph cluster bootstrap workflow. • Rook configures the PoD spec of the `mon` and launch the PoD with `makeDeployment()` • Launch the ceph `mgr` and `osd` PoD with similar method. • Finally we’ll get a Ceph cluster like this(on a 2-node cluster): 7 © 2017 Arm Limited

8 .Flex Volume Driver -- rookflex • `rookflex` exists in form of a binary file and has been deployed into volume-plugin-dir by Rook Agent on each node. • `rookflex` implements ‘mount’ and ‘umount’ methods required by FlexVolume Spec • For a specific YAML file of a workload, the storage related part looks like: Storage Provisioning Storage Consuming 8 © 2017 Arm Limited

9 .Flex Volume Driver -- rookflex • When that workload PoD is scheduled to one node and begin to run, the kubelet will interacts with the driver to mount the volume into the `mountPath` in the YAML. To do so, kubelet needs to: 1. Lookup the right Flexvolume driver. The look up flow is: PVC name à StorageClass à provisioner name: ceph.rook.io/block à Flex volume vendor name: "ceph.rook.io“ à figure out the driver folder and driver name: rookflex 2. Call `mount` method of rookflex like: `$(volume-plugin-dir)/rookflex mount` 3. The above `mount` will call the corresponding function in Rook Agent via UDS. 4. Local Rook Agent will attach the volume into its node(a ‘rbd map’ operation). 9 © 2017 Arm Limited

10 .Ceph RBD volume operations kubectl create –f workload.yaml Node 0 Node n Workload Agent PoD kubelet kubectl create –f my-pvc.yaml Operator PoD PoD Flexvolume /var/www/html server mount rook flexvolume /dev/rbd0 Driver Mount UDS 1. Provisioning part. createVolume Attach GetPV `rbd create` a volume in Ceph cluster. 2. Attach and Mount part. `rbd map` the volume to a specified volume node as a block device then mount to the dir path in workload pod. PV Cluster 10 © 2017 Arm Limited

11 .Status Update on AArch64 • Participating the Rook community (https://github.com/rook/rook) actively • 14 patches from Arm have been merged into the upstream mainline code so far • Rook can be built and run smoothly on AArch64 now • CSI support in the future (csi volume driver, csi provision controller, etc) 11 © 2017 Arm Limited

3点赞

0收藏

1下载