展开查看详情
1 . 从 到
如何应对资源管理及作业调度的挑战
杨巍威
4 .“Kubernetes is a portable, extensible, open-source
platform for managing containerized workloads and
services, that facilitates both declarative configuration
and automation.”
●
●
●
●
6 .“CDP delivers powerful self-service analytics across hybrid and multi-cloud
environments, along with sophisticated and granular security and governance policies
that IT and data leaders demand.”
“Open Data Hub is a blueprint for building an AI as a service platform on Red Hat's
Kubernetes-based OpenShift® Container Platform and Ceph Object Storage. It inherits
from upstream efforts such as Kafka/Strimzi and Kubeflow, and is the foundation for
Red Hat's internal data science and AI platform.“
“The launch of Cloud Dataproc on Kubernetes is significant in that it provides
customers with a single control plane for deploying and managing Apache Spark jobs
on Google Kubernetes Engine in both public cloud and on-premises environments.”
7 .Separate Compute Resource Job Scheduling
and Storage Management
8 .Apache YuniKorn (Incubating) is a light-weight,
universal resource scheduler for container
orchestrator systems. It provides fine-grained
resource sharing for various workloads efficiently
on a large scale, multi-tenant, and cloud-native
environment.
10 .DECOUPLED An abstraction of
scheduler-interface to decouple
the scheduler-core with the
underneath platforms
SCHEDULING Built-in with
advanced scheduling capabilities
to support both batch and
long-running workloads.
CLOUD-NATIVE Highly
extendable, scalable, natively
works on-prem and cloud.
11 .Feature Default YUNIKORN Note
Scheduler
Scheduling at app App is the 1st class citizen in YuniKorn, YuniKorn schedules apps with respect
dimension to, e,g their submission order, priority, resource usage, etc.
Job ordering YuniKorn supports FIFO/FAIR/Priority (WIP) job ordering policies
Fine-grained resource Manage cluster resources with hierarchy queues, queue provides the
capacity management guaranteed resources (min) and the resource quota (max).
Resource fairness Inter-queue resource fairness
Natively support Big Data The default scheduler is main for long-running services. YuniKorn is designed
workloads for Big Data app workloads, it natively supports Spark/Flink/Tensorflow, etc.
Scale & Performance YuniKorn is optimized for performance, it is suitable for high throughput and
large scale environments.
12 . master
ETCD
Resource
Scheduler
Apps 1
API
Server
2 3
Kubelet
Default
Scheduler YUNIKORN
Queues
Sort
Filter Apps
Score Request Queue
Sort
Extensions
Nodes App
Sort
Node
Sort
Pluggable
Policies
14 . Compute
Pool
Manager
Central
Resource
Compute Pool
Manager
Storage
Job YUNIKORN
API
Server
Storage
Control Plane
Metadata Security Governance
15 .Schedule 50,000 pods on 50k pods on 2k nodes 50k pods on 4k nodes
2,000/4,000 nodes.
Compare Scheduling throughput
(Pods per second allocated by
scheduler)
Red line (YuniKorn)
Green line (Default Scheduler)
↑
↑
Detail report:
https://github.com/apache/incubator-yunikorn-core/blob/master/docs/evaluate-perf-function-with-Kubemark.md
17 .THANKS!
如果对 YUNICORN 技术感兴趣请扫描入群
或者发邮件至 dev@yunikorn.apache.org
热门城市
