- 微博 QQ QQ空间 贴吧
- 视频嵌入链接 文档嵌入链接
Alluxio2.5 + Bring data locality to ML and AI workload
1 .What’s new in Alluxio 2.5 Alluxio Day II 2021/03/11- Jiacheng Liu
2 .About me ● Jiacheng Liu ● Master of CS @ Columbia University ● Software Engineer @ Alluxio ● Core maintainer
4 .Feature List Community Enterprise ● JNI Based POSIX API ● AWS STS Support ● S3 Northbound API v2 ● Hybrid Quickstart with Alluxio Hub ● ADLS Gen 2 UFS Connector ● Compatibility with Ranger 2.0 ● Native GCS Connector* ● Remote Logging in K8s Environments
5 .JNI Based POSIX API Description Alluxio 2.5 introduces a new JNI-based FUSE integration to support POSIX data access. This new JNI-based FUSE integration improves the performance by 3x to 5x for workloads of high-performance and high-concurrency such as AI/ML training. Speciﬁcation Envs - Community Testing - K8s Scale - Community Testing - up to 200 nodes. Multi-Tenancy - Single user support. Workﬂows - Community Testing - AI/ML workloads. Conceptually - Traditional POSIX workloads. Impact Fundamental module for supporting POSIX workloads going forward. Success with community will lead to reference architecture and product deﬁnition to expand the use cases. Expect this in 2.6 development cycle.
6 .S3 Northbound API v2 Description Support for S3Browser to browse the Alluxio namespace through the S3 Northbound API. Speciﬁcation Scale - Administrative QPS (< 100 / s). Compatibility - Supports select subset of S3 interface to enable basic browsing, modiﬁcation, and downloading/uploading capabilities. Multi-Tenancy/Authentication - Multi-user, Simple (server trusts client) Impact S3Browser is an administrative tool, especially for Windows platforms. Availability of a REST management interface greatly improves Alluxio compatibility.
7 .ADLS Gen 2/GCS Connector Description Alluxio has a new connector to ADLS Gen 2, and an updated connector for GCS. Speciﬁcation Environments - ADLS Gen 2, GCS Impact Users can run Alluxio with ADLS Gen 2. GCS is also better supported because the native SDK has more optimizations and the latest features. One important feature for GCS is support for JSON based authentication.
8 .Remote Logging in K8s Environments Description Alluxio 2.5 supports remote log server in K8s environment. One challenge users have in a containerized environment is the logs getting disposed or overwritten when a container is killed or restarted. With the remote logger, the logs will be sent to a centralized location (a dedicated pod). Speciﬁcation Environments - Cloud/On-Prem K8s Impact Enables us to effectively gather logs in K8s environments.
9 .AWS STS Support Description Alluxio supports connecting to S3 using the secure token service as opposed to traditional authentication methods such as specifying access key and secret key. STS is AWS’s recommended authentication paradigm and has beneﬁts such as all credentials are temporary, cross account bucket sharing, and ﬁne grained privilege control. Speciﬁcation Environments - AWS S3 as under storage Impact This is a common requirement for user environments which are AWS cloud native and already use STS.
10 .Hybrid Quickstart with Alluxio Hub - 2.4 ● Alluxio Hub was added in Alluxio 2.4 ● Management console for Alluxio clusters ● Wizards for connecting to external storage and validating integrations ● System monitoring ● Conﬁguration management
11 .Hybrid Quickstart with Alluxio Hub - 2.5 Description Hub is now supported on Kubernetes to aid cluster conﬁguration and connectivity across private data centers or public clouds. AWS users now also have access to a quickstart using Terraform to deploy an Alluxio cluster with Amazon EMR in minutes. Once an Alluxio cluster is deployed, either using the new Terraform or helm on Kubernetes, the Hub is available to manage subsequent changes. Speciﬁcation Environments - Cloud PaaS (ie. AWS EC2) with K8s, On-prem K8s, AWS EMR
12 .Compatibility with Ranger 2.0 Description Alluxio 2.5 supports integration with Ranger 2.0 for third party authorization. Ranger 2.0 includes ﬁner grained security policies and is the default for CDP 7. Speciﬁcation Environments - Ranger 2.0 Impact Customers on CDH 7 will be able to integrate with Ranger 2.0 and provide access to data which is guarded by Ranger.
14 .Metrics We added a few key metrics to show Alluxio’s value and troubleshoot system limitations. Expect a more comprehensive metrics/instrumentation story in 2.6. ● Better Web UI metric graphs ● Alluxio data cache hit rate ● Alluxio metadata cost savings ● Lock utilization (troubleshoot out of locks issue) ● Inode cache effectiveness (troubleshoot RocksDB)
15 .Logging / Recovery We’ve made several improvements to the cycle of informing errors, attempting to gracefully recover, and cold restarts. ● Make error messages more user friendly (10+ changes, constant effort) ● Allow journal replay without ZK connection (reduces chance of master restarts failing due to GC) ● Improve tiered storage out of space handling
16 .Welcome to Join Alluxio Community! Alluxio DingTalk Group