Milvus Community Conf 2020 开场演讲_The Journey of Milvus

Zilliz 的创始人兼 CEO 星爵与 Milvus 核心研发团队出席了本场大会,与大家分享 Milvus 产品演进路线图,共同回顾 Milvus 开源一周年以来的的风风雨雨,畅想未来发展蓝图。
Zilliz 金海老师的分享主要从我们面临的搜索挑战、为什么选择 Milvus 以及 Milvus 的最新版本三个方向出发,为参会观众更加细致的解读 Milvus 的功能。在过去的一年中, Milvus 一共经历了 11 次大版本更新,每一次都带来了全新的功能。

展开查看详情

1.The journey of Milvus 金海 Zilliz

2.Speaker bio 金海 R&D Director Project Leader, Committer Career history Education © 2020 Milvus. All rights reserved.

3.Why Milvus What do we have? • Faiss, HNSW, Annoy, LSH, NGT, SPTAG, ScaNN …… What do we need? Vector Search Engine / Vector DBMS • Supporting different indexes • Supporting large-scale dataset and different datasets • Supporting CRUD operations • Utilizing Computation resources efficiently • Scalability, High availability • …… © 2020 Milvus. All rights reserved.

4.Challenges – Data Management Training data {0.32, 0.21, 0.23, ...... 0.51} Faiss - IVF {0.52, 0.11, 0.72, ...... 0.60} Flat / SQ / PQ {. . . . . ..... . .} {0.91, 0.33, 0.54, ...... 0.13} Centroids …… Vector data {0.32, 0.21, 0.23, ...... 0.51} {0.52, 0.11, 0.72, ...... 0.60} {. . . . . ..... . .} {0.91, 0.33, 0.54, ...... 0.13} …… Clusters Memor y © 2020 Milvus. All rights reserved.

5.Challenges – Data Management {0.32, 0.21, 0.23, ...... 0.51} {0.52, 0.11, 0.72, ...... 0.60} {. . . . . ..... . .} Segment {0.91, 0.33, 0.54, ...... 0.13} Segment Segment Segment Segment Segment Segment with Index Segment Collection <<Milvus在大规模向量检索场景下的数据管理>> © 2020 Milvus. All rights reserved.

6.Challenges – Multiple GPU support Faiss - IndexReplicas Faiss - IndexShards GPU0 GPU1 GPU0 GPU1 © 2020 Milvus. All rights reserved.

7.Challenges – Multiple-GPU support Segment0 Segment1 Segment2 Segment3 11GB/ s Segment4 Segment5 Segment3 Segment0 Segment4 Segment1 Segment2 Segment5 CPU Memory GPU0 memory GPU1 Memory <<Milvus 向量搜索引擎:查询任务调度原理>> © 2020 Milvus. All rights reserved.

8.Challenges – CPU-GPU co- processing Faiss – IVF Query Index {0.32, 0.21, 0.23, ...... 0.51} Step1 Centroids …… Query {0.32, 0.21, 0.23, ...... 0.51} Step2 …… Clusters © 2020 Milvus. All rights reserved.

9.Challenges – CPU-GPU co- processing Faiss – IVF Index IVF-SQ8-Hybrid nlist 2x-5x speed up! nlist * dimension * 4 Centroids …… GPU bytes …… CPU Clusters © 2020 Milvus. All rights reserved.

10.Challenges – Entity Deletion Faiss – IVF Index Centroids …… …… Clusters © 2020 Milvus. All rights reserved.

11.Challenges – Entity Deletion Milvus – IVF Index Centroids …… bitmap bitmap bitmap bitmap …… Clusters <<Milvus数据删除的实现原理>> © 2020 Milvus. All rights reserved.

12.Milvus Releases v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 C++ Implemented Supports IP metric Milvus CPU-only Implementation Adds Multiple-GPU Supports IVF-SQ8H Supports data loading Supports IVF SQ Supports IVF-PQ based on Faiss support index and computation index Index Python APIs Adds Java SDK pipeline Mishards open sourced v0.7 v0.8 v0.9 v0.10 v0.11 Supports entity Supports Annoy Index Supports selecting CPU Query request merge Scalar Field Filtering deletion instruction set during Supports large topk Meta Snapshot runtime search © 2020 Milvus. All rights reserved.

13.0.11.0 – Scalar Field Filtering Milvus DSL AST AST Parser Validator Optimizer Listener Plan Request Query Client DSL Context Executor Connection Connection Connection Resul Context Context Response Context t Resul Data t Cache Data Data on disk © 2020 Milvus. All rights reserved.

14.0.11.0 – Meta Snapshot New Merg Q1 Req Q2 Req Q3 Req Q1 Rep Q2 Rep Q3 Rep Segment e Timeline Collection A Snapshots Segment: S1 Versio Q1 Segments n Latest snapshot Segment: S2 Q1 v1 S1, S2 Latest snapshot Q2 Q2 v2 S1, S2, S3 Segment: S3 Latest snapshot Q3 v3 S3, S4 Q3 Segment: S4 © 2020 Milvus. All rights reserved.

15.Next Step • Milvus Technical Steering Committee • How to contribute • Milvus github Discussions (https://github.com/milvus-io/milvus/discussions) © 2020 Milvus. All rights reserved.

16.Fin Speaker Contact WeChat: 13917405881 Mail: hai.jin@zilliz.com