Milvus 数据管理

大规模场景下的向量数据管理

展开查看详情

1.Milvus Data Management Yihua

2.General view © 2020 Zilliz. All rights reserved.

3.Hierarchy © 2020 Zilliz. All rights reserved.

4.Meta © 2020 Zilliz. All rights reserved.

5.Insert © 2020 Zilliz. All rights reserved.

6.Merge Why? Avoid large number of small files Query performance How? 1. Create new segment, combine data in memory 2. Persist new segment to storage 3. Change meta: new record for new segment mark old segments as ‘delete’ within one transaction 4. Remove old segments from storage © 2020 Zilliz. All rights reserved.

7.Merge Write amplification Optimize © 2020 Zilliz. All rights reserved.

8.Query © 2020 Zilliz. All rights reserved.

9.Query queue Sequence Queue: Combined Queue: © 2020 Zilliz. All rights reserved.

10.Combine Conditions Conditions: • Same collection • Same partition • Combine NQ < 200 • Topk distinct < 200 • Same nprobe(for IVF index) © 2020 Zilliz. All rights reserved.

11.Q&A © 2020 Zilliz. All rights reserved.