申请试用
HOT
登录
注册
 
3 Milvus——开源向量搜索引擎-顾钧
0 点赞
0 收藏
9下载
AI 检索技术社区
/
发布于
/
173
人观看

议题简介:

随着深度学习技术的成熟,人们尝试利用AI技术挖掘非结构化数据(图片,视频,自然语言文本等)中潜藏的价值。由此,人们对特征向量数据的分析处理需求大幅增长。然而通过现有的数据库组件和大数据技术来支撑这样的新型应用场景,却面临开发困难、运行成本高昂的挑战。
为了帮助克服现有技术的局限性,我们发起了 Milvus 开源向量数据库项目。作为一个开源AI基础组件,Milvus 加快了企业开发AI应用的速度、大幅降了AI应用的部署成本。

嘉宾简介:

顾钧,Zilliz高级架构师&合伙人
北大毕业16年以来始终专注于数据库、大数据技术,尤其对OLTP平台与场景有着丰富的经验。顾钧现后任职于工商银行,IBM,摩根士丹利,华为等企业。加入Zilliz以后,顾钧的工作重心在于开源社区的构建与推广。同时,顾钧代表Zilliz出席LF AI & Data基金会中的技术咨询委员会。

展开查看详情

1.Milvus 开源向量搜索引擎 顾钧

2.Speaker bio Jun Gu Database engineer, SME Voting member in Technical Advisory Council Partner, Chief Evangelist

3.About Zilliz • Open-source software company based in Shanghai • Mission: Reinvent Data Science • Main contributor of Milvus project

4.The era of Software 2.0 Unstructured Deep learning Embedding Knowledge, data models vectors insight, $

5.How Milvus supports Software 2.0 Model Service Model Service Training, AutoML, Inference, etc Milvus Vector Database Data Service ANNS, Streaming, Scalability, etc Vector Vector Raw Data Persistence Index Data Data

6.Vectors are different Numbers Vectors Arithmetic operation Similarity (eg. Euclidean distance) Operation Number comparison Similarity comparison 1–10 1–5 6–10 Organization 1 2 3 4 5 6 7 8 9 10

7.Milvus v1.0 single node Query Processing Engine Buffer Pool Scheduler ANNS Collaborative Query Mi-FAISS, Mi-Annoy tag/structured data Index SDK / Web API Result Files top-K result Reducer Multi-modal Scoring app specific query Segment Segment Metadata obj Selection insert obj X86: supports SSE4.2, AVX2, AVX512 GPU: Pascal microarchitecture or later, CUDA 10.0 or later x86 ARM GPU New Index Arm: requires aarch64 Index Kunpeng: tested on Kunpen 920 with Centos 7.x Files Loongson: tested on Loongson with docker File container Kunpeng Loongson RISC-V RSIC-V: in early development Various Processors Storage Tier

8.Milvus v1.0 scalability Proxy / Service Router Query Query Query node�1 node�N …… node�N Writer Cloud Persistent Storage (S3, NFS, etc)

9.Milvus future scalability Proxy / Service Router Query Service Query Query node�1 …… node�N Data Persistence Data Data node�1 …… node�N Cloud Persistent Storage (S3, MinIO, etc)

10.The proprietary software ecosystem 3rd party The software vendor vendor Professionals Users

11.Align the interests of different participants The OSS project The community •Contributors •Professionals •3rd parties The users

12.Supportive and inclusive community WG 1 SIG Toolchain • Reputation comes from SIG Testing SIG Engine SIG ANNS SIG … contribution • Guidelines rather than policies WG 2 • Consensus rather than authority • Collaboration rather than solo TSC

13.Find the Milvus community Performance benchmark: https://milvus.io/docs/benchmarks_azure https://milvus.io Live demo: https://milvus.io/scenarios https://github.com/milvus-io/milvus • Content-based image retrieval system (以图搜图) https://twitter.com/milvusio • Q&A chatbot powered by NLP (智能客服机器人) • Molecular analysis (化合物分析) https://medium.com/unstructured-data-service https://zhuanlan.zhihu.com/ai-search Follow us on Wechat >>>>>

14.Thanks

0 点赞
0 收藏
9下载
相关文档