基于Intel OpenVINO Model Server的推理部署及案例分享

1 .Mar 2021 赵朝卿 

2 . Legal notices & disclaimers This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Performance estimates were obtained prior to implementation of recent software patches and firmware updates intended to address exploits referred to as "Spectre" and "Meltdown." Implementation of these updates may make these results inapplicable to your device or system. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Intel, the Intel logo, Pentium, Celeron, Atom, Core, Xeon, Movidius, Saffron and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2019 Intel Corporation. 2 

3 .议题 1. OpenVINO™ Model Server介绍 2. OpenVINO™ Model Server模型部署 3. OpenVINO™ Model Server在贝壳搜房的应用 

4 .OpenVINO™ Model Server介绍 

5 .深度学习如何工作？ 人 自行车 前向 计算量 “草莓” 训练 ? “自行车” 大 反向 误差 草莓 模型权重 前向 小 推理 “自行车”? 5 ?????? 5 

6 .推理模型部署 研究 & 开发 生产环境 有了训练好的模型，部署才正式开始 在生产环境中产生实际价值 6 

7 .深度学习部署挑战 INTEL® DISTRIBUTION OF OPENVINO™ TOOLKIT 独特的推理需求 集成挑战 没有一刀切方法 训练和部署模型之间存在性能 没有简化的端到端开发工 对多种用例的不同要求需要独 和精度差异 作流程 特的方法 低性能，低精度的模型部署 解决方案慢，部署上线慢 不能满足用例特定的需求 7 

8 .AI算力思考 How do you determine the right computing for your AI needs? 工作负载 要求 需求 ASIC/ 1 FPGA/ Utilization GPU 2 CPU 3 Scale 工作负载配置是什么？ 用例需求是什么？ AI在我的环境中有多普遍？ 8 

9 . 英特尔®OpenVINO™工具包 高性能，深度学习推理工具套件 OpenVINO™是一个高性能，快速部署深度学习的工具套件，将更准确的真实结果部署到横跨英特尔®架 构从边缘到云的生产环境中 高性能深度学习推理 简化开发，易于使用 只写一次，随地部署 For public use – OK for non-NDA disclosure 9 

10 .英特尔®OpenVINO™工具包 简化深度学习部署的高级能力 1. 构建 2. 优化 3. 部署 训练模型 读取 装载 推理 CPU Plugin Model Optimizer IR 转换和优化训练模型 Inference Engine GPU Plugin 中间表示(.xml,.bin) 通用API，是对每一种 GNA Plugin 设备的低级语言的抽象 训练后优化 OpenCV OpenCL C C++ Python HDDL Plugin 工具 OpenVINO Model Zoo Code Samples & Demos FPGA Plugin 深度学习 100+开源和优化的预训练 工作台 e.g. 部署管理 benchmark_app 模型 Accuracy Checker 80+公开模型 Model Downloader 10 

11 .英特尔®AI硬件: Accelerate with Purpose CLOUD-to-EDGE CLOUD EDGE only only CPU + XPU DL Custom DL Training DL Inference For when your compute is or Inference dominated by DL Performance/Watt CPU + GPU DL & More For when your compute is dominated by DL, HPC, graphics & real-time media CPU only For when AI is a portion of your 1000+ workloads Workloads 

12 .AI推理工作流程 OpenVINO™简化推理服务部署流程 优化 测试 部署 扩展 Dev Cloud for the Edge Available today https://software.intel.com/content/www/cn/zh/develop/tools/devcloud.html 

13 . 英特尔®OpenVINO™ Model Server 一个英特尔开发的，开源的，工业产品级的模型服 务推理平台 OpenVINO™ Model Server • 通过便捷的推理API接口来部署一组模型 gRPC RESTful 是什么？ • 推理引擎库来自OpenVINO™工具套件 • 扩展工作负载到英特尔的各种硬件，如CPU，GPU，VPU等 13 

14 .英特尔®OpenVINO™ Model Server 客户端 Model Server 推理 API ?????? 14 

15 .英特尔®OpenVINO™ Model Server Model Server 客户端 API 客户端 客户端 推理 客户端 推理 客户端 推理 客户端 推理 Model Server 推理 推理 API ?????? ?????? ?????? Model Server ?????? ?????? API ?????? Model Server API 15 

16 .英特尔®OpenVINO™ Model Server 为什么不直接使用OpenVINO™工具套件? 标准网络 算力 API 托管 横向扩展 一句话：部署简单 负载均衡 K8S集成 原生云 16 

17 .英特尔®OpenVINO™ Model Server OpenVINO™ Model Server架构 客户端 gRPC endpoint REST endpoint 调度器 配置 模型 管理 管理 容器 OpenVINO 推理 引擎 OpenVINO CPU GPU FPGA HDDL ... 设备插件 OS 处理器 17 

18 .英特尔®OpenVINO™ Model Server 主要特性 本地/远程模型 模型版本策略 - 支持本地仓库，也可接远程仓库：GCS，S3，Mino和Azure Blob - 指定一个或多个版本 兼容TF Serving 模型Reshape - 客户端无需修改，可无缝迁移 -支持auto，单个固定shape，和多个固定shape列表 后端无关 Pipeline策略 - 只训练一次，到处运行，如CPU，Myriad， iGPU和HDDL - 可以将多个模型组成一个Pipeline，以整体对外提供推理服务 框架无关 在线配置更新 - 可部署多种框架训练的模型，如Tensorflow, PyTorch等 - 推理线程数，请求处理线程数，BatchSize和算力分配方式等 18 

19 .OpenVINO™ Model Server模型部署 

20 .英特尔®OpenVINO™ Model Server AI推理的部署流程 简单推理 高级部署 模型仓库 1 下载模型 1 配置文件 2 3 4 开始推理 启动容器 2 开始推理 启动容器 3 模型 5 控制 

21 .英特尔®OpenVINO™ Model Server 准备模型和配置 模型目录结构 IR模型目录结构 • <model>/<version#> 输入/输出节点名字可重新定义(可选) • 创建mapping_config.json mapping_config.json 原始名字 友好名字 

22 .英特尔®OpenVINO™ Model Server 部署多个模型 启动命令 model-config.json 模型目录结构 

23 .英特尔®OpenVINO™ Model Server 异构部署 model-config.json iGPU需要特殊的镜像 • openvino/model_server:latest-gpu 启动容时器设置适当的选项 • MYRIAD: 蓝色框 • iGPU：绿色框 Windows不能作为主机 • 没有/dev目录 GPU MYRIAD 

24 .英特尔®OpenVINO™ Model Server Pipeline推理服务部署 模型目录结构 准备模型 定义pipeline 启动服务 定义pipeline 请求服务 

25 .

26 .OpenVINO™ Model Server在贝壳找房的应用 

27 .贝壳AI应用场景 融合OpenVINO到生产环境 图像识别、检测 推荐 ASR OCR 视频合成 

28 .贝壳搜房应用场景 搜房AI的pipeline方案 

29 .模型优化案例 户型图分割 正常 分割 热力值