戴建辉-Intel® CS for WebRTC 媒体服务器架构设计与实践

1 .Intel® CS for WebRTC 

2 .第七届 全球软件案例研究峰会 AI/ AIOps DevOps AI 2018 11 30 -12 3 | 100+ 

3 . 上海精品公开课 管理3.0认证课程 时间：12月22-23日 | 地点：上海 | 讲师：林伟丹 大数据及AI挖掘技术 时间：12月22-23日 | 地点：上海 | 讲师：风清扬 高可用架构与设计 时间：01月12-13日 | 地点：上海 | 讲师：沈老师 K8S与service mesh 时间：01月12-13日 | 地点：上海 | 讲师：Jim 备注：扫码查看课程详情，两人以上报名有优惠，详情咨询：15802217295 

4 .Agenda • What is WebRTC? • API • Architecture • Multi-party • Use cases • Intel® CS for WebRTC • Intel® Media Server Studio • OpenVINO™ toolkit 

5 .What is WebRTC? WebRTC is a free, open project that provides browsers and mobile applications with Real-Time Communications (RTC) capabilities via simple APIs. The WebRTC initiative is a project supported by Google, Mozilla and Opera, amongst others. WebRTC 1.0 W3C Candidate Recommendation. 

6 .WebRTC APIs WebRTC implements three APIs: • MediaStream (aka getUserMedia) • RTCPeerConnection • RTCDataChannel Safari Edge 

7 .WebRTC Architecture Video • VP8/VP9/H264/AV1 • Simulcast/SVC • Image enhancements Audio • iSAC/iLBC/Opus • Acoustic Echo Canceler (AEC) • NetEQ Transport • ICE/STUN/TURN • RTP/RTCP/SRTP/SRTCP • DTLS/SCTP • QUIC/UDP Security 

8 .P2P sample 

9 .Mesh 

10 .SFU (Selective Forwarding Unit) 

11 .MCU (Multipoint Control Unit) 

12 .Comparison WebRTC RTMP HLS Developed by Google Adobe Apple Proprietary Standardized Open standard protocol Open standard Supported All mainstream Flash Safari Browsers browsers Transport RTP/UDP TCP HTTP Interactive YES NO NO Latency ~200ms ~3s ~10s Scalability 2~8 CDN CDN 

13 .Multiple protocols collaboration 

14 .WebRTC • • • • • ( ) • • • • • • WebRTC+AI • WebRTC+AR/VR 

15 . WebRTC ® Intel® Collaboration Suite for WebRTC RESTful Control Control Protocols E5/ E3 Client SDK Server Capability System Framework Server JavaScript Wide Deployment Deployment Targets Streaming Servers Transport Protocols Streaming Conferencing Desktop/ Android Large Scalability NUC iOS Media Server Virtual High Availability Machine Windows Transcoding Analytics High Efficiency Container Linux MSS FFMPEG OpenVINO Strong Validation Customized WebRTC Enterprise Intel®Collaboration Suite for WebRTC Gateway • Mesh/SFU/MCU • Hardware accelerated video codec • VP8/VP9/H264/HEVC • WebRTC/SIP/RTSP/RTMP/HLS • AI+ 

16 .Connections with Different Protocols Intel® CS for WebRTC / Server / C 

17 .Distributed Deployment Signaling Servers C Access Node Media Servers Access Node (WebRTC) Access Endpoints (WebRTC) Audio Servers Access Node Access Node (Streaming) (Streaming) Access Node (Others) Ø Docker based distributed deployment Ø High availability Ø Large scalability 

18 . Multiple video process models • Forwarding • Transcoding am 1 (T) stre • Trans-codec Intel® CS Server (F) streams 1,2,3 • Trans-resolution (M ) • Trans-rate stre ams 1+ 2 (S • Mixing D) (M (V • Customized Layout )s A) str stream 1 tre am ea • Multiple views m s1 4 stream 2 stream 3 +2 +3 • Video Analytics stream 4 (H D) • Boundary box C • Event 

19 . Video Mixing Layout Configuration 1920x1080 30 FPS HEVC Rate Scaler Encoder Decoder Converter 30 FPS MixedFrame 1280x720 15 FPS H264 Decoder Generator Rate Scaler Encoder Converter Composer Decoder 1280x720 24 FPS VP8 48 FPS Rate MixedFrame Scaler Encoder Decoder Generator Converter VP9 Encoder • Reusable modularity design supports both mixing and transcoding pipeline • Hybrid codec processing with multiple HW and SW codec engine units • High efficient zero-frame-copy through whole video pipeline • Up to 3+ times performance boost on IA Graphics comparing with SW pipeline 

20 .Intel® Media Server Studio for Linux* Deliver Fast, Efficient, High Quality Video Transcoding 

21 .Video Analytics Decoder Overlay Encoder Boundary Box Info Event Resize/CSC Inference GPU FPGA CPU VPU 

22 . OpenVINO™ toolkit (Open Visual Inference & Neural Network Optimization) Intel® Deep Learning Deployment Toolkit Traditional Computer Vision Tools & Libraries Optimized Libraries Model Optimizer Inference Engine Photography Convert & Optimize IR Optimized Inference OpenCV* OpenVX* Vision 20+ Pre-trained Computer Vision Code Samples Samples Models Algorithms For Intel® CPU & CPU with integrated graphics IR = Intermediate Representation file Increase Media/Video/Graphics Performance Intel® Media SDK OpenCL™ Open Source version Drivers & Runtimes For CPU with integrated graphics Optimize Intel® FPGA FPGA RunTime Environment Bitstreams (from Intel® FPGA SDK for OpenCL™) FPGA – Linux* only OS Support CentOS* 7.4 (64 bit) Ubuntu* 16.04.3 LTS (64 bit) Microsoft Windows* 10 (64 bit) Yocto Project* version Poky Jethro v2.0.3 (64 bit) Intel® Architecture-Based Platforms Support 

23 . Intel® Deep Learning Deployment Toolkit Take Full Advantage of the Power of Intel® Architecture Model Optimizer Inference Engine § What it is: Preparation step -> imports trained models § What it is: High-level inference API § Why important: Optimizes for performance/space with § Why important: Interface is implemented as dynamically loaded conservative topology transformations; biggest boost is from plugins for each hardware type. Delivers best performance for conversion to data types matching hardware. each type without requiring users to implement and maintain multiple code pathways. Trained Extendibility C++ Model CPU Plugin Convert & optimize to Caffe* fit all targets Extendibility OpenCL™ GPU Plugin TensorFlow* Inference Engine Model Common API (C++ / Extendibility OpenCL/TBD MxNet* Optimizer IR IR .data Python) FPGA Plugin Convert & Load, infer Optimized cross- Extendibility TBD ONNX* Optimize platform inference Myriad Plugin IR = Intermediate Kaldi* Representation format Extendibility TBD GNA Plugin GPU = Intel CPU with integrated graphics processing unit/Intel® Processor Graphics 

24 .Resources OpenVINO™ toolkit Intel® CS for WebRTC Free Download } Free Download } Free Download } software.intel.com/en-us/ software.intel.com/en-us/ software.intel.com/en-us/ intel-media-server-studio webrtc-sdk openvino-toolkit 

25 .Reference • https://webrtc.org/ • https://webrtc.org/architecture/ • https://www.w3.org/TR/webrtc/ • https://www.html5rocks.com/en/tutorials/webrtc/basics/ • https://www.html5rocks.com/en/tutorials/webrtc/infrastructure/ • http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/The-State-of- WebRTC-and-Streaming-Media-2018-124068.aspx • https://software.intel.com/en-us/webrtc-sdk • https://docbox.etsi.org/Workshop/2015/201510_STQWORKSHOP/S02_SPEECH_AU DIO_VIDEO_QUAL_2_INSTRUMENTAL_TESTING/BLACK_BOX_DELAY_TESTING_WEBR TC_MELVIN_UNIV_IRELAND.pdf 

26 .Legal Disclaimer & Optimization Notice Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. The cost reduction scenarios described in this document are intended to enable you to get a better understanding of how the purchase of a given Intel product, combined with a number of situation-specific variables, might affect your future cost and savings. Nothing in this document should be interpreted as either a promise of or contract for a given level of costs. Intel, the Intel logo, Xeon, Core, Iris Pro, and VTune are trademarks of Intel Corporation in the U.S. and other countries. OpenCL and theNotice Optimization OpenCL logo are trademarks of Apple Inc. used by permission by Khronos. Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 

27 .