JStorm Based Network Analytics Platform

下载 0

圆圆

发布于

2101

人观看

#信息技术

This tutorial will introduce a JStorm based solution to analyze network in real-time. An application based on JStorm is built to consume huge amount of network metrics data as stream. To handle the complexity of analysis, an analytic modeling system is built to work with the JStorm application so that dimension or topology could be recognized on the fly. The analyzed stream data is then sent to batch-data-processing system for further analysis such as high-level statistics or OLAP.

展开查看详情

1 .JStorm Based Network Analytics Platform Alibaba Cloud Senior Technical Manager, Biao Lyu

2 . Overview of Alibaba Cloud 18 Regions 150+ Products 1Million+ Customers

3 . Comprehensive Networking Product Family 12 products for 5 scenarios Self-Defined Cloud Internet Access Saving BGP Cost Building Hybrid Cloud Global Connection Networking Environment SLB VPC Shared Flow Package Express Connect Cloud Enterprise Network NAT Gateway Shared Bandwidth Package Cloud Hosting Global Acceleration EIP VPN Gateway Smart Access Gateway From Data Center to Global Connection, for every enterprise networking scenarios

4 . Networking is getting complicated 25G 10W+ NIC Underlay VxLAN Neutron 5G 400G DCI Hybrid Cloud IoT vCPE Overlay SD-WAN vRouter RDMA NFV Vitrual Private Cloud Virtual switch NSH SDN Segment Routing Large Scale! High Throughput! Low Latency ! Rich Features!

5 . Challenges from Growing Scale Tenants Servers VMs 1,000,000,000 1,000,000,000 10,000,000,000 Multi-Dimension ! Random Failures! Daily Maintenance! Profiling!

6 . What we want to achieve Network as a Service Minimum Impact Precise Diagnose Network Update Network Issue Realtime Monitor Deep Understanding Network Failure Network Business

7 . Automation to Intelligence Network Update Tool Disaster Network Script Controller Monitor Recovery Network Config Network Network Diagnose Profiling Automation Intelligence

8 . QiTian - JStorm Based Network Analytic Platform Flow API WEB Robot Output Analytic Result Basic Monitor Diagnose Scheduling Profiling DB ETL Math Policy Event Notification Packet Route Connectivity LB Recovery User/Device Real-Time Analytic JStorm Config flow, metric, log, topology, device status, etc product, customer, instance, etc SDN Controller Network Devices Business Services Data Source

9 . QiTian - Data Sources & Types QiTian operation log, topology operation, infrastructure SDN Controller Operation Services Streaming Log, DB, message Log Streaming Streaming Log DB Streaming customer, product, message Streaming access, ddos, waf Business Services instance, income/cost Security Streaming Log Streaming Log, DB flow, metric, device status, access log, etc Streaming Log Overlay Network Device Underlay Network Device Gateway Router Switch Host VSwitch vBorder Device

10 . QiTian - High Performance Network Analytics Network Devices Stream Dispatch Streams Local Data ETL Local-Shuffling Merge Telemetry Local-Shuffling Fields-Grouping 100 Million 1 Million 100 Million Records/min Records/min Records/min 1 Million Records/min per Stream

11 . QiTian - High Performance Network Analytics Metrics TPS Data 500+ Metrics 2,000,000 100T/Day Tuples/Sec

12 . QiTian - Real-Time Network Monitor Network Devices Basic Steam Policy Steam Event Merge Event Notification Processing Processing Steam Steam Processing Processing metric vibration mail Merged Rich Meta Network Network Raw Data Event Data Event sms flow, metric, user, instance, predicted interval issue, failure log cluster app msg event statistics

13 . QiTian - Real-Time Network Monitor metric vibration Major Network Issue Detection abnormal event statistics 90% predicted interval

14 . QiTian - Streaming Based Network Diagnose Packet Analytic Config Packets Steam Network Devices Processing Packet Sort flow route Route Calculation Result Packets device latency Latency Calculation device loss rate Loss Rate Calculation Packets Collection Packets Processing Spout Bolt

15 . QiTian - Streaming Based Network Diagnose Packet Loss/Latency Diagnose Package Analyze Config Flow Route Packet Analyze

16 . QiTian - Intelligent Network Traffic Scheduling Traffic Scheduling Stream Network Devices Reliability Stream Processing Processing Topology Analyze Network Load Calculation Network Metric Event Cluster Selection issue, failure Device Selection Scheduling-Aware Traffic Scheduling Event Spout Processing Bolt SDN Controller Traffic Schedule Config

17 . QiTian - Multi-Dimension Network Profiling Business Services Business Data Rich ETL Steam Steam Processing Processing Biz Data Biz Data Biz Data Bolt Group 1 Stream Batch Processing DB In Memory ETL Traffic Pattern Rich Meta msg Data Hash Biz Data Bolt Group 2 Business Cost Biz Data Spout customer, Network Devices In Memory ETL instance, cluster User Profiling Telemetry Biz Data Bolt Group 2 In Memory ETL Telemetry Data Rich ETL Bolt Spout

18 .QiTian - Multi-Dimension Network Profiling Income Analyze by Industries Customer Resource Layout BGP Cost

19 . What’s Next Deployment Elastic Monitor real-time network deployment real-time network expansion new detection policy policy policy based on machine- learning integrated

20 .Thanks

1点赞

0收藏

0下载