Sliwinski@cern.ch. Beams Department, Controls Group .... Today vital part of Beam Operations .... Middleware by Facebook (moved to Apache).

Hafenkranich发布于2011/01/26 00:10

注脚

1.The role of messaging solutions in complex systems, based on experience in the accelerator controls at cern JINR/CERN/MEPHI Computing school DUBNA , 26 th October 2011 Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

2.Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure Middleware Review for CERN Acc . Controls Conclusions 2 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

3.Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure Middleware Review for CERN Acc . Controls Conclusions 3 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

4.Why Middleware? How do we build/connect applications over a network? How do we facilitate Distributed C omputing (2-tier, …, n-tier )? How to support the heterogeneous environments ? 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 4

5.What is Middleware? Software , which allows an application to interoperate with other software, without requiring the user to understand and to code the low-level operations required to achieve interoperability S oftware layer between OS and the applications Hides complexity & heterogenity of distributed system Handles issues related to OS, Network Protocols & hardware platforms 5 Application Operating System1 Communication system MW Application Operating System2 MW Application Operating SystemN MW Standard API Standard API Specific API Specific API Standard API Specific API 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

6.Introduction to Middleware Middleware provides support for: Naming, Location, Service discovery, Replication Protocol handling, Communication faults, QoS Synchronisation, Concurrency, Failover , Scalability Access control, Authentication Middleware dimensions: Request - Reply vs. Asynchronous Messaging Language-specific vs. Language-independent Proprietary vs. Standards-based Small-scale vs. Large-scale Tightly-coupled vs. Loosely-coupled components Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 6 Middleware is everywhere. You might think you can leave it but it just finds you again :-) 26th October 2011

7.Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure Middleware Review for CERN Acc . Controls Conclusions 7 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

8.Middleware – large domain … which to choose ? 8 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 direct p2p broker req /rep pub/sub rpc /orb message LAN WAN/ Internet Network Embedded

9.Types of Middleware Remote Procedure Call (RPC) Object-Oriented Middleware (OOM) Message-Oriented Middleware (MOM) Event-Based Middleware a.k.a Publish-Subscribe 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 9

10.Remote Procedure Call - RPC Masks remote function calls as being local Client-Server model with Request - Reply paradigm S ynchronous c ommunication Marshalling of function parameters and return value Provides L ocation & P latform transparency Technologies: DCE - RPC XML-RPC 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 10 Client App. IDL Stub Network Server App. IDL Skeleton Network Req / Reply

11.RPC - Disadvantages Synchronous R equest -R eply interaction T ight coupling between client and server C lient may block for a long time if server loaded M ulti -threaded programming at client S low/failed clients may delay servers when replying M ulti-thre aded programming a t servers RPC paradigm is not object-oriented I nvoke functions on servers as opposed to methods on objects 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 11

12.Object-Oriented Middleware - OOM Client-Server model with Request - Reply paradigm S ynchronous c ommunication Marshalling of function parameters and return value Provides L ocation & P latform transparency Objects can be local or remote R emote objects accesible via proxy objects Technologies: OMG CORBA MS DCOM Java RMI SOAP 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 12 Client App. IDL Stub IIOP Client ORB Server App. IDL Skeleton IIOP Server ORB POA Adapter Req / Reply

13.Object-Oriented MW - Disadvantages Synchronous R equest -R eply interaction T ight coupling between client and server C lient may block for a long time if server loaded M ulti -threaded programming at client S low/failed clients may delay servers when replying M ulti-thre aded programming a t servers Distributed garbage collection Releasing memory for unused remote objects Error-prone and difficult to debug S tatic mappings and heavy-weight development Difficult to extend an operational system 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 13

14.Message-Oriented Middleware - MOM Data exchange via Messages & Message Queues Asynchronous c ommunication ( Point-to-point  One-to-One ) Client and Server are loosely coupled Message Servers may do filtering , transforming , logging … Technologies: IBM WebSphereMQ , TIBCO Rendez - vous , JMS , SOAP 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 14 Client App. local message queues Server App. local message queues message queues Network Network Network Message Servers

15.Message-Oriented MW - Disadvantages Poor programming abstraction (but has evolved) Rather low-level Request -R eply more difficult to achieve, but can be done Message formats originally unknown to middleware No type checking ( solved in JMS) Queue abstraction only gives O ne-to- O ne communication Limits scalability ( solved in JMS with Publish-Subscribe ) 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 15

16.What we lack, so far General interaction patterns W e have R equest- R eply and O ne-to- O ne Publish-Subscribe ? O ne-to- M any? M any - to -M any? D ynamic joining and leaving? Location transparency A nonymity of communicating entitie s 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 16

17.Event-Based Middleware a.k.a Publish-Subscribe Publishers (advertise and) publish E vents ( M essages ) Subscribers express interest in E vents with S ubscriptions Event Service notifies interested S ubscribers of published E vents Events can have arbitrary content (typed) Technologies ( Brokers ): JMS ( ActiveMQ , RabbitMQ ), AMQP (…) 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 17 Publisher1 Event Service (Broker) PublisherN … Publisher2 topics publish publish publish Subscriber1 SubscriberN … Subscriber2 subscribe notify subscribe notify subscribe notify

18.Properties of Publish - Subscribe Asynchronous communication Publishers and S ubscribers are loosely coupled Many-to- M any interaction between P ubs and S ubs Scalable scheme for large-scale systems Publishers do not need to know S ubscribers, and vice-versa Dynamic join and leave of P ubs & S ubs ( thanks to B rokers) Topic based Publish-Subscribe very expressive Filtered information delivered only to interested parties 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 18

19.Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure CERN Acc . Controls Middleware Review Conclusions 19 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

20.CERN Controls Middleware – in all SW tiers … 20 CMW client (C++/Java) GUIs , LabView , RADE CMW client (Java servers ) Logging , LSA, InCA , SIS CMW server (C++/Java) Proxy, DIP, AlarmMon , AQ CMW server (C++) FESA, FGC, GM CMW server (C++) PVSS ( Cryo , Vacuum ) JMS client (Java) GUIs JMS client (Java) Servers : Logging , InCA , SIS Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

21.JMS for Accelerator Controls J ava M essaging S ervice Purpose Reliable transport of data between Java Processes Controls Alarm System Data Rendering Services (Control) GUIs Logging Services Beam Security System(s) No JMS - No Beam ! 26th October 2011 21 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

22.JMS for Accelerator Controls Devices / Hardware JMS C/C++ Process JAVA Process JAPC CMW JMS Front End GUI Process JAPC CMW JMS CMW CMW ~50 Processes ~ 100 Processes ~46K Subscriptions 280 Connections 40 00 servers @ 1 7 00 machines 26th October 2011 22 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

23.Examples of JMS ” aware ” systems 26th October 2011 23 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

24.History of JMS in Accelerator Controls Usage of SonicMQ JMS for LHC Alarm System since 2001 ActiveMQ introduced in 2005 Other project started to use JMS service Today vital part of Beam Operations 26th October 2011 24 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

25.Technical Aspects Broker Configuration Non-persistent messages on topics No subscription recovery policy No durable subscribers No exclusive consumers No replyTo mechanism Broker is persistent Clients Pure JAVA - Wireformat Protocol 26th October 2011 25 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26.General JMS Deployment Situation SonicMQ : 12 Brokers TIM, LASER, DIAMON Always as two node cluster ActiveMQ : 16 Brokers One main default Broker Service Otherwise project dedicated Brokers All Brokers on Technical Network 26th October 2011 26 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

27.Overall Broker Service Jan 2011 26th October 2011 27 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

28.JMS Service Data Handling Central JMS Service (276 days in 2010) 58 Million msg / day 120 TB IN 650 TB OUT Additional handling of Heavy Connection and Subscription handling Bursts loads Connected subscribers grew from 2K to 12K in 6 months => 440 GB /day => 2.5 TB /day YES, ActiveMQ can! 26th October 2011 28 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

29.Data Handling during LHC startup 2010 Total Data Out Increase in data demand 2.57 TB/day 1TB/day LHC startup 26th October 2011 29 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

30.JMS Service Availability 2010* Total hours : 6624 ActiveMQ : 99.99% = 8 minutes downtime (due to kernel update) SoniqMQ : 99.87 % = 8 hours downtime * November 2010 26th October 2011 30 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

31.Monitoring and Troubleshooting Tools DIAMON & Lemon JConsole / JVisualVM Dump of broker information to SQLite DB Traffic Monitoring Tool Various Message Listeners P R I O R I T Y 26th October 2011 31 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

32.Easy Identification of Bad Producers Fixed Problematic Producer 26th October 2011 32 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

33.JMS Service – Lessons Learnt ActiveMQ was a good choice Running reliably Performs well Provides many metrics for diagnostic and monitoring Good integration in BE-CO monitoring tools via JMX Projects are not always aware of sending so much data Service Level Agreements with projects required More analyzing tools for end users required 26th October 2011 33 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

34.JMS Service – Lessons Learnt Monitoring is a vital part of operations Message reception speed test Memory Percent usage JMX attributes (#topics, #consumers, #subscriptions) Broker Configuration is not always obvious 26th October 2011 34 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

35.JMS Service – Lessons Learnt Clustering Good for failover and load balancing Additional complexity Updating clients for new cluster members Automatic rebalancing of clients Memory is important Tuning of Java GC options 26th October 2011 35 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

36.Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure CERN Acc . Controls Middleware Review Conclusions 36 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

37.Hardware Devices GUI operator consoles Front ends “virtual devices” CERN Controls Middleware – CMW 37 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Controls Middleware Embedded Middleware

38.CMW Mandate & Scope Standard set of MW solutions & libraries Centrally managed services Track & optimize runtime parameters Scope: CERN Accelerator Complex Operational 24*7*365 Must be Reliable & High Quality ~80 ’000 HW devices, ~4 ’ 000 servers ~1700 machines , ~2’000’000 IO points Huge infrastructure ! 38 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

39.CMW Data Flow 39 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

40.Linux/Windows CERN Controls Middleware 40 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 C++/Java process Controls Middleware API TCP/IP CORBA Controls Middleware RT Linux/ LynxOS C++ process Controls Middleware API CORBA Problems: - C++ and Java implementations differ  Heavy in memory usage  Complex error prone API  No direct support for pub/sub  Blocking issues ( JacORB )  Shrinking community  Lack of new releases and bug fixes  Long LHC shutdown at the end of 2012 A unique possibility! Instead of patching let’s provide a new one!

41.CERN Middleware Requirements 41 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Desirable Mandatory Fundamental Lightweight Active community Friendly API, documentation Request/reply & pub/sub patterns Open source, redistributable license QoS Stability, Maturity & Longevity Performance & Scalability Asynchronous C++/Java Linux/Windows Over TCP/IP LAN

42.How did we evaluate –> our criteria 42 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 CRITERIA QoS resources, binary size, memory performance Communications patterns API, look & feel, documentation Community, maturity Appearance Creators specification documentation Users forums bug reports Internet Simple usage Download licensing Compile Linux LynxOS Run examples Testing Communication patterns Performance QoS Exceptional situations

43.Surprising how many great new products! 43 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 2011 1969 ARPANET TODAY 1974 TCP 1998 IP v6 1980 UDP 1981 IP v4 1983 1991 2011

44.Evaluated middleware products 44 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Ice Thrift omniORB YAMI OpenSpliceDDS OpenAMQ CoreDX RTI DDS ZeroMQ QPid MQtt RSMB JacORB Mosquito All opinions are based only on our knowledge and evaluation . Each of the products, depending on the requirements, may constitute a good solution. RabbitMQ

45.CORBA ( omniORB , JacORB ) 45 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Object-oriented communication platform standardized by OMG. It is over 20 years old, and the standard is well established but... Complex, hard to learn and use  Differences between implementations  Big memory footprint  Shrinking community  Seldom updates and bug fixes  Complex Heavy Shrinking community Dying out

46.Object-oriented middleware by ZeroC , people who left CORBA to fix its problems. Conceptually similar to CORBA but... Better design and implementation  Modern easier to use API and IDL mapping  Concise C++ and Java implementation  Support for versioning  Active community  Ice 46 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Well designed Active community

47.Thrift 47 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Middleware by Facebook (moved to Apache) Lightweight, free of unnecessary dependencies  Modern, clean API  Active community  Ongoing development, still incomplete  No documentation, empty tutorial  Poor documentation Work still in progress

48.Message-oriented middleware developed by one of our former colleagues. Used by some CERN Controls services. Lightweight, free of unnecessary dependencies  Well designed, modern API  Asynchronous req /rep and pub/sub patterns  Dynamic type specification (paid in performance)   Small community  YAMI4 48 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Lightweight Well designed Small community

49.Middleware where messages are distributed by a broker. A few independent implementations of the broker and clients. Big community  Designed to support pub/sub  Protocol standard issues  Broker – single point of failure  Broker – additional hop, slower communication  Broker – additional, non compliant monitoring tools  AMQP – OpenAMQ , Qpid , RabbitMQ 49 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Slower for req /rep Broker issues Active community

50.Data-oriented middleware with p2p communication. Standard defined by OMG, with a few compatible implementations. Big industrial and military community  Possible compatibility with CORBA  Designed to support pub/sub  Req /rep possible but needs 2 channels  Steep learning curve  Complex API  DDS – RTI, CoreDX , OpenSplice 50 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Heavy Complex Big industry presence

51.Message-oriented middleware by iMatix (they dropped OpenAMQ .) API resembles BSD sockets but it is so much more. Simple yet powerful API  Support for in-proc, inter-proc, TCP, PGM  Lightweight, free of unnecessary dependencies  Many communication patterns  May be used as a concurrency framework  Active community  ZeroMQ 51 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Lightweight Well designed Active community

52.Performance requirements reviewed 52 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Instead of estimating We asked our users what they will need Gathered statistics from the current system Most demanding users identified Definition of a few performance tests

53.C++ server with client in: Performance tests, reliable R eq uest-Reply 53 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 4000 msg /sec Payload = 4B 5 msg /sec Payload = 10MB Many small messages A few big messages

54.Performance/Scalability tests, reliable P ub -S ub 54 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 400 msg x 8 B 10 clients, <50ms 30 msg x 8 B 10 clients, <20ms Throughput Latency

55.Products comparison (according to the criteria) 55 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 Sync, async & msg patterns QoS Dependencies & memory f-p Performance Look & feel, API, docs Community & maturity Score ZeroMQ       6 Ice       5 YAMI4       4 RTI       3 Qpid       3 CORBA       2 Thrift       2

56.Middleware Review Summary Several good middleware solutions available. The choice is dictated by the most critical requirements for any given application. Not easy  QoS policies and performance matter, but also ease of use, community, … 56 Concerning CERN Controls Middleware… P rototype with the most promising candidates: Deploy the new middleware before the long accelerator shutdown at the end of 2012 ZeroMQ Ice YAMI Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

57.Conclusions Middleware is an important abstraction for building the distributed systems Synchronous vs. asynchronous communication One-to-one vs . many-to-many communication Better performance with direct communication Better scalability and less coupling with Brokers System requirements  Messaging patterns to apply 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 57

user picture
  • Hafenkranich
  • Independent consultant specializing in database, web and M2M applications

相关Slides