Middleware Progress report and plans

1 .The role of messaging solutions in complex systems, based on experience in the accelerator controls at cern JINR/CERN/MEPHI Computing school DUBNA , 26 th October 2011 Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

2 .Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure Middleware Review for CERN Acc . Controls Conclusions 2 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

3 .Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure Middleware Review for CERN Acc . Controls Conclusions 3 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

4 .Why Middleware? How do we build/connect applications over a network? How do we facilitate Distributed C omputing (2-tier, …, n-tier )? How to support the heterogeneous environments ? 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 4

5 .What is Middleware? Software , which allows an application to interoperate with other software, without requiring the user to understand and to code the low-level operations required to achieve interoperability S oftware layer between OS and the applications Hides complexity & heterogenity of distributed system Handles issues related to OS, Network Protocols & hardware platforms 5 Application Operating System1 Communication system MW Application Operating System2 MW Application Operating SystemN MW Standard API Standard API Specific API Specific API Standard API Specific API 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

6 .Introduction to Middleware Middleware provides support for: Naming, Location, Service discovery, Replication Protocol handling, Communication faults, QoS Synchronisation, Concurrency, Failover , Scalability Access control, Authentication Middleware dimensions: Request - Reply vs. Asynchronous Messaging Language-specific vs. Language-independent Proprietary vs. Standards-based Small-scale vs. Large-scale Tightly-coupled vs. Loosely-coupled components Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 6 Middleware is everywhere. You might think you can leave it but it just finds you again :-) 26th October 2011

7 .Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure Middleware Review for CERN Acc . Controls Conclusions 7 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

8 .Middleware – large domain … which to choose ? 8 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011 direct p2p broker req /rep pub/sub rpc /orb message LAN WAN/ Internet Network Embedded

9 .Types of Middleware Remote Procedure Call (RPC) Object-Oriented Middleware (OOM) Message-Oriented Middleware (MOM) Event-Based Middleware a.k.a Publish-Subscribe 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 9

10 .Remote Procedure Call - RPC Masks remote function calls as being local Client-Server model with Request - Reply paradigm S ynchronous c ommunication Marshalling of function parameters and return value Provides L ocation & P latform transparency Technologies: DCE - RPC XML-RPC 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 10 Client App. IDL Stub Network Server App. IDL Skeleton Network Req / Reply

11 .RPC - Disadvantages Synchronous R equest -R eply interaction T ight coupling between client and server C lient may block for a long time if server loaded M ulti -threaded programming at client S low/failed clients may delay servers when replying M ulti-thre aded programming a t servers RPC paradigm is not object-oriented I nvoke functions on servers as opposed to methods on objects 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 11

12 .Object-Oriented Middleware - OOM Client-Server model with Request - Reply paradigm S ynchronous c ommunication Marshalling of function parameters and return value Provides L ocation & P latform transparency Objects can be local or remote R emote objects accesible via proxy objects Technologies: OMG CORBA MS DCOM Java RMI SOAP 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 12 Client App. IDL Stub IIOP Client ORB Server App. IDL Skeleton IIOP Server ORB POA Adapter Req / Reply

13 .Object-Oriented MW - Disadvantages Synchronous R equest -R eply interaction T ight coupling between client and server C lient may block for a long time if server loaded M ulti -threaded programming at client S low/failed clients may delay servers when replying M ulti-thre aded programming a t servers Distributed garbage collection Releasing memory for unused remote objects Error-prone and difficult to debug S tatic mappings and heavy-weight development Difficult to extend an operational system 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 13

14 .Message-Oriented Middleware - MOM Data exchange via Messages & Message Queues Asynchronous c ommunication ( Point-to-point  One-to-One ) Client and Server are loosely coupled Message Servers may do filtering , transforming , logging … Technologies: IBM WebSphereMQ , TIBCO Rendez - vous , JMS , SOAP 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 14 Client App. local message queues Server App. local message queues message queues Network Network Network Message Servers

15 .Message-Oriented MW - Disadvantages Poor programming abstraction (but has evolved) Rather low-level Request -R eply more difficult to achieve, but can be done Message formats originally unknown to middleware No type checking ( solved in JMS) Queue abstraction only gives O ne-to- O ne communication Limits scalability ( solved in JMS with Publish-Subscribe ) 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 15

16 .What we lack, so far General interaction patterns W e have R equest- R eply and O ne-to- O ne Publish-Subscribe ? O ne-to- M any? M any - to -M any? D ynamic joining and leaving? Location transparency A nonymity of communicating entitie s 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 16

17 .Event-Based Middleware a.k.a Publish-Subscribe Publishers (advertise and) publish E vents ( M essages ) Subscribers express interest in E vents with S ubscriptions Event Service notifies interested S ubscribers of published E vents Events can have arbitrary content (typed) Technologies ( Brokers ): JMS ( ActiveMQ , RabbitMQ ), AMQP (…) 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 17 Publisher1 Event Service (Broker) PublisherN … Publisher2 topics publish publish publish Subscriber1 SubscriberN … Subscriber2 subscribe notify subscribe notify subscribe notify

18 .Properties of Publish - Subscribe Asynchronous communication Publishers and S ubscribers are loosely coupled Many-to- M any interaction between P ubs and S ubs Scalable scheme for large-scale systems Publishers do not need to know S ubscribers, and vice-versa Dynamic join and leave of P ubs & S ubs ( thanks to B rokers) Topic based Publish-Subscribe very expressive Filtered information delivered only to interested parties 26th October 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 18

19 .Outline Defining Middleware Types of Middleware CERN Acc . Controls Messaging Infrastructure CERN Acc . Controls Middleware Review Conclusions 19 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

20 .CERN Controls Middleware – in all SW tiers … 20 CMW client (C++/Java) GUIs , LabView , RADE CMW client (Java servers ) Logging , LSA, InCA , SIS CMW server (C++/Java) Proxy, DIP, AlarmMon , AQ CMW server (C++) FESA, FGC, GM CMW server (C++) PVSS ( Cryo , Vacuum ) JMS client (Java) GUIs JMS client (Java) Servers : Logging , InCA , SIS Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 26th October 2011

21 .JMS for Accelerator Controls J ava M essaging S ervice Purpose Reliable transport of data between Java Processes Controls Alarm System Data Rendering Services (Control) GUIs Logging Services Beam Security System(s) No JMS - No Beam ! 26th October 2011 21 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

22 .JMS for Accelerator Controls Devices / Hardware JMS C/C++ Process JAVA Process JAPC CMW JMS Front End GUI Process JAPC CMW JMS CMW CMW ~50 Processes ~ 100 Processes ~46K Subscriptions 280 Connections 40 00 servers @ 1 7 00 machines 26th October 2011 22 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

23 .Examples of JMS ” aware ” systems 26th October 2011 23 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

24 .History of JMS in Accelerator Controls Usage of SonicMQ JMS for LHC Alarm System since 2001 ActiveMQ introduced in 2005 Other project started to use JMS service Today vital part of Beam Operations 26th October 2011 24 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

25 .Technical Aspects Broker Configuration Non-persistent messages on topics No subscription recovery policy No durable subscribers No exclusive consumers No replyTo mechanism Broker is persistent Clients Pure JAVA - Wireformat Protocol 26th October 2011 25 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26 .General JMS Deployment Situation SonicMQ : 12 Brokers TIM, LASER, DIAMON Always as two node cluster ActiveMQ : 16 Brokers One main default Broker Service Otherwise project dedicated Brokers All Brokers on Technical Network 26th October 2011 26 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

27 .Overall Broker Service Jan 2011 26th October 2011 27 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

28 .JMS Service Data Handling Central JMS Service (276 days in 2010) 58 Million msg / day 120 TB IN 650 TB OUT Additional handling of Heavy Connection and Subscription handling Bursts loads Connected subscribers grew from 2K to 12K in 6 months => 440 GB /day => 2.5 TB /day YES, ActiveMQ can! 26th October 2011 28 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

29 .Data Handling during LHC startup 2010 Total Data Out Increase in data demand 2.57 TB/day 1TB/day LHC startup 26th October 2011 29 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...