Managing Smart Meter Data with DataStax DSE

• Who are First Utility • Creating a high availability platform • Enabling customer control and insights through analytics • Providing a self healing platform with event history • Monitoring and alerting across the platform
展开查看详情

1.Managing Smart Meter Data with DataStax DSE Satish Ramjee Principal Software Engineer First Utility June 2017

2. • Who are First Utility • Creating a high availability platform • Enabling customer control and insights through analytics • Providing a self healing platform with event history • Monitoring and alerting across the platform • Conclusion Contents

3.Who are First Utility

4. ● Largest independent energy supplier ● Supply almost 1 million UK homes ● One of the first to provide Smart meters ● Provide broadband and home services as well ● Technology driven company First Utility

5.Creating a high availability platform

6.● National smart meter rollout ○ Aim is to have all 53 million residential premises in England, Wales, Scotland with Smart Meters by 2020 ● Energy suppliers are required to offer smart meters to all their customers ● Our immediate need to manage a high volume of meter reads with high availability ● Meter access devices can potentially provide data every 5- 10 seconds from the total meter estate Creating a high availability platform

7.● Smart reads ideal for an event sourced type database ● Saw the need to add resilience, high volume and linear scaling capability into our key information base ● Opportunity to manage business expectation through service layers ● Providing a meter agnostic business layer “One of the things DataStax offered to us was the ability to really compress that learning curve. So we had some of their key experts come in and spend a few weeks with us over a period of time and provide that technical leadership” Bill Wilkins, CIO, First Utility. http://diginomica.com/2017/02/02/first-utility-makes-smart-meters-even-smarter-datastax-platform/ Creating a high availability platform

8. From Monolithic to Microservices Architecture Application Server Application Server Docker Container Docker Container application application application application (instance 1) 1) application (instance application (instance 1) (instance 2) (instance 1) 2) (instance store store multiple node single node distributed store rdms Creating a high availability platform

9. Data Stores Customer Usage View Insights Meter Type 1 Billing View Billing spark Meter Type Meter Data 1 2 Meter Data 2 spark Industry Industry View Meter Type n Meter Data 3 spark Data Warehouse Creating a high availability platform

10. ● Multiple DCs ● Appropriate RF ● DCs split between Business Critical access and Analytics Creating a high availability platform

11.Enabling Customer Control and Insights

12.● Provide customers with an insightful view ● Help customers better manage their usage ● My energy comparison of usage ● Allow control of read submission frequency ● In home display reflects current tariff and usage Enabling Customer Control and Insights

13. Data Storage Customer Insights read Usage View Meter read Meter Preferences IHD preference/price Tariff Meter Data Enabling Customer Control and Insights

14.Enabling Customer Control and Insights

15.Enabling Customer Control and Insights

16.Enabling Customer Control and Insights

17.Providing a self healing platform

18.● Record commands sent to the meter ● Detect failures ● Replay commands (with retry limit) meter state Event response Meter command commands spark replay reconcile Providing a self healing platform

19.Monitoring and alerting across the platform

20.● Eyes and ears ● Dashboards of platform behaviour ● Operational: is the platform working ● Business information: reporting and analytics Monitoring and alerting across the platform

21.Monitoring and alerting across the platform

22.Conclusion

23.● Gone from read only system to ability control of meter (read/write) ● Self healing ● Real time monitoring and alerting ● Enduring solution can scale both processing and persistence ● Lessons learnt ○ Denormalise - many copies with different PK ○ Know your queries in advance ○ Reporting and aggregation - duplicate, spark or counters Conclusion

24.That’s all folks