Lessons Learned from Microservice at scale

1 .Lessons Learned from Microservices at Scale Mike Calvo Target Corp

2 .a bout this talk Speaker: Director Engineering Target – target.com MSSE Adjunct faculty 25 Years programming and leadership experience Basis: Lessons learned from building services to support target.com and Target Mobile apps Experience from building non-commerce (non-PCI) micro-services in the cloud

3 .Defining m icroservices Small, single function endpoints Loosely coupled services that collaborate to provide a product or experience Each service performs a specific thing Single domain Follows OO Single Responsibility Principle Often depends on supporting infrastructure Load balancers, proxies, databases, caching stores, CDN, configuration

4 .Scalability Credit: https:// cloud.google.com /solutions/scalable-and-resilient-apps

5 .Real Life Adventure Story Mongo and the Monolith

6 .Lessons from Treachery on Mongo Independent service deployments can save you from total failure External Dependencies: Monitor all access times Circuit break every remote call

7 .Independent Deployments Failures are more isolated Independent scaling Capacity matches demand Auto-scaling independently CPU-based Network-based Memory-based Credit "Building Microservices " by Sam Newman

8 .Monitor Access Times Database Caching layers Dependent services Options Log - ELK Emit metrics – Prometheus or Dropwizard Visualize Grafana Kibana Splunk

9 .Circuit Breaker Pattern for preventing cascading errors from remote resources Wrap a call to a remote resource Define thresholds when circuit should be “open” Timeout Error percentages When circuit is open no calls are made to remote resource Configure circuit to be retried after a period of time Alerts can fire when circuits are open

10 .Netflix Hystrix and Turbine Open source implementation of circuit-breaker pattern Java library for latency and fault-tolerant logic Wrap remote calls with Hystrix Fallback responses Configure timeouts and error thresholds Turbine: Web-base dashboard for displaying connection health and circuit status

11 .Turbine Example Credit Netflix

12 .Other Lessons Stateless services Asynchronous I/O Caching patterns Deploy frequently Protect uptime

13 .Stateless Services Each request allocates and frees memory required to process the request Avoid problem of session affinity Allows any instance to respond to any request Java: avoid using HTTP Session object Solutions for session-based problems Time-expired key value stores ( Memcached or Redis ) Fast read access data stores ( MongoDB )

14 .Asynchronous I/O Traditional forms of system I/O calls are blocking C urrent thread waits for all data to be read or written before continuing More complex forms of I/O enable notification when operation has completed Enables non-blocking I/O More complex programming model Frees CPU to do more work while waiting for I/O to complete Asynchronous I/O enables higher throughput for I/O-bound processes Network calls Database access File access

15 .Credit: https:// www.toptal.com /back-end/server-side- io -performance-node- php -java-go

16 .Single Event Loop Model Main application thread processes events in a queue Inbound HTTP requests generate events If the handling of a request requires I/O use asynchronous I/O When I/O completes an event is placed on the queue with the result Offer better performance for I/O-bound services

17 .Credit : https :// medium.com /@ tigranbs /concurrency-vs-event-loop-vs-event-loop-concurrency-eb542ad4067b

18 .Asychronous I/O Server Platforms Node Javascript I/O performed on background threads Netty / Vertx / Ratpack JVM-based implementation I/O performed on background threads Go Go language developed by Google Natively compiled Go-routines can be suspended based on activity Callback structure abstracted away from developer

19 .Caching Time-to-live (TTL) Soft Hard Micro caching Grace period Clearing considerations Composite keys Edge caching CDN – Akamai, Fastly Proxies – Varnish

20 .Deploy Frequently Team of four engineers should deploy several times per week Deploy small things frequently Requires automated testing User acceptance testing still required Continuous Integration/Continuous Deployment Deployments are Automated and fast

21 .Favoring Speed Though microservices support building with diverse platforms this hasn’t been strong feature for our teams Building on a well-understood tech stack (JVM) leverages efficiencies of deployment, scale and troubleshooting Ruthlessly pursue reducing time from code change to deploy

22 .Common Cloud Deployment Models Virtual Machines Containers Mesos Docker / Kubernettes Serverless

23 .Virtual Machines Each application server runs on its own dedicated VM Full OS instance Select UNIX or Windows flavors Supported natively via cloud provider Provision using cloud console or API Spinnaker Abstraction layer enabling cross-cloud provisioning and deployment Netflix

24 .Containers App runs within a container which is managed by an environment Environment optimizes usage of infrastructure Free memory and processors will be allocated during scaling Disk access managed by environment Examples: Docker and Mesos

25 .Serverless Cloud provider enables deployments of code into platform Ephemeral containers Function as a service API developer not concerned with provisioning of compute resources Typically limited platform options Common options: JavaScript, Python and Java Examples AWS Lambda Google Cloud Functions Microsoft Azure Functions

26 .Protect Uptime Need to be available around the clock Measured in 9s (99.999% up) Monitoring and Alerting Key System metrics Health checks/Aliveness checks Response codes Response times Non functional requirements of all services must include monitoring

27 .Microservices Advantages Supports technology diversity Can be developed with any language on any platform Experimentation Correct tool for the job Resilience Scaling Faster to develop and release Organizational alignment Composability Optimize for replace ability

28 .Microservices Disadvantages/Challenges Complicated deployment and management Sharing of code is harder Technology/Architecture consistency is harder to obtain Decisions on how to partition system

29 .More Details ”Building Microservices ” by Sam Newman Martin Fowler Microservices https://martinfowler.com/articles/microservices.html 12 Factor Apps https://12factor.net/