16/06 - Running Cassandra on Amazon's ECS

1 .Running Cassandra on Amazon’s ECS
 
 Anirvan Chakraborty 
 @anirvan_c 
 

2 . Agenda • Motivation • Docker • Cassandra • Cassandra on Docker best practices • EC2 Container Service ECS • Cassandra on ECS © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

3 . Motivation © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

4 . Motivation • Ease of development • Support polyglot languages, frameworks and components • Operational simplicity • Quick feedback loop © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

5 . Docker © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

6 . Docker history • Came out of dotCloud, a PaaS company • Was originally written in Python • Got re-written in Golang in Feb, 2013 • Docker 0.1 was released on Mar, 2013 • Docker 1.10 is the latest release © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

7 . Docker tag line Build, ship and run any app, anywhere © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

8 . Docker tag line • Build: package your application in a container • Ship: move it between machines • Run: execute that container with your application • Any application: as long as it runs on Linux • Anywhere: local VM, bare metal, cloud instances © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

9 . Why Docker? • Deploy reliably & consistently • Execution is fast and light weight • Simplicity • Developer friendly workflow • Fantastic community © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

10 . Apache Cassandra © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

11 . What is Apache Cassandra? • Fast distributed database • High Availability • Linear Scalability • Predictable performance • No single point of failure • Multi-DC • Easy to manage • Can use commodity hardware • Not a drop in replacement for RDBMS © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

12 . Hash ring • Data is partitioned around the ring • Location of data in ring is determined by partition key • Data is replicated to N servers based on RF • All nodes hold data and can answer read or write queries source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

13 . CAP Tradeoff • During network partition it is impossible to be both consistent and highly available • Latency between data centres also makes consistency impractical • Cassandra chooses Availability & Partition tolerance over Consistency © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

14 . Replication • Choose “replication factor” or RF • Data is always replicated to each replica • If node is down, missing data is replayed via hinted handoff source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

15 . Consistency level • Per query consistency • ALL, QUORUM, ONE • How many replicas to respond OK for query to succeed source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

16 . Cassandra on Docker © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

17 . Dockerize C* Dev Environment • Make it run as slow, but as stable as possible! • Super low memory settings in cassandra-env.sh • MAX_HEAP_SIZE=“128M” • HEAP_NEWSIZE=“24M” • Remove caches in dev mode in cassandra.yml • key_cache_size_in_mb: 0 • reduce_cache_sizes_at: 0 • reduce_cache_capacity_to: 0 © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

18 . Dockerize C* Production • Use host networking (—net=host) for better network performance • Put data, commitlog and saved_caches in volume mount folders to the underlying host • Run cassandra on the foreground using (-f) • Tune JVM heap for optimal size • Tune JVM garbage collector for your workload © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

19 . Amazon EC2 Container Service © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

20 . What is ECS? …is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

21 . What is ECS? Amazon Docker as a Service https://www.expeditedssl.com/aws-in-plain-english © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

22 . How does ECS work? http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

23 . Cluster http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

24 . Container Instance http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

25 . ECS Agent http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

26 . Task http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

27 . Task { "containerDefinitions": [ { "name": "wordpress", "links": [ "mysql" ], "image": "wordpress", "essential": true, Docker Container "portMappings": [ { "containerPort": 80, "hostPort": 80 } ], "memory": 500, "cpu": 10 }, { "name": "mysql", "image": "mysql", "cpu": 10, Docker Container "memory": 500, "essential": true } ], Task definition } "family": "hello_world" http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

28 . Service http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 

29 . ECS Service http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0