Vitess on Kubernetes

1 .Vitess on Kubernetes …followed by a demo of VReplication Jiten Vaidya jiten@planetscale.com 

2 .A word about me ... Jiten Vaidya - Managed teams that operationalized Vitess at Youtube CEO at PlanetScale Founded in early 2018 to help operationalize Vitess • Sugu Sugumaran (CTO, Vitess community leader) Offerings • Open Source Vitess Support • Custom Vitess Development • Kubernetes Deployment Manager • Cross-cloud DBaaS 

3 .Show of hands ... ● How many of you know what Vitess is? ● How many of you are facing scaling problems? ● How many of you want to run mysql in Cloud? 

4 .What does Vitess solve? ● Database Scalability ● MySQL Guardrails ● Operation in Cloud 

5 .Vitess Architecture app server vtctld lockserver shard vttablet app server mysqld vtgate master app server vtgate vttablet vttablet vttablet vttablet mysqld mysqld mysqld mysqld big data replicas big data replicas App Vitess 

6 .Major users using Vitess In production Evaluating Quiz of Kings 

7 .Vtgate in Kubernetes ● Stateless proxy ● Vtgates can be created as load ● Accepts connections as a MySQL increase compatible server + over GRPC ● Start n vtgates as a Replica Set ● Supports an http endpoint + Web ● For co-located workloads start one UI for status vtgate per node and expose with a ● Computes target shards ClusterIP ● Sends queries to vttablets for targeted shards ● Receives, collates and serves response to application 

8 .Vtctld in Kubernetes ● Vitess Control Plane ● Start one or two vtctld processes ● Serves a Web UI per cell ○ Operational commands ● Start them as a Deployments ○ Status ● Expose them behind a Service ○ Topology browser ● Serves an API over GRPC ○ Used by vtctlclient tool ● Supports resharding workflows 

9 .lockserver (etcd) in Kubernetes ● Knits the Vitess cluster together ● One global cluster ● Backing store for metadata ● One cluster per cell (optional) ○ Service discovery ● Use etcd-operator to spin out a ○ Topology cluster ○ VSchema ● Expose etcd cluster behind a ● Not used for query serving Service ● Needed for any change in topology ○ Add a keyspace ○ Add a shard to keyspace ○ Add a tablet to a shard ○ Change master for a shard 

10 .Tablet (vttablet + mysqld) in Kubernetes ● Vitess Tablet is a combination ● 2 containers in the same pod of a mysqld instance and a ● Communicate over Unix socket corresponding vttablet process ● Unix socket created in Shared ● Each tablet requires a unique Volume id in Vitess cluster ● Local Persistent Volume for ● Tablets can be of type: master, data replica, rdonly ● One master, 2 replicas with ● Tablets of type “replica” can semi-sync replication enabled be promoted to master and for high availability should have low replica lag ● Replicas should not be co-located with other members of shard ( Anti-Affinity ) 

11 .Authentication/Secrets management ● What secrets are needed? ○ Application -> Vtgate authentication ○ Vttablet -> mysqld authentication for various roles that Vitess supports (app, dba, replication, filtered replication etc). ○ TLS certs and keys for GRPC traffic over TLS (optional) ○ TLS certs and keys for binary logs over TLS (optional) ○ TLS certs and keys for client authorization and authentication over TLS (optional) ● Use Kubernetes Secrets and mount them in pods 

12 .High Availability ● Planned reparent ○ Coordinated via lockserver ○ Existing transactions are allowed to complete ○ New transactions are buffered by vtgate ○ New master is made writable ○ Replicas are made slaves of the new master ○ Query serving is resumed ● Unplanned reparent ○ Orchestrator ○ TabletExternallyReparented ● Resharding ○ No interruption to query traffic during resharding 

13 .Supporting multiple cells ● Vitess cell is the equivalent of a failure domain (e.g. AWS availability zones or regions) ● Not necessarily the same as Kubernetes failure domain. ● Choice to use global lockserver cluster OR use one lockserver cluster per cell. ● Global lockserver cluster typically outside of Kubernetes. ● Expose lockserver behind a service definition. ● If using etcd use etcd-operator to start per cell cluster. 

14 .Vitess clusters spanning Kubernetes clusters ● Global lockserver cluster should be outside of Kubernetes ● While creating each cell designate which Kubernetes cluster it resides in ● Must have a non-overlapping ip space and all addresses must be routable. ● pod to pod communication ○ Needed for mysql replication and query serving ○ Achieved by Peering and Routing 

15 .Part II: Vitess in Action ● Vitess super power: filtered replication aka VReplication ● Ability to consume binary replication log, and apply sharding or other logic to it ● Shard splits (aka resharding) ● Sharded Materialized Views 

16 . Splitting a Shard in Vitess Step 1 - Vitess in-front of app 1 shard. Vttablet (Shard -) app server vtgate mysqld 

17 . Live Resharding Filtered Replication Step 2 - Split Shard Vttablet (Shard 00-80) mysqld Vttablet (Shard 0) app server vtgate mysqld Vttablet (Shard 80-FF) mysqld 17 

18 . Live Resharding Step 3 - Split Shard Vttablet (Shard 00-80) mysqld app server vtgate Vttablet (Shard 80-FF) mysqld 

19 .Hot off the presses ... … Sharded Materialized Views (Ability to shard using more than one sharding key using replication streams) M shards using key1 -> N shards using key2 M * N replication streams 

20 .Materialized view example 

21 .Demo lookup user unsharded sharded merchant sharded name_user_idx user name id merchant uid name name product order Id id description uid pid mname 

22 .Questions? jiten@planetscale.com