Mailchimp Scale:a MySQL Perspective

1 . Mailchimp Scale: A MySQL Perspective John Scott Mailchimp 

2 .What is Mailchimp’s secret sauce? Hint: It’s not much of a secret. 2 

3 .Focus on the small business “Empowering the Underdog” 3 

4 . “We give marketers production-ready software designed to help them grow…” Mailchimp Engineering Mission Statement https://mailchimp.com/culture/how-our-engineering-team-found-its-mission-statement/ 

5 .5 

6 .Another way to say it “We SCALE through togetherness, momentum, and pragmatism.” 6 

7 .Old Mentality: The 3 Disciplines of Data Administration ● OPS / KTLO ● Support ● Performance 

8 .Old Mentality: The 3 Disciplines of Data Administration ● OPS / KTLO ● Support ● Performance “I’m a DevOps DBA” 

9 .Old Mentality: The 3 Disciplines of Data Administration ● OPS / KTLO ● Support ● Performance 

10 .Old Mentality: The 3 Disciplines of Data Administration ● OPS / KTLO ● Support ● Performance “I help other departments work with databases” 

11 .Old Mentality: The 3 Disciplines of Data Administration ● OPS / KTLO ● Support ● Performance 

12 .Old Mentality: The 3 Disciplines of Data Administration ● OPS / KTLO ● Support ● Performance “over the fence” 

13 .New Mentality: “Ops is product” 

14 .Ops is Product “If you improve database performance resulting in 10% reduction in churn, you would create an additional <big revenue number>.” 

15 .Ops is Product “Developer Enablement” New paradigm “looking at ops through the lens of product” --Tyler Treat ● https://bravenewgeek.com/operations-in-the-world-of-developer-enablement/ ● https://www.youtube.com/watch?v=JUy3GYkPfto OR in the case of Mailchimp, ops actually developing software, too. 

16 .Developer Enablement Product Enablement In most organizations “Product enablement” is sales term with the “four Ps” ● Positioning ● Pitch ● Play ● Program 

17 .Developer Enablement Product Enablement 1000 employees 350+ engineers 0 salespeople 

18 .Mailchimp “Board Room” 

19 .

20 . Sounds great. But what does that mean for a database engineer? 

21 .#togetherness in action MySQL log analysis based on pt-query-digest and Elasticsearch / Kibana resulted in a Top 20 table activity graph 

22 .End of story? “Toss it over the wall.” “Not my problem.” “I don’t have commit rights.” 

23 .This is Mailchimp Engineering “We succeed through togetherness, Momentum, and Pragmatism” 

24 .We identified an N+1 pattern and fixed it, together. 

25 .But wait.... 

26 .What was the impact to the user experience? 

27 .265 billion queries per week 247 thousand unique query fingerprints 2200 Instances of mysql 

28 .Old Mentality: Effective Slow Query Log Analysis Across The Infrastructure FTW! “Query Macroeconomics” https://johnscott.net/2018/08/03/query-macroeconomics/ ● Prioritize query fixes by how much DB capacity you get back ○ MySQL not stressed with contention equals what? ■ A pretty innodb status? ■ Nice looking graphs? 

29 .Old Mentality: Effective Slow Query Log Analysis Across The Infrastructure FTW! “Query Macroeconomics” https://johnscott.net/2018/08/03/query-macroeconomics/ ● Prioritize query fixes by how much DB capacity you get back ○ MySQL not stressed with contention equals what? ■ A pretty innodb status? ■ Nice looking graphs?