Extending and Customizing Percona Monitoring and Management

1 . Extending and Customizing PMM Michael Coburn Percona 1 © 2019 Percona 

2 .Michael Coburn ▪ Product Manager for PMM (as well as for Percona Toolkit) ▪ At Percona for 6 years across multiple MySQL roles Principal Architect, Managing Consultant, Technical Account Manager 2 © 2019 Percona 

3 .What is PMM ▪ Free, Open Source database troubleshooting and performance optimization platform for MySQL, MongoDB, and PostgreSQL We also support: ▪ ProxySQL ▪ Amazon RDS MySQL & Aurora MySQL ▪ Remote MySQL & PostgreSQL instances ▪ Runs in your secure environment (this is not a SaaS product!) and on your equipment ▪ Secured with SSL between client and server 3 © 2019 Percona 

4 .Customizing Dashboards 4 © 2019 Percona 

5 .Grafana in a Nutshell ▪ Open Source data visualisation tool ▪ Dashboards are just JSON ▪ Popular datasources Prometheus CloudWatch Graphite Elasticsearch ▪ Templated Variables Define your graph metrics, and let the hosts get filled in automatically GREAT for large, dynamic environments where hosts are considered ephemeral 5 © 2019 Percona 

6 .Dashboard Example - MySQL Overview 6 © 2019 Percona 

7 .Editing a Dashboard ▪ Enter into Edit mode ▪ Review Metrics formulas & define graph characteristics 7 © 2019 Percona 

8 .Collect Metrics from anywhere 8 © 2019 Percona 

9 .External Exporters ▪ Prometheus is a very active community, hundreds of exporters exist ▪ First configure an Exporter to run, then notify PMM to scrape the new Exporter ▪ pmm-admin add external:service --service-port=9187 postgresql 9 © 2019 Percona 

10 .Adding Dashboards ▪ Create your own! ▪ Or use one from grafana.com/dashboards and edit as needed Datasource name (change to Prometheus) Edit instance variables to match PMM variable names (i.e. $host when most grafana.com Dashboards use $instance) 10 © 2019 Percona 

11 .Custom Queries Available for MySQL and PostgreSQL 11 © 2019 Percona 

12 .Custom Queries ▪ MySQL and PostgreSQL support the concept of Custom Queries - run SELECT statements in order to create Prometheus metric series that you can then plot on a graph ▪ Configuration is enabled by default and executed by the Exporter ▪ Runs every 60 seconds 12 © 2019 Percona 

13 .Users table - MySQL view 13 © 2019 Percona 

14 .Users table - queries-mysqld.yml ▪ Configuration is stored in /usr/local/pmm-client/ 14 © 2019 Percona 

15 .Users table - Advanced Data Exploration 15 © 2019 Percona 

16 .Customizing Exporter Options 16 © 2019 Percona 

17 .Exporter Options ▪ Exporters have multiple collectors - specific code to collect specific metrics from the service being monitored ▪ Percona balances maximum data collection with minimum impact on the observed system ▪ This leads to some Exporter collectors being disabled 17 © 2019 Percona 

18 .node_exporter Example ▪ 48 total collectors: 32 are enabled by default, 16 disabled ▪ We want to enable monitoring of NTP pmm-admin remove linux:metrics pmm-admin add linux:metrics -- --collectors.enabled=...,ntp ps aux | grep node_exporter 18 © 2019 Percona 

19 .Using Grafana Alerting 19 © 2019 Percona 

20 .Grafana Alerting ▪ Rules are added to graphs that react based on defined thresholds ▪ Alerts are sent to pre-defined Notification Channels You can have multiple channels defined 20 © 2019 Percona 

21 .Grafana Alerting 21 © 2019 Percona 

22 .Grafana Notification Channels ▪ Defines how you will learn about new alerts ▪ Over a dozen are available: Email Slack PagerDuty AlertManager ... 22 © 2019 Percona 

23 .Using MySQL Data Source 23 © 2019 Percona 

24 .MySQL Data Source ▪ Grafana feature which PMM inherits from upstream ▪ Results of a SELECT statement can draw graphs ▪ Our example is using Percona Production data 24 © 2019 Percona 

25 .PMM Versions over Time 25 © 2019 Percona 

26 .Do's and Don’t’s 26 © 2019 Percona 

27 .PMM Best Practices ▪ Grafana Alerting works great for < 50 instances Over that it becomes prohibitive to manage the variable -> string mappings ▪ Don't edit Percona-provided Dashboards - your changes will be overwritten at next upgrade! Instead copy the dashboard and edit your copy, and it will survive upgrades ▪ Send us your ideas, and even better, your Dashboards Percona loves Pull Requests! Your changes can make it into a future release ▪ Participate in the percona.com/forums ! 27 © 2019 Percona