Writing Prometheus exporters

Writing Prometheus exporters in theory and practice

展开查看详情

1. Writing Prometheus exporters In theory and practice Alexey Palazhchenko PMM developer / Prometheus contributor / Gopher Percona University Kiev November 11, 2017 1 © 2016 Percona

2.2 © 2017 Percona

3.OpenMetrics ▪ Prometheus developers ▪ Google ▪ InfluxData ▪ SolarWinds ▪ Okmeter 3 © 2017 Percona

4.Exposition Format # HELP http_requests_total The total number of HTTP requests. # TYPE http_requests_total counter http_requests_total{method="post",code="200"} 1027 1395066363000 http_requests_total{method="post",code="400"} 3 1395066363000 {__name__="http_requests_total",method="post",code="400"} 4 © 2017 Percona

5.Theory 5 © 2016 Percona

6.Instrumentation Methods 6 © 2017 Percona

7.Instrumentation Methods ▪ Direct 6 © 2017 Percona

8.Instrumentation Methods ▪ Direct ▪ Custom whitebox exporter 6 © 2017 Percona

9.Instrumentation Methods ▪ Direct ▪ Custom whitebox exporter ▪ blackbox_exporter 6 © 2017 Percona

10.Instrumentation Methods ▪ Direct ▪ Custom whitebox exporter ▪ blackbox_exporter 6 © 2017 Percona

11.Instrumentation Methods ▪ Direct ▪ Custom whitebox exporter ▪ blackbox_exporter ▪ Ephemeral and batch jobs: Pushgateway or node_exporter's Textfile collector 6 © 2017 Percona

12.Tradeoffs 7 © 2017 Percona

13.Tradeoffs ▪ Perfect data: proper metric names, labeld, types, descriptions, stateless, no caching, etc 7 © 2017 Percona

14.Tradeoffs ▪ Perfect data: proper metric names, labeld, types, descriptions, stateless, no caching, etc ▪ Quick and dirty implementation: SELECT * FROM i_s, convert columns to UNTYPED metrics without labels 7 © 2017 Percona

15.Tradeoffs ▪ Perfect data: proper metric names, labeld, types, descriptions, stateless, no caching, etc ▪ Quick and dirty implementation: SELECT * FROM i_s, convert columns to UNTYPED metrics without labels ▪ Something in between 7 © 2017 Percona

16.Metrics mysql_perf_schema_table_io_waits_seconds_total {name="sbtest1",operation="insert",schema="innodb"} 8 © 2017 Percona

17.Metric Names: Namespaces ▪ Exporter name, service name, or some common namespace ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ mysql_up ▪ go_gc_duration_seconds, go_goroutines ▪ process_cpu_seconds_total, process_open_fds ▪ scrape_duration_seconds, scrape_samples_scraped 9 © 2017 Percona

18.Metric Names: Main Part 10 © 2017 Percona

19.Metric Names: Main Part ▪ mysql_perf_schema_table_io_waits_seconds_total 10 © 2017 Percona

20.Metric Names: Main Part ▪ mysql_perf_schema_table_io_waits_seconds_total 10 © 2017 Percona

21.Metric Names: Main Part ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ Exactly one subsystem 10 © 2017 Percona

22.Metric Names: Main Part ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ Exactly one subsystem ▪ Should represent the same logical thing-being-measured across all label dimensions 10 © 2017 Percona

23.Metric Names: Main Part ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ Exactly one subsystem ▪ Should represent the same logical thing-being-measured across all label dimensions ▪ As a rule of thumb, aggregation over all label dimensions of a given metric should be meaningful (though not necessarily useful) 10 © 2017 Percona

24.Metric Names: Main Part ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ Exactly one subsystem ▪ Should represent the same logical thing-being-measured across all label dimensions ▪ As a rule of thumb, aggregation over all label dimensions of a given metric should be meaningful (though not necessarily useful) ▪ Extra care must be taken if names (and types, and descriptions) are generated programmatically (IOWaits -> i_o_waits) 10 © 2017 Percona

25.Metric Names: Units 11 © 2017 Percona

26.Metric Names: Units ▪ mysql_perf_schema_table_io_waits_seconds_total 11 © 2017 Percona

27.Metric Names: Units ▪ mysql_perf_schema_table_io_waits_seconds_total 11 © 2017 Percona

28.Metric Names: Units ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ Use base units in plural form 11 © 2017 Percona

29.Metric Names: Units ▪ mysql_perf_schema_table_io_waits_seconds_total ▪ Use base units in plural form ▪ Ratios, not percentages (but two counters are even better) 11 © 2017 Percona