PingCAP-Infra-Meetup-101-wangpenghan-log-observability

本次分享王鹏翰老师首先阐述了 Log 和 observability 是什么,以及建立可观察性(observability)的三个主要手段:logging, metrics, tracing。然后对 Filebeat 的设计架构,实现细节,以及如何对 filebeat 进行合理的监控进行了介绍。最后分享了在生产环境中 50TB/day 日志传输量下的 filebeat 调优细节。
展开查看详情

1.Log, Observability and Filebeat Dashbase 王鹏翰

2.

3.Log ?

4. Raft is a con mana sensu ging a s algo replic r ithm ated l for og. o ra Log ? r Ap pl Au E — ic AS at AB T io DA n E Lo IS TH g OG Binlog E L TH

5. “A log is perhaps the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time.” – Jay Kreps [1] [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

6.Log Immutable Order by time Timestamp + Data

7.Observability ?

8.long long time ago ……

9.Log with Grep

10. Monitor what’s broken, and why? 

11.Monitor StatsD Graphite Count Timing Gauges Sets ——— Samping

12.

13.

14.Monolithic MicroService

15.Cloud and Cloud Native

16.

17.

18.https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

19.LOG

20.Trace

21. Gauge Counter Histogram Metric

22. Spec OpenMetric OpenTracing OpenLog ? Not exists … Elastic Common Schema maybe a little like openLog

23.TSDB Prometheus InfluxDB OpenTSDB

24. Log Logstash Elasticsearch Beats [Filebeat] Loki Fluent FluentBit

25. Trace Zipkin Elastic apm agent Jeager Skywalking

26. Filebeat Lightweight Shipper for Logs

27.Head First Log Shipper Send log stable and fast to a search engine

28.

29.

TiDB 是一款定位于在线事务处理/在线分析处理( HTAP: Hybrid Transactional/Analytical Processing)的融合型数据库产品,实现了一键水平伸缩,强一致性的多副本数据安全,分布式事务,实时 OLAP 等重要特性。