Text
Page: 1
Cooperative works for Fluentd Community Kenji Okimoto Hiroshi Hatake ClearCode Inc. 2018-02-15 Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 2
Summary ✓ kafka-connect-fuentd and its benckmarking result ✓ maintaining fuentd plugins Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 3
kafka-connect-fuentd? ✓ Fluentd Forward Protocol Server implemented with Kafka Connect API ✓ kakfa-connect-fuentd Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 4
kafka-connect-fuentd? ✓ Use infuent for server implementation ✓ Partial SSL/TLS support ✓ Aim to replace output plugins in fuent-plugin-kafka Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 5
Kafka Connect? Kafka Connect is a framework included in Apache Kafka that integrates Kafka with other systems.Its purpose is to make it easy to add new systems to your scalable and secure stream data pipelines. [「Kafka Connect」より引用] Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 6
Kafka Connect? ✓ Kafka Connector Source -> Kafka Producer ✓ I'll talk about this implementation. ✓ Kafka Connector Sink -> Kafka Consumer ✓ This implementation also exists. Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 7
fuent-plugin-kafka Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 8
kafka-connect-fuentd Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 9
Difference language Run on Cooperative works for Fluentd Community kafka-connect- fuentd fuent-plugin-kafka Java Ruby Kafka side Fluentd side Powered by Rabbit 2.2.1
Page: 10
Requirements of benchmarking ✓ Easy to use benchmark test tool ✓ Reproducible ✓ Compare performance with same basis ✓ Easy to confrm results Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 11
Easy to use benchmarking test tool ✓ There is no tool which is easy to use from command line. ✓ Dummer + in_tail exist, but it is hard to apply high load. Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 12
Crated benchmark test tool ✓ fuent-benchmark-client ✓ Implemented by Kotlin language ✓ Sending log relies on fuency ✓ SSL/TLS is not supported for now. Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 13
Reproducible ✓ Built with Terraform + Ansible ✓ Developing specifc branch on fuentd- benchmark ✓ Need to tidy up implementation Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 14
Compare performance with same basis ✓ kafka-fuent-metrics-reporter ✓ Kafka plugin which sends Kafka metrics into Fluentd ✓ Write a script which parses result of pidstat and sends into Fluentd ✓ To measure CPU and memory usage Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 15
Benchmark environment GCP n1-standard-2 (vCPUx2, memory 7.5GB) Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 16
out_kafka CPU usage 10000 events/sec out_kafka CPU usage is 40-60% Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 17
kafka-connect-fuentd CPU usage 10000 events/sec kafka-connect-fuentd CPU usage is less than 20% Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 18
out_kafka_buffered 30000 events/sec out_kafka_buffered cannot process 30k events/sec Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 19
kafka-connect-fuentd 50000 events/sec kafka-connect-fuentd can process about 50k events/sec Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 20
Benchmark results ✓ kafka-connect-fuentd can handle more 50,000 events/sec ✓ output plugin of fuent-plugin-kafka can handle 10,000 events/sec ✓ cannot handle 30,000 events/sec in this environment Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 21
Fluentd Community Fluentd community has lots of plugins Over 700+ plugins Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 22
Fluentd Community ✓ Fluentd community has... ✓ lots of up-to-date plugins ✓ lots of outdated plugins Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 23
Fluentd Community ✓ Fluentd community has... ✓ lots of up-to-date plugins ✓ lots of outdated plugins Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 24
How does handle outdated plugins? ✓ Send PRs ✓ Report issues ✓ Become a maintainer Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 25
What means outdated? ✓ out-of-date dependencies ✓ c.f. client libraries ✓ out-of-date class inheritance ✓ Fluent::Input, Fluent::HttpInput etc. Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 26
What means outdated? ✓ Using old API ✓ Gap between v0.12 and v1.0 API ✓ Fluent::Engine.emit directly ✓ Lack of requires etc. Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 27
Maintaining Plugins ✓ Normally, same as other rubygems Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 28
Maintaining Plugins ✓ Report issues ✓ Send PRs Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 29
Maintaining Plugins ✓ 130+ PRs still opened... ✓ 1400+ PRs merged ✓ in 2.5 years Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 30
Maintaining Plugins ✓ But, sometimes plugin authors are busy.... Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 31
Let's take over Fluentd plugins project ✓ Ask plugin authors ✓ to add project collaborator ✓ to add gem owner ✓ Create a cooperative working organization Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 32
Become project collaborator ✓ Maintaining only ✓ takus/fuent-plugin-ec2-metadata ✓ y-ken/fuent-plugin-anonymizer Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 33
Become project collaborator and gem owner ✓ Still exists in original place, but maintaining ourselves ✓ ✓ uken/fuent-plugin-elasticsearch ✓ fuent/fuent-plugin-rewrite-tag-flter y-ken/fuent-plugin-geoip Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 34
Create cooperative working organization ✓ ✓ ✓ Take over outdated fuentd plugin project New plugins which will be widely used ✓ c.f. fuent-plugin-concat And some collaborations Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 35
Create cooperative working organization ✓ fuent-plugins-nursery Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 36
fuent-plugins-nursery ✓ For Fluentd plugins that are not maintained by original authors ✓ Aim to provide a sustainable maintenance system for Fluentd community Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 37
fuent-plugins-nursery ✓ Almost plugins are taken over maintaining from original author. ✓ fuent-plugin-map ✓ fuent-plugin-redis ✓ fuent-plugin-irc etc. Cooperative works for Fluentd Community Powered by Rabbit 2.2.1
Page: 38
fuent-plugins-nursery ✓ Let's maintain fuentd plugins by community! Cooperative works for Fluentd Community Powered by Rabbit 2.2.1