Plazma OSS Day slide - Hiroshi Hatake - Rabbit Slide Show

Cooperative works
for
Fluentd Community
Kenji Okimoto
Hiroshi Hatake
ClearCode Inc.
2018-02-15
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Summary
✓ kafka-connect-fuentd and its
benckmarking result
✓ maintaining fuentd plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd?
✓ Fluentd Forward Protocol Server
implemented with Kafka Connect
API
✓ kakfa-connect-fuentd
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd?
✓ Use infuent for server
implementation
✓ Partial SSL/TLS support
✓ Aim to replace output plugins in
fuent-plugin-kafka
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Kafka Connect?
Kafka Connect is a framework included in Apache
Kafka that integrates Kafka with other systems.Its
purpose is to make it easy to add new systems to your
scalable and secure stream data pipelines.
[「Kafka Connect」より引用]
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Kafka Connect?
✓ Kafka Connector Source ->
Kafka Producer
✓ I'll talk about this implementation.
✓ Kafka Connector Sink ->
Kafka Consumer
✓ This implementation also exists.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugin-kafka
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Difference
language
Run on
Cooperative works for Fluentd Community
kafka-connect-
fuentdfuent-plugin-kafka
JavaRuby
Kafka sideFluentd side
Powered by Rabbit 2.2.1

Requirements of benchmarking
✓ Easy to use benchmark test tool
✓ Reproducible
✓ Compare performance with same basis
✓ Easy to confrm results
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Easy to use benchmarking test tool
✓ There is no tool which is easy to
use from command line.
✓ Dummer + in_tail exist, but it is hard to apply high load.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Crated benchmark test tool
✓ fuent-benchmark-client
✓ Implemented by Kotlin language
✓ Sending log relies on fuency
✓ SSL/TLS is not supported for now.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Reproducible
✓ Built with Terraform + Ansible
✓ Developing specifc branch on fuentd-
benchmark
✓ Need to tidy up implementation
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Compare performance
with same basis
✓ kafka-fuent-metrics-reporter
✓ Kafka plugin which sends Kafka metrics into Fluentd
✓ Write a script which parses result of pidstat
and sends into Fluentd
✓ To measure CPU and memory usage
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Benchmark environment
GCP n1-standard-2 (vCPUx2, memory 7.5GB)
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

out_kafka
CPU usage
10000 events/sec
out_kafka CPU usage is 40-60%
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd
CPU usage
10000 events/sec
kafka-connect-fuentd CPU usage is less than 20%
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

out_kafka_buffered
30000 events/sec
out_kafka_buffered cannot process 30k events/sec
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd
50000 events/sec
kafka-connect-fuentd can process about 50k events/sec
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Benchmark results
✓ kafka-connect-fuentd can handle more
50,000 events/sec
✓ output plugin of fuent-plugin-kafka can
handle 10,000 events/sec
✓ cannot handle 30,000 events/sec in this environment
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Fluentd Community
Fluentd community has lots of plugins
Over 700+ plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Fluentd Community
✓ Fluentd community has...
✓ lots of up-to-date plugins
✓ lots of outdated plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Fluentd Community
✓ Fluentd community has...
✓ lots of up-to-date plugins
✓ lots of outdated plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

How does handle outdated
plugins?
✓ Send PRs
✓ Report issues
✓ Become a maintainer
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

What means outdated?
✓ out-of-date dependencies
✓ c.f. client libraries
✓ out-of-date class inheritance
✓ Fluent::Input, Fluent::HttpInput etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

What means outdated?
✓ Using old API
✓ Gap between v0.12 and v1.0 API
✓ Fluent::Engine.emit directly
✓ Lack of requires etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ Normally, same as other rubygems
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ Report issues
✓ Send PRs
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ 130+ PRs still opened...
✓ 1400+ PRs merged
✓ in 2.5 years
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ But, sometimes plugin authors are
busy....
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Let's take over
Fluentd plugins project
✓ Ask plugin authors
✓ to add project collaborator
✓ to add gem owner
✓ Create a cooperative working
organization
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Become project collaborator
✓ Maintaining only
✓ takus/fuent-plugin-ec2-metadata
✓ y-ken/fuent-plugin-anonymizer
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Become project collaborator and gem owner
✓ Still exists in original place, but
maintaining ourselves
✓ uken/fuent-plugin-elasticsearch
✓ y-ken/fuent-plugin-geoip
✓ fuent/fuent-plugin-rewrite-tag-flter
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Create cooperative working organization
✓ Take over outdated fuentd plugin project
✓ New plugins which will be widely used
✓ c.f. fuent-plugin-concat
✓ And some collaborations
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Create cooperative working organization
✓ fuent-plugins-nursery
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugins-nursery
✓ For Fluentd plugins that are not maintained
by original authors
✓ Aim to provide a sustainable maintenance
system for Fluentd community
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugins-nursery
✓ Almost plugins are taken over
maintaining from original author.
✓ fuent-plugin-map
✓ fuent-plugin-redis
✓ fuent-plugin-irc etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugins-nursery
✓ Let's maintain fuentd plugins by
community!
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Cooperative works
for
Fluentd Community
Kenji Okimoto
Hiroshi Hatake
ClearCode Inc.
2018-02-15
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Summary
✓ kafka-connect-fuentd and its
benckmarking result
✓ maintaining fuentd plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd?
✓ Fluentd Forward Protocol Server
implemented with Kafka Connect
API
✓ kakfa-connect-fuentd
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd?
✓ Use infuent for server
implementation
✓ Partial SSL/TLS support
✓ Aim to replace output plugins in
fuent-plugin-kafka
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Kafka Connect?
Kafka Connect is a framework included in Apache
Kafka that integrates Kafka with other systems.Its
purpose is to make it easy to add new systems to your
scalable and secure stream data pipelines.
[「Kafka Connect」より引用]
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Kafka Connect?
✓ Kafka Connector Source ->
Kafka Producer
✓ I'll talk about this implementation.
✓ Kafka Connector Sink ->
Kafka Consumer
✓ This implementation also exists.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugin-kafka
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Difference
language
Run on
Cooperative works for Fluentd Community
kafka-connect-
fuentdfuent-plugin-kafka
JavaRuby
Kafka sideFluentd side
Powered by Rabbit 2.2.1

Requirements of benchmarking
✓ Easy to use benchmark test tool
✓ Reproducible
✓ Compare performance with same basis
✓ Easy to confrm results
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Easy to use benchmarking test tool
✓ There is no tool which is easy to
use from command line.
✓ Dummer + in_tail exist, but it is hard to apply high load.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Crated benchmark test tool
✓ fuent-benchmark-client
✓ Implemented by Kotlin language
✓ Sending log relies on fuency
✓ SSL/TLS is not supported for now.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Reproducible
✓ Built with Terraform + Ansible
✓ Developing specifc branch on fuentd-
benchmark
✓ Need to tidy up implementation
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Compare performance
with same basis
✓ kafka-fuent-metrics-reporter
✓ Kafka plugin which sends Kafka metrics into Fluentd
✓ Write a script which parses result of pidstat
and sends into Fluentd
✓ To measure CPU and memory usage
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Benchmark environment
GCP n1-standard-2 (vCPUx2, memory 7.5GB)
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

out_kafka
CPU usage
10000 events/sec
out_kafka CPU usage is 40-60%
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd
CPU usage
10000 events/sec
kafka-connect-fuentd CPU usage is less than 20%
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

out_kafka_buffered
30000 events/sec
out_kafka_buffered cannot process 30k events/sec
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

kafka-connect-fuentd
50000 events/sec
kafka-connect-fuentd can process about 50k events/sec
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Benchmark results
✓ kafka-connect-fuentd can handle more
50,000 events/sec
✓ output plugin of fuent-plugin-kafka can
handle 10,000 events/sec
✓ cannot handle 30,000 events/sec in this environment
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Fluentd Community
Fluentd community has lots of plugins
Over 700+ plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Fluentd Community
✓ Fluentd community has...
✓ lots of up-to-date plugins
✓ lots of outdated plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Fluentd Community
✓ Fluentd community has...
✓ lots of up-to-date plugins
✓ lots of outdated plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

How does handle outdated
plugins?
✓ Send PRs
✓ Report issues
✓ Become a maintainer
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

What means outdated?
✓ out-of-date dependencies
✓ c.f. client libraries
✓ out-of-date class inheritance
✓ Fluent::Input, Fluent::HttpInput etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

What means outdated?
✓ Using old API
✓ Gap between v0.12 and v1.0 API
✓ Fluent::Engine.emit directly
✓ Lack of requires etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ Normally, same as other rubygems
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ Report issues
✓ Send PRs
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ 130+ PRs still opened...
✓ 1400+ PRs merged
✓ in 2.5 years
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Maintaining Plugins
✓ But, sometimes plugin authors are
busy....
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Let's take over
Fluentd plugins project
✓ Ask plugin authors
✓ to add project collaborator
✓ to add gem owner
✓ Create a cooperative working
organization
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Become project collaborator
✓ Maintaining only
✓ takus/fuent-plugin-ec2-metadata
✓ y-ken/fuent-plugin-anonymizer
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Become project collaborator and gem owner
✓ Still exists in original place, but
maintaining ourselves
✓ uken/fuent-plugin-elasticsearch
✓ y-ken/fuent-plugin-geoip
✓ fuent/fuent-plugin-rewrite-tag-flter
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Create cooperative working organization
✓ Take over outdated fuentd plugin project
✓ New plugins which will be widely used
✓ c.f. fuent-plugin-concat
✓ And some collaborations
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Create cooperative working organization
✓ fuent-plugins-nursery
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugins-nursery
✓ For Fluentd plugins that are not maintained
by original authors
✓ Aim to provide a sustainable maintenance
system for Fluentd community
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugins-nursery
✓ Almost plugins are taken over
maintaining from original author.
✓ fuent-plugin-map
✓ fuent-plugin-redis
✓ fuent-plugin-irc etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

fuent-plugins-nursery
✓ Let's maintain fuentd plugins by
community!
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

0 / 38