Rabbit Slide Show

Plazma OSS Day slide

Description

Plazma OSS Day 2018/2/15 に使用したスライドです。

Text

Page: 1

Cooperative works
for
Fluentd Community
Kenji Okimoto
Hiroshi Hatake
ClearCode Inc.
2018-02-15
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 2

Summary
✓ kafka-connect-fuentd and its
benckmarking result
✓ maintaining fuentd plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 3

kafka-connect-fuentd?
✓ Fluentd Forward Protocol Server
implemented with Kafka Connect
API
✓ kakfa-connect-fuentd
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 4

kafka-connect-fuentd?
✓ Use infuent for server
implementation
✓ Partial SSL/TLS support
✓ Aim to replace output plugins in
fuent-plugin-kafka
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 5

Kafka Connect?
Kafka Connect is a framework included in Apache
Kafka that integrates Kafka with other systems.Its
purpose is to make it easy to add new systems to your
scalable and secure stream data pipelines.
[「Kafka Connect」より引用]
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 6

Kafka Connect?
✓ Kafka Connector Source ->
Kafka Producer
✓
I'll talk about this implementation.
✓ Kafka Connector Sink ->
Kafka Consumer
✓
This implementation also exists.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 7

fuent-plugin-kafka
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 8

kafka-connect-fuentd
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 9

Difference
language
Run on
Cooperative works for Fluentd Community
kafka-connect-
fuentd fuent-plugin-kafka
Java Ruby
Kafka side Fluentd side
Powered by Rabbit 2.2.1

Page: 10

Requirements of benchmarking
✓ Easy to use benchmark test tool
✓ Reproducible
✓ Compare performance with same basis
✓
Easy to confrm results
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 11

Easy to use benchmarking test tool
✓ There is no tool which is easy to
use from command line.
✓
Dummer + in_tail exist, but it is hard to apply high load.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 12

Crated benchmark test tool
✓ fuent-benchmark-client
✓ Implemented by Kotlin language
✓ Sending log relies on fuency
✓ SSL/TLS is not supported for now.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 13

Reproducible
✓ Built with Terraform + Ansible
✓ Developing specifc branch on fuentd-
benchmark
✓ Need to tidy up implementation
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 14

Compare performance
with same basis
✓ kafka-fuent-metrics-reporter
✓
Kafka plugin which sends Kafka metrics into Fluentd
✓ Write a script which parses result of pidstat
and sends into Fluentd
✓
To measure CPU and memory usage
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 15

Benchmark environment
GCP n1-standard-2 (vCPUx2, memory 7.5GB)
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 16

out_kafka
CPU usage
10000 events/sec
out_kafka CPU usage is 40-60%
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 17

kafka-connect-fuentd
CPU usage
10000 events/sec
kafka-connect-fuentd CPU usage is less than 20%
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 18

out_kafka_buffered
30000 events/sec
out_kafka_buffered cannot process 30k events/sec
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 19

kafka-connect-fuentd
50000 events/sec
kafka-connect-fuentd can process about 50k events/sec
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 20

Benchmark results
✓ kafka-connect-fuentd can handle more
50,000 events/sec
✓ output plugin of fuent-plugin-kafka can
handle 10,000 events/sec
✓ cannot handle 30,000 events/sec in this environment
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 21

Fluentd Community
Fluentd community has lots of plugins
Over 700+ plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 22

Fluentd Community
✓ Fluentd community has...
✓ lots of up-to-date plugins
✓ lots of outdated plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 23

Fluentd Community
✓ Fluentd community has...
✓ lots of up-to-date plugins
✓ lots of outdated plugins
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 24

How does handle outdated
plugins?
✓ Send PRs
✓ Report issues
✓ Become a maintainer
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 25

What means outdated?
✓ out-of-date dependencies
✓ c.f. client libraries
✓ out-of-date class inheritance
✓ Fluent::Input, Fluent::HttpInput etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 26

What means outdated?
✓ Using old API
✓ Gap between v0.12 and v1.0 API
✓ Fluent::Engine.emit directly
✓ Lack of requires etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 27

Maintaining Plugins
✓ Normally, same as other rubygems
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 28

Maintaining Plugins
✓ Report issues
✓ Send PRs
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 29

Maintaining Plugins
✓ 130+ PRs still opened...
✓ 1400+ PRs merged
✓ in 2.5 years
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 30

Maintaining Plugins
✓ But, sometimes plugin authors are
busy....
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 31

Let's take over
Fluentd plugins project
✓ Ask plugin authors
✓ to add project collaborator
✓ to add gem owner
✓ Create a cooperative working
organization
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 32

Become project collaborator
✓ Maintaining only
✓ takus/fuent-plugin-ec2-metadata
✓ y-ken/fuent-plugin-anonymizer
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 33

Become project collaborator and gem owner
✓ Still exists in original place, but
maintaining ourselves
✓
✓ uken/fuent-plugin-elasticsearch
✓ fuent/fuent-plugin-rewrite-tag-flter
y-ken/fuent-plugin-geoip
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 34

Create cooperative working organization
✓
✓
✓
Take over outdated fuentd plugin project
New plugins which will be widely used
✓
c.f. fuent-plugin-concat
And some collaborations
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 35

Create cooperative working organization
✓ fuent-plugins-nursery
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 36

fuent-plugins-nursery
✓ For Fluentd plugins that are not maintained
by original authors
✓ Aim to provide a sustainable maintenance
system for Fluentd community
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 37

fuent-plugins-nursery
✓ Almost plugins are taken over
maintaining from original author.
✓ fuent-plugin-map
✓ fuent-plugin-redis
✓ fuent-plugin-irc etc.
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Page: 38

fuent-plugins-nursery
✓ Let's maintain fuentd plugins by
community!
Cooperative works for Fluentd Community
Powered by Rabbit 2.2.1

Other slides