Text
Page: 1
New Chapter of Fluentd Rebranding and New Release Cycle (LTS) Daijiro Fukuda, Kentaro Hayashi ClearCode Inc. Open Source Summit JAPAN 2023 2023-12-05 New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 2
About us ✓ Maintainers of Fluentd Project ✓ Software Engineers at ClearCode Inc ✓ Driving forward Free Software and Make the business New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 3
Fluentd and us https://github.com/fluent/fluentd/graphs/contributors? from=2021-01-01&to=2023-11-30&type=c New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 4
Agenda ✓ What is Fluentd ✓ History of Fluentd ✓ What’s new Fluent Package LTS ✓ Remarks of updating ✓ Major changes New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 5
Agenda ✓ What is Fluentd ✓ History of Fluentd ✓ What’s new Fluent Package LTS ✓ Remarks of updating ✓ Major changes New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 6
What’s Fluentd Fluentd is an open source data collector that unifies data collection and consumption. New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 7
Image of data collection without Fluentd New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 8
What Fluentd does New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 9
Why use Fluentd? ✓ Flexibility ✓ Scalability ✓ Open source and community New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 10
Flexibility ✓ Over 1000 plugins for many data sources and outputs ✓ By combining plugins, you can achieve what you want Main functions such as data collection, filtering, formatting, and sending are performed by each plugin ✓ You can easily make your plugin with Ruby New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 11
Scalability New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 12
Open source and community ✓ You can join to development and make what you want ✓ Plugins are maintained in the community ✓ https://github.com/fluent-plugins-nursery New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 13
Agenda ✓ What is Fluentd ✓ History of Fluentd ✓ What’s new Fluent Package LTS ✓ Remarks of updating ✓ Major changes New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 14
History of Fluentd 2011 2016 v0.9 Released by Treasure Data Accepted to CNCF 2017 v1.0 Released 2019 Graduated CNCF project New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 15
Needs of distribution package ✓ Fluentd is general purpose ✓ Plugins allow specific features ✓ Ruby is required to use Users want convenience. We need distribution package. New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 16
History of distribution package Year 2011 Package td-agent v1 Fluentd v0.9 2014 td-agent v2 v0.10 2017 td-agent v3 v0.14 2020 td-agent v4 v1.11 2023 fluent- package v5 v1.16 New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 17
Reason of rebranding ✓ The name td-agent does not associate with Fluentd The name of distribution package is better to associate or describe the main software ✓ td-agent is so called all-in-one package, not limited for Treasure Data CDP New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 18
Fluent Package ✓ The new package name reflects community driven development! New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 19
Agenda ✓ What is Fluentd ✓ History of Fluentd ✓ What’s new Fluent Package LTS ✓ Remarks of updating ✓ Major changes New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 20
Fluent Package LTS ✓ v5.0.2 had been released at Nov 2023 ✓ Recommended for stable production use ✓ Scheduled support lifecycle ✓ Security fixes, bug fixes for long term New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 21
Standard and LTS Channel Standard version LTS (Long term support) New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Support lifecycle Every 3 months Every 2 year or so Powered by Rabbit 3.0.4
Page: 22
Planned release cycle https://www.fluentd.org/blog/fluent-package-scheduled-lifecycle New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 23
Profit for updating to LTS ✓ Secure and stable operation! Long Term Support for updates, ready for stable production use ✓ Less hustle for major update! Advanced announcement for scheduled major update Next major update is scheduled in April, 2025 (T.B.D.) New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 24
Agenda ✓ What is Fluentd ✓ History of Fluentd ✓ What’s new Fluent Package LTS ✓ Remarks of updating ✓ Major changes New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 25
How to update from td-agent v4 ✓ In most cases, you just install Fluent Package! ✓ Compatible with td-agent v4! ✓ In some cases, manual migration is necessary New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 26
Compatibility You can still use … ✓ your config files as is ✓ /etc/td-agent/td-agent.conf ✓ commands with their old names ✓ td-agent, td-agent-gem ✓ td-agent user and group New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 27
Manual migration cases ✓ General case ✓ Specific to Linux ✓ Specific to Windows See https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5 for details! New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 28
Manual migration: general ✓ Manually installed plugins ✓ Reinstall those plugins manually ✓ Setting of monitoring Fluentd’s log file ✓ Update the filepath /var/log/fluent/fluentd.log New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 29
Manual migration: Linux ✓ Environment file ✓ ✓ RHEL: /etc/sysconfig/fluentd Debian: /etc/default/fluentd ✓ Log rotate configuration ✓ /etc/logrotate.d/fluentd ✓ Systemd unit file New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 30
Systemd unit file Update custom unit files (If you use) e.g. /etc/systemd/system/td-agent.service ✓ Update Fluentd bin path ✓ /opt/td-agent/bin/fluentd -> /opt/fluent/bin/fluentd ✓ Update environmental variables ✓ ✓ LD_PRELOAD: /opt/fluent/lib/libjemalloc.so GEM_HOME, GEM_PATH: /opt/fluentd/lib/ruby/gems/3.2.0/ New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 31
Manual migration: Windows ✓ Command line arguments (If you use) ✓ Set them again $ fluentd –reg-winsvc-fluentdopt … ✓ Automatic start as Windows Service ✓ Enable it manually if you need $ sc config fluentdwinsvc start=delayed-auto New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 32
Fluentd’s log rotation setting on Windows ✓ No longer need to use command line argument, --log-rotate-age, --log- rotate-size Since td-agent v4.4.2, you can use system configuration ✓ Use system configuration https://docs.fluentd.org/deployment/logging#log-rotation-setting New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 33
Agenda ✓ What is Fluentd ✓ History of Fluentd ✓ What’s new Fluent Package LTS ✓ Remarks of updating ✓ Major changes New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 34
for Fluent Command name ✓ Changes Package If updated from td-agent, you can still use the old name ✓ td-agent => fluentd, td-agent-gem => fluent-gem ✓ Linux: systemd unit name You can still use the old name ✓ td-agent => fluentd Windows: no automatic start New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 35
Improvements ✓ Fluentd: Fix in_tail bug ✓ Many other improvements! See https://github.com/fluent/fluentd/blob/master/CHANGELOG.md! ✓ Fluent Package: Update embedded Ruby to 3.2 from 2.7 ✓ Many other improvements! See https://github.com/fluent/fluent-package-builder/blob/master/ CHANGELOG.md! Powered by Rabbit 3.0.4 New Chapter of Fluentd - Rebranding and New Release Cycle (LTS)
Page: 36
Fix in_tail bug https://www.fluentd.org/blog/fluentd-v1.16.2-v1.16.3-have-been-released New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 37
Fix in_tail bug ✓ Finally, the bug is fixed! ✓ With collaboration of many users, the cause was found, and it is fixed! New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 38
Fix in_tail bug New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 39
Fix in_tail bug New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 40
Update embedded Ruby to 3.2 ✓ You can try Ruby’s performance improvements YJIT! https://docs.fluentd.org/deployment/system-config#enable_jit- experimental ✓ It is still experimental in Fluentd Please try it and give us feedback! New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 41
Fluent Package and future ✓ What happened? ✓ What will happen? New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 42
The fact from statistics (1) ✓ Recent Top 5 requests: ✓ United States (44.74%) ✓ Netherlands (9%) ✓ Japan (6.79%) ✓ France (5.96%) ✓ Ireland (5.58%) New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 43
The fact from statistics (2) Recent trends: ✓ Top 5 location Before LTS release ✓ United States,Netherlands,Japan,France,Ireland ✓ Top 5 location After LTS release ✓ United States,Netherlands,Japan,France,Taiwan New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 44
The fact from statistics (3) ✓ The undesirable facts: ✓ The majority of td-agent users: 3.x (unmaintained) or 4.x series. ✓ There is still an access for 2.x (unmaintained) New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 45
The fact from statistics (4) ✓ The most widely used package: ✓ td-agent 4.x (aarch64) for Amazon Linux 2. New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 46
The fact from statistics (5) ✓ GitHub Discussions low activity New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 47
The future ✓ No more old package users ✓ Renewal about distribution channel (Sponsored by CNCF?) ✓ Encourage users to be a fluentd supporter Join GitHub discussions! ✓ Continuous release New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
Page: 48
Wrap up ✓ Interested in and understand Fluent Package LTS! ✓ Encouraged switching to fluent- package LTS further more! ✓ Encouraged joining Fluentd community! New Chapter of Fluentd - Rebranding and New Release Cycle (LTS) Powered by Rabbit 3.0.4
