Fluentd Blog

Fluentd v0.14.0 has been released

Hi users!

We have released Fluentd version 0.14.0, which is the first release of major version 0.14, including many improvements and new APIs. This post shows some of major changes. See ChangeLog for the full list of changes.

Windows support

Fluentd v0.14 core supports Windows environment! Process management layer of Fluentd v0.14.0 was rewritten with ServerEngine to make it possible to spawn processes in Windows environments.

Some plugins might not work well with Windows services, environments or file systems. Please send us your feedback!

Sub-second time

In Fluentd v0.12 or earlier, time was represented as seconds from Epoch. Events generated in a second had same time. Currently, Fluentd v0.14 generates time in nanosecond resolution. 3rd party plugins which uses Fluent::Engine.now also generates these new time objects without any updates in v0.14.

This new time object behaves as Integer for compatibility with v0.12. If you want to get sub-second data from these time, update your plugins to call time.nsec to get sub-second values (and write it to anywhere you want).

This feature is implemented only on Fluentd v0.14, and Fluentd v0.12 will crash if it receives binary data including nanosecond resolution time forwarded from Fluentd v0.14. So users should be sure to specify time_as_integer true in configuration for forward output plugin if you want to transfer data from Fluentd v0.14 to Fluent v0.12 (It is true in default at v0.14.0, but will be turned into false in default at any point of future).

New Plugin APIs

Fluentd v0.14 has brand-new APIs for input, output, filter, parser, formatter and buffer plugins. The new APIs can make plugin authors to write/maintain their own plugins much more easily and safely. All new APIs uses Fluent::Plugin::* namespace instead of Fluent::*, and "Plugin Helpers" are available from plugins based on new APIs.

This new APIs make it possible to:

• implement both of buffered and non-buffered output in a plugin
• support formatting configurations dynamically (e.g., path /my/dest/${tag}/mydata.%Y-%m-%d.log)
• buffer/flush events along with time, tag, any fields in records and any combination of those
• flush buffers anytime (e.g., 1hour, 15minutes or 30seconds) or per bytes or number of records
• commit flushing buffers asynchronously
• create threads, event loops, child processes managed by Fluentd core

These features are enabled for plugins which are implemented on v0.14 plugin APIs. See douments of each plugins for details.

Storage Plugin

Storage plugin is a plugin type newly added, which is to provide functionality to store key-value pairs for other plugins (Input, Output and Filter). Fluentd v0.14.0 has built-in 'local' plugin, to store values as JSON on local storages.

As same with other plugins, 3rd party plugin authors can publish plugins which use any file systems or database systems as backend (e.g., redis, DynamoDB, RDBMS).

Compatibility layers

Fluentd v0.14 has compatibility layers to load/execute existing Input/Output/Filter plugins which uses v0.12 APIs. These plugins and configurations SHOULD work well without any modification on v0.14. If anything doesn't work, it is a bug. Please let us know about it on issues.

Bad news: Buffer plugin APIs were completely rewritten, and we couldn't make it compatible with v0.12's one. Existing buffer plugins in v0.12 way will not work on Fluentd v0.14.

Enforcing to use router

Fluentd v0.12 had introduced a feature to control data flow using <label> sections, and plugins were expected to support it by calling router.emit instead of Fluent::Engine.emit when emitting events into Fluentd core. But some of plugins doesn't support this feature yet. Users cannot use <label> feature with such plugins.

Fluentd v0.14 disables Fluent::Engine.emit API and raises error. It is to make it sure to enable <label> feature with all plugins. Please report on each repository of plugins if any plugin raises error with message "BUG: use router.emit instead of Engine.emit".

Supported Ruby version

Fluentd v0.14 supports Ruby 2.1 or later (2.2, 2.3). Ruby 2.0 is not maintained and supported by Ruby developer community (link). Use newer versions.

Plans for v0.14.x

We have plans to add some fascinating features at versions in v0.14.x. Here is the list in that plan:

• Symmetric multi process execution to use multi core
• Socket/Server plugin helpers
• Counter API
• TLS and authentication/autorization support in forward plugins
• Plugin generator and templates

Looking forward to introduce these features to make the world more happy. Enjoy logging!