Fluentd Blog

Fluentd v0.14.2 has been released

Hi users,

After a month and 10 days from release of 0.14.1, we have been released Fluentd version 0.14.2. That version includes many feature updates and bug fixes, which solves many problems reported for 0.14.1. Thank you reporters! However, Fluentd 0.14.2 is not stable version yet. We have some known issues, mainly for Windows environment, and some features to be added in stable Fluentd major version. These will be added in 0.14.3.

Anyway, 0.14.2 has many good things for Fluentd 0.14 users. Please try it!

Here are major changes (full ChangeLog is available here):

Core: Splitting events into some chunks

Fluentd 0.12 (, 0.14.0 and 0.14.1) stores all events in a bunch are stored in a buffer chunk, even if its bunch bytesize is bigger than buffer_chunk_limit. In such cases, buffer_chunk_limit is ignored. This is typically seen when Fluentd log forwarder's buffer_chunk_limit is larger than one of log aggregator node. This behavior may generate many larger buffer chunks than the size output plugin expects, and it may cause any unexpected troubles.

Fluentd 0.14.2 splits such large bunch of events into buffer chunks, according to configured chunk_limit_size. There will be no more chunks larger than our expectation with this improvement, but splitting events into chunks requires a bit additional CPU time. Roughly speaking, splitting chunks consumes 45% more CPU time in worst case (see benchmark result).

The way to avoid this is to configure chunk_limit_size (or buffer_chunk_limit) with the same value in all Fluentd processes, or use smaller configuration value in forwarder processes (using forward output plugin) than aggregator processes (using forward input plugin).

Core: compat_parameters plugin helper now supports parsers and filters

This is the last missing peace to support v0.12 style configurations in v0.14 API plugins. This plugin helper converts traditional style configurations (such like one using format parameter in <source> or <match> sections directly) into v0.14 style configuration (using <parse> @type ... </parse> or <format> @type ... </format>). #1079

Core: New RPC call to do both of flushing buffers and shutdown

A new RPC call /api/processes.flushBuffersAndKillWorkers was added to flush buffers and then shtudown Fluentd process. #1134

parser_csv: New option to specify delimiter

parser_csv is a parser plugin to parse CSV... it was originally "Comma-Separated Values" in good old days, but now it is "Character-Separated Values". The newly added option delimiter is to allow users to specify a delimiter (column separator) for various use cases. #1108

in_syslog: New option to receive longer syslog messages

in_syslog has message_length_limit option now (default is 2048). It's useful for syslog users who transfers web access logs or other large log messages via syslog protocol. #1127

Major bug fixes

• fix a bug to read non-buffer-chunk files as buffer chunks, and Fluentd process crashes #1098 #1124
• fix a bug not to load Filter plugins (even if configuration has <filter> sections) #1118

Enjoy logging!