Fluentd Blog

Fluentd v0.14.7 has been released

Hi users!

We have just shipped Fluentd v0.14.7. It's the release after a month from last release, including some major feature updates and bug fixes.

Here are major changes (full ChangeLog is available here):

Data compression support in buffers and forward plugin

Now buffer plugins support to write events into chunk as compressed data! It can save memory/disk spaces (especially under retrying situation), using additional CPU time. This feature is available with all buffer plugin types (both of memory and file).

<match mydata.*>
  @type any_output_plugin
  <buffer>
    @type memory
    compress gzip
    chunk_limit_size 16m  # 16MB compressed events
    chunk_total_size 512m # 512MB compressed events in total
  </buffer>
</match>

Fluentd compresses events before writing these into buffer chunk, and extract these data before passing these to output plugins. Output plugins can handle these data without any modification. After some very rough evaluation, we expect that compressed buffer consumes about 30% of memory/disk space against non-compressed buffers. It's larger than we expects for normal gzip, because Fluentd's buffer compression are done on relatively small amount of data units.

Moreover, forward input/output plugins also supports compression, to trasfer data in compressed format.

# In edge/forwarder node
<match mydata.*>
  @type forward
  compress gzip
  <server>
    host myserver.local # this Fluentd must be v0.14.7 or later!
    port 24224
  </server>
  # buffer compression is automatically turned on.
</match>

# In aggregator node
<source>
  @type forward
  port 24224
</source>

Forward output plugin automatically turn buffer compression on, and transfer the raw (compressed) bytes to destination (forward input plugin) with compressed option in forward protocol.

The common and symmetric time parser/formatter parameters

There were many fragmented configuration parameters to control time format and timezone. At v0.14.7, these parameters were unified and just a set of parameters are provided in formatter, parser and input/output/filter plugins which supports inject/extract plugin helpers.

Available configuration parameters are:

• time_type: type of time representation [string, unixtime, float]
• time_format: time format of string representation (like %Y-%m-%d %H:%M:%S.%N %z)
• localtime: boolean value to specify to use local time zone
• utc: boolean value to specify to use UTC
• timezone: string to specify timezone, in any format of [+-]HH:MM, [+-]HHMM, [+-]HH or Region/Zone (e.g., Asia/Tokyo)

These parameters are available in all parser/formatter plugins (if that plugin supports time parsing/formatting).

Log rotation support about Fluentd's log file

Two new command line options were introduced to make it possible to rotate logs of Fluentd itself!

-o, --log PATH                   log file path
    --log-rotate-age AGE         generations to keep rotated log files
    --log-rotate-size BYTES      sets the byte size to rotate log files

There is -o(--log) option for a long time to write Fluentd's log into a file, but it can't be rotated automatically (There were just one way to do it - use SIGUSR1). On Fluentd v0.14.7, you can use --log-rotate-age and/or --log-rotate-size to rotate log files per specified size, and leave old log files within specified ages. Log rotation is enabled when at least one of these parameters are specified: --log-rotate-age(5 if not specified), --log-rotate-size(1MB if not specified).

Secondary plugin to dump backup data, and a tool to load it

When buffered output plugin continues to fail writing data to destination, we can use <secondary> section to dump that data as backup... but there were no best practice or good way to open/read/modify these dumped files.

Fluentd v0.14.7 has a new output plugin, out_secondary_file, which is only for secondary output. That plugin has just few configuration parameters, to dump buffered data to a specified directory.

<match failing.data>
  @type forward
  <buffer tag>
    flush_interval 1s
  </buffer>
  <server>
    host unreachable.destination.local
  </server>
  <secondary>
    @type secondary_file
    directory /my/backup/${tag}
    # basename dump.bin # expected full path is /my/backup/failing.data/dump.bin.0
    # append   false
    # compress text     # or gzip
  </secondary>
</match>

When forward plugin fails to send data in enough times, Fluentd uses secondary_file plugin to write buffered data into /my/backup/failing.data directory.

The format of dumped data depends on the format method of primary output plugin. If the primary output plugin formats data into JSON, you can get a JSON object per lines. But many plugin uses Fluentd's standard format, MessagePack, as a format of buffer chunk data. It's just binary data, not human readable. Now we have a tool new introduced, fluent-binlog-reader, which is to read such binary data from files, and format to reuse these data in our own scripts or tools.

$ fluent-binlog-reader head dump.bin.0
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}

The default format is just same with stdout output plugin, and all formats supported in Fluentd are available via -f option.

Major bug fixes

• in_tail: fix a bug to crash to read large amount logs #1229
• out_forward: fix a bug that UDP heartbeat doesn't work #1238
• Fix a bug not to stop enqueue/flush threads correctly #1264
• Fix a bug to call shutdown method (and some others) twice #1161, #1222

Enjoy logging!