Fluentd Blog

Fluentd v0.12.33 has been released

Hi users!

We have released Fluentd version 0.12.33. Here are the changes:

New features / Enhancement

• parser_syslog: Backport 'Support rfc5424 syslog format' https://github.com/fluent/fluentd/pull/1495
• in_forward: Add source_address_key and fix source_hostname_key parameters https://github.com/fluent/fluentd/pull/1490
• in_tail: Add limit_recently_modified parameter to limit watching files https://github.com/fluent/fluentd/pull/1474
• in_tail: Skip the refresh of watching list on startup https://github.com/fluent/fluentd/pull/1487
• parser: Allow escape sequence in Apache access log https://github.com/fluent/fluentd/pull/1479
• log: Add Fluent::Log#<< to support some SDKs https://github.com/fluent/fluentd/pull/1478

Bug fixes

• config: Set encoding forcefully to avoid UndefinedConversionError https://github.com/fluent/fluentd/pull/1477
• in_tail: Skip directories when use**/* in path https://github.com/fluent/fluentd/pull/1464

parser_syslog: Support RFC5424 format

parser_syslog now supports RFC5424 format. Here is the result of RFC5424 format message:

# Incoming message
<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!
# Parsed result
{"pri" => 16, "host" => "192.168.0.1", "ident" => "fluentd", "pid" => "11111", "msgid" => "ID24224", "extradata" => "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]", "message" => "Hi, from Fluentd!"}

By default, parser_syslog assumes incoming message is RFC3614 format. You can parse RFC5424 format by setting message_format rfc5424 in parser configuration.
In addition, you can use message_format auto to handle two message formats in one stream. parser_syslog detects message format automatically and parse it.

Of course, you can use this feature in in_syslog because in_syslog uses parser_syslog internally.

<source>
  @type syslog
  @id in_syslog
  message_format rfc5424
  tag system.syslog
</source>

in_tail: Add 2 features

We added limit_recently_modified parameter. This is useful when use * in the path. If target files are not updated within limit_recently_modified, such files are ignored from watching list.

Another one is adding skip_refresh_on_startup parameter. By default, in_tail reads file content during startup when you set read_from_head true or after restarted.
The problem is this is done by main thread, so it blocks other startup routines. This is no problem when unread lines are small. But it is huge, consuming lines takes long time. skip_refresh_on_startup parameter delays the watching list update so useful for above case.

Lastly, v0.12.33 docker image has also been available on Docker Hub.

Happy logging!