Fluentd Blog

Fluentd v0.12.19 has been released

Hi users!

We have released Fluentd version 0.12.19. Here are the changes:

New features / Enhancement

• input: Standardize logging level for source configs: 743
• buffer: Add buffer_queue_full_action parameter: 745
• parser/formatter: Use oj when oj is installed in json parser/formatter: 748

Bug fixes

• fluent-cat: Fix the regression of --msgpack option: 747

buffer_queue_full_action parameter in buffers

By default, Fluentd's buffer raises BufferQueueLimitError when buffer queue becomes full. This is fit for streaming manner, e.g. in_forward returns error response to out_forward and out_forward can handle such error.

On the other hand, blocking action instead of raising error is sometimes useful, e.g. in_tail and out_elasticsearch for mini-batch use.

To support above 2 cases, we introduce buffer_queue_full_action parameter in buffers. Currently, there are 2 values.

buffer_queue_full_action exception # default. existing behaviour
buffer_queue_full_action block # block input processing

If you set buffer_queue_full_action block in buffered output configuration, input blocks emit records until buffer full is resolved.

We support buffer_queue_full_action but we don't recommend to use block action first. Please consider improving destination setting to resolve BufferQueueLimitError or use @ERROR label for routing overflowed events to another backup destination. If you sometimes hit BufferQueueLimitError, it means your destination capacity is not fit for your traffic.

Lastly, we are now considering back-pressure support in the future. It will resolve above problem smartly.

Use oj for performance improvement when it is installed

oj is very fast json library in Ruby. Fluentd uses yajl by default but if you install oj in your rubygems, Fluentd uses oj instead of yajl for better parsing / formatting performance. Using oj is 1.5x faster in in_tail case(See PR).

In our tests, there are no big differences between oj and yajl. But if you hit some troubles or want to keep to use yajl, set json_parser yajl in your plugin configuration.

Lastly, v0.12.19 docker image has also been available on Docker Hub.

Happy logging!