Plugins extend Fluentd’s functionality.
The user can use the standard plugins packaged with Fluentd, or can write their own plugins. For further information on writing additional plugins, please refer to Writing plugins.
For Fluentd to function properly, the user must enable at least one input plugin and at least one output plugin using the configuration file. For further information on the configuration file, please refer to Configuration File.
Fluentd has 3 types of plugins:
- Input plugin
Defines an input source of events.
An input plugin typically creates a thread and a listen socket. It can also be written to periodically pull data from data sources. Please note that Fluentd does not restrict the user from creating input plugins with alternative implementations.
- Output plugin
Defines an output destination for events.
An output plugin is typically buffered. A buffered output plugin instructs Fluentd to accumulate event logs in a buffer before writing chunks of data out to a file or network. The buffer’s behavior is defined by a separate buffer plugin. Different buffer plugins can be chosen for each output plugin.
Some output plugins are fully customized and do not use buffers.
- Buffer plugin
Defines the implementation of a buffer.
The user should choose the appropriate buffer plugin for the desired combination of performance and reliability.
The http input plugin allows Fluentd to listen to HTTP clients. The URL becomes the tag of the Fluentd event log, and the POSTed body element becomes the record of the Fluentd event log.
| Parameter | Description | Required? |
|---|---|---|
| json | body of the event in JSON format | either ‘msgpack’ or ‘json’ |
| msgpack | body of the event in MessagePack format | either ‘msgpack’ or ‘json’ |
| time | time of the event in integer (UNIX time) | no |
example:
POST /myapp.access HTTP/1.1
Content-Length: 21
Content-Type: application/x-www-form-urlencoded
json={"event":"body"}
configuration:
<source>
type http
port 9880
bind 0.0.0.0
body_size_limit 32m
keepalive_timeout 10s
</source>
..I think an example would help. The tail input plugin allows Fluentd to read events from the tail of text files. Its read behavior is analogous to the tail -f command.
configuration:
<source>
type tail
path /var/log/httpd-access.log
tag apache.access
format apache
</source>
Format of the log. It’s name of a template or regexp surround by ‘/’.
Regexp must have at least one named captures (?<NAME>PATTERN). If the regexp has capture named ‘time’, it is used as a time of the event. You can specify format of the time using time_format parameter. If the regexp has capture named ‘tag’, tag parameter + captured tag is used as the tag of the event.
Following templates are supported:
Reads apache’s log file host, user, time, method, path, code, size, referer and agent fields. This template is same as following configuration:
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
time_format %d/%b/%Y:%H:%M:%S %z
Reads syslog’s output file (e.g. /var/log/syslog) time, host, ident, message fields. This template is same as following configuration:
format /^(?<time>[^ ]* [^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?[^\:]*\: *(?<message>.*)$/
time_format %b %d %H:%M:%S
forward input plugin listens TCP socket to receive event stream, and also listens UDP socket to receive heartbeat message. This is used to receive event logs from other fluentd, fluent-cat command or client libraries.
configuration:
<source>
type forward
port 24224
bind 0.0.0.0
</source>
This plugin uses MessagePack for the protocol:
stream:
message...
message:
[tag, time, record]
or
[tag, [[time,record], [time,record], ...]]
example:
["myapp.access", [1308466941, {"a"=>1}], [1308466942, {"b"=>2}]]
exec input plugin executes external program to receive or pull event logs. This reads TSV (tab separated values) from the stdout of the program.
You can run the program periodically or permanently. To run periodically, use run_interval parameter.
configuration:
<source>
type exec
command cmd arg arg
keys k1,k2,k3
tag_key k1
time_key k2
time_format %Y-%m-%d %H:%M:%S
run_interval 10s
</source>
Most of output plugins are buffered which accumulates new events on memory or files.
The structure of the buffer is a queue of chunks like following:
queue
+---------+
| |
| chunk <-- write events to the top chunk
| |
| chunk |
| |
| chunk |
| |
| chunk --> write out the bottom chunk
| |
+---------+
When chunk size exceeds limit (buffer_chunk_limit) or specified time elapsed (flush_interval), new empty chunk is pushed. The bottom chunk is written out immediately when new chunk is pushed.
If it failed to write, the chunk is left in the queue and retried to write after seconds (retry_wait). If the retry count is exceeds limit (retry_limit), the chunk is trashed. The wait time before retrying increases twice and twice (1.0sec, 2.0sec, 4.0sec, ...). If the length of the queue exceeds limit (buffer_queue_limit), new events are rejected.
All buffered output plugins supports following parameters described above:
<match pattern>
buffer_type memory
buffer_chunk_limit 256m
buffer_queue_limit 128
flush_interval 60s
retry_limit 17
retry_wait 1s
</match>
buffer_type specifies the type of buffer plugin. Default is memory.
Suffixes “s” (seconds), “m” (minutes), “h” (hours) can be used for flush_interval and retry_wait. retry_wait can be a decimal.
Suffixes “k” (KB), “m” (MB), “g” (GB) can be used for buffer_chunk_limit.
file buffered output plugin writes events to files. By default, it writes into the file in daily basis (almost at 00:10). Before that, no files are created. If you want to output the logs hourly or minutely, please modify ‘time_slice_format’ value.
configuration:
<match pattern>
type file
path /var/log/fluent/myapp
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%dT%H%M%S%z
compress gzip
utc
</match>
| %Y | Year with century |
| %m | Month of the year (01..12) |
| %d | Day of the month (01..31) |
| %H | Hour of the day, 24-hour clock (00..23) |
| %M | Minute of the hour (00..59) |
| %S | Second of the minute (00..60) |
Default is %Y%m%d which splits files every day. Use %Y%m%d%H to split files every hour.
Note that this output plugin uses file buffer by default.
forward buffered output plugin forwards events to other fluentd servers.
This plugin supports load-balancing and automatic fail-over (a.k.a. active-active backup). If you want replication, use copy plugin described below.
It detects fault of a server using “φ accrual failure detector” algorithm. You can customize parameter of the algorithm.
When a fault server recovers, the plugin makes it available automatically after several seconds.
configuration:
<match pattern>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 8
hard_timeout 60s
<server>
name myserver1
host 192.168.1.3
port 24224
weight 60
</server>
<server>
name myserver2
host 192.168.1.4
port 24224
weight 60
</server>
...
<secondary>
type file
path /var/log/fluent/forward-failed
</secondary>
</match>
The exec buffered output plugin passes events to an external program as a tab-separated value (TSV) file.
The command is passed the location of a TSV file containing incoming events as its last argument.
configuration:
<match pattern>
type exec
command cmd arg arg
keys k1,k2,k3
tag_key k1
time_key k2
time_format %Y-%m-%d %H:%M:%S
</match>
exec_filter buffered output plugin executes an external program, using events as input and reading new events from the program output.
It passes tab-separated values (TSV) to stdin and reads TSV from stdout.
configuration:
<match pattern>
type exec_filter
command cmd arg arg
in_keys k1,k2,k3
out_keys k1,k2,k3,k4
tag_key k1
time_key k2
time_format %Y-%m-%d %H:%M:%S
</match>
copy output plugin copies events to multiple outputs.
configuration:
<match pattern>
type copy
<store>
type file
path /var/log/fluent/myapp1
...
</store>
<store>
...
</store>
<store>
...
</store>
</match>
roundrobin output plugin distributes events to multiple outputs using round-robin algorithm.
configuration:
<match pattern>
type roundrobin
<store>
type tcp
host 192.168.1.21
...
</store>
<store>
...
</store>
<store>
...
</store>
</match>
stdout output plugin prints event to the console.
configuration:
<match pattern>
type stdout
</match>
This output plugin is for debugging.
memory buffer plugin provides fast buffer implementation. It uses memory to store buffer chunks. Buffered events which can’t be written soon are deleted when fluentd is shut down.
configuration:
<match pattern>
buffer_type memory
</match>
file buffer plugin provides persistent buffer implementation. It uses file to store buffer chunks.
configuration:
<match pattern>
buffer_type file
buffer_path /var/log/fluent/myapp.*.buffer
</match>
You can find plugins released on RubyGems at the Fluentd plugins page.
You can also use following command to search plugins:
$ fluent-gem search -rd fluent-plugin
Type following command to install it:
$ sudo fluent-gem install fluent-plugin-scribe
Next step: Writing plugins