influx-spout 2.1
Posted on
influx-spout 2.1 has just been released and it includes a bunch of exciting new features. Here's the highlights...
Downsampler
The biggest addition is a new "downsampler" component which is useful for creating rolled up versions of metrics streams for long term archive. Generating rolled up versions of measurements in real-time is more straightforward than trying to create them from already stored measurements later, especially at large volumes. This approach also eliminates the extra load on the short-term database caused by batch rollup operations.
The downsampler reads measurements once they have been processed by influx-spout's filter and averages them over a configured period. The averaged measurements are emitted at the end of each sampling period at clean time boundaries. For example if the sampling period is 10 minutes then measurements will be emitted at 0 minutes, 10 minutes, 20 minutes and so on past the hour.
Downsampled lines are emitted for consumption by an influx-spout writer which will deliver them to an InfluxDB instance.
New "tags" filter rule type
influx-spout's filter component now supports a new filter rule type which allows matching against one or more tags in measurement lines. Using this rule type is much faster than using regex rules to achieve the same kinds of matches - 80-1000% faster depending on number tags to match and line size. This approach is also much safer as matching is independent of tag order and matches tag keys and values precisely.
Tag sorting
The filter component now orders tag keys in all measurement lines passing through it. This ensures the best performance when measurements are inserted into InfluxDB. Predictable tag ordering is also required by the downsampler.
Filter performance
Various optimisations have been made in the filter to avoid unnecessary processing and allocations around line unescaping.
The hashes of measurement names are now only computed once which speeds up matching when multiple "basic" rules are in use.
Get It!
If you or your organisation is a power user of InfluxDB you should really take a look at influx-spout. Version 2.1 is available via the project's Releases page on Github.
Also, if you're into Go development and performance sensitive software, influx-spout's code is worth studying.