[[releasenotes]] == Release Notes This section summarizes the changes in the following releases: * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> [[logstash-8-16-1]] === Logstash 8.16.1 Release Notes [[notable-8-16-1]] ==== Notable issues fixed * PipelineBusV2 deadlock proofing: We fixed an issue that could cause a deadlock when the pipeline-to-pipeline feature was in use, causing pipelines (and consequently) {ls} to never terminate https://github.com/elastic/logstash/pull/16680[#16680] ==== Plugins *Elastic_integration Filter - 0.1.16* * Reflect the Elasticsearch GeoIP changes into the plugin and sync with Elasticsearch 8.16 branch https://github.com/elastic/logstash-filter-elastic_integration/pull/170[#170] *Xml Filter - 4.2.1* * patch rexml to improve performance of multi-threaded xml parsing https://github.com/logstash-plugins/logstash-filter-xml/pull/84[#84] *Beats Input - 6.9.1* * Upgrade netty to 4.1.115 https://github.com/logstash-plugins/logstash-input-beats/pull/507[#507] *Http Input - 3.9.2* * Upgrade netty to 4.1.115 https://github.com/logstash-plugins/logstash-input-http/pull/183[#183] *Tcp Input - 6.4.4* * Upgrade netty to 4.1.115 https://github.com/logstash-plugins/logstash-input-tcp/pull/227[#227] *Http Output - 5.7.1* * Added new development `rackup` dependency to fix tests [[logstash-8-16-0]] === Logstash 8.16.0 Release Notes [[known-issues-8-16-0]] ==== Known issue **{ls} may fail to shut down under some circumstances when you are using <>.** Check out issue https://github.com/elastic/logstash/issues/16657[#16657] for details. Workaround: Add `-Dlogstash.pipelinebus.implementation=v1` to `config/jvm.options`. This change reverts the `PipelineBus` to `v1`, a version that does not exhibit this issue, but may impact performance in pipeline-to-pipeline scenarios. [[health-api-8-16-0]] ==== Announcing the new {ls} Health Report API The new Health Report API (`GET /_health_report`) is available starting with {ls} `8.16.0`. This API uses indicators capable of detecting the degraded status of pipelines and providing actionable insights https://github.com/elastic/logstash/pull/16520[#16520], https://github.com/elastic/logstash/pull/16532[#16532]. **Upgrading from earlier versions.** If your existing automation relies on liveliness scripts that expect the {ls} API status to be unavailable or to return a hardcoded `green` status, you can set a property to preserve pre-8.16.0. To maintain existing behavior for API responses, add the `-Dlogstash.forceApiStatus=green` property to your `config/jvm.options` file. This setting prevents the new Health API status from affecting the top-level `status` field of existing {ls} API responses, forcing other APIs to return the previous hard-coded `green` value. https://github.com/elastic/logstash/pull/16535[#16535] Check out the <> docs more for info. [[featured-8-16-0]] ==== New features and enhancements * {ls} now gracefully handles `if` conditionals in pipeline definitions that can't be evaluated (https://github.com/elastic/logstash/pull/16322[#16322]), either by dropping the event or by sending it to the pipeline's DLQ if enabled. https://github.com/elastic/logstash/pull/16423[#16423] [[core-8-16-0]] ==== Other changes to Logstash core * Added deprecation logs for modules `netflow`, `fb_apache` and `azure`. https://github.com/elastic/logstash/pull/16548[#16548] * Added deprecation logs for users that doesn't explicitly select a value for `pipeline.buffer.type` forcing them to proactively make a choice before version `9.0` when this setting will default to heap. https://github.com/elastic/logstash/pull/16498[#16498] * The flag `--event_api.tags.illegal` was deprecated and will be removed in version 9. This flag remains available throughout all version 8.x releases. Users who rely on this flag to allow non strings assignment to `tags` field should update their pipeline. https://github.com/elastic/logstash/pull/16507[#16507] [[dependencies-8.16.0]] ==== Updates to dependencies * Updated JRuby to 9.4.9.0 https://github.com/elastic/logstash/pull/16638[#16638] [[plugins-8-16-0]] ==== Plugins *Cef Codec - 6.2.8* * [DOC] Added missing documentation of the `raw_data_field` option https://github.com/logstash-plugins/logstash-codec-cef/pull/105[#105] *Json_lines Codec - 3.2.2* * Raised the default value of the `decode_size_limit_bytes` option to 512 MB https://github.com/logstash-plugins/logstash-codec-json_lines/pull/46[#46] * Added the `decode_size_limit_bytes` option to limit the maximum size of JSON lines that can be parsed. https://github.com/logstash-plugins/logstash-codec-json_lines/pull/43[#43] *Elastic_integration Filter - 0.1.15* * Use Elasticsearch code from its `8.16` branch and adapt to changes in Elasticsearch GeoIP processor https://github.com/elastic/logstash-filter-elastic_integration/pull/170[#170] *Geoip Filter - 7.3.1* * Fixed a pipeline crash when looking up a database with customised fields https://github.com/logstash-plugins/logstash-filter-geoip/pull/225[#225] *Azure_event_hubs Input - 1.5.0* * Updated Azure Event Hub client library to version `3.3.0` https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/96[#96] *Beats Input - 6.9.0* * Improved plugin's shutdown process and fixed a crash when a connection is terminated while processing messages https://github.com/logstash-plugins/logstash-input-beats/pull/500[#500] *Http Input - 3.9.1* * Fixed an issue where the value of `ssl_enabled` during `run` wasn't correctly logged https://github.com/logstash-plugins/logstash-input-http/pull/180[#180] * Separated Netty boss and worker groups to improve the graceful shutdown https://github.com/logstash-plugins/logstash-input-http/pull/178[#178] *Tcp Input - 6.4.3* * Updated dependencies for TCP input https://github.com/logstash-plugins/logstash-input-tcp/pull/224[#224] *Jdbc Integration - 5.5.0* * Added support for SQL `DATE` columns to jdbc static and streaming filters https://github.com/logstash-plugins/logstash-integration-jdbc/pull/171[#171] *Rabbitmq Integration - 7.4.0* * Removed obsolete `verify_ssl` and `debug` options https://github.com/logstash-plugins/logstash-integration-rabbitmq/pull/60[#60] [[logstash-8-15-4]] === Logstash 8.15.4 Release Notes [[known-issues-8-15-4]] ==== Known issue **{ls} may fail to shut down under some circumstances when you are using <>.** Check out issue https://github.com/elastic/logstash/issues/16657[#16657] for details. Workaround: Add `-Dlogstash.pipelinebus.implementation=v1` to `config/jvm.options`. This change reverts the `PipelineBus` to `v1`, a version that does not exhibit this issue, but may impact performance in pipeline-to-pipeline scenarios. [[notable-8-15-4]] ==== Notable issues fixed * Fixed an issue where Logstash could not consume lines correctly when a codec with a delimiter is in use and the input buffer becomes full https://github.com/elastic/logstash/pull/16482[#16482] [[dependencies-8-15-4]] ==== Updates to dependencies * Updated JRuby to 9.4.9.0 https://github.com/elastic/logstash/pull/16638[#16638] [[plugins-8-15-4]] ==== Plugins *Cef Codec - 6.2.8* * [DOC] Added `raw_data_field` to docs https://github.com/logstash-plugins/logstash-codec-cef/pull/105[#105] *Elastic_integration Filter - 0.1.15* * Fixed the connection failure where SSL verification mode is disabled over SSL connection https://github.com/elastic/logstash-filter-elastic_integration/pull/165[#165] *Geoip Filter - 7.3.1* * Fixed issue causing pipelines to crash during lookup when a database has custom fields https://github.com/logstash-plugins/logstash-filter-geoip/pull/225[#225] *Tcp Input - 6.4.3* * Updated dependencies https://github.com/logstash-plugins/logstash-input-tcp/pull/224[#224] [[logstash-8-15-3]] === Logstash 8.15.3 Release Notes [[known-issues-8-15-3]] ==== Known issue **{ls} may fail to shut down under some circumstances when you are using <>.** Check out issue https://github.com/elastic/logstash/issues/16657[#16657] for details. Workaround: Add `-Dlogstash.pipelinebus.implementation=v1` to `config/jvm.options`. This change reverts the `PipelineBus` to `v1`, a version that does not exhibit this issue, but may impact performance in pipeline-to-pipeline scenarios. [[notable-8.15.3]] ==== Notable issues fixed * Improved the pipeline bootstrap error logs to include the cause's backtrace, giving a hint where the issue occurred https://github.com/elastic/logstash/pull/16495[#16495] * Fixed Logstash core compatibility issues with `logstash-input-azure_event_hubs` versions `1.4.8` and earlier https://github.com/elastic/logstash/pull/16485[#16485] ==== Plugins *Elastic_integration Filter - 0.1.14* * Enabled the use of org.elasticsearch.ingest.common.Processors in Ingest Pipelines, resolving an issue where some integrations would fail to load https://github.com/elastic/logstash-filter-elastic_integration/pull/162[#162] *Azure_event_hubs Input - 1.4.9* * Fixed issue with `getHostContext` method accessibility, causing plugin not to be able to run https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/93[#93] * Fixed connection placeholder replacements errors with Logstash `8.15.1` and `8.15.2` https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/92[#92] *Kafka Integration - 11.5.2* * Updated avro to 1.11.4 and confluent kafka to 7.4.7 https://github.com/logstash-plugins/logstash-integration-kafka/pull/184[#184] [[logstash-8-15-2]] === Logstash 8.15.2 Release Notes [[known-issues-8-15-2]] ==== Known issue **{ls} may fail to shut down under some circumstances when you are using <>.** Check out issue https://github.com/elastic/logstash/issues/16657[#16657] for details. Workaround: Add `-Dlogstash.pipelinebus.implementation=v1` to `config/jvm.options`. This change reverts the `PipelineBus` to `v1`, a version that does not exhibit this issue, but may impact performance in pipeline-to-pipeline scenarios. [[notable-8.15.2]] ==== Notable issues fixed * Fixed a https://github.com/elastic/logstash/issues/16437[regression] from {ls} 8.15.1 in which {ls} removes all quotes from docker env variables, possibly causing {ls} not to start https://github.com/elastic/logstash/pull/16456[#16456] ==== Plugins *Beats Input - 6.8.4* * Fix to populate the `@metadata` fields even if the source's metadata value is `nil` https://github.com/logstash-plugins/logstash-input-beats/pull/502[#502] *Dead_letter_queue Input - 2.0.1* * Fix NullPointerException when the plugin closes https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/53[#53] *Elastic_serverless_forwarder Input - 0.1.5* * [DOC] Fix attributes to accurately set and clear default codec values https://github.com/logstash-plugins/logstash-input-elastic_serverless_forwarder/pull/8[#8] *Logstash Integration - 1.0.3* * [DOC] Fix attributes to accurately set and clear default codec values https://github.com/logstash-plugins/logstash-integration-logstash/pull/23[#23] *Elasticsearch Output - 11.22.9* * Vendor ECS template for Elasticsearch 9.x in built gem https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1188[#1188] * Added ECS template for Elasticsearch 9.x https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1187[#1187] [[logstash-8-15-1]] === Logstash 8.15.1 Release Notes [[known-issues-8-15-1]] ==== Known issues * **{ls} may fail to start under some circumstances.** Single and double quotes are stripped from a pipeline configuration if the configuration includes environment or keystore variable references. If this situation occurs, {ls} may fail to start or some plugins may use a malformed configuration. Check out issue https://github.com/elastic/logstash/issues/16437[#16437] for details. + Workaround: Downgrade to {ls} 8.15.0, or temporarily avoid using environment and keystore variable references. * **{ls} may fail to shut down under some circumstances when you are using <>.** Check out issue https://github.com/elastic/logstash/issues/16657[#16657] for details. + Workaround: Add `-Dlogstash.pipelinebus.implementation=v1` to `config/jvm.options`. This change reverts the `PipelineBus` to `v1`, a version that does not exhibit this issue, but may impact performance in pipeline-to-pipeline scenarios. [[notable-8.15.1]] ==== Performance improvements and notable issues fixed * Fixed issue where `${VAR}` incorrectly treated array as a string https://github.com/elastic/logstash/pull/16375[#16375] [[core-8.15.1]] ==== Other changes to Logstash core * Emit warning message for java version less than 17. Starting from Logstash 9.0, the minimum required version of Java will be Java 17 https://github.com/elastic/logstash/pull/16370[#16370] [[plugins-8.15.1]] ==== Plugin releases *Elasticsearch Input - 4.20.4* * Fixed issue where the `index` parameter was being ignored when using `response_type => aggregations` https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/209[#209] *Redis Input - 3.7.1* * [Doc] Added `threads` option https://github.com/logstash-plugins/logstash-input-redis/pull/95[#95] *Aws Integration - 7.1.8* * Added support to all storage classes for s3 output https://github.com/logstash-plugins/logstash-integration-aws/pull/49[#49] *Kafka Integration - 11.5.1* * [Doc] Added only headers with UTF-8 encoded values are supported in `decorate_events` https://github.com/logstash-plugins/logstash-integration-kafka/pull/174[#174] [[logstash-8-15-0]] === Logstash 8.15.0 Release Notes [[known-issues-8-15-0]] ==== Known issue **{ls} may fail to shut down under some circumstances when you are using <>.** Check out issue https://github.com/elastic/logstash/issues/16657[#16657] for details. Workaround: Add `-Dlogstash.pipelinebus.implementation=v1` to `config/jvm.options`. This change reverts the `PipelineBus` to `v1`, a version that does not exhibit this issue, but may impact performance in pipeline-to-pipeline scenarios. [[snmp-ga-8.15.0]] ==== Announcing the new {ls} SNMP integration plugin The new `logstash-integration-snmp` plugin is available and bundled with {ls} 8.15.0 (and later) by default. This plugin combines our classic `logstash-input-snmp` and `logstash-input-snmptrap` plugins into a single Ruby gem at v4.0.0 and later. The `logstash-integration-snmp` plugin combines the `logstash-input-snmp` and `logstash-input-snmptrap` plugins into one integrated plugin that encompasses the capabilities of both. This integrated plugin package provides better alignment in snmp processing, better resource management, easier package maintenance, and a smaller installation footprint. [[breaking-8.15.0]] ===== Breaking change with release of `logstash-integration-snmp` Prior to 8.15.0, {ls} bundled stand-alone versions of the `input-snmp` and `input-snmptrap` plugins by default. When you upgrade to 8.15.0, the stand-alone plugins are replaced by the 4.0.0+ version contained in the new integration. IMPORTANT: Before you upgrade to {ls} 8.15.0, be aware of {logstash-ref}/plugins-integrations-snmp.html#plugins-integrations-snmp-migration[behavioral and mapping differences] between stand-alone plugins and the new versions included in `integration-snmp`. If you need to maintain current mappings for the `input-snmptrap` plugin, you have options to {logstash-ref}/plugins-integrations-snmp.html#plugins-integrations-snmp-input-snmptrap-compat[preserve existing behavior]. [[featured-8.15.0]] ==== New features and enhancements [[wolfi-8.15.0]] ===== Wolfi-flavored Docker Image Starting with 8.15.0, {ls} provides an extra Docker image flavor based on the Wolfi Linux distribution: https://github.com/wolfi-dev. (https://github.com/elastic/logstash/pull/16189[#16189]). To use the image append "-wolfi" to the image name: ``` docker run docker.elastic.co/logstash/logstash-wolfi:8.15.0 ``` This new image flavor builds on top of a smaller and more secure base image, and is planned to become the default flavor in the next major version of {ls}. [[notable-8.15.0]] ==== Performance improvements and notable issues fixed * Improved performance in pipeline-to-pipeline workloads by removing lock contention in event serialization https://github.com/elastic/logstash/pull/16194[#16194] * Fixed pqcheck handling of paths with spaces on Windows https://github.com/elastic/logstash/pull/16205[#16205] [[core-8.15.0]] ==== Other changes to Logstash core * Changed rollover strategy in log4j2.properties to delete compressed files older than 7 days https://github.com/elastic/logstash/pull/16179[#16179] * Fixed handling and reporting of compilation error when a pipeline didn't fit the JVM's ThreadStackSize https://github.com/elastic/logstash/pull/16336[#16336] * Fixed missing deprecation logging in deprecated Logstash core settings https://github.com/elastic/logstash/pull/16348[#16348] * Fixed innacurate accounting of `queue_size_in_bytes` metric when using `clean_consumed` option in dead_letter_queue input plugin https://github.com/elastic/logstash/pull/16195[#16195] [[dependencies-8.15.0]] ==== Updates to dependencies * Updated bundled JDK to 21 https://github.com/elastic/logstash/pull/16055[#16055] * Updated JRuby to 9.4.8.0 https://github.com/elastic/logstash/pull/16278[#16278] [[docs-8.15.0]] ==== Documentation enhancements * Logstash {logstash-ref}/monitoring-with-ea.html[monitoring doc] improvements https://github.com/elastic/logstash/pull/16208[#16208] * Add ecs and datastream requirement for {logstash-ref}/ea-integrations.html#es-tips[integration filter] https://github.com/elastic/logstash/pull/16268[#16268] * Remove reference to puppet {ls} module https://github.com/elastic/logstash/pull/12356[#12356] * Add section to describe intended usage of {logstash-ref}/jvm-settings.html#reducing-off-heap-usage[`pipeline.buffer.type`] https://github.com/elastic/logstash/pull/16083[#16083] * Reposition {logstash-ref}/node-stats-api.html#pipeline-stats[`worker-utilization`] stat for better placement and flow https://github.com/elastic/logstash/pull/16337[#16337] * Add {logstash-ref}/performance-troubleshooting.html[tuning guidance] based on Flow metrics https://github.com/elastic/logstash/pull/16289[#16289] [[plugins-8.15.0]] ==== Plugin releases *De_dot Filter - 1.1.0* * Add support for recursively searching sub-fields with the new `recusive =>` config option https://github.com/logstash-plugins/logstash-filter-de_dot/pull/24[#24] *Elastic_integration Filter - 0.1.13* * Update default elasticsearch tree branch to 8.15 https://github.com/elastic/logstash-filter-elastic_integration/pull/156[#156] *Geoip Filter - 7.3.0* * Added support for MaxMind GeoIP2 Enterprise and Anonymous-IP databases https://github.com/logstash-plugins/logstash-filter-geoip/pull/223[#223] * Updated MaxMind dependencies. *Http Filter - 1.6.0* * Added new `ssl_enabled` setting for enabling/disabling the SSL configurations https://github.com/logstash-plugins/logstash-filter-http/pull/52[#52] *Http_poller Input - 5.6.0* * Added new `ssl_enabled` setting for enabling/disabling the SSL configurations https://github.com/logstash-plugins/logstash-input-http_poller/pull/146[#146] *Aws Integration - 7.1.7* * Remove empty temporary dirs at plugin close https://github.com/logstash-plugins/logstash-integration-aws/pull/46[#46] *Kafka Integration - 11.5.0* * Add "auto_create_topics" option to allow disabling of topic auto creation https://github.com/logstash-plugins/logstash-integration-kafka/pull/172[#172] *Snmp Integration - 4.0.4* * Introduce Snmp Integration by default, replacing Snmp Input v1.3.3 and Snmptrap Input v3.1.0 https://github.com/logstash-plugins/logstash-integration-snmp/pull/67[#67] *Http Output - 5.7.0* * Added new `ssl_enabled` setting for enabling/disabling the SSL configurations https://github.com/logstash-plugins/logstash-output-http/pull/144[#144] *Redis Output - 5.2.0* * Added support to SSL/TLS configurations https://github.com/logstash-plugins/logstash-output-redis/pull/69[#69] * `ssl_enabled` * `ssl_certificate_authorities` * `ssl_certificate` * `ssl_key` * `ssl_verification_mode` * `ssl_supported_protocols` * `ssl_cipher_suites` * Added basic support for SSL https://github.com/logstash-plugins/logstash-output-redis/pull/59[#59] * Fixed documentation of required settings https://github.com/logstash-plugins/logstash-output-redis/pull/61[#61] [[logstash-8-14-3]] === Logstash 8.14.3 Release Notes [[notable-8.14.3]] ==== Enhancements and notable issues fixed * Ensure pipeline metrics are cleared on shutdown https://github.com/elastic/logstash/pull/16264[#16264] [[logstash-8-14-2]] === Logstash 8.14.2 Release Notes [[notable-8.14.2]] ==== Notable issues fixed * Fixes a regression from Logstash 8.7.0 that prevented pipelines from starting when they included plugins with unicode ids https://github.com/elastic/logstash/pull/15971[#15971] * Fixes a regression from Logstash 8.12.0 that prevented pipelines from starting when they included a geoip filter that used the managed databases feature after the databases had been updated https://github.com/elastic/logstash/pull/16222[#16222] * Fixes an issue with the dead-letter queue that could cause it to emit superfluous warning messages when age-based retention was enabled while determining whether a fully-consumed segment that had already been removed was also expired https://github.com/elastic/logstash/pull/16204[#16204] ==== Plugins *Fluent Codec - 3.4.3* * Fix: reduce overhead of unpacking packforward-payloads by reusing a single instance https://github.com/logstash-plugins/logstash-codec-fluent/pull/32[#32] *Elastic_integration Filter - 0.1.10* * Fixes handling of array-type event fields by treating them as lists https://github.com/elastic/logstash-filter-elastic_integration/pull/146[#146] * Syncs with Elasticsearch 8.14, including support for new user-provided GeoIP database types `ConnectionType`, `Domain` and `Isp` https://github.com/elastic/logstash-filter-elastic_integration/pull/147[#147] *Elasticsearch Input - 4.20.3* * [DOC] Update link to bypass redirect, resolving directly to correct content https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/206[#206] *Elasticsearch Output - 11.22.7* * [DOC] `ssl_key` requires PKCS#8 format https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1181[#1181] [[logstash-8-14-1]] === Logstash 8.14.1 Release Notes * Fixes JSON serialization of payloads that are not UTF-8, eliminating an issue where the Elasticsearch Output could get stuck in a retry loop (#16072) https://github.com/elastic/logstash/pull/16168[#16168] * Fixes Persistent Queue bug in which a PQ configured with `queue.max_bytes` equal to its `queue.page_capacity` could become permanently blocked when _precisely_ full https://github.com/elastic/logstash/pull/16178[#16178] * Fixes a regression in multi-local pipeline loader that caused variable-references in a configured `pipelines.yml` to not be replaced by their values in the environment and/or keystore https://github.com/elastic/logstash/pull/16201[#16201] ==== Plugins *Elastic_integration Filter - 0.1.9* - [DOC] Removes Tech Preview label and adds link to extending integrations topic in LSR https://github.com/elastic/logstash-filter-elastic_integration/pull/142[#142] *Azure_event_hubs Input - 1.4.7* * [DOCS] Clarify examples for single and multiple event hubs https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/90[#90] * [DOCS] Add outbound port requirements for Event Hub https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/88[#88] *Jdbc Integration - 5.4.11* * Fixes an issue in which any one instance of a JDBC input plugin using `jdbc_default_timezone` changes the behaviour of plugin instances that do _not_ use `jdbc_default_timezone`, ensuring that timezone offsets remain consistent for each instance of the plugin _as configured_ https://github.com/logstash-plugins/logstash-integration-jdbc/pull/151[#151] * Fixes an exception that could occur while reloading `jdbc_static` databases when the underlying connection to the remote has been broken https://github.com/logstash-plugins/logstash-integration-jdbc/pull/165[#165] *Kafka Integration - 11.4.2* * Add default client_id of logstash to kafka output https://github.com/logstash-plugins/logstash-integration-kafka/pull/169[#169] *Http Output - 5.6.1* * Added body logging for non 2xx responses https://github.com/logstash-plugins/logstash-output-http/pull/142[#142] *Tcp Output - 6.2.1* * Document correct default plugin codec https://github.com/logstash-plugins/logstash-output-tcp/pull/54[#54] [[logstash-8-14-0]] === Logstash 8.14.0 Release Notes [[known-issues-8-14-0]] ==== Known issues * Environment and Keystore variable references for pipeline settings are not replaced in pipelines loaded from `pipelines.yml`. This regression is fixed in <> [[featured-8-14-0]] ==== Announcing the new SNMP integration plugin (Technical Preview) The new {logstash-ref}/plugins-integrations-snmp.html#plugins-integrations-snmp-migration[`logstash-integration-snmp`] plugin is now available in _Technical Preview_, and can be installed on {ls} 8.13. experimental[] The `logstash-integration-snmp` plugin combines our classic `logstash-input-snmp` and `logstash-input-snmptrap` plugins into a single Ruby gem at v4.0.0. Current 1.x versions of the `input-snmp` plugin are bundled with {ls} by default, and will soon be replaced by the 4.0.0+ version contained in this new integration. If you want to try the new `integration-snmp` plugin while it is in Technical Preview, run `bin/logstash-plugin install logstash-integration-snmp`. IMPORTANT: Before you install the new integration, be aware of {logstash-ref}/plugins-integrations-snmp.html#plugins-integrations-snmp-migration[behavioral and mapping differences] between current stand-alone plugins and the new versions included in `integration-snmp`. [[notable-8.14.0]] ==== Enhancements and notable issues fixed * Fixed a bug that created duplicated `io.netty.allocator.maxOrder` system property when using environment variable `LS_JAVA_OPTS` in Docker https://github.com/elastic/logstash/pull/16079[#16079] ==== Plugins *Jdbc Integration - 5.4.10* * [DOC] Added database-specific considerations https://github.com/logstash-plugins/logstash-integration-jdbc/pull/167[#167] *Kafka Integration - 11.4.1* * Added `message_headers` option to set headers of record for Kafka output https://github.com/logstash-plugins/logstash-integration-kafka/pull/162[#162] [[dependencies-8.14.0]] ==== Updates to dependencies * Update JRuby to 9.4.7.0 https://github.com/elastic/logstash/pull/16125[#16125] [[logstash-8-13-4]] === Logstash 8.13.4 Release Notes No user-facing changes in Logstash core. ==== Plugins *Elasticsearch Output - 11.22.6* * [DOC] Logstash output.elasticsearch index can be alias or datastream https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1179[#1179] [[logstash-8-13-3]] === Logstash 8.13.3 Release Notes No user-facing changes in Logstash core. ==== Plugins *Beats Input - 6.8.3* * Updated netty to 4.1.109 https://github.com/logstash-plugins/logstash-input-beats/pull/495[#495] *Http Input - 3.8.1* * Updated netty to 4.1.109 https://github.com/logstash-plugins/logstash-input-http/pull/173[#173] *Tcp Input - 6.4.2* * Updated netty to 4.1.109 https://github.com/logstash-plugins/logstash-input-tcp/pull/220[#220] *Multiline Codec - 3.1.2* * Fixed a race condition in periodic runner that blocks clean up process and pipeline shutdown https://github.com/logstash-plugins/logstash-codec-multiline/pull/72[#72] [[logstash-8-13-2]] === Logstash 8.13.2 Release Notes * Update Dockerfile to fix publishing of docker images to Docker Hub registry https://github.com/elastic/logstash/pull/16059[#16059] ==== Plugins *Elasticsearch Output - 11.22.5* * [DOC] Correctly document default data stream name (`logs-generic-default`) https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1140[#1140] * [DOC] Add note that ecs-compatibility is required for data streams to work properly https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1174[#1174] [[logstash-8-13-1]] === Logstash 8.13.1 Release Notes * Inject variable reference instead of variable value in `logstash.yaml` to ensure keystore values take precedence over Environment Variables https://github.com/elastic/logstash/pull/16037[#16037] [[logstash-8-13-0]] === Logstash 8.13.0 Release Notes [[featured-8-13-0]] ==== New {ls} plugin: Elastic_integration filter With the {ls} Elastic_integration filter, you can take advantage of the features in {integrations-docs}[Elastic integrations] without missing out on the advanced processing capabilities offered by {ls}. Together, Elastic integrations and {ls} can help you solve a wide range of use cases, including air gapped environments, data privacy processing, and multiple destinations. Your {ls} pipeline would include the elastic_agent input, the elastic_integration filter, and the elasticsearch output, in addition to any other plugins you want to use. Check out <> and the <> plugin docs for details. [[notable-8.13.0]] ==== Enhancements and notable issues fixed * Flow metrics added `worker_utilization` to pipeline-level https://github.com/elastic/logstash/pull/15912[#15912] * Ensure that the order of JVM options from both files and environment variables is respected https://github.com/elastic/logstash/pull/15997[#15997] [[plugin-changes-8.13.0]] ==== Changes to bundled plugins * Packaged `filter-elastic_integration` as a bundled plugin in preparation for GA. https://github.com/elastic/logstash/pull/15769[#15769] * Unbundled `logstash-input-imap` plugin https://github.com/elastic/logstash/pull/15985[#15985] ==== Plugins *Elastic_integration Filter - 0.1.6* * The elastic_integration filter is generally available and bundled with Logstash 8.13.0. Using this filter, {ls} can process data collected by {integrations-docs}[Elastic integrations]. + When you configure the filter to point to an {es} cluster, it auto-detects the event’s data-stream to determine what integration processing (if any) should be executed for each event. It performs that processing inside {ls} without transmitting the event to {es}. Events that are successfully handled are tagged so that any downstream {es} output in the {ls} pipeline will not re-run the integration in {es}. * Fixes issue where configured `username`/`password` credentials was not sent to Elasticsearch instances that had anonymous access enabled https://github.com/elastic/logstash-filter-elastic_integration/pull/127[#127] *Beats Input - 6.8.0* * Added a new configuration option `event_loop_threads` to control the threads count of netty event loop https://github.com/logstash-plugins/logstash-input-beats/pull/490[#490] ** When multiple beats-input pipelines are defined in a single machine, sometimes the number of thread may lead to a resource problem. ** By default, netty event loop creates CPU * 2 thread count. This configuration intends to limit or increase the number of threads to be created for the event loop. *Elasticsearch Input - 4.20.1* * Added support to aggregations with a new configuration option `response_type` https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/202[#202] ** `hits`, the default, is to generate one event per returned document, which is the current behavior. ** `aggregations` is to allow processing result of aggregations. A single Logstash event will be generated with the contents of the aggregations object of the query's response. *Tcp Output - 6.2.0* * Deprecated SSL settings to comply with Logstash's naming convention https://github.com/logstash-plugins/logstash-output-tcp/pull/53[#53] ** Deprecated `ssl_enable` in favor of `ssl_enabled` ** Deprecated `ssl_cert` in favor of `ssl_certificate` ** Deprecated `ssl_verify` in favor of `ssl_client_authentication` when mode is `server` ** Deprecated `ssl_verify` in favor of `ssl_verification_mode` when mode is `client` ** Added `ssl_cipher_suites` configuration ** Added SSL configuration validations *Elasticsearch Output - 11.22.3* * Fixes an issue where events containing non-unicode strings could fail to serialize correctly when compression is enabled https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1169[#1169] [[logstash-8-12-2]] === Logstash 8.12.2 Release Notes * Set Netty's maxOrder options to previous default value of 11 https://github.com/elastic/logstash/pull/15928[#15928] * Add "openssl" to UBI docker images https://github.com/elastic/logstash/pull/15929[#15929] [[known-issues-8.12.2]] ==== Known issues * The order of JVM flags is broken, leading to the inability to customize certain JVM configurations with the environment variable `LS_JAVA_OPTS`. https://github.com/elastic/logstash/issues/15996[15996] ** A workaround is to specify JVM flags in the `config/jvm.options` file rather than relying on the `LS_JAVA_OPTS` environment variable. ==== Plugins *Jdbc Integration - 5.4.8* * Update Derby to 10.15.2.1 (built from source) https://github.com/logstash-plugins/logstash-integration-jdbc/pull/155[#155] * Update sequel version to >= 5.74.0, that allows the generic jdbc adapter to better handle disconnect errors https://github.com/logstash-plugins/logstash-integration-jdbc/pull/153[#153] *Kafka Integration - 11.3.4* * Fix "retries" and "value_serializer" error handling in output plugin https://github.com/logstash-plugins/logstash-integration-kafka/pull/160[#160] [[logstash-8-12-1]] === Logstash 8.12.1 Release Notes * Updates bundled JDK https://github.com/elastic/logstash/pull/15840[#15840] ==== Plugins *Http Filter - 1.5.1* * Don't process response when the body is empty. https://github.com/logstash-plugins/logstash-filter-http/pull/50[#50] *Syslog_pri Filter - 3.2.1* * Remove spurious leftover text from "use_labels" docs https://github.com/logstash-plugins/logstash-filter-syslog_pri/pull/15[#15] *Logstash Integration - 1.0.2* * Fix: input plugin now correctly applies common event decorators `type`, `tags`, and `add_field` to events after receiving them https://github.com/logstash-plugins/logstash-integration-logstash/pull/21[#21] [[logstash-8-12-0]] === Logstash 8.12.0 Release Notes [[features-8.12.0]] ==== New features and enhancements * Add support for adding and removing multiple keystore keys in a single operation https://github.com/elastic/logstash/pull/15739[#15739] * Docker: Update Iron Bank base image to ubi9.2 https://github.com/elastic/logstash/pull/15490[#15490] * Internal: extract GeoIP database manager to stand-alone feature https://github.com/elastic/logstash/pull/15348[#15348] [[notable-8.12.0]] ==== Notable issues fixed * Add missing method of logger wrapper for puma https://github.com/elastic/logstash/pull/15640[#15640] * Fix logstash-keystore multiple keys operations with command flags https://github.com/elastic/logstash/pull/15737[#15737] * Separate scheduling of segments flushes from time https://github.com/elastic/logstash/pull/15697[#15697] * Add system properties to configure Jackson's stream read constraints https://github.com/elastic/logstash/pull/15763[#15763] * Fix issue with Jackson 2.15: Can not write a field name, expecting a value https://github.com/elastic/logstash/pull/15564[#15564] [[dependencies-8.12.0]] ==== Updates to dependencies * Add bigdecimal > 3.1 dependency. https://github.com/elastic/logstash/pull/15384[#15384] * Update Guava dependency to 32.1.2 https://github.com/elastic/logstash/pull/15394[#15394] * Swap dataformat-yaml with snakeyaml https://github.com/elastic/logstash/pull/15606[#15606] * Bump Puma to 6.4.2+ https://github.com/elastic/logstash/pull/15776[#15776] * Update jackson to 2.15.3 https://github.com/elastic/logstash/pull/15477[#15477] [[docs-8.12.0]] ==== Documentation enhancements * Add https://www.elastic.co/guide/en/logstash/8.12/running-logstash-kubernetes.html[info and link] to {ls} running on a {k8s} cluster through {eck} (ECK) https://github.com/elastic/logstash/pull/15565[#15565] * Add info for sending https://www.elastic.co/guide/en/logstash/current/serverless-monitoring-with-elastic-agent.html[{ls} monitoring data to Elastic {serverless-short}] https://github.com/elastic/logstash/pull/15636[#15636] * Add docs for https://www.elastic.co/guide/en/logstash/current/ea-integrations.html[extending integrations with filter-elastic_integration] https://github.com/elastic/logstash/pull/15674[#15674] * Update https://www.elastic.co/guide/en/logstash/current/introduction.html[Logstash intro] and https://www.elastic.co/guide/en/logstash/current/ls-security.html#serverless[security overview] for {serverless-short} https://github.com/elastic/logstash/pull/15663[#15663] * Update the https://www.elastic.co/guide/en/logstash/8.12/ls-to-ls-native.html#configure-upstream-logstash-output[{ls}-to-{ls} communication] docs to reflect the multiple hosts usage https://github.com/elastic/logstash/pull/15512[#15512] ==== Plugins *Elasticsearch Input - 4.19.1* * Plugin version bump to pick up docs fix in https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/199[#199] required to clear build error in docgen. https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/200[#200] * Add `search_api` option to support `search_after` and `scroll` https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/198[#198] * The default value `auto` uses `search_after` for Elasticsearch >= 8, otherwise, fall back to `scroll` *Http Input - 3.8.0* * Fixed SSL Java KeyStore support https://github.com/logstash-plugins/logstash-input-http/pull/171[#171] * Added `ssl_keystore_type` configuration * Added SSL Java TrustStore configurations (`ssl_truststore_type`, `ssl_truststore_path` and `ssl_truststore_password`) *Elastic_enterprise_search Integration - 3.0.0* * [BREAKING] Swiftype endpoints are no longer supported for both plugins App Search and Workplace Search * Bumped Enterprise Search clients to version `>= 7.16`, `< 9` https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/18[#18] * Added support to SSL configurations (`ssl_certificate_authorities`, `ssl_truststore_path`, `ssl_truststore_password`, `ssl_truststore_type`, `ssl_verification_mode`, `ssl_supported_protocols` and `ssl_cipher_suites`) * The App Search deprecated options `host` and `path` were removed *Kafka Integration - 11.3.3* * Fixed: "Can't modify frozen string" error when record value is `nil` (tombstones) https://github.com/logstash-plugins/logstash-integration-kafka/pull/155[#155] *Logstash Integration - 1.0.1* * Fixed: improves throughput by allowing pipeline workers to share a plugin instance _concurrently_ instead of _sequentially_ https://github.com/logstash-plugins/logstash-integration-logstash/pull/19[#19] * Introduced load balancing mechanism to distribute the requests among the `hosts` https://github.com/logstash-plugins/logstash-integration-logstash/pull/16[#16] *Elasticsearch Output - 11.22.2* * Fixed: avoid to populate `version` and `version_type` attributes when processing integration metadata and datastream is enabled. https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1161[#1161] * Added support for propagating event processing metadata when this output is downstream of an Elastic Integration Filter and configured _without_ explicit `version`, `version_type`, or `routing` directives https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1158[#1158] * Added support for propagating event processing metadata when this output is downstream of an Elastic Integration Filter and configured _without_ explicit `index`, `document_id`, or `pipeline` directives https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1155[#1155] * Changed the register to initiate pipeline shutdown upon bootstrap failure instead of simply logging the error https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1151[#1151] * Doc: Replace `document_already_exist_exception` with `version_conflict_engine_exception` in the `silence_errors_in_log` setting example https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1159[#1159] * Doc: Add content for sending data to Elasticsearch on serverless https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1164[#1164] [[logstash-8-11-4]] === Logstash 8.11.4 Release Notes No user-facing changes in Logstash core. ==== Plugins *Netflow Codec - 4.3.2* * Updates the milliseconds rounding for IPFIX start/end milliseconds fields. * Fix the test to run on Logstash 8 with microseconds precision. https://github.com/logstash-plugins/logstash-codec-netflow/pull/206[#206] * Fixed unable to initialize the plugin with Logstash 8.10+ https://github.com/logstash-plugins/logstash-codec-netflow/pull/205[#205] *Json Filter - 3.2.1* * Fix tag on failure test https://github.com/logstash-plugins/logstash-filter-json/pull/52[#52] *File Input - 4.4.6* * Change read mode to immediately stop consuming buffered lines when shutdown is requested https://github.com/logstash-plugins/logstash-input-file/pull/322[#322] *Twitter Input - 4.1.1* * Bumped `public_suffix` gem version to `> 4` `< 6` https://github.com/logstash-plugins/logstash-input-twitter/pull/77[#77] *Csv Output - 3.0.10* * Extend `spreadsheet_safe` prefix guard to '-', '+', and '@' https://github.com/logstash-plugins/logstash-output-csv/pull/27[#27] [[logstash-8-11-3]] === Logstash 8.11.3 Release Notes [[docs-8.11.3]] ==== Documentation enhancements * Document how to further transform events processed by the filter-elastic_integration plugin https://github.com/elastic/logstash/pull/15675[#15675] [[dependencies-8.11.3]] ==== Updates to dependencies * Update JRuby to 9.4.5.0 https://github.com/elastic/logstash/pull/15670[#15670] [[logstash-8-11-2]] === Logstash 8.11.2 Release Notes * Added missing method of logger wrapper for puma https://github.com/elastic/logstash/pull/15642[#15642] * Prevent calling shutdown on the DLQ segments flusher if it hasn't been started yet https://github.com/elastic/logstash/pull/15656[#15656] * Remove dependency on jackson-dataformat-yaml https://github.com/elastic/logstash/pull/15599[#15599] ==== Plugins *Mutate Filter - 3.5.8* * Fix "Can't modify frozen string" error when converting boolean to `string` https://github.com/logstash-plugins/logstash-filter-mutate/pull/171[#171] *Beats Input - 6.7.2* * Restore Lumberjack event parsing source code https://github.com/logstash-plugins/logstash-input-beats/pull/486[#486] *Elastic_serverless_forwarder Input - 0.1.4* * [DOC] Adds tips for using the logstash-input-elastic_serverless_forwarder plugin with the Elasticsearch output plugin https://github.com/logstash-plugins/logstash-input-elastic_serverless_forwarder/pull/7[#7] *Validator_support Mixin - 1.1.1* * Allow single-word host names such as "localhost". This addresses the inability to set "hosts" to "localhost" in the logstash-filter-elastic_integration plugin. https://github.com/logstash-plugins/logstash-mixin-validator_support/pull/7[#7] [[logstash-8-11-1]] === Logstash 8.11.1 Release Notes * Downgrade jackson to avoid serialization issues when log.format is set to "json" https://github.com/elastic/logstash/pull/15549[#15549] [[logstash-8-11-0]] === Logstash 8.11.0 Release Notes [[known-issues-8.11.0]] ==== Known issues * Input imap plugin's behavior with Logstash 8.10+ versions, is broken after upgrading its `mail` dependency https://github.com/logstash-plugins/logstash-input-imap/issues/61[#61] [[notable-8.11.0]] ==== Notable issues fixed * Update callsite syntax for i18n.t method to avoid deprecated and prohibited format https://github.com/elastic/logstash/pull/15500[#15500] [[docs-8.11.0]] ==== Documentation enhancements * Add native Logstash to Logstash documentation https://github.com/elastic/logstash/pull/15346[#15346] * Expand description of how to size the JVM memory https://github.com/elastic/logstash/pull/15210[#15210] [[dependencies-8.11.0]] ==== Updates to dependencies * Update Guava dependency to 32.1.2 https://github.com/elastic/logstash/pull/15394[#15394] * Downgrade jruby, keep updated default-gem dependencies https://github.com/elastic/logstash/pull/15369[#15369] * Pin psych 5.1.0 https://github.com/elastic/logstash/pull/15433[#15433] * Update JDK to 17.0.9+9 and jackson to 2.15.3 https://github.com/elastic/logstash/pull/15510[#15510] ==== Plugins *Elasticsearch Filter - 3.16.1* * Version bump to pick up doc fix in https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/172[#172] * Add request header `Elastic-Api-Version` for serverless https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/174[#174] *Http Filter - 1.5.0* * Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts https://github.com/logstash-plugins/logstash-filter-http/pull/49[#49] ** Introduce new `ssl_truststore_path`, `ssl_truststore_password`, and `ssl_truststore_type` settings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate their `truststore`, `truststore_password`, and `truststore_type` counterparts. ** Introduce new `ssl_certificate_authorities` setting for configuring SSL-trust using a PEM-formatted list certificate authorities, deprecate its `cacert` counterpart. ** Introduce new `ssl_keystore_path`, `ssl_keystore_password`, and `ssl_keystore_type` settings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate their `keystore`, `keystore_password`, and `keystore_type` counterparts. ** Introduce new `ssl_certificate` and `ssl_key` settings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate their `client_cert` and `client_key` counterparts. ** Introduce the `ssl_cipher_suites` option. *Beats Input - 6.7.0* * Add explicit support for receiving a 0-length window to encapsulate an empty batch. Empty batches are acknowledged with the same 0-sequence ACK's that are used as keep-alives during processing https://github.com/logstash-plugins/logstash-input-beats/pull/479[#479] *Elasticsearch Input - 4.18.0* * Add request header `Elastic-Api-Version` for serverless https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/195[#195] *Http_poller Input - 5.5.0* * Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts https://github.com/logstash-plugins/logstash-input-http_poller/pull/141[#141] ** Introduce new `ssl_truststore_path`, `ssl_truststore_password`, and `ssl_truststore_type` settings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate their `truststore`, `truststore_password`, and `truststore_type` counterparts. ** Introduce new `ssl_certificate_authorities` setting for configuring SSL-trust using a PEM-formatted list certificate authorities, deprecate its `cacert` counterpart. ** Introduce new `ssl_keystore_path`, `ssl_keystore_password`, and `ssl_keystore_type` settings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate their `keystore`, `keystore_password`, and `keystore_type` counterparts. ** Introduce new `ssl_certificate` and `ssl_key` settings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate their `client_cert` and `client_key` counterparts. ** Introduce the `ssl_cipher_suites` option. *Imap Input - 3.2.1* * Upgrade `email` dependency so that supports Ruby 3.1. This also fixes the `net-smtp` load regression https://github.com/logstash-plugins/logstash-input-imap/pull/60[#60] *Jdbc Integration - 5.4.6* * Update sequel version to >= 5.73.0. The ibmdb and jdbc/db2 adapters were fixed to properly handle disconnect errors, removing the related connection from the pool https://github.com/logstash-plugins/logstash-integration-jdbc/pull/144[#144] *Logstash Integration - 0.0.5* * Logstash 8.11 version embeds the `logstash-integration-logstash` plugin. *Http_client Mixin - 7.3.0* * Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts https://github.com/logstash-plugins/logstash-mixin-http_client/pull/42[#42] ** Introduce new `ssl_truststore_path`, `ssl_truststore_password`, and `ssl_truststore_type` settings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate their `truststore`, `truststore_password`, and `truststore_type` counterparts. ** Introduce new `ssl_certificate_authorities` setting for configuring SSL-trust using a PEM-formated list certificate authorities, deprecate its `cacert` counterpart. ** Introduce new `ssl_keystore_path`, `ssl_keystore_password`, and `ssl_keystore_type` settings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate their `keystore`, `keystore_password`, and `keystore_type` counterparts. ** Introduce new `ssl_certificate` and `ssl_key` settings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate their `client_cert` and `client_key` counterparts. ** Introduce a way for plugin maintainers to include this mixin _without_ supporting the now-deprecated SSL options. ** Introduce the `ssl_cipher_suites` option. *Validator_support Mixin - 1.1.0* * Introduce `:required_host_optional_port` validator https://github.com/logstash-plugins/logstash-mixin-validator_support/pull/4[#4] *Elasticsearch Output - 11.18.0* * Add request header `Elastic-Api-Version` for serverless https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1147[#1147] * Add support to http compression level. Deprecate `http_compression` in favour of `compression_level` and enable compression level 1 by default https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1148[#1148] *Email Output - 4.1.3* * Upgrade `email` dependency so that supports Ruby 3.1. This also fixes the `net-smtp` load regression https://github.com/logstash-plugins/logstash-output-email/pull/69[#69] *Http Output - 5.6.0* * Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts https://github.com/logstash-plugins/logstash-output-http/pull/140[#140] ** Introduce new `ssl_truststore_path`, `ssl_truststore_password`, and `ssl_truststore_type` settings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate their `truststore`, `truststore_password`, and `truststore_type` counterparts. ** Introduce new `ssl_certificate_authorities` setting for configuring SSL-trust using a PEM-formatted list certificate authorities, deprecate its `cacert` counterpart. ** Introduce new `ssl_keystore_path`, `ssl_keystore_password`, and `ssl_keystore_type` settings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate their `keystore`, `keystore_password`, and `keystore_type` counterparts. ** Introduce new `ssl_certificate` and `ssl_key` settings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate their `client_cert` and `client_key` counterparts. ** Introduce the `ssl_cipher_suites` option. *Webhdfs Output - 3.1.0* * Fix: remove snappy gem as a dependency in favor of directly vendoring the snappy jar. https://github.com/logstash-plugins/logstash-output-webhdfs/pull/46[#46] [[logstash-8-10-4]] === Logstash 8.10.4 Release Notes [[known-issues-8.10.4]] ==== Known issues These plugins may fail in Logstash 8.10.4: * *Avro codec plugin.* The avro codec plugin may fail with an `(Errno::ENOENT) No such file or directory` error https://github.com/logstash-plugins/logstash-codec-avro/issues/42[#42]. This issue has been resolved in plugin v3.4.1 https://github.com/logstash-plugins/logstash-codec-avro/pull/43[#43]. * *Imap input plugin.* Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in https://github.com/logstash-plugins/logstash-input-imap/issues/61[GitHub issue #61]. * *Email output plugin.* Plugin raises `LoadError: no such file to load -- net/smtp` runtime error. See the issue details and work around in https://github.com/logstash-plugins/logstash-output-email/issues/68[GitHub issue #68]. [[dependencies-8.10.4]] ==== Updates to dependencies * Update Guava dependency to 32.1.2 https://github.com/elastic/logstash/pull/15419[#15419] [[logstash-8-10-3]] === Logstash 8.10.3 Release Notes [[known-issues-8.10.3]] ==== Known issues These plugins may fail in Logstash 8.10.3: * *Avro codec plugin.* The avro codec plugin may fail with an `(Errno::ENOENT) No such file or directory` error https://github.com/logstash-plugins/logstash-codec-avro/issues/42[#42]. This issue has been resolved in plugin v3.4.1 https://github.com/logstash-plugins/logstash-codec-avro/pull/43[#43]. * *Imap input plugin.* Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in https://github.com/logstash-plugins/logstash-input-imap/issues/61[GitHub issue #61]. * *Email output plugin.* Plugin raises `LoadError: no such file to load -- net/smtp` runtime error. See the issue details and work around in https://github.com/logstash-plugins/logstash-output-email/issues/68[GitHub issue #68]. ==== Plugins *Elasticsearch Filter - 3.15.3* * Fixes a memory leak that occurs when a pipeline containing this filter terminates, which could become significant if the pipeline is cycled repeatedly https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/173[#173] *Useragent Filter - 3.3.5* * Upgrade `snakeyaml` dependency https://github.com/logstash-plugins/logstash-filter-useragent/pull/89[#89] *Beats Input - 6.6.4* * [DOC] Fix misleading `enrich/source_data` input beats documentation about the Logstash host. https://github.com/logstash-plugins/logstash-input-beats/pull/478[#478] *Elastic_serverless_forwarder Input - 0.1.3* * Deprecates the `ssl` option in favor of `ssl_enabled` https://github.com/logstash-plugins/logstash-input-elastic_serverless_forwarder/pull/6[#6] * Bumps `logstash-input-http` gem version to `>= 3.7.2` (SSL-normalized) *Aws Integration - 7.1.6* * Clean up plugin created temporary dirs at startup https://github.com/logstash-plugins/logstash-integration-aws/pull/39[#39] *Jdbc Integration - 5.4.5* * Pin sequel to < 5.72.0 due to ruby/bigdecimal#169 https://github.com/logstash-plugins/logstash-integration-jdbc/pull/141[#141] *Kafka Integration - 11.3.1* * Fix: update snappy dependency https://github.com/logstash-plugins/logstash-integration-kafka/pull/148[#148] [[logstash-8-10-2]] === Logstash 8.10.2 Release Notes [[known-issues-8.10.2]] ==== Known issues These plugins may fail in Logstash 8.10.2: * *Avro codec plugin.* The avro codec plugin may fail with an `(Errno::ENOENT) No such file or directory` error https://github.com/logstash-plugins/logstash-codec-avro/issues/42[#42]. This issue has been resolved in plugin v3.4.1 https://github.com/logstash-plugins/logstash-codec-avro/pull/43[#43]. * *Imap input plugin.* Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in https://github.com/logstash-plugins/logstash-input-imap/issues/61[GitHub issue #61]. * *Email output plugin.* Plugin raises `LoadError: no such file to load -- net/smtp` runtime error. See the issue details and work around in https://github.com/logstash-plugins/logstash-output-email/issues/68[GitHub issue #68]. [[logstash-8-10-1]] === Logstash 8.10.1 Release Notes [[known-issues-8.10.1]] ==== Known issues These plugins may fail in Logstash 8.10.1: * *Avro codec plugin.* The avro codec plugin may fail with an `(Errno::ENOENT) No such file or directory` error https://github.com/logstash-plugins/logstash-codec-avro/issues/42[#42]. This issue has been resolved in plugin v3.4.1 https://github.com/logstash-plugins/logstash-codec-avro/pull/43[#43]. * *Imap input plugin.* Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in https://github.com/logstash-plugins/logstash-input-imap/issues/61[GitHub issue #61]. * *Email output plugin.* Plugin raises `LoadError: no such file to load -- net/smtp` runtime error. See the issue details and work around in https://github.com/logstash-plugins/logstash-output-email/issues/68[GitHub issue #68]. [[logstash-8-10-0]] === Logstash 8.10.0 Release Notes [[known-issues-8.10.0]] ==== Known issues These plugins may fail in Logstash 8.10.0: * *Avro codec plugin.* The avro codec plugin may fail with an `(Errno::ENOENT) No such file or directory` error https://github.com/logstash-plugins/logstash-codec-avro/issues/42[#42]. This issue has been resolved in plugin v3.4.1 https://github.com/logstash-plugins/logstash-codec-avro/pull/43[#43]. * *Imap input plugin.* Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in https://github.com/logstash-plugins/logstash-input-imap/issues/61[GitHub issue #61]. * *Email output plugin.* Plugin raises `LoadError: no such file to load -- net/smtp` runtime error. See the issue details and work around in https://github.com/logstash-plugins/logstash-output-email/issues/68[GitHub issue #68]. [[notable-8.10.0]] ==== Notable issues fixed * Fixed issues in Dead Letter Queue (DLQ): ** `java.nio.file.NoSuchFileException` when finalizing the segment https://github.com/elastic/logstash/pull/15233[#15233] ** DLQ file using wrong sort order https://github.com/elastic/logstash/pull/15246[#15246] [[dependencies-8.10.0]] ==== Updates to dependencies * Updated JRuby to 9.4.2.0 https://github.com/elastic/logstash/pull/15283[#15283] * Removed custom bundler and used JRuby bundled bundler https://github.com/elastic/logstash/pull/15066[#15066] ==== Plugins *Elasticsearch Output - 11.16.0* * Added support to Serverless Elasticsearch https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1145[#114] *Elastic_serverless_forwarder input 0.1.2* * Introduces a dedicated input plugin for receiving events from Elastic Serverless Forwarder. This plugin is in late Technical Preview, which means that as we iterate toward a stable API both configuration options and implementation details may change in subsequent releases without the usual deprecation warnings. If you use this plugin, please keep your eye on the relevant changelogs when upgrading. https://github.com/logstash-plugins/logstash-input-elastic_serverless_forwarder/pull/4[#4] *Kafka Integration - 11.3.0* * Changed Kafka client to 3.4.1 https://github.com/logstash-plugins/logstash-integration-kafka/pull/145[#145] *Tcp Input - 6.3.5* * Standardized SSL settings https://github.com/logstash-plugins/logstash-input-tcp/pull/213[#213] ** deprecated `ssl_enable` in favor of `ssl_enabled` ** deprecated `ssl_cert` in favor of `ssl_certificate` ** deprecated `ssl_verify` in favor of `ssl_client_authentication` when mode is `server` ** deprecated `ssl_verify` in favor of `ssl_verification_mode` when mode is `client` [[logstash-8-9-2]] === Logstash 8.9.2 Release Notes No user-facing changes in Logstash core and plugins. [[logstash-8-9-1]] === Logstash 8.9.1 Release Notes [[notable-8.9.1]] ==== Notable issues fixed * Fix pipeline to pipeline communication when upstream pipeline is terminated and events is written to a closed queue in downstream. https://github.com/elastic/logstash/pull/15173[#15173] * Fix DLQ unable to finalize segment error https://github.com/elastic/logstash/pull/15241[#15241] [[dependencies-8.9.1]] ==== Updates to dependencies * Update JDK to 17.0.8+7 https://github.com/elastic/logstash/pull/15237[#15237] ==== Plugins *Elasticsearch Filter - 3.15.2* * Added checking to ensure either `query` or `query_template` is non empty https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/171[#171] *Snmp Input - 1.3.3* * Silence warnings when loading dictionary MIB files https://github.com/logstash-plugins/logstash-input-snmp/pull/118[#118] *Aws Integration - 7.1.5* * Fix external documentation links https://github.com/logstash-plugins/logstash-integration-aws/pull/35[#35] [[logstash-8-9-0]] === Logstash 8.9.0 Release Notes [[notable-8.9.0]] ==== Notable issues fixed * Fixed an https://github.com/elastic/logstash/issues/15130[issue] where installs and updates of certain {ls} plugins could fail when located behind a proxy https://github.com/elastic/logstash/pull/15131[#15131]. This issue surfaced after `logstash-filter-translate` was updated to require that the `jar-dependencies` gem be used to retrieve artifacts from maven when the plugin was installed. This requirement could prevent the plugin update when a proxy was in use. * Improved logging when {ls} is stalled on shutdown https://github.com/elastic/logstash/pull/15056[#15056]. We now provide additional information about the main thread if it is causing the shutdown to stall. * Improved SSL settings for connection to {es} for central management and monitoring https://github.com/elastic/logstash/pull/15045[#15045]. This commit adds settings support for file-based certificates and cipher suites for management and monitoring settings, and removes the deprecation warnings from the logs that have been in since SSL configuration settings were revamped in the {es} output. [[dependencies-8.9.0]] ==== Updates to dependencies * Update Bundler to version 2.4 https://github.com/elastic/logstash/pull/14995[#14995] ==== Plugins *Azure_event_hubs Input - 1.4.5* * Update multiple dependencies such as `gson`, `log4j2`, `jackson` https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/83[#83] *Beats Input - 6.6.3* * [DOC] Updated the `ssl_client_authentication` and `ssl_verify_mode` documentation explaining that CN and SAN are not validated. https://github.com/logstash-plugins/logstash-input-beats/pull/473[#473] * Update netty to 4.1.94 and jackson to 2.15.2 https://github.com/logstash-plugins/logstash-input-beats/pull/474[#474] *Http Input - 3.7.2* * Update netty to 4.1.94 https://github.com/logstash-plugins/logstash-input-http/pull/167[#167] *Snmp Input - 1.3.2* * [DOC] Add troubleshooting help for "failed to locate MIB module" error when using smidump to convert MIBs https://github.com/logstash-plugins/logstash-input-snmp/pull/112[#112] *Tcp Input - 6.3.5* * Update netty to 4.1.94 and other dependencies https://github.com/logstash-plugins/logstash-input-tcp/pull/216[#216] * Fix: reduce error logging (to info level) on connection resets https://github.com/logstash-plugins/logstash-input-tcp/pull/214[#214] *Tcp Output - 6.1.2* * Changed the client mode to write using the non-blocking method. https://github.com/logstash-plugins/logstash-output-tcp/pull/52[#52] [[logstash-8-8-2]] === Logstash 8.8.2 Release Notes No user-facing changes in Logstash core. ==== Plugins *Translate Filter - 3.4.2* * Fix JRuby 9.4 compatibility issue https://github.com/logstash-plugins/logstash-filter-translate/pull/98[#98] *Aws Integration - 7.1.4* * Fix `use_aws_bundled_ca` to use bundled ca certs per plugin level instead of global https://github.com/logstash-plugins/logstash-integration-aws/pull/33[#33] * Add an option `use_aws_bundled_ca` to use bundled ca certs that ships with AWS SDK to verify SSL peer certificates https://github.com/logstash-plugins/logstash-integration-aws/pull/32[#32] * Fix JRuby 9.4 compatibility issue https://github.com/logstash-plugins/logstash-integration-aws/pull/29[#29] *Jdbc Integration - 5.4.4* * Fix: adaptations for JRuby 9.4 https://github.com/logstash-plugins/logstash-integration-jdbc/pull/125[#125] *Rabbitmq Integration - 7.3.3* * Fix the cancellation flow to avoid multiple invocations of basic.cancel https://github.com/logstash-plugins/logstash-integration-rabbitmq/pull/55[#55] *Csv Output - 3.0.9* * Fix JRuby 9.4 compatibility issue https://github.com/logstash-plugins/logstash-output-csv/pull/25[#25] *Elasticsearch Output - 11.15.8* * Fix a regression introduced in 11.14.0 which could prevent Logstash 8.8 from establishing a connection to Elasticsearch for Central Management and Monitoring core features https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1141[#1141] [[logstash-8-8-1]] === Logstash 8.8.1 Release Notes * Remove obsolete notice when using plugins with version < 1.0.0 https://github.com/elastic/logstash/pull/15077[#15077] * Docs: Add instructions to verify Docker install images https://github.com/elastic/logstash/pull/15064[#15064] * Fixes a regression introduced in 8.8.0 which could prevent Monitoring or Central Management from establishing a connection to Elasticsearch in some SSL configurations https://github.com/elastic/logstash/issues/15068[#15068] ==== Plugins *Cef Codec - 6.2.7* * Fix: when decoding in an ecs_compatibility mode, timestamp-normalized fields now handle provided-but-empty values https://github.com/logstash-plugins/logstash-codec-cef/issues/102[#102] *Anonymize Filter - 3.0.7* * Pin murmurhash3 to 0.1.6 https://github.com/logstash-plugins/logstash-filter-anonymize/pull/16[#16] *Elasticsearch Filter - 3.15.1* * Fixes a regression introduced in 3.15.0 which could prevent a connection from being established to Elasticsearch in some SSL configurations https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/169[#169] *Fingerprint Filter - 3.4.3* * Pin murmurhash3 to 0.1.6 https://github.com/logstash-plugins/logstash-filter-fingerprint/pull/74[#74] *Mutate Filter - 3.5.7* * Docs: Clarify that `split` and `join` also support strings https://github.com/logstash-plugins/logstash-filter-mutate/pull/164[#164] *Translate Filter - 3.4.1* * Fix the limitation of the size of yaml file that exceeds 3MB https://github.com/logstash-plugins/logstash-filter-translate/pull/97[#97] *Truncate Filter - 1.0.6* * Make 0.0.8 the lower bound for flores dependency https://github.com/logstash-plugins/logstash-filter-truncate/pull/9[#9] *Beats Input - 6.6.1* * Update netty to 4.1.93 and jackson to 2.13.5 https://github.com/logstash-plugins/logstash-input-beats/pull/472[#472] *Elasticsearch Input - 4.17.2* * Fixes a regression introduced in 4.17.0 which could prevent a connection from being established to Elasticsearch in some SSL configurations https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/193[#193] * Fixes scroll slice high memory consumption https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/189[#189] *Http Input - 3.7.1* * Update netty to 4.1.93 https://github.com/logstash-plugins/logstash-input-http/pull/166[#166] *Tcp Input - 6.3.3* * Update netty to 4.1.93 https://github.com/logstash-plugins/logstash-input-tcp/pull/212[#212] *Jdbc Integration - 5.4.3* * Fix: crash when metadata file can't be deleted after moving under path.data https://github.com/logstash-plugins/logstash-integration-jdbc/pull/136[#136] * Add new settings `statement_retry_attempts` and `statement_retry_attempts_wait_time` for retry of failed sql statement execution https://github.com/logstash-plugins/logstash-integration-jdbc/pull/123[#123] * Doc: described default_hash and tag_on_default_use interaction filter plugin https://github.com/logstash-plugins/logstash-integration-jdbc/pull/122[#122] *Rabbitmq Integration - 7.3.2* * Change `tls_certificate_password` type to `password` to protect from leaks in the logs https://github.com/logstash-plugins/logstash-integration-rabbitmq/pull/54[#54] *Elasticsearch Output - 11.15.7* * Fixes a regression introduced in 11.14.0 which could prevent a connection from being established to Elasticsearch in some SSL configurations https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1138[#1138] * Fixes possiblity of data loss when pipeline terminates very quickly after startup https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1132[#1132]. * Fixes `undefined 'shutdown_requested' method` error when plugin checks if shutdown request is received https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1134[#1134] * Improves connection handling under several partial-failure scenarios https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1130[#1130] ** Ensures an HTTP connection can be established before adding the connection to the pool ** Ensures that the version of the connected Elasticsearch is retrieved _successfully_ before the connection is added to the pool. ** Fixes a crash that could occur when the plugin is configured to connect to a live HTTP resource that is _not_ Elasticsearch * Removes the ECS v8 unreleased preview warning https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1131[#1131] * Restores DLQ logging behavior from 11.8.x to include the action-tuple as structured https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1105[#1105] *Email Output - 4.1.2* * Change `password` config type to `Password` to prevent leaks in debug logs https://github.com/logstash-plugins/logstash-output-email/pull/65[#65] [[logstash-8-8-0]] === Logstash 8.8.0 Release Notes [[known-issues-8.8.0]] ==== Known issues Logstash 8.8.0 may fail to start when SSL/TLS is enabled in monitoring and/or central management, due to a change introduced in version 11.14.0 of the https://github.com/logstash-plugins/logstash-output-elasticsearch[logstash-output-elasticsearch] plugin. When impacted by this issue, Logstash fails to start and logs an error similar to the following: ``` [logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#} ``` **Resolution** A successful Elasticsearch output plugin update to version `11.15.8` or higher will resolve this issue: ``` bin/logstash-plugin update logstash-output-elasticsearch ``` OR Specify the `ca_trusted_fingerprint` setting in the `logstash.yml`. The certificate fingerprint can be extract with: ``` cat your_ca.cert | openssl x509 -outform der | sha256sum | awk '{print $1}' ``` Then set the following on `logstash.yml` using the output from the previous command: ``` xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: "" xpack.management.elasticsearch.ssl.ca_trusted_fingerprint: "" ``` [[notable-8.8.0]] ==== Notable issues fixed * Fix a race condition that prevents Logstash from updating a pipeline's configuration with in-flight events experiencing connection errors. https://github.com/elastic/logstash/issues/14739[#14739] This issue primarily manifests following the update of Elasticsearch credentials through Central Management, after credentials expired while events were in-flight. It causes the Elasticsearch Output to get stuck attempting to send events with the expired credentials instead of using the updated ones. To address this problem, Logstash has improved the pipeline shutdown phase functionality to allow an output plugin to request the termination of the in-flight batch of events; hence preventing the need for administrators to manually restart Logstash. Furthermore, when used in combination with a persistent queue to prevent data loss, the batch is eligible for reprocessing on pipeline restart. Plugin developers can now decide whether to make use of such functionality on output plugins. https://github.com/elastic/logstash/pull/14940[#14940] [[dependencies-8.8.0]] ==== Updates to dependencies * Updates Bundler to version 2.4 https://github.com/elastic/logstash/pull/14995[#14995] ==== Plugins *Elasticsearch Filter - 3.15.0* * Standardize SSL settings to comply with Logstash's naming convention https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/168[#168] * Added support for configurable retries with new `retry_on_failure` and `retry_on_status` options https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/160[#160] *Memcached Filter - 1.2.0* * Upgrade Dalli to 3.x https://github.com/logstash-plugins/logstash-filter-memcached/pull/33[#33] *Beats Input - 6.6.0* * Standardize SSL settings to comply with Logstash's naming convention https://github.com/logstash-plugins/logstash-input-beats/pull/470[#470] *Elasticsearch Input - 4.17.0* * Standardize SSL settings to comply with Logstash's naming convention https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/185[#185] *Http Input - 3.7.0* * Standardize SSL settings to comply with Logstash's naming convention https://github.com/logstash-plugins/logstash-input-http/pull/165[#165] *Kafka Integration - 11.2.1* * Fix nil exception to empty headers of record during event metadata assignment https://github.com/logstash-plugins/logstash-integration-kafka/pull/140[#140] * Added TLS truststore and keystore settings specifically to access the schema registry https://github.com/logstash-plugins/logstash-integration-kafka/pull/137[#137] * Added config `group_instance_id` to use the Kafka's consumer static membership feature https://github.com/logstash-plugins/logstash-integration-kafka/pull/135[#135] * Changed Kafka client to 3.3.1, requires Logstash >= 8.3.0. * Deprecated `default` value for setting `client_dns_lookup` forcing to `use_all_dns_ips` when explicitly used https://github.com/logstash-plugins/logstash-integration-kafka/pull/130[#130] * Changed the consumer's poll from using the one that blocks on metadata retrieval to the one that doesn't https://github.com/logstash-plugins/logstash-integration-kafka/pull/133[#136] *Normalize_config_support Mixin - 1.0.0* *Elasticsearch Output - 11.15.1* * Fixed race condition during plugin registration phase https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1125[#1125] * Added the ability to negatively acknowledge the batch under processing if the plugin is blocked in a retry-error-loop and a shutdown is requested. https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1119[#1119] * Standardize SSL settings to comply with Logstash's naming convention https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1118[#1118] [[logstash-8-7-1]] === Logstash 8.7.1 Release Notes [[notable-8.7.1]] ==== Performance improvements and notable issues fixed * Fix inversion of pluginId and pluginType parameteres in DLQ entry creation https://github.com/elastic/logstash/pull/14906[#14906] * Fix pipeline crash when reopening empty DLQ for writing https://github.com/elastic/logstash/pull/14981[#14981] * Fix value of TimeoutStopSec on older systemd versions https://github.com/elastic/logstash/pull/14984[#14984] [[docs-8.7.1]] ==== Documentation enhancements * Document meaning of infinite flow metric rates https://github.com/elastic/logstash/pull/14999[#14999] [[dependencies-8.7.1]] ==== Updates to dependencies * Update JDK to 17.0.7+7 https://github.com/elastic/logstash/pull/15015[#15015] ==== Plugins *Fluent Codec - 3.4.2* * Fix: Convert LogStash::Timestamp values to iso-8601 to resolve crash issue with `msgpack` serialization https://github.com/logstash-plugins/logstash-codec-fluent/pull/30[#30] *Http Filter - 1.4.3* * DOC: add clarification on sending data as json https://github.com/logstash-plugins/logstash-filter-http/pull/48[#48] * Fix: resolve content type when a content-type header contains an array https://github.com/logstash-plugins/logstash-filter-http/pull/46[#46] *Useragent Filter - 3.3.4* * Upgrade `snakeyaml` dependency to 1.33 https://github.com/logstash-plugins/logstash-filter-useragent/pull/84[#84] *Aws Integration - 7.1.1* * Fix failure to load Java dependencies making v7.1.0 unusable https://github.com/logstash-plugins/logstash-integration-aws/pull/24[#24] [[logstash-8-7-0]] === Logstash 8.7.0 Release Notes [[features-8.7.0]] ==== New features and enhancements * Added `xpack.geoip.downloader.enabled` setting to manage auto-update GeoIP database https://github.com/elastic/logstash/pull/14823[#14823] * Flow metrics improvements: `throughput` for input plugins, `worker_utilization` and `worker_millis_per_event` (worker cost per event) for filter and output plugin metrics are implemented https://github.com/elastic/logstash/pull/14743[#14743] [[notable-8.7.0]] ==== Performance improvements and notable issues fixed * Fix: DLQ writer isn't properly created due to inversion of parameteres https://github.com/elastic/logstash/pull/14900[#14900] * Logstash fails to start on OracleLinux7 https://github.com/elastic/logstash/pull/14890[#14890] * Fix: DLQ age policy isn't executed if the current head segment haven't receives any write https://github.com/elastic/logstash/pull/14878[#14878] * Fixes an issue during process shutdown in which the stalled shutdown watcher incorrectly reports `inflight_count` as `0` even when there are events in-flight https://github.com/elastic/logstash/pull/14760[#14760] [[core-8.7.0]] ==== Other changes to Logstash core * Allow `dead_letter_queue.retain.age` usage in pipeline settings https://github.com/elastic/logstash/pull/14954[#14954] * Improved logging behavior in a docker container https://github.com/elastic/logstash/pull/14949[#14949] * `snakeyaml` upgraded to `1.33` version https://github.com/elastic/logstash/pull/14881[#14881] * Update bundeld JDK to `17.0.6+10` https://github.com/elastic/logstash/pull/14849[#14849] * `jrjackson` to `0.4.17` and `jackson` to `2.14.1` versions upgraded https://github.com/elastic/logstash/pull/14845[#14845] * Pins `murmurhash3` to `0.1.6` version https://github.com/elastic/logstash/pull/14832[#14832] * Guard reserved tags field against incorrect use https://github.com/elastic/logstash/pull/14822[#14822] * Remove unnecessary pipeline configuration logging https://github.com/elastic/logstash/pull/14779[#14779] * Developers using the Ruby- or Java-based Plugin APIs will have access to a new API boundary for acquiring a timer object to track execution time https://github.com/elastic/logstash/pull/14748[#14748] [[docs-8.7.0]] ==== Documentation enhancements * Describe how to use Elastic Agent to monitor Logstash https://github.com/elastic/logstash/pull/14959[#14959] * Update Debian/Ubuntu instructions following apt-key deprecation https://github.com/elastic/logstash/pull/14835[#14835] [[plugins-8.7.0]] ==== Plugin releases *Dns Filter - 3.2.0* * Add tag(s) on DNS lookup times outs, defaults to `["_dnstimeout"]` https://github.com/logstash-plugins/logstash-filter-dns/pull/67[#67] *Syslog_pri Filter - 3.2.0* * Add tag on unrecognized `facility_label` code https://github.com/logstash-plugins/logstash-filter-syslog_pri/pull/11[#11] *Beats Input - 6.5.0* * Added `enrich` enrichment option to control ECS passthrough. `ssl_peer_metadata` and `include_codec_tag` configurations are deprecated and can be managed through the `enrich` https://github.com/logstash-plugins/logstash-input-beats/pull/464[#464] *Aws Integration - 7.1.0* * Restore and upload corrupted GZIP files to AWS S3 after abnormal termination https://github.com/logstash-plugins/logstash-integration-aws/pull/20[#20] *Elasticsearch Output - 11.13.1* * Avoid a crash by ensuring ILM settings are injected in the correct location depending on the default (or custom) template format, `template_api` setting and ES version https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1102[#1102] * Technology preview support for allowing events to individually encode a default pipeline with `[@metadata][target_ingest_pipeline]` (as part of a technology preview, this feature may change without notice) https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1113[#1113] [[logstash-8-6-2]] === Logstash 8.6.2 Release Notes [[dependencies-8.6.2]] ==== Updates to dependencies * Updated JRuby to 9.3.10.0 https://github.com/elastic/logstash/pull/14865[#14865] * Updated bundled JDK to 17.0.6+10 https://github.com/elastic/logstash/pull/14855[#14855] ==== Plugins *Fingerprint Filter - 3.4.2* * Key config type changed to `password` type for better protection from leaks. https://github.com/logstash-plugins/logstash-filter-fingerprint/pull/71[#71] *Aws Integration - 7.0.1* * Resolved race conditions in the S3 Output plugin when handling temporary files https://github.com/logstash-plugins/logstash-integration-aws/pull/19[#19] *Elasticsearch Output - 11.12.4* * Changed `manage_template` default value to `false` when data streams is enabled https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1111[#1111] [[logstash-8-6-1]] === Logstash 8.6.1 Release Notes [[dependencies-8.6.1]] ==== Updates to dependencies * Updated snakeyaml to 1.33 https://github.com/elastic/logstash/pull/14848[#14848] [[logstash-8-6-0]] === Logstash 8.6.0 Release Notes [[features-8.6.0]] ==== New features and enhancements * Extends the flow rates introduced to the Node Stats API in 8.5.0 (which included windows for `current` and `lifetime`) to include a Technology Preview of several additional windows such as `last_15_minutes`, `last_24_hours`, etc.. https://github.com/elastic/logstash/pull/14571[#14571] * Logstash introduced instance and pipeline level flow metrics, `growth_bytes` and `growth_events` for persisted queue to provide a better visibility about how fast pipeline queue is growing. https://github.com/elastic/logstash/pull/14554[#14554] [[notable-8.6.0]] ==== Notable issues fixed * Adds new `close` method to Java's Filter API to be used to clean shutdown resources allocated by the filter during registration phase. https://github.com/elastic/logstash/pull/14485[#14485] * Improved JRuby runtime startup avoiding to compile ahead each Ruby code encountered. https://github.com/elastic/logstash/pull/14284[#14284] * Fixed issue in pipeline compilation. https://github.com/elastic/logstash/pull/13621[#13621] [[docs-8.6.0]] ==== Documentation enhancements * Crafted a guide on how to configure and troubleshooting Logstash on Kubernetes. ** Getting started https://github.com/elastic/logstash/pull/14655[#14655] ** Persistent Storage https://github.com/elastic/logstash/pull/14714[#14714] ** Stack Monitoring https://github.com/elastic/logstash/pull/14696[#14696] ** Securing Logstash https://github.com/elastic/logstash/pull/14737[#14737] [[plugins-8.6.0]] ==== Plugin releases *Netflow Codec - 4.3.0* * Added Gigamon ipfix definitions https://github.com/logstash-plugins/logstash-codec-netflow/pull/199[#199] *Elasticsearch Filter - 3.13.0* * Added support for this plugin identifying itself to Elasticsearch with an SSL/TLS client certificate using a new `keystore` option https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/162[#162] *Jdbc Integration - 5.4.1* * Bugfix leak which happened in creating a new Database pool for every query. The pool is now crated on registration and closed on plugin's `stop` https://github.com/logstash-plugins/logstash-integration-jdbc/pull/119[#119] * Ambiguous Timestamp Support https://github.com/logstash-plugins/logstash-integration-jdbc/pull/92[#92] ** FIX: when encountering an ambiguous timestamp, the JDBC Input no longer crashes ** Added support for disambiguating timestamps in daylight saving time (DST) overlap periods *Elasticsearch Output - 11.12.1* * Log bulk request response body on error, not just when debug logging is enabled https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1096[#1096] * Add legacy template API support for Elasticsearch 8 https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1092[#1092] * When using an `api_key` along with either `cloud_id` or https `hosts`, you no longer need to also specify `ssl => true` https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1065[#1065] * Feature: expose `dlq_routed` document metric to track the documents routed into DLQ https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1090[#1090] [[logstash-8-5-3]] === Logstash 8.5.3 Release Notes No user-facing changes in Logstash core. [[plugins-8-5-3]] ==== Plugins No user-facing changes in Logstash plugins. [[logstash-8-5-2]] === Logstash 8.5.2 Release Notes No user-facing changes in Logstash core. [[plugins-8-5-2]] ==== Plugins No user-facing changes in Logstash plugins. [[logstash-8-5-1]] === Logstash 8.5.1 Release Notes [[notable-8.5.1]] ==== Notable issues fixed * Fixes the reporting of configuration errors when using multiple-pipelines to make them more actionable https://github.com/elastic/logstash/pull/14713[#14713] [[dependencies-8.5.1]] ==== Updates to dependencies * The bundled JDK has been updated to 17.0.5+8 https://github.com/elastic/logstash/pull/14728[#14728] [[plugins-8-5-1]] ==== Plugins *Cef Codec - 6.2.6* * Fix: when decoding, escaped newlines and carriage returns in extension values are now correctly decoded into literal newlines and carriage returns respectively https://github.com/logstash-plugins/logstash-codec-cef/pull/98[#98] * Fix: when decoding, non-CEF payloads are identified and intercepted to prevent data-loss and corruption. They now cause a descriptive log message to be emitted, and are emitted as their own `_cefparsefailure`-tagged event containing the original bytes in its `message` field https://github.com/logstash-plugins/logstash-codec-cef/issues/99[#99] * Fix: when decoding while configured with a `delimiter`, flushing this codec now correctly consumes the remainder of its internal buffer. This resolves an issue where bytes that are written without a trailing delimiter could be lost https://github.com/logstash-plugins/logstash-codec-cef/issues/100[#100] *Json Codec - 3.1.1* * Fix: when decoded JSON includes an `[event][original]` field, having `ecs_compatibility` enabled will no longer overwrite the decoded field https://github.com/logstash-plugins/logstash-codec-json/pull/43[#43] *Grok Filter - 4.4.3* * Minor typos in docs examples https://github.com/logstash-plugins/logstash-filter-grok/pull/176[#176] *Tcp Input - 6.3.1* * Fixes a regression in which the ssl_subject was missing for SSL-secured connections in server mode https://github.com/logstash-plugins/logstash-input-tcp/pull/199[#199] *Unix Input - 3.1.2* * Fix: eliminate high CPU usage when data timeout is disabled and no data is available on the socket https://github.com/logstash-plugins/logstash-input-unix/pull/30[#30] *Rabbitmq Integration - 7.3.1* * DOCS: clarify the availability and cost of using the `metadata_enabled` option https://github.com/logstash-plugins/logstash-integration-rabbitmq/pull/52[#52] *Elasticsearch Output - 11.9.3* * DOC: clarify that `http_compression` option only affects _requests_; compressed _responses_ have always been read independent of this setting https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1030[#1030] * Fix broken link to Logstash Reference https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1085[#1085] * Fixes a possible infinite-retry-loop that could occur when this plugin is configured with an `action` whose value contains a <> that fails to be resolved for an individual event. Events in this state are routed to the pipeline's <> if the DLQ is enabled. Otherwise, these events are logged-and-dropped so that the remaining events in the batch can be processed. https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1080[#1080] [[logstash-8-5-0]] === Logstash 8.5.0 Release Notes [[known-issues-8.5.0]] ==== Known issues Due to a recent change in the Red Hat scan verification process, this version of Logstash is not available in the Red Hat Ecosystem Catalog. This bug will be fixed in the next release. Please use the https://www.docker.elastic.co/r/logstash/logstash[Elastic docker registry] to download the 8.5.0 Logstash image. [[features-8.5.0]] ==== New features and enhancements * It is often difficult to understand the health of a pipeline, including whether it is exerting or propagating back-pressure or otherwise staying reasonably “caught up” with its inputs. This release adds pipeline "flow" metrics to the node_stats API for each pipeline, which includes the current and lifetime rates for five key pipeline metrics: input_throughput, filter_throughput, output_throughput, queue_backpressure, and worker_concurrency. https://github.com/elastic/logstash/pull/14518[#14518] [[notable-8.5.0]] ==== Notable issues fixed * Added missing "monitoring.cluster_uuid" to the env2yaml list of accepted configurations and enables the user to set this configuration option via environment variable https://github.com/elastic/logstash/pull/14425[#14425] * Use COPY instruction instead of ADD in Dockerfiles https://github.com/elastic/logstash/pull/14423[#14423] [[docs-8.5.0]] ==== Documentation Improvements and Fixes * Add missing reference to full config of Logstash to Logstash over HTTP https://github.com/elastic/logstash/pull/14466[#14466] * Describe DLQ's age retention policy https://github.com/elastic/logstash/pull/14340[#14340] * Document the cleaning of consumed events from DLQ https://github.com/elastic/logstash/pull/14341[#14341] ==== Plugins *Translate Filter - 3.4.0* * Refactor: leverage scheduler mixin https://github.com/logstash-plugins/logstash-filter-translate/pull/93[#93] *Elasticsearch Input - 4.16.0* * Added `ssl_certificate_verification` option to control SSL certificate verification https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/180[#180] * Feat: add `retries` option. allow retry for failing query https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/179[#179] *Exec Input - 3.6.0* * Refactor: start using scheduler mixin https://github.com/logstash-plugins/logstash-input-exec/pull/33[#33] * Fix: behavior incompatiblity between (standalone) LS and LS in Docker https://github.com/logstash-plugins/logstash-input-exec/pull/30[#30] *File Input - 4.4.4* * Fixes gzip file handling in read mode when run on JDK12+, including JDK17 that is bundled with Logstash 8.4+ https://github.com/logstash-plugins/logstash-input-file/pull/312[#312] *Http_poller Input - 5.4.0* * Refactor: start using scheduler mixin https://github.com/logstash-plugins/logstash-input-http_poller/pull/134[#134] *Elasticsearch Output - 11.9.0* * Feature: force unresolved dynamic index names to be sent into DLQ. This feature could be explicitly disabled using `dlq_on_failed_indexname_interpolation` setting https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1084[#1084] * Feature: Adds a new `dlq_custom_codes` option to customize DLQ codes https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1067[#1067] * Feature: deprecates the `failure_type_logging_whitelist` configuration option, renaming it `silence_errors_in_log` https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1068[#1068] [[logstash-8-4-2]] === Logstash 8.4.2 Release Notes [[notable-8.4.2]] ==== Notable issues fixed * Fixed the inability to configure "monitoring.cluster_uuid" in docker https://github.com/elastic/logstash/pull/14496[#14496] * Disabled DES-CBC3-SHA cipher in some plugins that still supported it https://github.com/elastic/logstash/pull/14501[#14501] * Upgraded JRuby the CSV gem to fix a thread leak in Logstash 8.4.0 when using the CSV filter https://github.com/elastic/logstash/pull/14508[#14508] https://github.com/elastic/logstash/pull/14526[#14526] * Fixed Windows .bat scripts that prevented the use of the Plugin Manager and Keystore in Logstash 8.3.3/8.4.0 https://github.com/elastic/logstash/pull/14516[#14516] [[docs-8-4-2]] ==== Documentation improvements * Added https://www.elastic.co/guide/en/logstash/8.4/winlogbeat-modules.html[documentation for using Winlogbeat] with Logstash https://github.com/elastic/logstash/pull/14512[#14512] [[logstash-8-4-1]] === Logstash 8.4.1 Release Notes No user-facing changes in Logstash core. ==== Plugins *Beats Input - 6.4.1* * [DOC] Add direct memory example https://github.com/logstash-plugins/logstash-input-beats/pull/454[#454] *Gelf Input - 3.3.2* * Fix: avoid panic when handling very-large exponent-notation `_@timestamp` values https://github.com/logstash-plugins/logstash-input-gelf/pull/71[#71] *Tcp Output - 6.1.1* * Fixes an issue where payloads larger than a connection's current TCP window could be silently truncated https://github.com/logstash-plugins/logstash-output-tcp/pull/49[#49] [[logstash-8-4-0]] === Logstash 8.4.0 Release Notes [[features-8.4.0]] ==== New features and enhancements [[dlq-8.4.0]] ===== Improvements to the dead letter queue (DLQ) This release brings significant improvements to help users manage their dead letter queues, including: * A new `clean_consumed` option on the Dead Letter Queue input plugin. It can automatically delete segments from a dead letter queue after all events in the segment have been consumed by a Logstash pipeline. * A new age retention policy, enabling the automatic removal of segments from a dead letter queue based on the age of events within those segments. * Additional dead letter queue metrics available from the monitoring API https://github.com/elastic/logstash/pull/14324[#14324] [[aws-8.4.0]] ===== New AWS integration plugin Several AWS plugins are now bundled in a single {logstash-ref}/plugins-integrations-aws.html[AWS integration plugin], enabling easier maintenance and upgrades of AWS-based plugins. They all use version 3 of the AWS Ruby SDK. [[jdk-8.4.0]] ===== JDK17 support Logstash now comes bundled with JDK17, while still providing compatibility with user-supplied JDK11. The new JDK includes an update pertaining to a potential security vulnerability. Please see our link:https://discuss.elastic.co/c/announcements/security-announcements/31[security statement for details]. [[m1-8.4.0]] ===== Logstash M1 download Logstash is now available for download on M1 equipped MacOS devices, and comes bundled with M1 native JDK17. [[notable-8.4.0]] ==== Notable issues fixed * Remove `/etc/systemd/system/logstash.service` only when file is installed by Logstash https://github.com/elastic/logstash/pull/14200[#14200] * Fix Arcsight module compatibility with Elasticsearch `8.x` https://github.com/elastic/logstash/pull/13874[#13874] * Ensure that timestamp values are serialized with a minimum of 3 decimal places to guarantee that millisecond precision timestamps match those from Logstash `7.x` https://github.com/elastic/logstash/pull/14299[#14299] * Fix issue with native Java plugin thread-safety and concurrency https://github.com/elastic/logstash/pull/14360[#14360] * Allow the ability to use Ruby codecs inside native Java plugins https://github.com/elastic/logstash/pull/13523[#13523] [[dependencies-8.4.0]] ==== Updates to dependencies * The bundled JDK has been updated to 17.0.4+8 https://github.com/elastic/logstash/pull/14427[#14427] * The version of Sinatra has been updated to 2.2.2 https://github.com/elastic/logstash/pull/14454[#14454] * The version of Nokogiri has been updated to 1.13.8 https://github.com/elastic/logstash/pull/14454[#14454] [[plugins-8.4.0]] ==== Plugin releases *Dead Letter Queue Input - 2.0.0* * Introduce the boolean `clean_consumed` setting to enable the automatic removal of completely consumed segments. Requires Logstash 8.4.0 or above https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/43[#43] * Expose metrics about segments and events cleaned by this plugin https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/45[#45] *Xml Filter - 4.2.0* * Update Nokogiri dependency version https://github.com/logstash-plugins/logstash-filter-xml/pull/78[#78] *Aws Integration Plugin - 7.0.0* * This new integration plugin incorporates and replaces the use of the these individual plugins: individual plugins: ** logstash-input-s3 ** logstash-input-sqs ** logstash-mixin-aws ** logstash-output-cloudwatch ** logstash-output-s3 ** logstash-output-sns ** logstash-output-sqs * This replaces the use of the single combined aws `2.x` sdk gem, with the modularized aws `3.x` gems. [[logstash-8-3-3]] === Logstash 8.3.3 Release Notes [[notable-8.3.3]] ==== Notable issue fixed * We fixed an issue which occurred when users ran the plugin manager or the keystore with the bundled JVM. Some mandatory JVM options were not being picked up from the JvmOptionsParser, breaking compatibility with Windows on certain versions of the JDK. https://github.com/elastic/logstash/pull/14355[#14355] [[plugins-8.3.3]] ==== Plugin releases *Date Filter - 3.1.15* * Build: review build to be more reliable/portable https://github.com/logstash-plugins/logstash-filter-date/pull/139[#139] * Cleaned up Java dependencies *Fingerprint Filter - 3.4.1* * Added backward compatibility of timestamp format to provide consistent fingerprint https://github.com/logstash-plugins/logstash-filter-fingerprint/pull/67[#67] *Http_poller Input - 5.3.1* * Fix: Make sure plugin is closing the http client https://github.com/logstash-plugins/logstash-input-http_poller/pull/130[#130] *Scheduler Mixin - 1.0.1* * Refactor: Initialize time zone data eagerly https://github.com/logstash-plugins/logstash-mixin-scheduler/pull/2[#2] *Core Patterns - 4.3.4* * Fix: Correct typo in CISCOFW302013_302014_302015_302016 grok pattern https://github.com/logstash-plugins/logstash-patterns-core/pull/313[#313] [[logstash-8-3-2]] === Logstash 8.3.2 Release Notes No user-facing changes in this release. [[logstash-8-3-1]] === Logstash 8.3.1 Release Notes [[notable-8.3.1]] ==== Notable issues fixed * We fixed an event serializing incompatibility introduced in 8.3.0 https://github.com/elastic/logstash/pull/14314[#14314] If you're using dead letter queues or persistent queues we recommend that you do not use Logstash 8.3.0 and upgrade to 8.3.1. [[logstash-8-3-0]] === Logstash 8.3.0 Release Notes [[known-issue-8-3-0]] ==== Known issue An event serialization bug was discovered, which causes an issue when trying to read dead letter or persistent queues created in previous versions of Logstash. We recommend not upgrading to Logstash 8.3.0 if you are using dead letter or persistent queues. [[features-8.3.0]] ==== New features and enhancements * {ls} is more efficient at fetching pipelines as of 8.3.0. When a {ls} instance sends its pipeline IDs to {es} or central pipeline management in {kib}, it gets back only the pipeline configs that belong to that instance. These enhancements required changes to both {ls} https://github.com/elastic/logstash/pull/14076[(#14076)] and {es} https://github.com/elastic/elasticsearch/pull/85847[(#85847)]. + These improvements dramatically decrease network load while also giving users the ability to control pipelines dynamically using wildcards. * Dead Letter Queues can now be configured to drop older events instead of new ones when they're full. The setting "dead_letter_queue.storage_policy" has been introduced for this purpose, and new metrics - such as a counter for dropped events - are now exposed in the API to better monitor the DLQ behavior. https://github.com/elastic/logstash/pull/13923[#13923] https://github.com/elastic/logstash/pull/14058[#14058] * To improve security of Logstash deployments, 8.3.0 brings a new setting "allow_superuser" that defaults to false. When enabled it prevents Logstash from starting as super user ("root"). This setting will be enabled by default in the future. Consider explicitly enabling it. Otherwise a deprecation log entry will be emitted. https://github.com/elastic/logstash/pull/14046[#14046] https://github.com/elastic/logstash/pull/14089[#14089] * Continuing with the focus on security, we've introduced "api.auth.basic.password_policy.mode" to ensure the password used to guard Logstash's HTTP API has a minimum set of strength requirements. By default a warning will be emitted if the defined password doesn't meet the criteria, but in a future release the mode will be set to "ERROR". https://github.com/elastic/logstash/pull/14045[#14045] https://github.com/elastic/logstash/pull/14105[#14105] https://github.com/elastic/logstash/pull/14159[#14159] * Elasticsearch introduced "security on by default" back in 8.0.0, with TLS enabled by default in its HTTP and transport layers. To facilitate connecting to 8.x clusters, Elasticsearch displays the fingerprint of the Certificate Authority it generates on startup. This release of Logstash introduces support for setting "ca_trusted_fingerprint" in Elasticsearch input, filter and outputs plugins. https://github.com/elastic/logstash/pull/14120[#14120] * Technical Preview: Receiving events containing keys with characters that have special meaning to Logstash such as `[` and `]` (for field references) has always causes issues to data ingestion. A new setting in Technical Preview, disabled by default, called "config.field_reference.escape_style" was introduced to handle such special characters by escaping them. https://github.com/elastic/logstash/pull/14044[#14044] [[notable-8.3.0]] ==== Notable issues fixed * Don't display values of password type settings in conditionals https://github.com/elastic/logstash/pull/13997[#13997] * Introduce a retry mechanism in pipeline-to-pipeline instead of crashing https://github.com/elastic/logstash/pull/14027[#14027] * Add thread safety around Puma startup/shutdown https://github.com/elastic/logstash/pull/14080[#14080] * Add value converters for java.time classes https://github.com/elastic/logstash/pull/13972[#13972] * Correct the class reference to the MetricNotFound exception https://github.com/elastic/logstash/pull/13970[#13970] * Fix a possible corruption of Persistent Queue during a crash of the Logstash process https://github.com/elastic/logstash/pull/14165[#14165] [[dependencies-8.3.0]] ==== Updates to dependencies * The bundled JDK 11 has been updated to 11.0.15+10 https://github.com/elastic/logstash/pull/14031[#14031] * Jackson and jackson-databind have been updated to 2.13.3 https://github.com/elastic/logstash/pull/13945[#13945] [[plugins-8.3.0]] ==== Plugin releases *Avro Codec - 3.4.0* * Add `encoding` option to select the encoding of Avro payload, could be `binary` or `base64` https://github.com/logstash-plugins/logstash-codec-avro/pull/39[#39] *Elasticsearch Filter - 3.12.0* * Add support for `ca_trusted_fingerprint` when run on Logstash 8.3+ https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/158[#158] *Fingerprint Filter - 3.4.0* * Add support for 128bit murmur variant https://github.com/logstash-plugins/logstash-filter-fingerprint/pull/66[#66]. *Azure_event_hubs Input - 1.4.4* * Fix: Replace use of block with lambda to fix wrong number of arguments error on jruby-9.3.4.0 https://github.com/logstash-plugins/logstash-input-azure_event_hubs/pull/75[#75] *Beats Input - 6.4.0* * Feat: review and deprecate ssl protocol/cipher settings https://github.com/logstash-plugins/logstash-input-beats/pull/450[#450] *Elasticsearch Input - 4.14.0* * Refactor: switch to using scheduler mixin https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/177[#177] * Add support for `ca_trusted_fingerprint` when run on Logstash 8.3+ https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/178[#178] *Http Input - 3.6.0* * Feat: review and deprecate ssl protocol/cipher related settings https://github.com/logstash-plugins/logstash-input-http/pull/151[#151] *Jms Input - 3.2.2* * Fix: Remove usage of `java_kind_of?` to allow this plugin to be supported for versions of Logstash using jruby-9.3.x https://github.com/logstash-plugins/logstash-input-jms/pull/54[#54] *S3 Input - 3.8.4* * Refactoring, reuse code to manage `additional_settings` from mixin-aws https://github.com/logstash-plugins/logstash-input-s3/pull/237[#237] *Sqs Input - 3.3.2* * Fix an issue that prevented timely shutdown when subscribed to an inactive queue * Refactoring: used logstash-mixin-aws to leverage shared code to manage `additional_settings` https://github.com/logstash-plugins/logstash-input-sqs/pull/64[#64] *Tcp Input - 6.3.0* * Feat: ssl_supported_protocols (TLSv1.3) + ssl_cipher_suites https://github.com/logstash-plugins/logstash-input-tcp/pull/198[#198] *Jdbc Integration - 5.3.0* * Refactor: start using scheduler mixin https://github.com/logstash-plugins/logstash-integration-jdbc/pull/110[#110] * Fix: change default path of 'last_run_metadata_path' to be rooted in the LS data.path folder and not in $HOME https://github.com/logstash-plugins/logstash-integration-jdbc/pull/106[#106] *Kafka Integration - 10.12.0* * bump kafka client to 2.8.1 https://github.com/logstash-plugins/logstash-integration-kafka/pull/115[#115] * Feat: add connections_max_idle_ms setting for output https://github.com/logstash-plugins/logstash-integration-kafka/pull/118[#118] *Aws Mixin - 5.1.0* * Add support for 'additional_settings' configuration options used by S3 and SQS input plugins https://github.com/logstash-plugins/logstash-mixin-aws/pull/53[#53]. * Drop support for aws-sdk-v1 *Elasticsearch Output - 11.6.0* * Add support for `ca_trusted_fingerprint` when run on Logstash 8.3+ https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1074[#1074] * Feat: add ssl_supported_protocols option https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1055[#1055] * [DOC] Add `v8` to supported values for ecs_compatiblity defaults https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1059[#1059] *S3 Output - 4.3.7* * Refactor: avoid usage of ConcurrentHashMap (JRuby 9.3.4 work-around) https://github.com/logstash-plugins/logstash-output-s3/pull/248[#248] * Docs: more documentation on restore + temp dir https://github.com/logstash-plugins/logstash-output-s3/pull/236[#236] *Tcp Output - 6.1.0* * Feat: add support for TLS v1.3 https://github.com/logstash-plugins/logstash-output-tcp/pull/47[#47] * Fix: close server and client sockets on plugin close [[logstash-8-2-3]] === Logstash 8.2.3 Release Notes * Updated bundled JDK to 11.0.15+10 https://github.com/elastic/logstash/pull/14152[#14152] [[logstash-8-2-2]] === Logstash 8.2.2 Release Notes [[notable-8.2.2]] ==== Notable issues fixed * Avoid unnecessary thread synchronization when the Persistent Queue is full https://github.com/elastic/logstash/pull/14141[#14141] [[logstash-8-2-1]] === Logstash 8.2.1 Release Notes [[notable-8.2.1]] ==== Notable issues fixed * Added mandatory JVM option to avoid strict path checking introduced with recent JVM versions, starting from 11.0.15+10, 17.0.3+7. https://github.com/elastic/logstash/pull/14066[#14066] * Fixed Dead Letter Queue bug happening in position retrieval and restore. This happened when the DLQ input plugin used `commit_offset` feature. https://github.com/elastic/logstash/pull/14093[#14093] * Fixes an issue where custom java plugins were unable to be installed and run correctly when retrieved from rubygems.org. https://github.com/elastic/logstash/pull/14060[#14060] * Fixed no metrics update issue when PQ is draining. https://github.com/elastic/logstash/pull/13935[#13935] ==== Plugins *Cef Codec - 6.2.5* * [DOC] Update link to CEF implementation guide https://github.com/logstash-plugins/logstash-codec-cef/pull/97[#97] *Dns Filter - 3.1.5* * Fixed an issue where a non-string value existing in the resolve/reverse field could cause the plugin to crash https://github.com/logstash-plugins/logstash-filter-dns/pull/65[#65] *Grok Filter - 4.4.2* * Clarify the definition of matches that depend on previous captures https://github.com/logstash-plugins/logstash-filter-grok/pull/169[#169] *Http Filter - 1.4.1* * Fix: don't process response body for HEAD requests https://github.com/logstash-plugins/logstash-filter-http/pull/40[#40] *Beats Input - 6.3.1* * Fix: Removed use of deprecated `import` of java classes in ruby https://github.com/logstash-plugins/logstash-input-beats/pull/449[#449] *File Input - 4.4.2* * Doc: Fix attribute by removing extra character https://github.com/logstash-plugins/logstash-input-file/pull/310[#310] * Fix: update to Gradle 7 https://github.com/logstash-plugins/logstash-input-file/pull/305[#305] * [DOC] Add version attributes to doc source file https://github.com/logstash-plugins/logstash-input-file/pull/308[#308] *Http Input - 3.5.1* * Fix: codecs provided with `additional_codecs` now correctly run in the pipeline's context, which means that they respect the `pipeline.ecs_compatibility` setting https://github.com/logstash-plugins/logstash-input-http/pull/152[#152] *Jdbc Integration - 5.2.5* * Fix: do not execute more queries with debug logging https://github.com/logstash-plugins/logstash-integration-jdbc/pull/109[#109] *Core Patterns - 4.3.3* - Fix: parsing x-edge-location in CLOUDFRONT_ACCESS_LOG (ECS mode) https://github.com/logstash-plugins/logstash-patterns-core/pull/311[#311] [[logstash-8-2-0]] === Logstash 8.2.0 Release Notes ==== Breaking changes * Starting with Logstash 8.0 all supported and tested operating systems use system.d so this release removes leftover SysVinit scripts from .deb and .rpm packages https://github.com/elastic/logstash/pull/13954[#13954] https://github.com/elastic/logstash/pull/13955[#13955] [[notable-8.2.0]] ==== Notable issues fixed * Improved resiliency of Central Management requests when an Elasticsearch node is down https://github.com/elastic/logstash/pull/13689[#13689] https://github.com/elastic/logstash/pull/13941[#13941] * Ensure safe retrieval of queue stats that may not yet be populated https://github.com/elastic/logstash/pull/13942[#13942] * Print bundled JDK's version in launch scripts when `LS_JAVA_HOME` is provided https://github.com/elastic/logstash/pull/13880[#13880] * Updated jackson-databind to 2.13.2 in ingest-converter tool https://github.com/elastic/logstash/pull/13900[#13900] * Updated google-java-format dependency to 1.13.0 and guava to 31.0.1 in core https://github.com/elastic/logstash/pull/13700[#13700] * Multiple documentation improvements related to: Logstash to Logstash communication https://github.com/elastic/logstash/pull/13999[#13999], docker variable injection https://github.com/elastic/logstash/pull/12198[#12198], LS-ES security configuration https://github.com/elastic/logstash/pull/14012[#14012], JDK 11 Bundling https://github.com/elastic/logstash/pull/14022[#14022], and other overall documentation restructuring https://github.com/elastic/logstash/pull/14015[#14015]. ==== Plugins *Http Filter - 1.4.0* * Feat: added ssl_supported_protocols option https://github.com/logstash-plugins/logstash-filter-http/pull/38[#38] *Kv Filter - 4.7.0* * Allow attaching multiple tags on failure. The `tag_on_failure` option now also supports an array of strings https://github.com/logstash-plugins/logstash-filter-kv/issues/92[#92] *Beats Input - 6.3.0* * Added support for TLSv1.3. https://github.com/logstash-plugins/logstash-input-beats/pull/447[#447] *Elasticsearch Input - 4.12.3* * Fix: update Elasticsearch Ruby client to correctly customize 'user-agent' header https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/171[#171] *Http Input - 3.5.0* * Feat: TLSv1.3 support https://github.com/logstash-plugins/logstash-input-http/pull/146[#146] *Http_poller Input - 5.3.0* * Feat: added ssl_supported_protocols option https://github.com/logstash-plugins/logstash-input-http_poller/pull/133[#133] *Sqs Input - 3.3.0* * Feature: Add `additional_settings` option to fine-grain configuration of AWS client https://github.com/logstash-plugins/logstash-input-sqs/pull/61[#61] *Kafka Integration - 10.10.0* * Added config setting to enable 'zstd' compression in the Kafka output https://github.com/logstash-plugins/logstash-integration-kafka/pull/112[#112] *Http_client Mixin - 7.2.0* * Feat: add `ssl_supported_protocols` option https://github.com/logstash-plugins/logstash-mixin-http_client/pull/40[#40] *Http Output - 5.5.0* * Feat: added `ssl_supported_protocols` option https://github.com/logstash-plugins/logstash-output-http/pull/131[#131] * Fix retry indefinitely in termination process. This feature requires Logstash 8.1 https://github.com/logstash-plugins/logstash-output-http/pull/129[#129] * Docs: Add retry policy description https://github.com/logstash-plugins/logstash-output-http/pull/130[#130] * Introduce retryable unknown exceptions for "connection reset by peer" and "timeout" https://github.com/logstash-plugins/logstash-output-http/pull/127[#127] [[logstash-8-1-3]] === Logstash 8.1.3 Release Notes No user-facing changes in this release. [[logstash-8-1-2]] === Logstash 8.1.2 Release Notes [[notable-8.1.2]] ==== Notable issues fixed * Fixed issue where Logstash crashed if Central Management couldn't reach Elasticsearch https://github.com/elastic/logstash/pull/13689[#13689] ==== Plugins *Cef Codec - 6.2.4* * [DOC] Emphasize importance of delimiter setting for byte stream inputs https://github.com/logstash-plugins/logstash-codec-cef/pull/95[#95] *Geoip Filter - 7.2.12* * [DOC] Add `http_proxy` environment variable for GeoIP service endpoint. The feature is included in 8.1.0, and was back-ported to 7.17.2 https://github.com/logstash-plugins/logstash-filter-geoip/pull/207[#207] *Truncate Filter - 1.0.5* * Switches behavior of add_tag and add_field, now tags and fields are added only when the truncation happens on any field or nested field https://github.com/logstash-plugins/logstash-filter-truncate/pull/7[#7]. *Tcp Output - 6.0.2* * Fix: unable to start with password protected key https://github.com/logstash-plugins/logstash-output-tcp/pull/45[#45] [[logstash-8-1-1]] === Logstash 8.1.1 Release Notes [[notable-8.1.1]] ==== Notable issues fixed * The `bin/logstash-plugin uninstall ` command works as expected, successfully uninstalling the specified plugin https://github.com/elastic/logstash/pull/13823[#13823] * Logstash CLI tools are now able to use the selected JDK on Windows https://github.com/elastic/logstash/pull/13839[#13839] * Logstash can successfully locate the Windows JVM, even if the path includes spaces https://github.com/elastic/logstash/pull/13881[#13881] * The GeoIP database lookup will now respect a proxy defined with the http_proxy environment variable. https://github.com/elastic/logstash/pull/13840[#13840] ==== Updates to dependencies * The version of the bundled JDK has been updated to 11.0.14.1+1. https://github.com/elastic/logstash/pull/13869[#13869] ==== Plugins *Dissect Filter - 1.2.5* * Fix: bad padding `->` suffix with delimiter https://github.com/logstash-plugins/logstash-filter-dissect/pull/84[#84] *Elasticsearch Filter - 3.11.1* * Fix: hosts => "es_host:port" regression https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/156[#156] *Dead_letter_queue Input - 1.1.11* * Fix: pre-flight checks before creating DLQ reader https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/35[#35] * Fix: avoid Logstash crash on shutdown if DLQ files weren't created https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/33[#33] *Elasticsearch Input - 4.12.2* * Fix: hosts => "es_host:port" regression https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/168[#168] *Http_poller Input - 5.2.1* * Deps: unpin rufus-scheduler dependency https://github.com/logstash-plugins/logstash-input-http_poller/pull/132[#132] *Jdbc Integration - 5.2.4* * Fix: compatibility with all (>= 3.0) rufus-scheduler versions https://github.com/logstash-plugins/logstash-integration-jdbc/pull/97[#97] * Performance: avoid contention on scheduler execution https://github.com/logstash-plugins/logstash-integration-jdbc/pull/103[#103] *Tcp Output - 6.0.1* * Fix: logging fail retry to stdout https://github.com/logstash-plugins/logstash-output-tcp/pull/43[#43] * Fix: Use `reconnect_interval` when establish a connection [[logstash-8-1-0]] === Logstash 8.1.0 Release Notes [[known-issue-8-1-0]] ==== Known issue Uninstalling a plugin using `bin/logtash-plugin uninstall` may result in an error: ``` Gem::LoadError: You have already activated jruby-openssl 0.12.2, but your Gemfile requires jruby-openssl 0.12.1. Prepending `bundle exec` to your command may solve this. ``` Logstash should still run, and other plugin operations, such as `update` and `install`, should work as expected. NOTE: The `bin/logstash-plugin list` command may fail with the same error after a failed uninstallation. **Resolution** A successful plugin `update` will resolve this issue, and allow subsequent `uninstall` and `list` operations to work without issue. The `filter-dissect` plugin has recent changes available for update. Running `bin/logstash-plugin update logstash-filter-dissect` should mitigate this issue. ==== Logstash core No user-facing changes in Logstash core. ==== Plugins *Http Filter - 1.3.0* * Feat: support ssl_verification_mode option https://github.com/logstash-plugins/logstash-filter-http/pull/37[#37] *Kv Filter - 4.6.0* * Added `allow_empty_values` option https://github.com/logstash-plugins/logstash-filter-kv/pull/72[#72] *Http_poller Input - 5.2.0* * Feat: support ssl_verification_mode option https://github.com/logstash-plugins/logstash-input-http_poller/pull/131[#131] *Sqs Input - 3.2.0* * Feature: Add `queue_owner_aws_account_id` parameter for cross-account queues https://github.com/logstash-plugins/logstash-input-sqs/pull/60[#60] *Elastic_enterprise_search Integration - 2.2.1* * Fix, change implementation of connectivity check method to be compatible with version `v8.0+` of Workplace Search https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/16[#16] * Feature, switch the connection library to elastic-enterprise-search https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/3[#3] * [DOC] Added required parameters to Workplace Search example snippet and describe little better what's expected in url parameter https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/11[#11] *Http_client Mixin - 7.1.0* * Feat: add `ssl_verification_mode` https://github.com/logstash-plugins/logstash-mixin-http_client/pull/39[#39] *Http Output - 5.3.0* * Feat: support ssl_verification_mode option https://github.com/logstash-plugins/logstash-output-http/pull/126[#126] [[logstash-8-0-1]] === Logstash 8.0.1 Release Notes [[notable-8.0.1]] ==== Notable issues fixed * Fixed monitoring incompatibility on Windows where the CPU metric was not available. https://github.com/elastic/logstash/pull/13727[#13727] * Recently, users running `bin/logstash-plugin` to install or update plugins stumbled upon an issue that would prevent Logstash from starting due a third-party dependency update. The dependency was pinned to an older version. https://github.com/elastic/logstash/issues/13777[#13777] * Logstash startup and the `pqrepair`/`pqcheck` tools have been improved to handle corrupted files in case of an unexpected shutdown. https://github.com/elastic/logstash/pull/13692[#13692] https://github.com/elastic/logstash/pull/13721[#13721] ==== Plugins *Dissect Filter - 1.2.5* * Fix bad padding `->` suffix with delimiter https://github.com/logstash-plugins/logstash-filter-dissect/pull/84[#84] *Elasticsearch Filter - 3.11.1* * Fix: hosts => "es_host:port" regression https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/156[#156] *Beats Input - 6.2.6* * Update guidance regarding the private key format and encoding https://github.com/logstash-plugins/logstash-input-beats/pull/445[#445] *Dead_letter_queue Input - 1.1.10* * Fix, avoid Logstash crash on shutdown if DLQ files weren't created https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/33[#33] * Fix `@metadata` get overwritten by reestablishing metadata that stored in DLQ https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/34[#34] *Tcp Input - 6.2.7* * Build: skip shadowing jar dependencies https://github.com/logstash-plugins/logstash-input-tcp/pull/187[#187] ** plugin no longer shadows dependencies into its *logstash-input-tcp.jar* ** log4j-api is now a provided dependency and is no longer packaged with the plugin *Jdbc Integration - 5.2.3* * Performance: avoid contention on scheduler execution https://github.com/logstash-plugins/logstash-integration-jdbc/pull/103[#103] *Tcp Output - 6.0.1* * Fixed logging fail retry to stdout https://github.com/logstash-plugins/logstash-output-tcp/pull/43[#43] * Fixed to use `reconnect_interval` when establish a connection [[logstash-8-0-0]] === Logstash 8.0.0 Release Notes The following list are changes in 8.0.0 as compared to 7.17.0, and combines release notes from the 8.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases. [[breaking-8.0.0]] ==== Breaking changes * Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS). This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting. In Logstash 8, this compatibility mode will be on-by-default for all pipelines. If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`). * Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability. * Support for using `JAVA_HOME` to override the path to the JDK that Logstash runs with has been removed for this release. In the `8.x` release, users should set the value of `LS_JAVA_HOME` to the path of their preferred JDK if they wish to use a version other than the bundled JDK. The value of `JAVA_HOME` will be ignored. * The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. https://github.com/elastic/logstash/pull/12517[#12517] * We have added support for UTF-16 and other multi-byte-character when reading log files. https://github.com/elastic/logstash/pull/9702[#9702] * Setting `config.field_reference.parser` has been removed. The Field Reference parser interprets references to fields in your pipelines and plugins. Its behavior was configurable in 6.x, and since 7.0 allowed only a single option: `strict`. 8.0 no longer recognizes the setting, but maintains the same behavior as the `strict` setting. {ls} rejects ambiguous and illegal inputs as standard behavior. https://github.com/elastic/logstash/pull/12466[#12466] For a more detailed view of these changes please check <>. [[features-8.0.0]] ==== New features and enhancements * As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request https://github.com/elastic/logstash/pull/12797[#12797] allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity. * We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. https://github.com/elastic/logstash/pull/12437[#12437] ==== Plugins Logstash 8.0.0 includes the same versions of all bundled plugins as Logstash 7.17.0. If you upgrade to 7.17 before upgrading to 8.0 (as recommended), you won't see any changes to plugin versions. *Clone Filter - 4.2.0* * Added support for ECS v8 as alias for ECS v1 https://github.com/logstash-plugins/logstash-filter-clone/pull/27[#27] *Geoip Filter - 7.2.11* * Improved compatibility with the Elastic Common Schema https://github.com/logstash-plugins/logstash-filter-geoip/pull/206[#206] ** Added support for ECS's composite `region_iso_code` (`US-WA`), which _replaces_ the non-ECS `region_code` (`WA`) as a default field with City databases. To get the stand-alone `region_code` in ECS mode, you must include it in the `fields` directive ** [DOC] Improve ECS-related documentation * [DOC] Air-gapped environment requires both ASN and City databases https://github.com/logstash-plugins/logstash-filter-geoip/pull/204[#204] *Http Filter - 1.2.1* * Fix: do not set content-type if provided by user https://github.com/logstash-plugins/logstash-filter-http/pull/36[#36] * Feat: improve ECS compatibility https://github.com/logstash-plugins/logstash-filter-http/pull/35[#35] * Add support for PUT requests https://github.com/logstash-plugins/logstash-filter-http/pull/34[#34] *Ruby Filter - 3.1.8* * [DOC] Added doc to describe the option `tag_with_exception_message`https://github.com/logstash-plugins/logstash-filter-ruby/pull/62[#62] * Fix SyntaxError handling so other pipelines can shut down gracefully https://github.com/logstash-plugins/logstash-filter-ruby/pull/64[#64] *Useragent Filter - 3.3.3* * Docs: mention added fields in 3.3 with a note https://github.com/logstash-plugins/logstash-filter-useragent/pull/78[#78] *Exec Input - 3.4.0* * Feat: adjust fields for ECS compatibility https://github.com/logstash-plugins/logstash-input-exec/pull/28[#28] * Plugin will no longer override fields if they exist in the decoded payload (It no longer sets the `host` field if decoded from the command's output) *Gelf Input - 3.3.1* * Fix: safely coerce the value of `_@timestamp` to avoid crashing the plugin https://github.com/logstash-plugins/logstash-input-gelf/pull/67[#67] *Generator Input - 3.1.0* * Feat: adjusted fields for ECS compatibility https://github.com/logstash-plugins/logstash-input-generator/pull/22[#22] * Fix: do not override the host field if it's present in the generator line (after decoding) * Fix: codec flushing when closing input *Imap Input - 3.2.0* * Feat: ECS compatibility https://github.com/logstash-plugins/logstash-input-imap/pull/55[#55] * added (optional) `headers_target` configuration option * added (optional) `attachments_target` configuration option * Fix: plugin should not close `$stdin`, while being stopped *Jms Input - 3.2.1* * Fix: improve compatibility with MessageConsumer implementations https://github.com/logstash-plugins/logstash-input-jms/pull/51[#51], such as IBM MQ. * Test: Fix test failures due to ECS compatibility default changes in `8.x` of logstash https://github.com/logstash-plugins/logstash-input-jms/pull/53[#53] * Feat: event_factory support + targets to aid ECS https://github.com/logstash-plugins/logstash-input-jms/pull/49[#49] * Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event * Fix: when adding the `jms_reply_to` header to an event, a string representation is set instead of an opaque object. *Pipe Input - 3.1.0* * Feat: adjust fields for ECS compatibility https://github.com/logstash-plugins/logstash-input-pipe/pull/19[#19] *S3 Input - 3.8.3* * Fix missing `metadata` and `type` of the last event https://github.com/logstash-plugins/logstash-input-s3/pull/223[#223] * Refactor: read sincedb time once per bucket listing https://github.com/logstash-plugins/logstash-input-s3/pull/233[#233] *Snmp Input - 1.3.1* * Refactor: handle no response(s) wout error logging https://github.com/logstash-plugins/logstash-input-snmp/pull/105[#105] * Feat: ECS compliance + optional target https://github.com/logstash-plugins/logstash-input-snmp/pull/99[#99] * Internal: update to Gradle 7 https://github.com/logstash-plugins/logstash-input-snmp/pull/102[#102] *Snmptrap Input - 3.1.0* * Feat: ecs_compatiblity support + (optional) target https://github.com/logstash-plugins/logstash-input-snmptrap/pull/37[#37] *Syslog Input - 3.6.0* * Add support for ECS v8 as alias to v1 implementation https://github.com/logstash-plugins/logstash-input-syslog/pull/68[#68] *Twitter Input - 4.1.0* * Feat: optional target + ecs_compatibility https://github.com/logstash-plugins/logstash-input-twitter/pull/72[#72] *Unix Input - 3.1.1* * Fix: unable to stop plugin (on LS 6.x) https://github.com/logstash-plugins/logstash-input-unix/pull/29[#29] * Refactor: plugin internals got reviewed for `data_timeout => ...` to work reliably * Feat: adjust fields for ECS compatibility https://github.com/logstash-plugins/logstash-input-unix/pull/28[#28] *Jdbc Integration - 5.2.2* * Feat: name scheduler threads + redirect error logging https://github.com/logstash-plugins/logstash-integration-jdbc/pull/102[#102] * Refactor: isolate paginated normal statement algorithm in a separate handler https://github.com/logstash-plugins/logstash-integration-jdbc/pull/101[#101] * Added `jdbc_paging_mode` option to choose if use `explicit` pagination in statements and avoid the initial count query or use `auto` to delegate to the underlying library https://github.com/logstash-plugins/logstash-integration-jdbc/pull/95[#95] * Several improvements to Java driver loading ** Refactor: to explicit Java (driver) class name loading https://github.com/logstash-plugins/logstash-integration-jdbc/pull/96[#96]. The change is expected to provide a more robust fix for the driver loading issue https://github.com/logstash-plugins/logstash-integration-jdbc/issues/83[#83]. NOTE: A fatal driver error will no longer keep reloading the pipeline and now leads to a system exit. ** Fix: regression due returning the Java driver class https://github.com/logstash-plugins/logstash-integration-jdbc/pull/98[#98] *Kafka Integration - 10.9.0* * Refactor: leverage codec when using schema registry Previously using `schema_registry_url` parsed the payload as JSON even if `codec => 'plain'` was explicitly set, this is no longer the case. https://github.com/logstash-plugins/logstash-integration-kafka/pull/106[#106] *Cloudwatch Output - 3.0.10* * Fix: an old undefined method error which would surface with load (as queue fills up) * Deps: unpin rufus scheduler https://github.com/logstash-plugins/logstash-output-cloudwatch/pull/20[#20] *Elasticsearch Output - 11.4.1* * Feat: upgrade manticore (http-client) library https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1063[#1063] ** the underlying changes include latest HttpClient (4.5.13) ** resolves an old issue with `ssl_certificate_verification => false` still doing some verification logic * Updates ECS templates https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1062[#1062] ** Updates v1 templates to 1.12.1 for use with Elasticsearch 7.x and 8.x ** Updates BETA preview of ECS v8 templates for Elasticsearch 7.x and 8.x * Feat: add support for 'traces' data stream type https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1057[#1057] * Refactor: review manticore error handling/logging, logging originating cause in case of connection related error when debug level is enabled. Java causes on connection related exceptions will now be extra logged when plugin is logging at debug level https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1029[#1029] * ECS-related fixes https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1046[#1046] ** Data Streams requirement on ECS is properly enforced when running on Logstash 8, and warned about when running on Logstash 7. ** ECS Compatibility v8 can now be selected *Core Patterns - 4.3.2* - Fix: typo in `BIN9_QUERYLOG` pattern (in ECS mode) https://github.com/logstash-plugins/logstash-patterns-core/pull/307[#307] [[logstash-8-0-0-rc2]] === Logstash 8.0.0-rc2 Release Notes [[notable-8.0.0-rc2]] ==== Notable issues fixed * Fixed long-standing issue in which the `events.out` count incorrectly included events that had been dropped with the drop filter. Now the total out event count includes only events that reach the out stage. https://github.com/elastic/logstash/pull/13593[#13593] * Reduced scope and impact of a memory leak that can be caused by using UUIDs or other high-cardinality field names https://github.com/elastic/logstash/pull/13642[#13642] * Fixed an issue with the Azure input plugin that caused Logstash to crash when the input was used in a pipeline. https://github.com/elastic/logstash/pull/13603[#13603] ==== Plugin releases Plugins align with release 7.17.0 [[logstash-8-0-0-rc1]] === Logstash 8.0.0-rc1 Release Notes ==== Breaking changes [[rn-ecs-compatibility]] ===== ECS compatibility Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS). This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting. In Logstash 8, this compatibility mode will be on-by-default for all pipelines. If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8, you can set pipeline.ecs_compatibility: disabled to its definition in pipelines.yml (or globally in logstash.yml). ==== New features and enhancements Logstash Docker images are now based on Ubuntu 20.04. ==== Plugin releases Plugins align with release 7.16.2 [[logstash-8-0-0-beta1]] === Logstash 8.0.0-beta1 Release Notes ==== Breaking changes [[rn-java-11-minimum]] ===== Java 11 minimum Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability. See <> for a preview of additional breaking changes coming your way. ==== New features and enhancements [[rn-nanosecond-precision]] ===== Nanosecond precision As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request https://github.com/elastic/logstash/pull/12797[#12797] allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity. Timestamp precision is limited to the JVM and Platform's available granularity, which in many cases is microseconds. This change also grants users access to https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns[Java time's improved formatters], which include support fort ISO quarters, week-of-month, and a variety of timezone/offset-related format substitutions. For example: [source,json] -------------------------------------------------------------------------------- filter { mutate { add_field => {"nanos" => "Nanos: %{{n}}" } } } -------------------------------------------------------------------------------- Results in the following event: [source,json] -------------------------------------------------------------------------------- { "@timestamp" => 2021-10-31T22:32:34.747968Z, "host" => "logstash.lan", "nanos" => "Nanos: 747968000", "message" => "test", "type" => "stdin", "@version" => "1" } -------------------------------------------------------------------------------- ==== Plugin releases Plugins align with release 7.15.1 [[logstash-8-0-0-alpha2]] === Logstash 8.0.0-alpha2 Release Notes ==== Breaking changes [[java-home-breaking-change]] ===== Removed support for JAVA_HOME Support for using `JAVA_HOME` to override the path to the JDK that Logstash runs with has been removed for this release. In the `8.x` release, users should set the value of `LS_JAVA_HOME` to the path of their preferred JDK if they wish to use a version other than the bundled JDK. The value of `JAVA_HOME` will be ignored. ==== Plugin releases Plugins align with release 7.15.0 [[logstash-8-0-0-alpha1]] === Logstash 8.0.0-alpha1 Release Notes ==== Breaking changes [[ruby-engine]] ===== Ruby Execution Engine removed The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. https://github.com/elastic/logstash/pull/12517[#12517] [[utf-16]] ===== Support for UTF-16 We have added support for UTF-16 and other multi-byte-character when reading log files. https://github.com/elastic/logstash/pull/9702[#9702] [[field-ref-parser]] ===== Field Reference parser removed The Field Reference parser interprets references to fields in your pipelines and plugins. It was configurable in 7.x, with the default set to strict to reject inputs that are ambiguous or illegal. Configurability is removed in 8.0. Now {ls} rejects ambiguous and illegal inputs as standard behavior. https://github.com/elastic/logstash/pull/12466[#12466] ==== New features and enhancements **Option to pass custom data to the benchmark CLI** We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. https://github.com/elastic/logstash/pull/12437[#12437] ==== Plugin releases Plugins align with release 7.14.0