2.12.1

2025-08-05 Version 2.12.1

---

Bug Fixes

• Service Map does not rotate with multiple workers (#5901)
• Fixes a regression in core where @SingleThread annotated processors are only running the last instance. (#5904)
• geoip S3 download fails to handle existing files during download (#5898)
• geoip local file fails to handle existing files during download (#5899)

Security

• CVE-2025-46762 - Parquet 1.15.2 (#5923)
• CVE-2025-48734 - commons-beanutils 1.11.0 and Checkstyle 10.26.1 (#5923)
• CVE-2024-57699 - json-smart 2.5.2 (#5923)
• CVE-2025-24970 - Netty 4.1.123 (#5923)
• CVE-2025-27817 - Apache Kafka 3.9.1 and Confluent Kafka 7.9.1 (#5923)
• CVE-2024-7254 - protobuf-java 3.25.5 (#5924)
• CVE-2025-49146 - Postgresql JDBC driver 42.7.1 (#5936)
• CVE-2025-23206 - aws-cdk-lib 2.190.0 (#5925)
• CVE-2025-5889 - aws-cdk-lib 2.190.0 (#5925)
• CVE-2025-47273 - setuptools v78 (#5926)

Maintenance

• Updated the smoke tests scripts to use the end-to-end tests (#5913)

2.12.0

2025-06-26 Version 2.12.0

---

Breaking Changes

Features

• OTel telemetry unified source (#5596)
• Add SQS sink to Data Prepper (#5634)
• KDS cross account stream (#5687)
• Add otlp sink (for AWS X-Ray) (#5663)
• Adds the Modulus operator to Data Prepper expressions (#5685)
• Add API tokens as an Authc/z method for OpenSearch Sink (#5549)
• Add support for convering keys to lowercase/uppercase in RenameKeyProcessor (#5810)
• Add multi-line csv support (#5784)
• Add auto conversion option to convert_type processor (#5782)
• Add detect format processor (#5774)
• Add getEventType() expression function (#5686)

Enhancements

• Allow plugins to access the default pipeline role (#4958)
• Allow disabling metrics (#5431)
• Support enabling specific experimental plugins (#5675)
• Make opensearch source scroll timeout configurable and increase default value (#5679)

Bug Fixes

• [BUG] OpenTelemetry Spans are indexed using the span id causing collisions (#5370)
• [BUG] _default route no longer seems to exist in 2.11.0 (#5763)
• Fix kafka source with glue registry (#5765)

Security

• protobuf-3.19.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: 1 vulnerabilities (highest severity is: 7.5) (#5802)
• urllib3-2.2.2-py3-none-any.whl: 2 vulnerabilities (highest severity is: 5.3) (#5801)
• protobuf-3.20.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl: 1 vulnerabilities (highest severity is: 7.5) (#5800)
• urllib3-1.26.19-py2.py3-none-any.whl: 2 vulnerabilities (highest severity is: 5.3) (#5799)

Maintenance

• Refactor maven downloading logic to be dynamic (#5826)

2.11.0

2025-04-24 Version 2.11.0

---

Features

• Support AWS Aurora/RDS PostgreSQL as source (#5309)
• Support SQS as a Source (#1049)
• Jira Connector - to seamlessly sync all the ticket details to OpenSearch (#4754)
• AWS Lambda as Processor and Sink (#4699)
• Support AWS Aurora/RDS MySQL as source (#4561)
• AWS Lambda as Sink (#4170)
• Integrate OpenSearch Ml-Commons into Data Prepper (#5509)

Enhancements

• Update OTel Protobuf specification (#5434)
• Data Prepper support for dynamic renaming of keys (#4849)
• S3 DLQ should pass expected bucket owner to PutObject request (#5498)
• AWS Secrets Manager Plugin does not support sts_header_overrides (#5475)
• Support configurable stream read constraints max length in the JSON input codec (#5466)
• Support reading S3 object metadata only (#5433)
• Kafka local AWS Glue registry support (#5377)
• Address Scale Items for Lambda Processor and Sink (#5031)
• Add support to skip remote peer forwarding based on configuration (#5127)
• Add index_types for OTel logs and metrics #3148 (#3929)
• Send RetryInfo on OTel timeouts to improve or clarify backpressure on OTel gRPC sources (#4294, #4119)
• Enhance Lambda processor to retry based on certain class of exception (#5340)
• Support multiple delete_when condition in delete_entries processor (#5315)
• Add additional index_types (#3148)
• Add experimental feature concept (#2695)

Bug Fixes

• NullPointerException on S3 Delete Event Due to Null Object Size (#5448)
• Index Template with flat_object type field fails during deserialization (#5425)
• DynamoDB source with acknowledgements expires frequently (#5412)
• Kinesis source doesn't pass the given polling retrieval config to underlying KCL (#5269)
• UTF-8 Character Encoding Issues in opensearchproject/data-prepper container (#5238)
• Validate that routes configured in the sink exist on startup of Data Prepper (#5106)
• Escaping of "/" in json pointers (#5101)
• DynamoDB Source doesn't support parsing data with Control Characters (#5027)

Security

• [docker] Run as non root (#5311)
• werkzeug-3.0.3-py3-none-any.whl: 2 vulnerabilities (highest severity is: 7.5) (#5122)

Maintenance

• Migrate existing plugins to use POJO configuration classes. (#5246)
• Support examples in documentation (#5077)
• End-to-end log test failing when index name has DateTime pattern suffix in OpenSearch Sink Config (#984)

2.10.3

2025-04-16 Version 2.10.3

---

Security

• Updates Parquet to 1.15.1. Fixes CVE-2025-30065. (#5597)

2.10.2

2024-12-09 Version 2.10.2

---

Bug Fixes

• FIX: missed exception in plugin error (#5105)

Security

• Fix otel_logs_source server configuration for getHttpAuthenticationService. Fixes CVE-2024-55886. (#5215)

2.10.1

2024-10-21 Version 2.10.1

---

Bug Fixes

• [BUG] Kinesis source is failing on startup (#5084)

2.10.0

2024-10-15 Version 2.10.0

---

Features

• Kafka source: support SASL/SCRAM mechanisms (#4241)
• OpenSearch Bulk API Source (#248)
• Support AWS Kinesis Data Streams as a Source (#1082)
• Support OpenTelemetry logs in S3 source (#5028)

Enhancements

• Use HTML in JsonPropertyDescription instead of Markdown (#4984)
• Variable drain time when shutting down via shutdown API (#4966)
• Make max connections and acquire timeout configurable on S3 sink client (#4949)
• Support BigDecimal data type in expressions (#4817)
• Caching implementation of EventKeyFactory (#4843)
• Json codec changes with specific json input codec config (#5054)

Bug Fixes

• [BUG] Close Opensearch RestHighLevelClient in OpenSearchClientRefresher on shutdown and initialization failure (#4770)

Security

• CVE-2024-6345 (High) detected in setuptools-68.0.0-py3-none-any.whl (#4940)
• CVE-2023-46136 (High) detected in Werkzeug-2.2.3-py3-none-any.whl (#4938)
• CVE-2024-34069 (High) detected in Werkzeug-2.2.3-py3-none-any.whl (#4938)
• CVE-2024-37891 (Medium) detected in urllib3-2.0.7-py3-none-any.whl (#4937)
• CVE-2024-35195 (Medium) detected in requests-2.31.0-py3-none-any.whl (#4939)
• CVE-2024-5569 (Low) detected in zipp-3.15.0-py3-none-any.whl ([#4936]#4936))

Maintenance

• Fixes and improvements for AbstractSinkTest (#5021)
• Update the test logging to include the failed assertion (#4987)

2.9.0

2024-08-28 Version 2.9.0

---

Features

• Support sets and set operations in Data Prepper expressions (#3854)
• Add startsWith expression function (#4840)
• Support default route option for Events that match no other route (#4615)
• Delete input for processors which expand the event (#3968)
• Dynamic Rule Detection (#4600)
• Kafka Source should support message headers (#4565)
• Aggregate processor : add option to allow raw events (#4598)
• Add support for start and end times in count and histogram aggregate actions (#4614)
• Add an option to count unique values of specified key(s) to CountAggregateAction (#4644)
• Flatten processor: option for keys wihout brackets (#4616)
• Modify Key Value processor to support string literal grouping (#4599)
• Make AWS credential management available in data-prepper-config.yaml (#2570)

Enhancements

• Support enhanced configuration of the Kafka source and buffer loggers (#4126)
• Update the rename_keys and delete_entries processors to use EventKey (#4636)
• Update the mutate string processors to use the EventKey. (#4649)
• OpenSearch Sink add support for sending pipeline parameter in BulkRequest (#4609)
• Add support for Kafka headers and timestamp in the Kafka Source (#4566)

Bug Fixes

• [BUG] Visibility duplication protection fails when using S3 source for large files and receiving 10 messages from SQS queue (#4812)
• [BUG] ChangeVisibilityTimeout call failure during pipeline shutdown. (#4575)
• [BUG] Service-map relationship should be created regardless of missing traceGroupName (#4821)
• [BUG] Unable to create stateful processors with multiple workers. (#4660)
• [BUG] Routes: regex doesn't work (#4763)
• [BUG] Grok plugin CLOUDFRONT_ACCESS_LOG pattern does not compile (#4604)
• [BUG] The user_agent processor throws exceptions with multiple threads. (#4618)
• [BUG] DynamoDB source export converts Numbers ending in 0 to scientific notation (#3840)
• Fix null document in DLQ object (#4814)
• Fix KeyValue Processor value grouping bug (#4606)

Security

• CVE-2024-6345 (High) detected in setuptools-68.0.0-py3-none-any.whl (#4738)
• CVE-2024-39689 (High) detected in certifi-2023.7.22-py3-none-any.whl (#4715)
• CVE-2024-5569 (Low) detected in zipp-3.15.0-py3-none-any.whl (#4714)
• CVE-2024-3651 (High) detected in idna-3.3-py3-none-any.whl (#4713)
• CVE-2024-35195 (Medium) detected in requests-2.31.0-py3-none-any.whl (#4562)
• CVE-2024-37891 (Medium) detected in urllib3-2.0.7-py3-none-any.whl (#4641)

Maintenance

• Removes Zookeeper from Data Prepper (#4707)
• Tools to generate User Agent strings in the performance-test project (#4620)

2.8.1

2024-08-01 Version 2.8.1

---

Bug Fixes

• Jackson 2.17.0 LockFreePool causes memory issues (#4729)

Maintenance

• Updates Jackson to 2.17.2 (#4753)
• Updates to Armeria 1.29.0 (#4741)
• Parquet codec tests fix (#4742)

2.8.0

2024-05-16 Version 2.8.0

---

Features

• Support Full load and CDC from AWS DocumentDB [#4534] (#4534)
• Support conditional expression to evaluate based on the data type for a given field (#4478 #4523, #4500))
• Allow using event fields in s3 sink object_key [#3310] (#3310)
• Support ndjson with a codec [#2700] (#2700)
• Support S3 bucket ownership validation on the S3 sink (#4468)
• Support encoding JSON (#832 #4514)
• Support for Event Json input and output codecs (#4436)
• Add support for dynamic bucket and default bucket in S3 sink (#4402)
• Add support to export/full load MongoDB/DocumentDB collection with _id field of different data type (#4503)

Enhancements

• HTTP data chunking support for kafka buffer (#4475)
• ENH: automatic credential refresh in kafka source (#4258)
• Add creation and aggregation of dynamic S3 groups based on events (#4346)
• Truncate Processor: Add support to truncate all fields in an event (#4317)
• Provide validations of AWS accountIds (#4398)
• Better metrics on OpenSearch document errors (#4344)
• Better metrics for OpenSearch duplicate documents (#4343)
• Address route and subpipeline for pipeline tranformation (#4528)
• Add support for BigDecimal in ConvertType processor (#4316)
• Checkpoint records at an interval for TPS case when AckSet is enabled (#4526)
• Write stream events that timeout to write to internal buffer in separate thread (#4524)
• Key value processor enhancements (#4521)
• Add bucket owner support to s3 sink (#4504)
• Initial work to support core data types in Data Prepper (#4496)
• Changing logging level for config transformation and fixing rule (#4466)
• Add folder-based partitioning for s3 scan source (#4455)
• Pipeline Configuration Transformation (#4446)
• Added support for multiple workers in S3 Scan Source (#4439)
• Bootstrap the RuleEngine package (#4442)
• Make s3 partition size configurable and add unit test for S3 partition creator classes (#4437)
• Remove creating S3 prefix path partition upfront (#4432)
• Change s3 sink client to async client (#4425)
• Create new codec for each s3 group in s3 sink (#4410)
• Validate the AWS account Id in the S3 source using a new annotation (#4400)
• Add server connections metric to http and otel sources (#4393)
• Log the User-Agent when Data Prepper shuts down from POST /shutdown (#4390)
• Add aggregate_threshold with maximum_size to s3 sink (#4385)
• Refactor PipelinesDataFlowModelParser to take in an InputStream instead of a file path (#4289)
• Add support to use old ddb stream image for REMOVE events (#4275)

Bug Fixes

• Fix count aggregation exemplar data (#4341)
• Revert HTTP data chunking changes for kafka buffer done in PR 4266 (#4329)
• Fix Router performance issue (#4327)
• Do not require field_split_characters to not be empty for key_value processor (#4358)
• Do not write empty lists of DlqObject to the DLQ (#4403)
• Fix transient test failure for subpipelines (#4479)
• Fix JacksonEvent to propagate ExternalOriginalTime if its set at the time of construction (#4489)
• FIX: null certificate value should be valid in opensearch connection (#4494)
• [BUG]Incorrect Behavior of Obfuscate Processor with Predefined Pattern "%{CREDIT_CARD_NUMBER}" (#4340)
• [BUG] Empty DLQ entries when version conflicts occur (#4301)
• [BUG] otel sources should show a more clear exception when receiving data that cannot be processed based on the configured compression type (#4022)
• [BUG] : unable to set field_delimiter_regex (#2946)
• Fix aggregate processor local mode (#4529)
• Add long as a target type for convert_entry_type processor (#4120)
• Fix write json basic test (#4527)
• Fix depth field in template (#4509)
• Fix for S3PartitionCreatorScheduler ConcurrentModification Exception (#4473)
• Fix acknowledgements in DynamoDB (#4419)
• Fix DocumentDB source S3PathPrefix null or empty (#4472)
• Fix an issue that exception messages are masked (#4416)
• Fix bug where using upsert or update without routing parameter caused… (#4397)
• Fix bug in s3 sink dynamic bucket and catch invalid bucket message (#4413)
• Fix flaky PipelineConfigurationFileReaderTest (#4386)
• Aggregate Processor: local mode should work when there is no when condition (#4380)

Security

• CVE-2024-22201 on http2-common 9.4.51 version - autoclosed (#4452)
• CVE-2023-22102 (High) detected in mysql-connector-j-8.0.33.jar - autoclosed (#3920)

Maintenance

• Gradle 8.7 (#4417)
• Adds a Gradle convention plugin for Maven publication (#4421)
• MAINT: allow latest schema version if not specified in confluent schema (#4453)
• Publish expression and logstash-configuration to Maven (#4474)
• Create unit test report as html (#4384)
• Update Stream Ack Manager unit test and code refactor (#4383)
• Grpc exception handler: Modified to return BADREQUEST for some internal errors (#4387)
• Remove unexpected event handle message (#4388)
• Bump parquet version to 1.14.0. (#4520)
• Clear system property to disable s3 scan when stream worker exits, set s3 sink threshold to 15 s...