Transaction do_abort_tx request: not found, assuming already aborted.

[vsarunas]

Version & Environment

Redpanda version: (use rpk version):

rpk version
Version:   v24.2.10
Git ref:   74404e718c
Build date: 2024-11-08T13:25:02Z
OS/Arch:   darwin/arm64
Go version: go1.23.1
Redpanda Cluster
 node-0 v24.2.8 - a18efa7206b0266473842dab463e7a55298dad13

macOS, Linux VM running a Docker container with rpk 24.2.8.

What went wrong?

rdkafka client was getting rejects "Local: This instance has been fenced by a newer instance" when trying to abort a transaction; (2024-12-06T06:56:02.818413+01:00)

This section of logs look suspicious from the same time as the client reject:

INFO  2024-12-06 05:56:02,893 [shard 0:main] tx - [N:dc-1+loadBalancerNext+assigner S:Stable G:27] group.cc:3181 - attempting expiration of producer: {producer_identity: id=12, epoch=786} transaction
INFO  2024-12-06 05:56:02,918 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+loadBalancerNext+transactional] trying to abort transaction. pid: {producer_identity: id=12, epoch=786} sequence: 0
INFO  2024-12-06 05:56:02,918 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+loadBalancerNext+transactional] found transaction {id: dc-1@keres@core@default+loadBalancerNext+transactional, status: ongoing, pid: {producer_identity: id=12, epoch=786}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
INFO  2024-12-06 05:56:02,944 [shard 0:kafk] tx - [N:dc-1+loadBalancerNext+assigner S:Stable G:27] group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=12, epoch=786} not found, sequence: 0, assuming already aborted.

Transaction in CompleteAbort state:

rpk cluster txn describe
COORDINATOR TRANSACTIONAL-ID                          PRODUCER-ID PRODUCER-EPOCH STATE      START-TIMESTAMP      TIMEOUT TOPICS
0      dc-1@keres@core@default+loadBalancerNext+transactional       12      787       CompleteAbort  2024-12-06T06:56:02.943Z 1m0s

Restart of rpk cleared up the state and allowed the application to work further.

Full logs

INFO  2024-12-06 05:43:46,407 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,407 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,407 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,407 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,407 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,408 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,408 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:43:46,408 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=61} sequence: 0
INFO  2024-12-06 05:44:47,516 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:23] group.cc:3181 - attempting expiration of producer: {producer_identity: id=4, epoch=63} transaction
INFO  2024-12-06 05:44:47,517 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,517 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] found transaction {id: dc-1@keres@core@default+metadata-entry+Augmentation, status: ongoing, pid: {producer_identity: id=4, epoch=63}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
INFO  2024-12-06 05:44:47,519 [shard 0:kafk] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:23] group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=4, epoch=63} not found, sequence: 0, assuming already aborted.
INFO  2024-12-06 05:44:47,713 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] producer {producer_identity: id=4, epoch=63} is fenced of by {producer_identity: id=4, epoch=64}
INFO  2024-12-06 05:44:47,720 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,724 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/7}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3434, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60212, evicted: false, transaction_state: {first: 142, last: 142, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3436, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60212, evicted: false, transaction_state: {first: 81, last: 81, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3440, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60211, evicted: false, transaction_state: {first: 177, last: 177, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3430, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60212, evicted: false, transaction_state: {first: 143, last: 143, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3429, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60212, evicted: false, transaction_state: {first: 189, last: 189, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/11}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3438, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60212, evicted: false, transaction_state: {first: 172, last: 172, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:44:47,725 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/8}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=63}, group: 3435, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60212, evicted: false, transaction_state: {first: 77, last: 77, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
INFO  2024-12-06 05:44:47,725 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:47,730 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,725 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,726 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,726 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,726 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,726 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,726 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:44:57,726 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=63} sequence: 0
INFO  2024-12-06 05:53:21,900 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54663 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,900 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54522 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,900 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54519 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,900 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54489 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,900 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54463 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,900 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54439 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,904 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54426 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:21,904 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:54368 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:30,557 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:53632 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:30,557 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:53615 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:30,557 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:53610 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:30,557 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:53609 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:53:30,557 [shard 0:main] kafka - server.cc:167 - Disconnected 192.168.64.1:53605 (applying protocol, Connection reset by peer)
INFO  2024-12-06 05:56:02,783 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3181 - attempting expiration of producer: {producer_identity: id=4, epoch=65} transaction
INFO  2024-12-06 05:56:02,785 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=65} sequence: 1
INFO  2024-12-06 05:56:02,786 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] found transaction {id: dc-1@keres@core@default+metadata-entry+Augmentation, status: ongoing, pid: {producer_identity: id=4, epoch=65}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 1, partitions: } to abort
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3573, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 296, last: 296, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/22}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3593, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 228, last: 228, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3574, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 224, last: 224, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/11}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3582, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 268, last: 268, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/8}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3579, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 122, last: 122, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/7}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3578, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 218, last: 218, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3584, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 264, last: 264, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3580, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60009, evicted: false, transaction_state: {first: 138, last: 138, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/22}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3449, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 146, last: 146, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/7}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3434, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 144, last: 144, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/8}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3435, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60009, evicted: false, transaction_state: {first: 79, last: 79, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/11}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3438, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 174, last: 174, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3429, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 191, last: 191, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3430, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 145, last: 145, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3440, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 179, last: 179, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3436, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60008, evicted: false, transaction_state: {first: 83, last: 83, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3190 - producer {producer_identity: id=4, epoch=65} transaction expiration result: tx::errc::unknown_server_error
WARN  2024-12-06 05:56:02,789 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/23}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=65}, group: 3594, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60009, evicted: false, transaction_state: {first: 164, last: 164, sequence: 1, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
INFO  2024-12-06 05:56:02,893 [shard 0:main] tx - [N:dc-1+loadBalancerNext+assigner S:Stable G:27] group.cc:3181 - attempting expiration of producer: {producer_identity: id=12, epoch=786} transaction
INFO  2024-12-06 05:56:02,918 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+loadBalancerNext+transactional] trying to abort transaction. pid: {producer_identity: id=12, epoch=786} sequence: 0
INFO  2024-12-06 05:56:02,918 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+loadBalancerNext+transactional] found transaction {id: dc-1@keres@core@default+loadBalancerNext+transactional, status: ongoing, pid: {producer_identity: id=12, epoch=786}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
INFO  2024-12-06 05:56:02,944 [shard 0:kafk] tx - [N:dc-1+loadBalancerNext+assigner S:Stable G:27] group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=12, epoch=786} not found, sequence: 0, assuming already aborted.
INFO  2024-12-06 05:56:02,950 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+loadBalancerNext+transactional] producer {producer_identity: id=12, epoch=786} is fenced of by {producer_identity: id=12, epoch=787}
INFO  2024-12-06 05:56:02,971 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] producer {producer_identity: id=4, epoch=65} is fenced of by {producer_identity: id=4, epoch=66}
INFO  2024-12-06 05:56:52,199 [shard 0:main] storage - segment.cc:811 - Creating new segment /var/lib/redpanda/data/kafka/instrument-reference-data-pending-dc-1/0_329/2458979-4-v1.log
INFO  2024-12-06 05:57:04,170 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3181 - attempting expiration of producer: {producer_identity: id=4, epoch=67} transaction
INFO  2024-12-06 05:57:04,172 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=67} sequence: 0
INFO  2024-12-06 05:57:04,172 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] found transaction {id: dc-1@keres@core@default+metadata-entry+Augmentation, status: ongoing, pid: {producer_identity: id=4, epoch=67}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
INFO  2024-12-06 05:57:04,182 [shard 0:kafk] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=4, epoch=67} not found, sequence: 0, assuming already aborted.
INFO  2024-12-06 05:57:04,376 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=67} sequence: 0
INFO  2024-12-06 05:57:04,390 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=67} sequence: 0
WARN  2024-12-06 05:57:04,390 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3574, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 226, last: 226, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,390 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/11}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3582, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 270, last: 270, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,390 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/8}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3579, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 124, last: 124, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,390 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/7}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3578, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 220, last: 220, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3584, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 266, last: 266, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3580, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 140, last: 140, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/8}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3435, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 81, last: 81, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/11}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3438, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 176, last: 176, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3430, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 147, last: 147, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3440, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 181, last: 181, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3436, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 85, last: 85, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/23}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3594, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 166, last: 166, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3573, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 298, last: 298, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/22}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3449, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60220, evicted: false, transaction_state: {first: 148, last: 148, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/7}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3434, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60220, evicted: false, transaction_state: {first: 146, last: 146, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:57:04,391 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=67}, group: 3429, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60221, evicted: false, transaction_state: {first: 193, last: 193, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
INFO  2024-12-06 05:57:04,426 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] producer {producer_identity: id=4, epoch=67} is fenced of by {producer_identity: id=4, epoch=68}
INFO  2024-12-06 05:58:05,508 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3181 - attempting expiration of producer: {producer_identity: id=4, epoch=69} transaction
INFO  2024-12-06 05:58:05,512 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:05,512 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] found transaction {id: dc-1@keres@core@default+metadata-entry+Augmentation, status: ongoing, pid: {producer_identity: id=4, epoch=69}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
WARN  2024-12-06 05:58:05,514 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/22}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3593, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60006, evicted: false, transaction_state: {first: 232, last: 232, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,514 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3190 - producer {producer_identity: id=4, epoch=69} transaction expiration result: tx::errc::unknown_server_error
INFO  2024-12-06 05:58:05,716 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:05,717 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:05,717 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:05,735 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3584, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 268, last: 268, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/8}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3435, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 83, last: 83, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3429, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 195, last: 195, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/7}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3434, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 148, last: 148, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/22}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3449, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 150, last: 150, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/2}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3573, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 300, last: 300, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/23}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3594, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 168, last: 168, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/13}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3440, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 183, last: 183, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3430, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 149, last: 149, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/11}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3438, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 178, last: 178, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3580, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 142, last: 142, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 05:58:05,742 [shard 0:main] tx - [{kafka/metadata-entry-snapshots-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=69}, group: 3436, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60233, evicted: false, transaction_state: {first: 87, last: 87, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
INFO  2024-12-06 05:58:05,796 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] producer {producer_identity: id=4, epoch=69} is fenced of by {producer_identity: id=4, epoch=70}
INFO  2024-12-06 05:58:15,759 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:58:15,769 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=69} sequence: 0
INFO  2024-12-06 05:59:06,990 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3181 - attempting expiration of producer: {producer_identity: id=4, epoch=71} transaction
INFO  2024-12-06 05:59:06,991 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:06,991 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] found transaction {id: dc-1@keres@core@default+metadata-entry+Augmentation, status: ongoing, pid: {producer_identity: id=4, epoch=71}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
INFO  2024-12-06 05:59:06,994 [shard 0:kafk] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=4, epoch=71} not found, sequence: 0, assuming already aborted.
INFO  2024-12-06 05:59:07,149 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] producer {producer_identity: id=4, epoch=71} is fenced of by {producer_identity: id=4, epoch=72}
INFO  2024-12-06 05:59:07,178 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,179 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,179 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,179 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,182 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,183 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,183 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,183 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 05:59:07,183 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=71} sequence: 0
INFO  2024-12-06 06:00:08,305 [shard 0:main] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:3181 - attempting expiration of producer: {producer_identity: id=4, epoch=73} transaction
INFO  2024-12-06 06:00:08,306 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,306 [shard 0:main] tx - tx_gateway_frontend.cc:578 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] found transaction {id: dc-1@keres@core@default+metadata-entry+Augmentation, status: ongoing, pid: {producer_identity: id=4, epoch=73}, last_pid: {producer_identity: id=-1, epoch=-1}, etag: 10, seq: 0, partitions: } to abort
WARN  2024-12-06 06:00:08,306 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/3}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=73}, group: 3574, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60001, evicted: false, transaction_state: {first: 232, last: 232, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
WARN  2024-12-06 06:00:08,307 [shard 0:main] tx - [{kafka/metadata-entry-updates-dc-1/9}] - rm_stm.cc:1461 - state of transaction { id: {producer_identity: id=4, epoch=73}, group: 3580, requests: { inflight: 0, finished: 1 }, ms_since_last_update: 60001, evicted: false, transaction_state: {first: 146, last: 146, sequence: 0, timeout: {60000}, coordinator partition: 32, status: initialized }} is unknown:tx::errc::unknown_server_error
INFO  2024-12-06 06:00:08,308 [shard 0:kafk] tx - [N:dc-1+metadata-entryAugmentation S:Stable G:25] group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=4, epoch=73} not found, sequence: 0, assuming already aborted.
INFO  2024-12-06 06:00:08,468 [shard 0:main] tx - tx_gateway_frontend.cc:2497 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] producer {producer_identity: id=4, epoch=73} is fenced of by {producer_identity: id=4, epoch=74}
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:08,506 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:18,310 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0
INFO  2024-12-06 06:00:18,314 [shard 0:main] tx - tx_gateway_frontend.cc:513 - [tx_id=dc-1@keres@core@default+metadata-entry+Augmentation] trying to abort transaction. pid: {producer_identity: id=4, epoch=73} sequence: 0

[dotnwat]

@bharathv replication team?

[bharathv]

@vsarunas Based on my reading of the logs, everything is working as expected. There was an expired group transaction which triggered an internal abort of the transaction. Internal aborts (initiated by Redpanda, not client) are treated specially compared to client initiated aborts by bumping the epoch. This is done to fence all further client requests with previous epoch and the client has to init again to make progress.

So I believe that resulted in a "FENCED" error when the client attempted to abort again.

Restart of rpk

What exactly did you restart? Client or the broker? I'd expect the client to init_transactions() again in this case and it should unblock itself.

[vsarunas]

@bharathv, the application was restarted but could not complete a transaction. After restart of the broker, application started to work correctly.

The issue reproduced again on another system for the same user; but same version v24.2.8. Upgraded now to v24.3.1.

Same issue group.cc:2850 - do_abort_tx request:- producer/transaction {producer_identity: id=1007, epoch=25} not found, sequence: 0, assuming already aborted., did you see the full log when expanding Full logs ?

[bharathv]

did you see the full log when expanding Full logs ?

Yes, i did, this log line is not a problem.

the application was restarted but could not complete a transaction. After restart of the broker, application started to work correctly.

Thanks, do you have client logs (timestamps and error codes, pre and post RP restart) and a rough pseudo code of what it does? (exception handling)