Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing creation topic with few brokers on the same host in the docker swarm mode #24615

Open
dima-dmytruk23 opened this issue Dec 19, 2024 · 2 comments
Open

Failing creation topic with few brokers on the same host in the docker swarm mode #24615

dima-dmytruk23 opened this issue Dec 19, 2024 · 2 comments
Labels
kind/bug Something isn't working

Comments

@dima-dmytruk23
Copy link

dima-dmytruk23 commented Dec 19, 2024 •
edited
Loading

Version & Environment

Redpanda version: (use rpk version): v24.2.12

Versions of other components:

  • Operating System (e.g. from /etc/os-release): Ubuntu 24.04.1 LTS
  • Docker (use docker info): 27.4.0

What went wrong?

I can't create a topic (even with 1 partition) with brokers running on the same host in docker swarm mode.

What should have happened instead?

I want to run 3 brokers on one node. I also want to create a topic (ideally with 500 partitions).

How to reproduce the issue?

  1. Create stack.yaml file with 3 brokers
services:
  redpanda1:
    image: redpandadata/redpanda:v24.2.12
    container_name: redpanda1
    hostname: redpanda1
    ports:
      - "9644:9644"
    environment:
      - REDPANDA_NODE_ID=1
      - REDPANDA_KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
      - REDPANDA_ADVERTISED_KAFKA_LISTENERS=PLAINTEXT://redpanda1:9092
      - REDPANDA_ADMIN_API_LISTENERS=0.0.0.0:9644
      - REDPANDA_ADVERTISED_ADMIN_API_LISTENERS=redpanda1:9644
      - REDPANDA_LOG_DIR=/var/lib/redpanda/data
      - REDPANDA_LOG_LEVEL=warn
      - REDPANDA_ENABLE_RACK_AWARE_REPLICATION=false
      - REDPANDA_RETENTION_BYTES=536870912
      - REDPANDA_RETENTION_MS=60000
      - REDPANDA_SEGMENT_BYTES=104857600
      - REDPANDA_REPLICATION_FACTOR=3
      - REDPANDA_CONTROLLER_CONNECT=redpanda1:33145,redpanda2:33145,redpanda3:33145
      - REDPANDA_CONTROLLER_EXTERNAL_CONNECT=redpanda1:33145,redpanda2:33145,redpanda3:33145
      - REDPANDA_SEED_SERVERS=redpanda1:33145,redpanda2:33145,redpanda3:33145
    volumes:
      - redpanda_data1:/var/lib/redpanda/data
    networks:
      - backend-net
      - redpanda-net
    deploy:
      mode: replicated
      replicas: 1
      resources:
        limits:
          memory: 4G
          cpus: "2.0"

  redpanda2:
    image: redpandadata/redpanda:v24.2.12
    container_name: redpanda2
    hostname: redpanda2
    ports:
      - "9645:9644"
    environment:
      - REDPANDA_NODE_ID=2
      - REDPANDA_KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
      - REDPANDA_ADVERTISED_KAFKA_LISTENERS=PLAINTEXT://redpanda2:9092
      - REDPANDA_ADMIN_API_LISTENERS=0.0.0.0:9644
      - REDPANDA_ADVERTISED_ADMIN_API_LISTENERS=redpanda2:9644
      - REDPANDA_LOG_DIR=/var/lib/redpanda/data
      - REDPANDA_LOG_LEVEL=warn
      - REDPANDA_ENABLE_RACK_AWARE_REPLICATION=false
      - REDPANDA_RETENTION_BYTES=536870912
      - REDPANDA_RETENTION_MS=60000
      - REDPANDA_SEGMENT_BYTES=104857600
      - REDPANDA_REPLICATION_FACTOR=3
      - REDPANDA_CONTROLLER_CONNECT=redpanda1:33145,redpanda2:33145,redpanda3:33145
      - REDPANDA_CONTROLLER_EXTERNAL_CONNECT=redpanda1:33145,redpanda2:33145,redpanda3:33145
      - REDPANDA_SEED_SERVERS=redpanda1:33145,redpanda2:33145,redpanda3:33145
    volumes:
      - redpanda_data2:/var/lib/redpanda/data
    networks:
      - backend-net
      - redpanda-net
    deploy:
      mode: replicated
      replicas: 1
      resources:
        limits:
          memory: 4G
          cpus: "2.0"

  redpanda3:
    image: redpandadata/redpanda:v24.2.12
    container_name: redpanda3
    hostname: redpanda3
    ports:
      - "9646:9644"
    environment:
      - REDPANDA_NODE_ID=3
      - REDPANDA_KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
      - REDPANDA_ADVERTISED_KAFKA_LISTENERS=PLAINTEXT://redpanda3:9092
      - REDPANDA_ADMIN_API_LISTENERS=0.0.0.0:9644
      - REDPANDA_ADVERTISED_ADMIN_API_LISTENERS=redpanda3:9644
      - REDPANDA_LOG_DIR=/var/lib/redpanda/data
      - REDPANDA_LOG_LEVEL=warn
      - REDPANDA_ENABLE_RACK_AWARE_REPLICATION=false
      - REDPANDA_RETENTION_BYTES=536870912
      - REDPANDA_RETENTION_MS=60000
      - REDPANDA_SEGMENT_BYTES=104857600
      - REDPANDA_REPLICATION_FACTOR=3
      - REDPANDA_CONTROLLER_CONNECT=redpanda1:33145,redpanda2:33145,redpanda3:33145
      - REDPANDA_CONTROLLER_EXTERNAL_CONNECT=redpanda1:33145,redpanda2:33145,redpanda3:33145
      - REDPANDA_SEED_SERVERS=redpanda1:33145,redpanda2:33145,redpanda3:33145
    volumes:
      - redpanda_data3:/var/lib/redpanda/data
    networks:
      - backend-net
      - redpanda-net
    deploy:
      mode: replicated
      replicas: 1
      resources:
        limits:
          memory: 4G
          cpus: "2.0"

volumes:
  redpanda_data1:
  redpanda_data2:
  redpanda_data3:

networks:
  backend-net:
    external: true
  redpanda-net:
    external: true

When I try create topic from redpanda1 container I get error

INFO  2024-12-19 20:28:10,656 [shard  1:main] cluster - topics_frontend.cc:210 - Create topics [{configuration: { topic: {kafka/atr}, partition_count: 1, replication_factor: 3, is_migrated: false, properties: {compression: {nullopt}, cleanup_policy_bitflags: {nullopt}, compaction_strategy: {nullopt}, retention_bytes: {}, retention_duration_ms: {}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {}, retention_local_target_ms: {}, remote_delete: true, segment_ms: {}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy:  {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt},  record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {}, initial_retention_local_target_ms: {}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}}}, custom_assignments: {}},]

2024-12-19T20:28:10.656623161Z infra_redpanda1.1.yx4mq8vckkyn@dev    | WARN  2024-12-19 20:28:10,656 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor

I tried different partitions (1, 3, 150, 500 etc)

I see following logs after startup container

2024-12-19T20:22:20.499172662Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,499 [shard  0:main] main - application.cc:2664 - Started Schema Registry listening at {{:{host: 0.0.0.0, port: 8081}:<nullopt>}}
2024-12-19T20:22:20.499316311Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,499 [shard  0:main] main - application.cc:2999 - Waiting for cluster membership
2024-12-19T20:22:20.499322243Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,499 [shard  0:main] main - application.cc:3017 - Waiting for Cluster ID to initialize...
2024-12-19T20:22:20.502204557Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,502 [shard  0:main] cluster - members_manager.cc:164 - Node configuration updated successfully
2024-12-19T20:22:20.502795144Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,502 [shard  0:main] main - application.cc:3028 - Started Kafka API server listening at {{:{host: 0.0.0.0, port: 9092}:{nullopt}}}
2024-12-19T20:22:20.503196617Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,503 [shard  0:main] cluster - metrics_reporter.cc:409 - Initialized cluster_id to aca7dfc6-ba85-4319-90b8-47128d2bb90e
2024-12-19T20:22:20.503568734Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:20,503 [shard  0:main] main - application.cc:2707 - Successfully started Redpanda!
2024-12-19T20:22:23.484328900Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:23,484 [shard  0:main] cluster - partition_balancer_planner.cc:1972 - counts rebalancing objective in domain -1: 1 -> 1
2024-12-19T20:22:23.484354698Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:23,484 [shard  0:main] cluster - partition_balancer_planner.cc:1972 - counts rebalancing objective in domain 0: 1 -> 1
2024-12-19T20:22:23.484358876Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:23,484 [shard  0:main] cluster - partition_balancer_backend.cc:423 - last status: in_progress; violations: unavailable nodes: 0, full nodes: 0; nodes to rebalance count: 1; on demand rebalance requested: false; updates in progress: 0; action counts: reassignments: 0, cancellations: 0, failed: 0; counts rebalancing finished: true, force refresh health report: false
2024-12-19T20:22:23.484563841Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:22:23,484 [shard  0:main] cluster - members_manager.cc:375 - applying finish_reallocations_cmd, offset: 8, node id: 0
2024-12-19T20:23:19.963522505Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:23:19,963 [shard  0:main] main - memory_sampling.cc:143 - 34812723 bytes of available memory left - Top-N alloc sites: size: 33554432 count: 2 at: 0xa66e23a 0xa2aadd6 0xa2b43d6 0xa2a6bce 0x96b50f7 0x96b5e46 0x96b49d3 0x32ff1a3 0x335f9a2 0xa49ffeb|size: 21000259 count: 7 at: 0xa66e23a 0xa2aadd6 0xa2b34cf 0xa2bf9c8 0x969843f 0x344c8da 0xa39f22f 0xa3a2951 0xa39fb16 0xa27de00 0xa27c1f8 0x32fc17d 0xa8ea329 /opt/redpanda/lib/libc.so.6+0x2a087 /opt/redpanda/lib/libc.so.6+0x2a14a 0x32f4624|size: 18000222 count: 6 at: 0xa66e23a 0xa2cb4a0 0xa2b3b57 0xa2a6bce 0x3656fe1 0x365780c 0xa39f22f 0xa3a2951 0xa39fb16 0xa27de00 0xa27c1f8 0x32fc17d 0xa8ea329 /opt/redpanda/lib/libc.so.6+0x2a087 /opt/redpanda/lib/libc.so.6+0x2a14a 0x32f4624|size: 8388608 count: 1 at: 0xa66e23a 0xa2aadd6 0xa2b43d6 0xa2a6bce 0x96c469d 0x96c4b7f 0x32ff1bc 0x335f9a2 0xa49ffeb|size: 4194304 count: 1 at: 0xa66e23a 0xa2aadd6 0xa2b43d6 0xa2a6bce 0x96b4d34 0x96b547b 0x96b49d3 0x32ff1a3 0x335f9a2 0xa49ffeb
2024-12-19T20:23:20.481709784Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:23:20,481 [shard  0:main] cluster - controller_stm.cc:136 - created snapshot at offset 8 in 0 ms
2024-12-19T20:23:20.492409816Z infra_redpanda1.1.yx4mq8vckkyn@dev    | INFO  2024-12-19 20:23:20,492 [shard  0:main] storage - disk_log_impl.cc:2486 - Removing "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log" (remove_prefix_full_segments, {offset_tracker:{term:1, base_offset:0, committed_offset:8, dirty_offset:8}, compacted_segment=0, finished_self_compaction=0, finished_windowed_compaction=0, generation=18, reader={/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log, (1545 bytes)}, writer={no_of_chunks:64, closed:0, fallocation_offset:33554432, stable_offset:1545, flushed_offset:1545, committed_offset:1545, inflight:0, dispatched:0, merged:0, bytes_flush_pending:0}, cache={cache_size=9, dirty tracker: {min: -9223372036854775808, max: -9223372036854775808}}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.base_index, offsets:0, index:{header_bitflags:0, base_offset:0, max_offset:8, base_timestamp:{timestamp: 1734639740355}, max_timestamp:{timestamp: 1734639743484}, batch_timestamps_are_monotonic:1, with_offset:false, non_data_timestamps:0, broker_timestamp:{{timestamp: 1734639743484}}, num_compactible_records_appended:{8}, index(1,1,1)}, step:32768, needs_persistence:1}})
2024-12-19T20:28:10.656542950Z infra_redpanda1.1.yx4mq8vckkyn@dev    |

Also I see next errors logs

WARN  2024-12-19 22:25:32,305 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,305 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,305 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,305 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,305 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,305 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,307 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,307 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,307 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,307 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,307 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,308 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,308 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,308 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,309 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,309 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,309 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,310 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,310 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,310 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,310 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,311 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,311 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,311 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,311 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,313 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,313 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,313 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,313 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
INFO  2024-12-19 22:25:32,321 [shard  0:main] cluster - config_manager.cc:510 - Can't load config cache: std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or directory ["/var/lib/redpanda/data/config_cache.yaml"])
INFO  2024-12-19 22:25:32,321 [shard  0:main] cluster - config_manager.cc:450 - Can't load config bootstrap file: std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or directory ["/etc/redpanda/.bootstrap.yaml"])

INFO  2024-12-19 22:25:32,325 [shard  0:main] main - application.cc:523 - Setting abort_on_allocation_failure (abort on OOM): true
ERROR 2024-12-19 22:25:32,330 [shard  0:main] syschecks - Memory: '174063616' below recommended: '1073741824'

WARN  2024-12-19 22:26:36,192 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
WARN  2024-12-19 22:26:43,377 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
WARN  2024-12-19 22:26:50,994 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
WARN  2024-12-19 22:27:03,188 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
@dima-dmytruk23 dima-dmytruk23 added the kind/bug Something isn't working label Dec 19, 2024
@dima-dmytruk23
Copy link
Author

Also I see next errors logs

WARN  2024-12-19 22:25:32,305 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,305 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,305 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,305 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,305 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,305 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,306 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,306 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,307 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,307 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,307 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,307 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,307 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,308 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,308 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,308 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,309 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,309 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,309 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,310 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,310 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,310 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,310 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,311 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,311 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,311 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,311 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,312 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,312 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,313 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,313 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
WARN  2024-12-19 22:25:32,313 seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2024-12-19 22:25:32,313 cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
INFO  2024-12-19 22:25:32,321 [shard  0:main] cluster - config_manager.cc:510 - Can't load config cache: std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or directory ["/var/lib/redpanda/data/config_cache.yaml"])
INFO  2024-12-19 22:25:32,321 [shard  0:main] cluster - config_manager.cc:450 - Can't load config bootstrap file: std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or directory ["/etc/redpanda/.bootstrap.yaml"])

INFO  2024-12-19 22:25:32,325 [shard  0:main] main - application.cc:523 - Setting abort_on_allocation_failure (abort on OOM): true
ERROR 2024-12-19 22:25:32,330 [shard  0:main] syschecks - Memory: '174063616' below recommended: '1073741824'

WARN  2024-12-19 22:26:36,192 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
WARN  2024-12-19 22:26:43,377 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
WARN  2024-12-19 22:26:50,994 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor
WARN  2024-12-19 22:27:03,188 [shard  1:main] kafka - create_topics.cc:159 - Failed to create topic(s) {{kafka/atr}} error_code observed: cluster::errc::topic_invalid_replication_factor

@dima-dmytruk23
Copy link
Author

All logs
https://gist.github.com/dima-dmytruk23/68f74bc03b3c9638ca78fca0632824f5

rpk cluster health:

redpanda@redpanda1:/$ rpk cluster health
CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          true
Unhealthy reasons:                []
Controller ID:                    0
All nodes:                        [0]
Nodes down:                       []
Leaderless partitions (0):        []
Under-replicated partitions (0):  []

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dima-dmytruk23