Missing Kafka topics in Console

[Aransh]

What happened?

While console summary shows a large number of messages (over 50000), the console view shows "no messages".

What did you expect to happen?

I expected the messages in the topic to appear.

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
  namespace: redpanda
spec:
  chartRef: {}
  clusterSpec:
    auth:
      sasl:
        enabled: true
        mechanism: SCRAM-SHA-512
        secretRef: kafka-credentials
        users: []
    connectors:
      enabled: true
      monitoring:
        enabled: true
    console:
      ingress:
        annotations:
          cert-manager.io/cluster-issuer: letsencrypt-production
        className: nginx
        enabled: true
    monitoring:
      enabled: true
    resources:
      cpu:
        cores: 4
      memory:
        container:
          max: 10Gi
          min: 10Gi
        enable_memory_locking: true
    statefulset:
      initContainers:
        configurator:
          resources:
            limits:
              cpu: 1
              memory: 1Gi
        setDataDirOwnership:
          enabled: true
        tuning:
          resources:
            limits:
              cpu: 1
              memory: 1Gi
      replicas: 3

Anything else we need to know?

This doesn't happen with all topics.
In this scenario I onlyy spotted this on this topics, while other topics on the same cluster were showing fine.
Also got reports of this from other team members, so not local cache.

Which are the affected charts?

Redpanda, console

Chart Version(s)

operator - 0.4.8
redpanda - 5.6.58

Cloud provider

Linode

JIRA Link: K8S-85

[alejandroEsc]

@Aransh could we please get a few logs from console and from chrome so we can spot what may be causing this. It could be a console specific issue which we will then want to file an issue in that repo.

[Aransh]

@alejandroEsc sure, attaching logs, let me know if this is what you meant.
Keep in mind these logs are for the same topic from the screenshot above, which now has 1,479,084 messages

Console logs:

{"level":"info","ts":"2023-12-17T06:01:48.053Z","msg":"started Redpanda Console","version":"v2.3.8","built_at":"1701900386"}
{"level":"info","ts":"2023-12-17T06:01:48.054Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2023-12-17T06:01:48.072Z","msg":"successfully connected to kafka cluster","advertised_broker_count":3,"topic_count":124,"controller_id":2,"kafka_version":"unknown custom version at least v0.11.0"}
{"level":"info","ts":"2023-12-17T06:01:48.072Z","msg":"creating schema registry client and testing connectivity"}
{"level":"info","ts":"2023-12-17T06:01:48.080Z","msg":"successfully tested schema registry connectivity"}
{"level":"info","ts":"2023-12-17T06:01:48.080Z","msg":"testing admin client connectivity","urls":["https://redpanda.redpanda.svc.cluster.local.:9644"]}
{"level":"info","ts":"2023-12-17T06:01:48.085Z","msg":"successfully tested the Redpanda admin connectivity","broker_count":3,"cluster_version":"Redpanda v23.2.18"}
{"level":"info","ts":"2023-12-17T06:01:48.085Z","msg":"creating Kafka connect HTTP clients and testing connectivity to all clusters"}
{"level":"info","ts":"2023-12-17T06:01:48.203Z","msg":"tested Kafka connect cluster connectivity","successful_clusters":1,"failed_clusters":0}
{"level":"info","ts":"2023-12-17T06:01:48.203Z","msg":"successfully create Kafka connect service"}
{"level":"info","ts":"2023-12-17T06:01:48.552Z","msg":"Server listening on address","address":"[::]:8080","port":8080}
{"level":"info","ts":"2023-12-19T15:27:53.044Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"}
{"level":"info","ts":"2023-12-19T15:27:53.053Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"csi_lke_perf_test.yondu-waf-triggers-dlr[140{51104 e-1 ce0} 293{50249 e-1 ce0} 292{49480 e-1 ce0} 159{49877 e-1 ce0} 0{50357 e-1 ce0} 82{49623 e-1 ce0} 107{50965 e-1 ce0} 9{50211 e-1 ce0} 133{50918 e-1 ce0} 118{49965 e-1 ce0} 250{50739 e-1 ce0} 235{50713 e-1 ce0} 100{49808 e-1 ce0} 246{49802 e-1 ce0} 79{51005 e-1 ce0} 200{49205 e-1 ce0} 150{50020 e-1 ce0} 213{50298 e-1 ce0} 50{50158 e-1 ce0} 183{50868 e-1 ce0} 265{50622 e-1 ce0} 153{50753 e-1 ce0} 199{50186 e-1 ce0} 174{50767 e-1 ce0} 254{50182 e-1 ce0} 123{50133 e-1 ce0} 102{50285 e-1 ce0} 204{50470 e-1 ce0} 16{51032 e-1 ce0} 28{49938 e-1 ce0} 229{49721 e-1 ce0} 248{50189 e-1 ce0} 68{50869 e-1 ce0} 195{50339 e-1 ce0} 70{49004 e-1 ce0} 141{50227 e-1 ce0} 60{50616 e-1 ce0} 167{50556 e-1 ce0} 42{50713 e-1 ce0} 242{50165 e-1 ce0} 117{50363 e-1 ce0} 148{50217 e-1 ce0} 162{50532 e-1 ce0} 270{51117 e-1 ce0} 171{49189 e-1 ce0} 147{49414 e-1 ce0} 1{50720 e-1 ce0} 95{50340 e-1 ce0} 273{49482 e-1 ce0} 297{50799 e-1 ce0}]"}
{"level":"info","ts":"2023-12-19T15:29:01.411Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"}
{"level":"info","ts":"2023-12-19T15:29:01.421Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"csi_lke_perf_test.yondu-waf-triggers-dlr[235{50713 e-1 ce0} 91{49567 e-1 ce0} 261{49194 e-1 ce0} 76{50671 e-1 ce0} 238{50588 e-1 ce0} 131{49856 e-1 ce0} 109{49256 e-1 ce0} 46{50303 e-1 ce0} 61{49607 e-1 ce0} 263{49895 e-1 ce0} 40{50722 e-1 ce0} 262{49779 e-1 ce0} 287{51348 e-1 ce0} 35{48972 e-1 ce0} 44{49782 e-1 ce0} 144{49280 e-1 ce0} 200{49215 e-1 ce0} 250{50743 e-1 ce0} 4{48956 e-1 ce0} 251{49158 e-1 ce0} 104{50021 e-1 ce0} 94{49577 e-1 ce0} 228{51204 e-1 ce0} 213{50298 e-1 ce0} 272{49607 e-1 ce0} 63{50595 e-1 ce0} 93{49488 e-1 ce0} 102{50295 e-1 ce0} 137{50336 e-1 ce0} 60{50616 e-1 ce0} 71{50542 e-1 ce0} 237{50431 e-1 ce0} 165{49282 e-1 ce0} 8{50656 e-1 ce0} 170{51337 e-1 ce0} 217{50290 e-1 ce0} 160{50615 e-1 ce0} 119{49886 e-1 ce0} 283{49622 e-1 ce0} 214{51024 e-1 ce0} 75{51208 e-1 ce0} 166{50870 e-1 ce0} 248{50189 e-1 ce0} 37{49643 e-1 ce0} 142{50168 e-1 ce0} 154{50145 e-1 ce0} 183{50878 e-1 ce0} 70{49004 e-1 ce0} 121{50442 e-1 ce0} 259{49912 e-1 ce0}]"}

Chrome console:

GET https://console.<domain-redacted>/api/users/me 405 (Method Not Allowed)

executeAction @ action.ts:69
ObservableObject.key @ action.ts:50
_callee3$ @ backendApi.ts:258
tryCatch @ regeneratorRuntime.js:44
(anonymous) @ regeneratorRuntime.js:125
(anonymous) @ regeneratorRuntime.js:69
asyncGeneratorStep @ asyncToGenerator.js:3
_next @ asyncToGenerator.js:22
(anonymous) @ asyncToGenerator.js:27
(anonymous) @ asyncToGenerator.js:19
refreshUserData @ backendApi.ts:257
executeAction @ action.ts:69
ObservableObject.key @ action.ts:50
UserProfile @ UserButton.tsx:29
Eh @ react-dom.profiling.min.js:162
Mj @ react-dom.profiling.min.js:215
Jl @ react-dom.profiling.min.js:322
Il @ react-dom.profiling.min.js:306
Hl @ react-dom.profiling.min.js:305
vl @ react-dom.profiling.min.js:305
tl @ react-dom.profiling.min.js:292
J @ scheduler.production.min.js:13
R @ scheduler.production.min.js:14

Backend reported new/unknown endpoints for endpointCompatibility:
"GET /api/users"

(anonymous) @ supportedFeatures.ts:66
executeAction @ action.ts:69
When-effect @ action.ts:50
(anonymous) @ when.ts:73
reactionRunner @ autorun.ts:88
trackDerivedFunction @ derivation.ts:183
track @ reaction.ts:137
(anonymous) @ autorun.ts:59
runReaction_ @ reaction.ts:102
runReactionsHelper @ reaction.ts:260
reactionScheduler @ reaction.ts:230
(anonymous) @ reaction.ts:270
El @ react-dom.profiling.min.js:298
reactionScheduler @ reaction.ts:270
runReactions @ reaction.ts:237
endBatch @ observable.ts:112
reportChanged @ atom.ts:76
setNewValue_ @ observablevalue.ts:141
setObservablePropValue_ @ observableobject.ts:166
set_ @ observableobject.ts:198
set @ dynamicobject.ts:42
_callee12$ @ backendApi.ts:699
tryCatch @ regeneratorRuntime.js:44
(anonymous) @ regeneratorRuntime.js:125
(anonymous) @ regeneratorRuntime.js:69
asyncGeneratorStep @ asyncToGenerator.js:3
_next @ asyncToGenerator.js:22
Promise.then (async)
asyncGeneratorStep @ asyncToGenerator.js:12
_next @ asyncToGenerator.js:22
(anonymous) @ asyncToGenerator.js:27
(anonymous) @ asyncToGenerator.js:19
refreshSupportedEndpoints @ backendApi.ts:695
executeAction @ action.ts:69
ObservableObject.key @ action.ts:50
(anonymous) @ config.ts:163
memoized @ memoize-one.esm.js:40
App @ App.tsx:51
(anonymous) @ observer.ts:104
(anonymous) @ useObserver.ts:126
trackDerivedFunction @ derivation.ts:183
track @ reaction.ts:137
useObserver @ useObserver.ts:124
observerComponent @ observer.ts:104
li @ react-dom.profiling.min.js:182
Hj @ react-dom.profiling.min.js:214
Gj @ react-dom.profiling.min.js:212
Ej @ react-dom.profiling.min.js:211
Jl @ react-dom.profiling.min.js:327
Il @ react-dom.profiling.min.js:306
Hl @ react-dom.profiling.min.js:305
vl @ react-dom.profiling.min.js:305
tl @ react-dom.profiling.min.js:292
J @ scheduler.production.min.js:13
R @ scheduler.production.min.js:14

[bfontaine]

Has there been any update on this? I get the /api/users/me 405 (Method Not Allowed) error as well

[plotzwi-eq]

Hi, is there any update regarding this 405? I have this issue for multiple endpoints:

I also can't view messages from any topic, despite ACLs being set correctly. There are no errors in redpanda console logs.

[weeco]

Hey @bfontaine and @plotzwi-eq , a colleague pinged me about this issue, since I'm not watching this repository for Console specific issues. I believe your questions are independent of OP's reported issue. At the time this was posted, the shown Console version was using WebSocket and since then we have switched to HTTP 1.1 streaming using ConnectRPC Streaming.

The reason for 405 response - these are somewhat expected. Console has a single binary that complements free and enterprise features. Enterprise features are only unlocked if you provide a valid enterprise license. The frontend application (React) doesn't know about it, so it just tries to call these enterprise endpoints (GET /users/me for example is asking for details about the currently logged-in user) and if it returns 405 it knows that this feature is not available. It won't request these endpoints again as it memorizes that.

If you can't read messages from topics - this is a different issue. You should check the ListMessages request that is being sent to the frontend. I encourage you to create a separate issue for your case where messages cannot be listed in the Console repo here: https://github.com/redpanda-data/console/issues . Recently we've got a report from this user who reported that specifically the Firefox browser is having issues with it: redpanda-data/console#1597 . Are you using Firefox as well? Please also mention the Console version you are using when creating a new issue. Thanks!