Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ResourceLogs emitted by experimental-otlp/stdout lack line breaks being incompatible with the OTelCol otlpjson connector #6836

Open
cyrille-leclerc opened this issue Oct 29, 2024 · 2 comments · May be fixed by #6848
Open

ResourceLogs emitted by experimental-otlp/stdout lack line breaks being incompatible with the OTelCol otlpjson connector #6836

cyrille-leclerc opened this issue Oct 29, 2024 · 2 comments · May be fixed by #6848
Labels
Bug Something isn't working

Comments

@cyrille-leclerc
Copy link
Member

Describe the bug

ResourceLogs emitted by OTEL_LOGS_EXPORTER="experimental-otlp/stdout" lack line breaks being incompatible with the OTel Collector filelog receiver and otlpjson connector.

Example of 2 consecutive log lines that are outputted in stdout without any character between them.

{"resourceLogs":[{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"staging"}},{"key":"host.arch","value":{"stringValue":"aarch64"}},{"key":"host.name","value":{"stringValue":"cyrille-le-clerc-macbook.local"}},{"key":"os.description","value":{"stringValue":"Mac OS X 15.1.24.0.1"}},{"key":"os.type","value":{"stringValue":"darwin"}},{"key":"process.command_args","value":{"arrayValue":{"values":[{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"},{"stringValue":"-javaagent:./../.otel/opentelemetry-javaagent-2.9.0.jar"},{"stringValue":"-Dotel.instrumentation.logback-appender.experimental-log-attributes=true"},{"stringValue":"-Dotel.instrumentation.logback-mdc.add-baggage=true"},{"stringValue":"-Dotel.instrumentation.micrometer.base-time-unit=s"},{"stringValue":"-Dotel.semconv-stability.opt-in=http/dup"},{"stringValue":"-DshippingServiceUrl=http://shipping.local:8088/"},{"stringValue":"-jar"},{"stringValue":"target/checkout-1.1-SNAPSHOT.jar"}]}}},{"key":"process.executable.path","value":{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"}},{"key":"process.pid","value":{"intValue":"92078"}},{"key":"process.runtime.description","value":{"stringValue":"Homebrew OpenJDK 64-Bit Server VM 21.0.5"}},{"key":"process.runtime.name","value":{"stringValue":"OpenJDK Runtime Environment"}},{"key":"process.runtime.version","value":{"stringValue":"21.0.5"}},{"key":"service.instance.id","value":{"stringValue":"01c0a650-fea5-48a8-b7a3-ffe6c256f18e"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"service.namespace","value":{"stringValue":"shop"}},{"key":"service.version","value":{"stringValue":"1.1"}},{"key":"telemetry.distro.name","value":{"stringValue":"opentelemetry-java-instrumentation"}},{"key":"telemetry.distro.version","value":{"stringValue":"2.9.0"}},{"key":"telemetry.sdk.language","value":{"stringValue":"java"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.43.0"}}]},"scopeLogs":[{"scope":{"name":"com.mycompany.checkout.CheckoutServiceServer","attributes":[]},"logRecords":[{"timeUnixNano":"1730209613443367000","observedTimeUnixNano":"1730209613446828000","severityNumber":9,"severityText":"INFO","body":{"stringValue":"GRPC server started, listening on 50051"},"attributes":[{"key":"thread.id","value":{"intValue":"1"}},{"key":"thread.name","value":{"stringValue":"main"}}]}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.24.0"}]}{"resourceLogs":[{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"staging"}},{"key":"host.arch","value":{"stringValue":"aarch64"}},{"key":"host.name","value":{"stringValue":"cyrille-le-clerc-macbook.local"}},{"key":"os.description","value":{"stringValue":"Mac OS X 15.0.1"}},{"key":"os.type","value":{"stringValue":"darwin"}},{"key":"process.command_args","value":{"arrayValue":{"values":[{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"},{"stringValue":"-javaagent:./../.otel/opentelemetry-javaagent-2.9.0.jar"},{"stringValue":"-Dotel.instrumentation.logback-appender.experimental-log-attributes=true"},{"stringValue":"-Dotel.instrumentation.logback-mdc.add-baggage=true"},{"stringValue":"-Dotel.instrumentation.micrometer.base-time-unit=s"},{"stringValue":"-Dotel.semconv-stability.opt-in=http/dup"},{"stringValue":"-DshippingServiceUrl=http://shipping.local:8088/"},{"stringValue":"-jar"},{"stringValue":"target/checkout-1.1-SNAPSHOT.jar"}]}}},{"key":"process.executable.path","value":{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"}},{"key":"process.pid","value":{"intValue":"92078"}},{"key":"process.runtime.description","value":{"stringValue":"Homebrew OpenJDK 64-Bit Server VM 21.0.5"}},{"key":"process.runtime.name","value":{"stringValue":"OpenJDK Runtime Environment"}},{"key":"process.runtime.version","value":{"stringValue":"21.0.5"}},{"key":"service.instance.id","value":{"stringValue":"01c0a650-fea5-48a8-b7a3-ffe6c256f18e"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"service.namespace","value":{"stringValue":"shop"}},{"key":"service.version","value":{"stringValue":"1.1"}},{"key":"telemetry.distro.name","value":{"stringValue":"opentelemetry-java-instrumentation"}},{"key":"telemetry.distro.version","value":{"stringValue":"2.9.0"}},{"key":"telemetry.sdk.language","value":{"stringValue":"java"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.43.0"}}]},"scopeLogs":[{"scope":{"name":"com.mycompany.checkout.CheckoutServiceServer$CheckoutServiceImpl","attributes":[]},"logRecords":[{"timeUnixNano":"1730209630540189000","observedTimeUnixNano":"1730209630540384000","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order successfully placed customerId=customer-94 shipping=ok durationInMillis=31"},"attributes":[{"key":"thread.id","value":{"intValue":"39"}},{"key":"thread.name","value":{"stringValue":"grpc-default-executor-0"}}],"flags":1,"traceId":"468a15931e800d1644dee4e0e62b1fa2","spanId":"8465d2938245281d"}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.24.0"}]}

Example failing OTel Collector pipeline, it expects line breaks between log records

receivers:
  filelog/otlp-json-logs:
    # debug: restart at the beginning of the file
    start_at: beginning
    include: [/path/to/otlp-json-logs.log]

processors:

connectors:
  otlpjson:

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs/raw:
      receivers: [filelog/otlp-json-logs]
      processors: []
      exporters: [otlpjson]
    logs/normalized:
      receivers: [otlpjson]
      processors: []
      exporters: [debug]
  telemetry:
    metrics:
      level: none

Steps to reproduce

  • Use a Java app that produces log lines through Logback
  • Instrument it with otel java instrumentation agent v2.9.0
  • Export logs through the new stdout/otlp-json instead of OTLP setting OTEL_LOGS_EXPORTER="experimental-otlp/stdout"
  • Launch the app and produce a few log lines,
  • Verify that they are outputted in stdout as JSON documents {"resourceLogs":[{"resource":...
  • Verify that the OTLP JSON log records ({"resourceLogs":[{"resource":...) lack line break delimiters

What did you expect to see?

OTLP JSON log records ({"resourceLogs":[{"resource":...) delimited by line breaks so it can be parsed by the OTel Collector filelog receiver and otlpjson connector

What did you see instead?

OTLP JSON log records ({"resourceLogs":[{"resource":...) lacking delilmiters

What version and what artifacts are you using?
Artifacts: OpenTelemetry Java Instrumentation Agent with the experimental-otlp/stdout exporter
Version: v2.9.0
How did you reference these artifacts? `java -javaagent:...

Environment
OS: MacOS 15.0.#1
Runtime (if different from JDK above): openjdk version "21.0.5" 2024-10-15 / OpenJDK 64-Bit Server VM Homebrew (build 21.0.5, mixed mode, sharing)

Additional context

cc @zeitlinger
@cyrille-leclerc cyrille-leclerc added the Bug Something isn't working label Oct 29, 2024
@zeitlinger
Copy link
Member

As per open-telemetry/opentelemetry-specification#4183 it it allowed to have multiple log lines in one resource logs. If the connector doesn't support this it should be fixed - not the Java part.
I didn't test that myself BTW.

@zeitlinger
Copy link
Member

Oh, I see the problem now in the Java part.

@zeitlinger zeitlinger linked a pull request Nov 4, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bugfix: add newline in stdout exporter zeitlinger/opentelemetry-java
2 participants
@cyrille-leclerc @zeitlinger