otlp: Add beta support to receive logs

[marclop]

Motivation/summary

Adds support to receive OTLP logs on the GRPC receiver. This feature is
beta and should be used sparsely until the OTel spec declares it as
stable.

Currently, it's challenging to test this in the system tests since the
opentelemetry-go SDK doesn't expose any APIs for logs.

Checklist

• Update CHANGELOG.asciidoc
• Update package changelog.yml (only if changes to apmpackage have been made)
• Documentation has been updated

How to test these changes

1. docker-compose up -d
2. make
3. ./apm-server -E output.elasticsearch.username=admin -E output.elasticsearch.password=changeme -e
4. Run the following go program: (Make sure that you're using go.opentelemetry.io/collector/[email protected]):

package main

import (
	"context"
	"time"

	"go.opentelemetry.io/collector/model/otlpgrpc"
	"go.opentelemetry.io/collector/model/pdata"
	semconv "go.opentelemetry.io/collector/model/semconv/v1.5.0"
	"google.golang.org/grpc"
)

func main() {
	conn, err := grpc.Dial("localhost:8200", grpc.WithInsecure(), grpc.WithBlock())
	if err != nil {
		panic(err)
	}
	defer conn.Close()

	client := otlpgrpc.NewLogsClient(conn)
	newLog := func(body interface{}) pdata.Logs {
		logs := pdata.NewLogs()
		resourceLogs := logs.ResourceLogs().AppendEmpty()
		logs.ResourceLogs().At(0).Resource().Attributes().InsertString(
			semconv.AttributeTelemetrySDKLanguage, "go",
		)
		instrumentationLogs := resourceLogs.InstrumentationLibraryLogs().AppendEmpty()
		otelLog := instrumentationLogs.Logs().AppendEmpty()
		otelLog.SetTraceID(pdata.NewTraceID([16]byte{1}))
		otelLog.SetSpanID(pdata.NewSpanID([8]byte{2}))
		otelLog.SetName("doOperation()")
		otelLog.SetSeverityNumber(pdata.SeverityNumberINFO)
		otelLog.SetSeverityText("Info")
		otelLog.SetTimestamp(pdata.NewTimestampFromTime(time.Now()))
		otelLog.Attributes().InsertString("key", "value")
		otelLog.Attributes().InsertDouble("key", 1234)

		switch b := body.(type) {
		case string:
			otelLog.Body().SetStringVal(b)
		case int:
			otelLog.Body().SetIntVal(int64(b))
		case float64:
			otelLog.Body().SetDoubleVal(float64(b))
		case bool:
			otelLog.Body().SetBoolVal(b)
		}
		return logs
	}

	_, err = client.Export(context.Background(), newLog("my log message"))
	if err != nil {
		panic(err)
	}
	_, err = client.Export(context.Background(), newLog(12334))
	if err != nil {
		panic(err)
	}
}

5. Install the APM Integration
6. Verify that the two log messages show up in the Observability > Logs UI.

Related issues

Closes #5491

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-12-07T05:01:50.035+0000

• Duration: 41 min 51 sec

• Commit: 457ce9b

Test stats 🧪

Test	Results
Failed	0
Passed	5616
Skipped	19
Total	5635

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /hey-apm : Run the hey-apm benchmark.

• /package : Generate and publish the docker images.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[simitt]

Couldn't this also refer to the transaction.ID. Is there any information available that would allow a similar check as https://github.com/elastic/apm-server/blob/master/processor/otel/traces.go#L211-L226?

[axw]

I don't think we can know that here, but happy to be told otherwise. We're going to need to do something like elastic/apm#413, renaming transaction.id to span.id.

I think it might be best to set span.id even though it might not be correct, i.e. because it's a transaction. At least then users can search for a transaction with the span.id value, and we'll be aligned for a future where transaction.id doesn't exist.

@felixbarny WDYT?

[felixbarny]

SGTM. We could also add a span.id to transactions and look for ways how to phase out transaction.id going forward.

[simitt]

sounds good, thanks

[axw]

I don't think we can know that here, but happy to be told otherwise. We're going to need to do something like elastic/apm#413, renaming transaction.id to span.id.

I think it might be best to set span.id even though it might not be correct, i.e. because it's a transaction. At least then users can search for a transaction with the span.id value, and we'll be aligned for a future where transaction.id doesn't exist.

@felixbarny WDYT?

[axw]

Looks great, thank you 🎉

Can you please add the fields to app_logs, unless there's a reason not to?

[mergify]

This pull request is now in conflicts. Could you fix it @marclop? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b f/support-otel-logs upstream/f/support-otel-logs
git merge upstream/master
git push upstream f/support-otel-logs

[cristianmad]

This is awesome! Is there a beta build we could try?

Thank you!

[marclop]

@cristianmad We'll publish a release candidate (8.0.0-rc1) at the end of December, that would be the safest way to beta test this feature before 8.0.0 is officially released.

[stuartnelson3]

confirmed with 2d1c270