docs: 8.0 release notes and breaking changes

changelogs/8.0.asciidoc

@@ -3,12 +3,77 @@
 
 https://github.com/elastic/apm-server/compare/7.17\...8.0[View commits]
 
+* <<release-notes-8.0.0>>
 * <<release-notes-8.0.0-rc2>>
 * <<release-notes-8.0.0-rc1>>
 * <<release-notes-8.0.0-beta1>>
 * <<release-notes-8.0.0-alpha2>>
 * <<release-notes-8.0.0-alpha1>>
 
+[float]
+[[release-notes-8.0.0]]
+=== APM version 8.0.0
+
+https://github.com/elastic/apm-server/compare/v7.16.3\...v8.0.0[View commits]
+
+These release notes include all changes made in the alpha, beta, and RC
+releases of 8.0.0.
+
+[float]
+==== Breaking Changes
+- Removed source map upload endpoint {pull}6447[6447]
+- Removed unsupported libbeat `processors` configuration {pull}6474[6474]
+- Removed `apm-server.aggregation.transactions.enabled` configuration option {pull}6495[6495]
+- Removed `apm-server.aggregation.service_destinations.enabled` configuration option {pull}6503[6503]
+- Removed legacy Jaeger gRPC/HTTP endpoints {pull}6417[6417]
+- Removed `apm-server.jaeger` configuration options {pull}6560[6560]
+- Removed `apm-server.instrumentation` configuration options in favor of `instrumentation` {pull}6560[6560]
+- Removed `apm-server.rum.{allowed_service,event_rate}` configuration option in favor of `apm-server.auth.anonymous.{allow_service,rate_limit}` {pull}6560[6560]
+- Removed `apm-server.{api_key,secret_token}` configuration options in favor of `apm-server.auth.{api_key,secret_token}` {pull}6560[6560]
+- Removed unused `span.start.us` field, and deprecated `span.http.*` fields {pull}6602[6602]
+- Removed `apm-server.register.ingest.pipeline` and `output.elasticsearch.pipeline` configuration options {pull}6575[6575]
+- Removed `logging.ecs` and `logging.json` config {pull}6613[6613]
+- Removed `apm-server.data_streams.enabled`, and `setup.*` configuration options {pull}6606[6606]
+- Removed `apm-server.sampling.keep_unsampled` configuration option; non-RUM unsampled transactions are always dropped {pull}6514[6514] {pull}6669[6669]
+- Removed interpretation of non-standard "component" attribute from OpenTelemetry spans; changed default span type and subtypes {pull}6834[6834]
+- Onboarding documents are no longer indexed {pull}6431[6431]
+- APM Server now responds with 403 (HTTP) and PermissionDenied (gRPC) for authenticated but unauthorized requests {pull}5545[5545]
+- `sourcemap.error` and `sourcemap.updated` are no longer set due to failing to find a matching source map {pull}5631[5631]
+- Removed `service.name` from dataset {pull}5451[5451]
+
+[float]
+==== Added
+- Introduced a delete phase for all data streams. Traces, errors and logs are kept for 10 days, metrics are kept for 90 days {pull}6480[6480]
+- Changed RUM traces to use a dedicated data stream (`traces-apm.rum`). RUM traces are kept for 90 days {pull}6480[6480]
+- Standalone apm-server can now fetch source maps uploaded to Kibana, when `apm-server.kibana` is configured {pull}6447[6447]
+- Added metrics for new Elasticsearch output: `libbeat.output.events.{acked,batches,toomany}`; added tracing and log correlation {pull}6630[6630]
+- The `labels` indexed field is now ECS compliant (string only) and added a new `numeric_labels` object that holds labels with numeric values {pull}6633[6633]
+- APM Server is now using a new Elasticsearch output implementation {pull}6656[6656]
+- Modify default standalone apm-server config values to be more in line with the default managed apm-server values {pull}6675[6675]
+- APM Server now has beta support to receive OpenTelemetry Logs on the OTLP/GRPC receiver {pull}6768[6768]
+- experimental: Add support for CloudWatch metric streams through firehose endpoint {pull}6380[6380]
+- Add HTTP span fields as top level ECS fields {pull}5396[5396]
+- Introduce `apm-server.auth.anonymous.*` config {pull}5623[5623]
+- Upgrade Go to 1.17.6 {pull}7207[7207]
+- Introduce ingest pipeline `apm_data_stream_migration` for migrating pre-data stream indices {pull}5768[5768]
+
+[float]
+==== Bug fixes
+- `host.ip` is now stored as an array, as specified by ECS {pull}6694[6694]
+- In accord with ECS, the server logs now set `source.address` to the immediate network peer's IP address, and `client.ip` to the originating client IP if known {pull}6690[6690]
+- Agent configuration metrics are now sent to the metrics-apm.internal* data stream {pull}6767[6767]
+- Improve logged message when APM Server blocks ingestion to wait for the integration to be installed {pull}6796[6796]
+
+[float]
+==== Intake API Changes
+- `transaction.name` was added to the error objects in the intake API {pull}6539[6539]
+- Add support for composite spans in the intake API {pull}5661[5661]
+- Allow to overwrite `service.name` and `service.version` for metricset objects  {pull}6407[6407]
+
+[float]
+==== Licensing Changes
+- Updated the `x-pack` source files license to the Elastic License 2.0 {pull}6524[6524]
+
 [float]
 [[release-notes-8.0.0-rc2]]
 === APM version 8.0.0-rc2

docs/apm-breaking.asciidoc

@@ -5,49 +5,147 @@
 === Breaking Changes
 
 // These tagged regions are required for the stack-docs repo includes
+// tag::81-bc[]
+// end::81-bc[]
 // tag::notable-v8-breaking-changes[]
 // end::notable-v8-breaking-changes[]
 
+This section describes the breaking changes and deprecations introduced in this release
+and previous minor versions.
+
 [float]
 [[breaking-changes-8.0]]
 === 8.0
 
-// todo: this is rough for now. still need to link to relevant docs
-// and pull in more from release notes.
-
 // tag::80-bc[]
-* All index management has been removed from APM Server;
+The following breaking changes are introduced in APM version 8.0.
+
+[float]
+==== Indices are now manged by {fleet}
+
+All index management has been removed from APM Server;
 {fleet} is now entirely responsible for setting up index templates, index lifecycle polices,
 and index pipelines.
-* APM Server now only writes to well-defined data streams;
-writing to classic indices is no longer supported.
-* APM Server has a new {es} output implementation with defaults that should be sufficient for
-most use cases.
 
-As a result of the above changes, the following index management and index tuning configuration variables
-have been removed:
+As a part of this change, the following settings have been removed:
 
-* `apm-server.ilm.*``
+* `apm-server.ilm.*`
 * `apm-server.register.ingest.pipeline.*`
-* `output.elasticsearch.bulk_max_size`
+* `setup.*`
+
+[float]
+==== Data streams by default
+
+APM Server now only writes to well-defined data streams;
+writing to classic indices is no longer supported.
+
+As a part of this change, the following settings have been removed:
+
+* `apm-server.data_streams.enabled`
 * `output.elasticsearch.index`
 * `output.elasticsearch.indices`
 * `output.elasticsearch.pipeline`
 * `output.elasticsearch.pipelines`
+
+[float]
+==== New {es} output
+
+APM Server has a new {es} output implementation with defaults that should be sufficient for
+most use cases; it should no longer be necessary to manually tune the output
+of APM Server.
+
+As a part of this change, the following settings have been removed:
+
+* `output.elasticsearch.bulk_max_size`
 * `output.elasticsearch.worker`
 * `queue.*`
-* `setup.*`
 
-The following configuration variables have been introduced:
+[float]
+==== New source map upload endpoint
+
+The source map upload endpoint has been removed from APM Server.
+Source maps should now be uploaded directly to {kib} instead.
+
+[float]
+==== Legacy Jaeger endpoints have been removed
+
+The legacy Jaeger gRPC and HTTP endpoints have been removed from APM Server.
+
+As a part of this change, the following settings have been removed:
+
+* `apm-server.jaeger`
+
+[float]
+==== All removed and changed settings
+
+Below is a list of all **removed settings** (in alphabetical order) for
+users upgrading a standalone (legacy) APM Server to {stack} version 8.0.
+
+[source,yml]
+----
+apm-server.data_streams.enabled
+apm-server.ilm.*
+apm-server.jaeger
+apm-server.register.ingest.pipeline.*
+apm-server.sampling.keep_unsampled
+output.elasticsearch.bulk_max_size
+output.elasticsearch.index
+output.elasticsearch.indices
+output.elasticsearch.pipeline
+output.elasticsearch.pipelines
+output.elasticsearch.worker
+queue.*
+setup.*
+----
+
+Below is a list of **renamed settings** (in alphabetical order) for
+users upgrading a standalone (legacy) APM Server to {stack} version 8.0.
+
+[source,yml]
+----
+previous setting                --> new setting
+
+apm-server.api_key              --> apm-server.auth.api_key
+apm-server.instrumentation      --> instrumentation
+apm-server.rum.allowed_service  --> apm-server.auth.anonymous.allow_service
+apm-server.rum.event_rate       --> apm-server.auth.anonymous.rate_limit
+apm-server.secret_token         --> apm-server.auth.secret_token
+----
+
+[float]
+==== Supported {ecloud} settings
 
-* `output.elasticsearch.flush_bytes`: threshold for bulk request size, in bytes (with suffix), before flushing to {es}. The value must have a suffix, e.g. `"1MB"`, and defaults to `5MB`.
-* `output.elasticsearch.flush_interval`: max duration for accumulating events for a bulk request before it is flushed to {es}. The value must have a duration suffix, e.g. `"5s"`, and defaults to `1s`.
+Below is a list of all **supported settings** (in alphabetical order) for
+users upgrading an {ecloud} standalone (legacy) cluster to {stack} version 8.0.
+Any previously supported settings not listed below will be removed when upgrading.
 
-// talk about no queue
-// see https://github.com/elastic/apm-server/issues/6662 for more
-// jaeger
-// instrumentation
-// source map upload endpoint
-// onboarding docs
+[source,yml]
+----
+apm-server.agent.config.cache.expiration
+apm-server.aggregation.transactions.*
+apm-server.auth.anonymous.allow_agent
+apm-server.auth.anonymous.allow_service
+apm-server.auth.anonymous.rate_limit.event_limit
+apm-server.auth.anonymous.rate_limit.ip_limit
+apm-server.auth.api_key.enabled
+apm-server.auth.api_key.limit
+apm-server.capture_personal_data
+apm-server.default_service_environment
+apm-server.max_event_size
+apm-server.rum.allow_headers
+apm-server.rum.allow_origins
+apm-server.rum.enabled
+apm-server.rum.exclude_from_grouping
+apm-server.rum.library_pattern
+apm-server.rum.source_mapping.enabled
+apm-server.rum.source_mapping.cache.expiration
+logging.level
+logging.selectors
+logging.metrics.enabled
+logging.metrics.period
+max_procs
+output.elasticsearch.flush_bytes
+output.elasticsearch.flush_interval
+----
 
 // end::80-bc[]