Skip to content

Commit

Permalink
Merge pull request #894 from mandy-chessell/code2024
Browse files Browse the repository at this point in the history
Rename open lineage server to lineage warehouse
  • Loading branch information
mandy-chessell committed Feb 7, 2024
2 parents 0dab618 + 83c5d90 commit b417cb5
Show file tree
Hide file tree
Showing 168 changed files with 1,978 additions and 1,562 deletions.
2 changes: 1 addition & 1 deletion site/docs/concepts/audit-log-destination-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The audit log destination connector implements a common interface to an audit lo

![Audit Log Destination Connector](/connectors/runtime/audit-log-destination-connector.svg)

An [OMAG Server](/concepts/omag-server) can have [multiple audit log destinations](/guides/admin/servers/configuring-an-integration-daemon/#configure-the-audit-log) configured. This configuration can control which severities of audit log record are sent to each destination.
An [OMAG Server](/concepts/omag-server) can have [multiple audit log destinations](/guides/admin/servers/by-section/respository-services-section/#configuring-the-audit-log) configured. This configuration can control which severities of audit log record are sent to each destination.

The implementation for the audit log destination connector can make further choices about how each log record is processed.

Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/audit-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@ hide:
??? education "Further information"

* There is more information on the use of audit logging in the [Diagnostic Guide](/guides/diagnostic/diagnostic-sources).
* [Configuring the audit Log](/guides/admin/servers/configuring-the-audit-log).
* [Configuring the audit Log](/guides/admin/servers/by-section/repository-services-section/#configuring-the-audit-log).

--8<-- "snippets/abbr.md"
2 changes: 1 addition & 1 deletion site/docs/concepts/cohort-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ This is shown in figure 1.
paid to upgrading the back-level server so it can use the three topics.

Details of configuring the different topic options can be found in the
[administration guide](/guides/admin/servers/configuring-a-metadata-access-store/#add-access-to-a-cohort).
[administration guide](/guides/admin/servers/by-section/repository-services-section/#registering-the-server-with-a-cohort).

## Enterprise event topic

Expand Down
3 changes: 0 additions & 3 deletions site/docs/concepts/configuration-document-structure.svg

This file was deleted.

6 changes: 3 additions & 3 deletions site/docs/concepts/configuration-document.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@

# Configuration Documents

A configuration document provides the configuration details for a single [OMAG Server](omag-server.md). It defines which subsystems are activated in the server and which connector implementations it should use.
A configuration document provides the configuration properties for a single [OMAG Server](omag-server.md). It defines which subsystems are activated in the server and which connector implementations it should use.

## Configuration document structure

An OMAG Server's configuration document is structured into elements that each describe the configuration properties for each of its desired capabilities.

![Structure of the configuration document](configuration-document-structure.svg)
![Structure of the configuration document](/guides/admin/servers/by-section/configuration-document-structure.svg)

The sections are as follows:

Expand Down Expand Up @@ -37,7 +37,7 @@ When the server is running, the following command returns the configuration docu
At the top of the configuration document are:

- [Local server URL root](omag-server.md#platform-url-root), which defines the root of the network address for the [OMAG Server Platform](omag-server-platform.md) where the OMAG Server will run.
- [Event bus config](/guides/admin/servers/configuring-a-metadata-access-point/#set-up-the-default-event-bus), which provides the configuration of the event bus (Apache Kafka or similar) where all the event topics that the server will use are located.
- [Event bus config](/guides/admin/servers/by-section/event-bus-config-section), which provides the configuration of the event bus (Apache Kafka or similar) where all the event topics that the server will use are located.

Both of these elements provide default values for other configuration elements.

Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/engine-host.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,6 @@ The engine host services have a REST API to query the status of the governance e
!!! education "Further information"

- The capabilities of each of the engine services are described in the [engine services](/services/omes).
- Configuring an engine host is described in the [Administration Guide](/guides/admin/servers/configuring-an-engine-host).
- Configuring an engine host is described in the [Administration Guide](/guides/admin/servers/by-server-type/configuring-an-engine-host).

--8<-- "snippets/abbr.md"
4 changes: 2 additions & 2 deletions site/docs/concepts/governance-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ When a governance engine is called, it is passed a [governance request type](/co
![Governance Engine Definition](/guides/developer/open-metadata-archives/governance-engine-definition.svg)
> The structure of a governance engine definition includes a list of governance request types. Each governance request type, along with optional request parameters, are linked to a governance service definition. This indicates which governance service to call when the governance request type is used on a call to the governance engine. The associated request parameters provide default values for the request parameters passed to the governance service when it is invoked. The request parameters can be added to and overridden by the caller. The governance request type and request parameters indicate to the governance service what its action should be.
The unique name of the governance engine defined in the governance engine definition is added to the [OMES configuration](/guides/admin/servers/configuring-an-engine-host/#configure-engine-service). When the [engine host](/concepts/engine-host) starts up the OMES, it reads the *governance engine definition* from its partner [metadata access server](/concepts/metadata-access-server) to complete its initialization. The engine host monitors changes in the governance engine definition so the services within the governance engine can be dynamically maintained.
The unique name of the governance engine defined in the governance engine definition is added to the [Engine Host Services configuration](/guides/admin/servers/by-section/engine-host-services-section). When the [engine host](/concepts/engine-host) starts up the OMES, it reads the *governance engine definition* from its partner [metadata access server](/concepts/metadata-access-server) to complete its initialization. The engine host monitors changes in the governance engine definition so the services within the governance engine can be dynamically maintained.

Calls to the governance engine are made by creating [engine actions](/concepts/engine-action) in the open metadata ecosystem. The engine host is monitoring for new engine actions for the governance engine. When one occurs, it claims the engine action and passes on the request to the governance engine to run. The claim process is used since there may be multiple engine hosts running that governance engine and only one of them should process the request.

Expand All @@ -26,7 +26,7 @@ Calls to the governance engine are made by creating [engine actions](/concepts/e
??? education "Further information"

Each governance engine has a unique name. A *governance engine definition* for this unique name is either created through the [Governance Engine OMAS](/services/omas/governance-engine/overview) API or as part of a [governance engine pack](/concepts/governance-engine-pack). The [governance engine is configured](/guides/admin/servers/configuring-an-engine-host/#configure-engine-service) in the [engine host](/concepts/engine-host) using this unique name.
Each governance engine has a unique name. A *governance engine definition* for this unique name is either created through the [Governance Engine OMAS](/services/omas/governance-engine/overview) API or as part of a [governance engine pack](/concepts/governance-engine-pack). The [governance engine is configured](/guides/admin/servers/by-section/engine-host-services-section) in the [engine host](/concepts/engine-host) using this unique name.

The open metadata types for this definition are in model [0461 - Governance Engines](/types/4/0461-Governance-Engines) - specifically notice **Governance Engine** and **GovernanceService** linked by the **SupportedGovernanceService** relationship.

Expand Down
8 changes: 4 additions & 4 deletions site/docs/concepts/governance-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ not integrate directly with open metadata.

These are the different types:

- [Engine hosts](engine-host.md) - host governance engines for active management of the open metadata ecosystem.
- [Integration daemons](integration-daemon.md) - manage the exchange of metadata with third party technologies.
- [Data engine proxy](data-engine-proxy.md) - captures information about processes and the data sources that they work with and catalogs them in open metadata.
- [Open lineage server](open-lineage-server.md) - accumulates lineage information to provide a comprehensive historical reporting service for lineage.
- [Engine hosts](/concepts/engine-host) - host governance engines for active management of the open metadata ecosystem.
- [Integration daemons](/concepts/integration-daemon) - manage the exchange of metadata with third party technologies.
- [Data engine proxy](/concepts/data-engine-proxy) - captures information about processes and the data sources that they work with and catalogs them in open metadata.
- [Lineage warehouse](/concepts/lineage-warehouse) - accumulates lineage information to provide a comprehensive historical reporting service for lineage.

--8<-- "snippets/abbr.md"
7 changes: 3 additions & 4 deletions site/docs/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,8 @@
- [Likes](/concepts/like)
- [Lineage](/features/lineage-management/overview)
- [Lineage Integrator OMIS](/services/omis/lineage-integrator/overview)
- [Lineage Warehouse](/concepts/lineage-warehouse)
- [Lineage Warehouse Services](/services/lineage-warehouse-services)
- [Local Metadata Collection Id](/concepts/metadata-collection-id)
- [Local Repository Services (OMRS)](/services/omrs)

Expand Down Expand Up @@ -276,8 +278,6 @@
- [Open Discovery Framework (ODF)](/frameworks/odf/overview)
- [Open Discovery Service](/concepts/open-discovery-service)
- [Open Integration Framework](/frmeworks/oif/overview)
- [Open Lineage Server](/concepts/open-lineage-server)
- [Open Lineage Services](/services/open-lineage-services)
- [Open Metadata and Governance (OMAG)](/introduction/overview)
- [Open Metadata Archives](/concepts/open-metadata-archive)
- [Open Metadata Cohort Member](/concepts/cohort-member)
Expand All @@ -289,8 +289,7 @@
- [Open Metadata Repository Cohort](/concepts/cohort-member)
- [Open Metadata Repository Services](/services/omrs)
- [Open Metadata Type Definitions (TypeDefs)](/concepts/open-metadata-type-definitions)
- [Open Lineage Server](/concepts/open-lineage-server)
- [OpenLineage Standard](/features/lineage-management/overview/#openlineage-standard)
- [Open Lineage Standard](/features/lineage-management/overview/#openlineage-standard)
- [Operational Lineage](/features/lineage-management/overview/#openlineage-standard)
- [Organization](/concepts/organization)
- [Organization Integrator OMIS](/services/omis/organization-integrator/overview)
Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/integration-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
!!! education "Further information"

- [Integration connector catalog](/connectors/#integration-connectors).
- [Configuring an integration connector](/guides/admin/servers/configuring-an-integration-daemon/#configure-the-integration-services).
- [Configuring an integration connector](/guides/admin/servers/by-section/integration-daemon-services-section).
- [Writing an integration connector](/guides/developer/integration-connectors/overview).
- [Open Connector Framework (OCF)](/frameworks/ocf/overview).

Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/integration-daemon.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ The integration connectors that run inside the integration daemon can be configu
!!! education "Further information"

- The capabilities of each of the integration services are described in the [integration services](/services/omis).
- Configuring an integration daemon is described in the [Administration Guide](/guides/admin/servers/configuring-an-integration-daemon).
- Configuring an integration daemon is described in the [Administration Guide](/guides/admin/servers/by-server-type/configuring-an-integration-daemon).

--8<-- "snippets/abbr.md"
4 changes: 2 additions & 2 deletions site/docs/concepts/integration-group.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ The integration connector can have multiple catalog log targets that can change

## Configuring integration groups

An [integration daemon is configured](/guides/admin/servers/configuring-an-integration-daemon/) with a list of qualified names for various integration groups. It retrieves the list of integration connectors in each of these groups from the open metadata ecosystem and starts to run them. It continues to monitor for changes to the integration groups and their linked integration connectors, changing the connectors it is running as needed.
An [integration daemon is configured](/guides/admin/servers/by-server-type/configuring-an-integration-daemon) with a list of qualified names for various integration groups. It retrieves the list of integration connectors in each of these groups from the open metadata ecosystem and starts to run them. It continues to monitor for changes to the integration groups and their linked integration connectors, changing the connectors it is running as needed.

![integration group operation](/services/omas/governance-engine/integration-group-use.svg)

???+ education "Further information"

Each integration group has a unique name. An *IntegrationGroup* entity with a qualifiedName set to this unique name is created through the [Governance Engine OMAS](/services/omas/governance-engine/overview) API. This API is also used to create and link the integration connector definitions into the integration group.

The [integration group's qualifiedName is configured](/guides/admin/servers/configuring-an-integration-daemon/#configure-dynamic-integration-groups) in the [integration daemon](/concepts/integration-daemon).
The [integration group's qualifiedName is configured](/guides/admin/servers/by-section/integration-daemon-services-section) in the [integration daemon](/concepts/integration-daemon).

The open metadata types for this definition are in model [0464 - Dynamic Integration Groups](/types/4/0464-Dynamic-Integration-Groups) - specifically notice that *IntegrationGroup* and *IntegrationConnector* are linked by the *RegisteredIntegrationConnector* relationship.

Expand Down
18 changes: 18 additions & 0 deletions site/docs/concepts/lineage-warehouse.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
hide:
- toc
---

<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project 2020. -->

# Lineage Warehouse

The lineage warehouse is a [governance server](/concepts/governance-server)
that manages a historical warehouse of lineage information.

![Lineage Warehouse](lineage-warehouse.svg)

Its behavior and configuration is described [in the administration guide](/guides/admin/servers/by-server-type/configuring-a-lineage-warehouse).

--8<-- "snippets/abbr.md"
4 changes: 4 additions & 0 deletions site/docs/concepts/lineage-warehouse.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion site/docs/concepts/omag-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The types of OMAG Server are shown in Figure 1. The hierarchy groups similar typ
- [Engine Host](/concepts/engine-host) - provides a runtime for a specific type of [governance engine](/concepts/governance-engine) that runs [governance services](/concepts/governance-service) to monitor, enhance and govern the open metadata ecosystem.
- [Integration Daemon](/concepts/integration-daemon) - manages the synchronization of metadata with third party technology that can not call the access services directly. through the [integration services](/services/omis).
- [Data Engine Proxy](/concepts/data-engine-proxy) - supports the capture of metadata from a data engine. This includes details of the processing of data that it is doing which is valuable when piecing together lineage.
- [Open Lineage Server](/concepts/open-lineage-server) - Manages the collation of lineage information and maintains it in a format for reporting. This includes the state of the lineage at different points in time.
- [Lineage Warehouse](/concepts/lineage-warehouse) - Manages the collation of lineage information and maintains it in a format for reporting. This includes the state of the lineage at different points in time.

## Server connectivity

Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/omag-subsystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Fixed services that are always present in the platform:
- [Data engine services](/services/data-engine-proxy-services) implements the services for the [Data Engine Proxy](/concepts/date-engine-proxy).
- [Engine host services](/services/engine-host-services) implements the services for the [Engine Host](/concepts/engine-host).
- [Integration daemon services](/services/integration-daemon-services) implements the services for the [Integration Daemon](/concepts/integration-daemon).
- [Open lineage services](/services/open-lineage-services) implements services for the [Open Lineage Server](/concepts/open-lineage-server).
- [Lineage warehouse services](/services/lineage-warehouse-services) implements services for the [Lineage Warehouse](/concepts/lineage-warehouse).

- [Connected Asset Services](/services/ocf-metadata-management) for supporting the server-side REST API behind the `ConnectedAssetProperties` interface of an [Open Connector Framework (OCF)](/frameworks/ocf/overview) connector.

Expand Down
Binary file not shown.
9 changes: 4 additions & 5 deletions site/docs/concepts/open-lineage-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,12 @@ hide:
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project 2020. -->

# Open lineage server
# Lineage Warehouse

The open lineage server is a [governance server](/concepts/governance-server)
that manages a historical warehouse of lineage information.
The Lineage Warehouse is a [governance server](/concepts/governance-server)that manages a historical warehouse of lineage information.

![Open Lineage Server](open-lineage-server.svg)
![Lineage Warehouse](lineage-warehouse.svg)

Its behavior and configuration is described [in the administration guide](/guides/admin/servers/configuring-an-open-lineage-server).
Its behavior and configuration is described [in the administration guide](/guides/admin/servers/by-server-type/configuring-a-lineage-warehouse).

--8<-- "snippets/abbr.md"
Loading

0 comments on commit b417cb5

Please sign in to comment.