Skip to content

Commit 8833924

Browse files
StewartJinggaStewartJingga
authored
doc: improve documentation (#155)
* doc: add tour sections * feat: add Tour section * doc: update docs * doc: update docs * doc: update docs * doc: update docs
1 parent f01ec6f commit 8833924

File tree

8 files changed

+426
-30
lines changed

8 files changed

+426
-30
lines changed

docs/docs/concepts/overview.md renamed to docs/docs/concepts/asset.mdx

Lines changed: 15 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,13 @@
1-
# Overview
1+
import Tabs from '@theme/Tabs';
2+
import TabItem from '@theme/TabItem';
23

3-
Compass has three major concept when it comes to data ingestion: Asset, Type, and Service.
4+
# Asset
45

5-
Asset is essentially an arbitrary JSON object that represent a metadata of a specific service with a specific type.
6+
In Compass, we call every metadata that you input as an Asset. All your tables, dashboards, topics, jobs are an example of assets.
67

7-
Type defines a ‘type’ of an asset and it is pre-defined. There are currently 4 supported types in Compass: `table`, `job`, `dashboard`, and `topic`.
8-
9-
Service defines the application name that the asset was coming from. For example: `biquery`, `postgres`, etc. If you wanted to push data for `bigquery` dataset\(s\) to Compass, you would need to first define the ‘`bigquery`’ service in compass.
8+
<Tabs>
9+
<TabItem value="table" label="Table View">
1010

11-
Some features that compass has:
12-
* Asset Tagging
13-
* User
14-
* Discussion
15-
* Starring
16-
17-
## Asset
18-
19-
An Asset is a JSON document that describes a metadata. Asset has a schema:
2011
| Field | Required | Type | Description |
2112
|---|---|---|---|
2213
| id | false | string | compass' auto-generated uuid |
@@ -29,7 +20,10 @@ An Asset is a JSON document that describes a metadata. Asset has a schema:
2920
| labels | false |json | labels of metadata, written in key-value string |
3021
| owners | false | []json | array of json, where each json contains `email` field |
3122

32-
```text
23+
</TabItem>
24+
<TabItem value="json" label="JSON">
25+
26+
```json
3327
{
3428

3529
"urn": "topic/order-log",
@@ -57,9 +51,13 @@ An Asset is a JSON document that describes a metadata. Asset has a schema:
5751
}
5852
```
5953

54+
</TabItem>
55+
</Tabs>
56+
57+
6058
Every asset that is pushed SHOULD have the required fields: `urn`, `type`, `service`, `name`. The value of these fields MUST be string, if present.
6159

62-
Asset ingestion API \(/v1beta1/assets\) is using HTTP PATCH method. The behavioud would be similar with how PATCH works. It is possible to patch one field only in an asset by sending the updated field to the ingestion API. This also works for the data in dynamic `data` field. The combination of `urn`, `type`, `service` will be the identifier to patch an asset.
60+
Asset ingestion API (`/v1beta1/assets`) is using HTTP PATCH method. The behavioud would be similar with how PATCH works. It is possible to patch one field only in an asset by sending the updated field to the ingestion API. This also works for the data in dynamic `data` field. The combination of `urn`, `type`, `service` will be the identifier to patch an asset.
6361
In case the `urn` does not exist, the asset ingestion PATCH API \(/v1beta1/assets\) will create a new asset.
6462

6563
## Lineage
@@ -146,13 +144,3 @@ If there is an update to the `environment` in the asset labels, here is the asse
146144
## Tagging an Asset
147145
Compass allows user to tag a specific asset. To tag a new asset, one needs to create a template of the tag. Tag's template defines a set of fields' tag that are applicable to tag each field in an asset.
148146
Once a template is created, each field in an asset is possible to be tagged by calling `/v1beta1/tags` API. More detail about [Tagging](../guides/tagging.md).
149-
150-
## User
151-
The current version of Compass does not have user management. Compass expect there is an external instance that manages user. Compass consumes user information from the configurable identity uuid header in every API call. The default name of the header is `Compass-User-UUID`.
152-
Compass does not make any assumption of what kind of identity format that is being used. The `uuid` indicates that it could be in any form (e.g. email, UUIDv4, etc) as long as it is universally unique.
153-
The current behaviour is, Compass will add a new user if the user information consumed from the header does not exist in Compass' database. More detail about [User](./user.md).
154-
## Discussion
155-
Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md).
156-
157-
## Starring
158-
Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md).

‎docs/docs/concepts/overview.mdx‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Overview
2+
3+
Compass has three major concept when it comes to data ingestion: Asset, Type, and Service.
4+
5+
Asset is the main model that represents a metadata of a specific service with a specific type.
6+
7+
Type defines a ‘type’ of an asset and it is pre-defined. Compass currently supports the following types:
8+
1. `table`
9+
2. `job`
10+
3. `dashboard`
11+
4. `topic`
12+
5. `feature_table`
13+
6. `model` (under development)
14+
7. `application` (under development)
15+
16+
Service defines the application or source that maintains or generates the asset. Examples would be `biquery`, `postgres`, etc.
17+
18+
Some features that compass has:
19+
* [Discovery](../tour/2-querying-assets.mdx)
20+
* [Lineage](../tour/3-asset-lineage.mdx)
21+
* [Asset Tagging](./asset#tagging-an-asset)
22+
* [User](./user.md)
23+
* [Discussion](../guides/discussion.md)
24+
* [Starring](../guides/starring.md)
25+
26+
## Discussion
27+
Compass supports discussion feature. User could drop comments in each discussion. Currently, there are three types of discussions `issues`, `open ended`, and `question and answer`. Depending on the type, the discussion could have multiple possible states. In the current version, all types only have two states: `open` and `closed`. A newly created discussion will always be assign an `open` state. More detail about [Discussion](../guides/discussion.md).
28+
29+
## Starring
30+
Compass allows a user to stars an asset. This bookmarking functionality is introduced to increase the speed of a user to get information. There is also an API to see which users star an asset (stargazers). More detail about [Starring](../guides/starring.md).

‎docs/docs/concepts/type.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Type
2+
3+
Each Asset will have a `Type` to represent the kind of metadata it represents. It is currently pre-defined by Compass, so no arbitrary types will be supported.
4+
5+
Compass currently supports the following types:
6+
1. `table`
7+
2. `job`
8+
3. `dashboard`
9+
4. `topic`
10+
5. `feature_table`
11+
6. `model` (under development)
12+
7. `application` (under development)
13+
14+
Type will be extremely useful for categorizing your assets and it will be really helpful during discovery.
15+
Check [this section on querying assets](../guides/querying#using-the-get-assets-api) on how to leverage `type` for your discovery.

docs/docs/guides/installation.md renamed to docs/docs/installation.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,6 @@ time="2022-04-27T09:18:08Z" level=info msg="server started"
7676
```
7777

7878
## Required Header/Metadata in API
79-
Compass has a concept of [User](../concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`.
79+
Compass has a concept of [User](./concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`.
8080

81-
Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](../concepts/user.md) section.
81+
Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](./concepts/user.md) section.

‎docs/docs/tour/1-my-first-asset.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# 1. My First Asset
2+
3+
Before starting the tour, make sure you have a running Compass instance. You can refer this [installation guide](../installation).
4+
5+
## 1.1 Introduction
6+
7+
In Compass, we call every metadata that you input as an [Asset](../concepts/asset). All your tables, dashboards, topics, jobs are an example of assets.
8+
9+
In this section, we will help you to build your first Asset and hopefully it will give your clear idea about what an Asset is in Compass.
10+
11+
## 1.2 Hello, ~~World~~ Asset!
12+
13+
Let's imagine we have a `postgres` instance that we keep referring to as our `main-postgres`. Inside it there is a database called `my-database` that has plenty of tables. One of the tables is named `orders`, and below is how you represent that `table` as an Compass' Asset.
14+
15+
```json
16+
{
17+
"urn": "main-postgres:my-database.orders",
18+
"type": "table",
19+
"service": "postgres",
20+
"name": "orders",
21+
"data": {
22+
"database": "my-database",
23+
"namespace": "main-postgres"
24+
}
25+
}
26+
```
27+
28+
- **urn** is a unique name you assign to an asset. You need to make sure you don't have a duplicate urns across all of your assets because Compass treats `urn` as an identifier of your asset. For this example, we use the following format to make sure our urn is unique, `{NAMESPACE}:{DB_NAME}.{TABLE_NAME}`. (more info about URN generation can be found [here](../guides/urn-generation))
29+
30+
- **type** is your Asset's type. The value for type has to be recognizable by Compass. More info about Asset's Type can be found [here](../concepts/type).
31+
32+
- **service** can be seen as the source of your asset. `service` can be anything, in this case since our `orders` table resides in `postgres`, we can just put `postgres` as the service.
33+
34+
- **name** is the name of your asset, it does not have to be unique. We don't need to worry to get mixed up if there are other tables with the same name, `urn` will be the main identifier for your asset, that is why we need to make it unique across all of your assets.
35+
36+
- **data** can hold your asset's extra details if there is any. In the example, we use it to store information of the **database name** and the **alias/namespace** that we use when referring the postgres instance.
37+
38+
## 1.3 Sending your first asset to Compass
39+
40+
Here is the asset that we built on previous section.
41+
42+
```json
43+
{
44+
"urn": "main-postgres:my-database.orders",
45+
"type": "table",
46+
"service": "postgres",
47+
"name": "orders",
48+
"data": {
49+
"database": "my-database",
50+
"namespace": "main-postgres"
51+
}
52+
}
53+
```
54+
Let's send this into Compass so that it would be discoverable.
55+
56+
As of now, Compass supports ingesting assets via `gRPC` and `http`. In this example, we will use `http` to send your first asset to Compass.
57+
Compass exposes an API `[PATCH] /v1beta1/assets` to upload your asset.
58+
59+
```bash
60+
curl --location --request PATCH 'http://localhost:8080/v1beta1/assets' \
61+
--header 'Content-Type: application/json' \
62+
--header 'Compass-User-UUID: [email protected]' \
63+
--data-raw '{
64+
"asset": {
65+
"urn": "main-postgres:my-database.orders",
66+
"type": "table",
67+
"service": "postgres",
68+
"name": "orders",
69+
"data": {
70+
"database": "my-database",
71+
"namespace": "main-postgres"
72+
}
73+
}
74+
}'
75+
```
76+
77+
There are a few things to notice here:
78+
1. The HTTP method used is `PATCH`. This is because Compass does not have a dedicated `Create` API, it uses a single API to `Patch / Create` an asset instead. So when updating or patching your asset, you can use the same API.
79+
80+
2. Compass requires `Compass-User-UUID` header to be in the request. More information about the identity header can be found [here](../concepts/user). To simplify this tour, let's just use `[email protected]`.
81+
82+
3. When sending our asset to Compass, we need to put our asset object inside an `asset` field as shown in the sample curl above.
83+
84+
On a success insertion, your will receive below response:
85+
86+
```json
87+
{ "id": "cebeb793-8933-434c-b38f-beb6dbad91a5" }
88+
```
89+
90+
**id** is an identifier of your asset. Unlike `urn` which is provided by you, `id` is auto generated by Compass if there was no asset found with the given URN.
91+
92+
## Conclusion
93+
94+
Now that you have successfully ingested your asset to Compass, we can now search and find it via Compass.
95+
96+
In the next section, we will see how Compass can help you in searching and discovering your assets.

0 commit comments

Comments
 (0)