docs: add namespace in cli and api specs

kushsharma · ravisuhag · commit 0a04c076ba0c · 2023-03-29T22:46:45.000-05:00
Signed-off-by: Kush Sharma &lt;thekushsharma@gmail.com&gt;
diff --git a/docs/docs/concepts/internals.md b/docs/docs/concepts/internals.md
@@ -4,9 +4,7 @@ This document details information about how Compass interfaces with elasticsearc
 
 ## Index Setup
 
-There is a migration command in compass to setup all storages. Once the migration is executed, all types are being created (if does not exist). When a type is created, an index is created in elasticsearch by it's name. All created indices are aliased to the `universe` index, which is used to run the search when all types need to be searched, or when `filter[type]` is not specifed in the Search API.
-
-The indices are also configured with a camel case tokenizer, to support proper lexing of some resources that use camel case in their nomenclature \(protobuf names for instance\). Given below is a sample of the index settings that are used:
+There is a migration command in compass to setup all storages. The indices are configured with a camel case tokenizer, to support proper lexing of some resources that use camel case in their nomenclature \(protobuf names for instance\). Given below is a sample of the index settings that are used:
 
 ```javascript
 // PUT http://${ES_HOST}/{index}
@@ -28,6 +26,30 @@ The indices are also configured with a camel case tokenizer, to support proper l
     }
 ```
 
+One shared index is created for all services and tenants but each request(read/write) is routed to a unique shard for each tenant. Compass categorize tenants into two tires, `shared` and `dedicated`. For shared tenants, all the requests will be routed by namespace id over a single shard in an index. For dedicated tenants, each tenant will have its own index. Note, a single index will have N number of `types` same as the number of `Services` supported in Compass. This design will ensure, all the document insert/query requests are only confined to a single shard(in case of shared) or a single index(in case of dedicated).
+Details on why we did this is available at [issue #208](https://github.com/odpf/compass/issues/208).
+
+## Postgres
+
+To enforce multi-tenant restrictions at the database level, [Row Level Security](https://www.postgresql.org/docs/current/ddl-rowsecurity.html) is used. RLS requires Postgres users used for application database connection not to be a table owner or a superuser else all RLS are bypassed by default. That means a Postgres user that is migrating the application and a user that is used to serve the app should both be different.
+
+To create a postgres user
+```sql
+CREATE USER "compass_user" WITH PASSWORD 'compass';
+GRANT CONNECT ON DATABASE "compass" TO "compass_user";
+GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "compass_user";
+GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO "compass_user";
+GRANT ALL ON ALL FUNCTIONS IN SCHEMA public TO "compass_user";
+
+ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT, INSERT, UPDATE, DELETE, REFERENCES
+ON TABLES TO "compass_user";
+ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT USAGE ON SEQUENCES TO "compass_user";
+ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT EXECUTE ON FUNCTIONS TO "compass_user";
+```
+
+A middleware for grpc looks for `x-namespace-id` header to extract tenant id if not found falls back to `default` namespace. 
+Same could be passed in a `jwt token` of Authentication Bearer with `namespace_id` as a claim.
+
 ## Search
 
 We use elasticsearch's `multi_match` search for running our queries. Depending on whether there are additional filter's specified during search, we augment the query with a custom script query that filter's the result set.
diff --git a/docs/docs/reference/api.md b/docs/docs/reference/api.md
@@ -1,7 +1,7 @@
-# API
+# Compass
 Documentation of our Compass API with gRPC and gRPC-Gateway.
 
-## Version: 0.2.1
+## Version: 0.3.0
 
 **License:** [Apache License 2.0](https://github.com/odpf/compass/blob/main/LICENSE)
 
@@ -317,6 +317,8 @@ API for querying documents. 'text' is fuzzy matched against all the available da
 | text | query | text to search for (fuzzy) | No | string |
 | rankby | query | descendingly sort based on a numeric field in the asset. the nested field is written with period separated field name. eg, "rankby[data.profile.usage_count]" | No | string |
 | size | query | number of results to return | No | long |
+| include_fields | query |  | No | [ string ] |
+| offset | query | offset parameter defines the offset from the first result you want to fetch | No | long |
 
 ##### Responses
 
@@ -1097,6 +1099,111 @@ Get all assets starred by a user
 
 ## default
 
+### /v1beta1/namespaces
+
+#### GET
+##### Summary
+
+List namespace
+
+##### Description
+
+List all created namespaces
+
+##### Responses
+
+| Code | Description | Schema |
+| ---- | ----------- | ------ |
+| 200 | A successful response. | [ListNamespacesResponse](#listnamespacesresponse) |
+| 400 | Returned when the data that user input is wrong. | [Status](#status) |
+| 404 | Returned when the resource does not exist. | [Status](#status) |
+| 409 | Returned when the resource already exist. | [Status](#status) |
+| 500 | Returned when theres is something wrong on the server side. | [Status](#status) |
+| default | An unexpected error response. | [Status](#status) |
+
+#### POST
+##### Summary
+
+Create a namespace
+
+##### Description
+
+Create a new namespace, throws error if already exists
+
+##### Parameters
+
+| Name | Located in | Description | Required | Schema |
+| ---- | ---------- | ----------- | -------- | ------ |
+| body | body |  | Yes | [CreateNamespaceRequest](#createnamespacerequest) |
+
+##### Responses
+
+| Code | Description | Schema |
+| ---- | ----------- | ------ |
+| 200 | A successful response. | [CreateNamespaceResponse](#createnamespaceresponse) |
+| 400 | Returned when the data that user input is wrong. | [Status](#status) |
+| 404 | Returned when the resource does not exist. | [Status](#status) |
+| 409 | Returned when the resource already exist. | [Status](#status) |
+| 500 | Returned when theres is something wrong on the server side. | [Status](#status) |
+| default | An unexpected error response. | [Status](#status) |
+
+### /v1beta1/namespaces/{urn}
+
+#### GET
+##### Summary
+
+Get namespace
+
+##### Description
+
+Fetch a namespace details, throws error if doesn't exists. Use id or name as urn.
+
+##### Parameters
+
+| Name | Located in | Description | Required | Schema |
+| ---- | ---------- | ----------- | -------- | ------ |
+| urn | path | set either id or name | Yes | string |
+
+##### Responses
+
+| Code | Description | Schema |
+| ---- | ----------- | ------ |
+| 200 | A successful response. | [GetNamespaceResponse](#getnamespaceresponse) |
+| 400 | Returned when the data that user input is wrong. | [Status](#status) |
+| 404 | Returned when the resource does not exist. | [Status](#status) |
+| 409 | Returned when the resource already exist. | [Status](#status) |
+| 500 | Returned when theres is something wrong on the server side. | [Status](#status) |
+| default | An unexpected error response. | [Status](#status) |
+
+#### PUT
+##### Summary
+
+Update namespace
+
+##### Description
+
+Update an existing namespace, throws error if doesn't exists. Use id or name as urn.
+
+##### Parameters
+
+| Name | Located in | Description | Required | Schema |
+| ---- | ---------- | ----------- | -------- | ------ |
+| urn | path | set either id or name | Yes | string |
+| body | body |  | Yes | { **"metadata"**: object, **"state"**: string } |
+
+##### Responses
+
+| Code | Description | Schema |
+| ---- | ----------- | ------ |
+| 200 | A successful response. | [UpdateNamespaceResponse](#updatenamespaceresponse) |
+| 400 | Returned when the data that user input is wrong. | [Status](#status) |
+| 404 | Returned when the resource does not exist. | [Status](#status) |
+| 409 | Returned when the resource already exist. | [Status](#status) |
+| 500 | Returned when theres is something wrong on the server side. | [Status](#status) |
+| default | An unexpected error response. | [Status](#status) |
+
+## default
+
 ### /v1beta1/search
 
 #### GET
@@ -1115,6 +1222,8 @@ API for querying documents. 'text' is fuzzy matched against all the available da
 | text | query | text to search for (fuzzy) | No | string |
 | rankby | query | descendingly sort based on a numeric field in the asset. the nested field is written with period separated field name. eg, "rankby[data.profile.usage_count]" | No | string |
 | size | query | number of results to return | No | long |
+| include_fields | query |  | No | [ string ] |
+| offset | query | offset parameter defines the offset from the first result you want to fetch | No | long |
 
 ##### Responses
 
@@ -1535,6 +1644,21 @@ Request to be sent to create a discussion
 | ---- | ---- | ----------- | -------- |
 | id | string |  | No |
 
+#### CreateNamespaceRequest
+
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| id | string | optional, if not specified will be auto generated | No |
+| metadata | object | key value pairs as metadata for the namespace | No |
+| name | string |  | No |
+| state | string |  | No |
+
+#### CreateNamespaceResponse
+
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| id | string |  | No |
+
 #### CreateTagAssetRequest
 
 Request to be sent to create a tag
@@ -1708,6 +1832,12 @@ Request to be sent to create a tag's template
 | ---- | ---- | ----------- | -------- |
 | data | [ [v1beta1.Asset](#v1beta1asset) ] |  | No |
 
+#### GetNamespaceResponse
+
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| namespace | [Namespace](#namespace) |  | No |
+
 #### GetTagByAssetAndTemplateResponse
 
 | Name | Type | Description | Required |
@@ -1742,6 +1872,21 @@ Request to be sent to create a tag's template
 | type | string |  | No |
 | urn | string |  | No |
 
+#### ListNamespacesResponse
+
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| namespaces | [ [Namespace](#namespace) ] |  | No |
+
+#### Namespace
+
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| id | string |  | No |
+| metadata | object | key value pairs as metadata for the namespace | No |
+| name | string |  | No |
+| state | string |  | No |
+
 #### NodeAttributes
 
 | Name | Type | Description | Required |
@@ -1851,6 +1996,12 @@ Request to be sent to create a tag's template
 | ---- | ---- | ----------- | -------- |
 | UpdateCommentResponse | object |  |  |
 
+#### UpdateNamespaceResponse
+
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| UpdateNamespaceResponse | object |  |  |
+
 #### UpdateTagAssetResponse
 
 | Name | Type | Description | Required |
diff --git a/docs/docs/reference/cli.md b/docs/docs/reference/cli.md
@@ -133,25 +133,55 @@ view discussion for the given ID
 
 observe the lineage of metadata
 
+## `compass namespace`
+
+Manage namespaces
+
+### `compass namespace create [flags]`
+
+create a new namespace
+
+```
+-n, --name string    namespace unique name
+-s, --state string   is namespace shared with existing tenants or a dedicated one (default "shared")
+````
+
+### `compass namespace list [flags]`
+
+lists all namespaces
+
+```
+-o, --out -o json   flag to control output viewing, for json -o json (default "table")
+````
+
+### `compass namespace view <id>`
+
+view namespace for the given uuid or name
+
 ## `compass search <text> [flags]`
 
 query the metadata available
 
 ```
--f, --filter string   --filter=field_key1:val1,key2:val2,key3:val3 gives exact match for values
--q, --query string    --query=--filter=field_key1:val1 supports fuzzy search
--r, --rankby string   --rankby=<numeric_field>
--s, --size uint32     --size=10 maximum size of response query
+-f, --filter string      --filter=field_key1:val1,key2:val2,key3:val3 gives exact match for values
+-n, --namespace string   namespace id or name (default "00000000-0000-0000-0000-000000000000")
+-q, --query string       --query=--filter=field_key1:val1 supports fuzzy search
+-r, --rankby string      --rankby=<numeric_field>
+-s, --size uint32        --size=10 maximum size of response query
 ````
 
 ## `compass server <command>`
 
 Run compass server
 
-### `compass server migrate`
+### `compass server migrate [flags]`
 
 Run storage migration
 
+```
+--down   rollback migration one step
+````
+
 ### `compass server start`
 
 Start server on default port 8080
