Skip to content

Commit f59cca2

Browse files
ronenhronenh
authored
Merge pull request #187 from aserto-dev/ds-load-docs
Add ds-load cli documentation
2 parents a8b1e33 + 755267c commit f59cca2

File tree

12 files changed

+924
-3
lines changed

12 files changed

+924
-3
lines changed

‎docs/command-line-interface/ds-load-cli/installation.mdx‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
sidebar_label: Installation
3+
title: Installing the ds-load CLI and plugins
4+
description: ds-load CLI installation instructions
5+
---
6+
7+
# Installation
8+
9+
The `ds-load` CLI and plugins are available on Linux, macOS, and Windows platforms.
10+
11+
- Via Homebrew for macOS or LinuxBrew for Linux
12+
13+
```shell
14+
brew tap aserto-dev/tap && brew install ds-load
15+
```
16+
17+
- Via WinGet for Windows
18+
19+
```
20+
winget install Aserto.DSLoad
21+
```
22+
23+
- Binaries for Linux, Windows, and Mac are available as tarballs on the GitHub [release](https://github.com/aserto-dev/ds-load/releases) page.

‎docs/command-line-interface/ds-load-cli/plugins/auth0.mdx‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
sidebar_label: Auth0
3+
---
4+
5+
# Auth0 plugin
6+
7+
## Usage
8+
9+
```sh
10+
Usage: ds-load-auth0 <command>
11+
12+
auth0 directory loader
13+
14+
Commands:
15+
version version information
16+
fetch fetch auth0 data
17+
transform transform auth0 data
18+
export-transform export default transform template
19+
exec fetch and transform auth0 data
20+
verify verify fetcher configuration and credentials
21+
22+
Flags:
23+
-h, --help Show context-sensitive help.
24+
-c, --config=CONFIG-FLAG Configuration file path
25+
-v, --verbosity=INT Use to increase output verbosity.
26+
27+
Run "ds-load-auth0 <command> --help" for more information on a command.
28+
```
29+
30+
## Arguments
31+
32+
The Auth0 plugin supports the following arguments:
33+
34+
```sh
35+
-d, --domain=STRING auth0 domain ($AUTH0_DOMAIN)
36+
-i, --client-id=STRING auth0 client id ($AUTH0_CLIENT_ID)
37+
-s, --client-secret=STRING auth0 client secret ($AUTH0_CLIENT_SECRET)
38+
--connection-name=STRING auth0 connection name ($AUTH0_CONNECTION_NAME)
39+
--user-pid=STRING auth0 user PID of the user you want to read ($AUTH0_USER_PID)
40+
--user-email=STRING auth0 user email of the user you want to read ($AUTH0_USER_EMAIL)
41+
--[no-]roles include roles ($AUTH0_ROLES)
42+
--[no-]rate-limit enable http client rate limiter
43+
```
44+
45+
## Transform
46+
47+
The Auth0 plugin can retrieve both users and roles, and transform these into directory user objects, identity objects, roles, and relationships between these.
48+
49+
To export the default transform, use:
50+
51+
`ds-load auth0 export-transform`
52+
53+
You can use this as the basis for your own transform, which can be tweaked for a different mapping between Auth0 and directory objects and relations.
54+
55+
To learn about the transformation language, refer to the [transform](/docs/command-line-interface/ds-load-cli/transform) docs.

‎docs/command-line-interface/ds-load-cli/plugins/azuread.mdx‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
sidebar_label: Azure AD
3+
---
4+
5+
# Azure AD plugin
6+
7+
## Usage
8+
9+
```sh
10+
Usage: ds-load-azuread <command>
11+
12+
AzureAD directory loader
13+
14+
Commands:
15+
version version information
16+
fetch fetch Azure AD data
17+
transform transform Azure AD data
18+
export-transform export default transform template
19+
exec fetch and transform Azure AD data
20+
verify verify fetcher configuration and credentials
21+
22+
Flags:
23+
-h, --help Show context-sensitive help.
24+
-c, --config=CONFIG-FLAG Configuration file path
25+
-v, --verbosity=INT Use to increase output verbosity.
26+
27+
Run "ds-load-azuread <command> --help" for more information on a command.
28+
```
29+
30+
## Arguments
31+
32+
The Azure AD plugin supports the following arguments:
33+
34+
```sh
35+
-a, --tenant=STRING AzureAD tenant ($AZUREAD_TENANT)
36+
-i, --client-id=STRING AzureAD Client ID ($AZUREAD_CLIENT_ID)
37+
-s, --client-secret=STRING AzureAD Client Secret ($AZUREAD_CLIENT_SECRET)
38+
-r, --refresh-token=STRING AzureAD Refresh Token ($AZUREAD_REFRESH_TOKEN)
39+
```
40+
41+
## Transform
42+
43+
The Azure AD plugin can retrieve both users and groups, and transform these into directory user objects, identity objects, groups, and relationships between these.
44+
45+
To export the default transform, use:
46+
47+
`ds-load azuread export-transform`
48+
49+
You can use this as the basis for your own transform, which can be tweaked for a different mapping between Azure AD and directory objects and relations.
50+
51+
To learn about the transformation language, refer to the [transform](/docs/command-line-interface/ds-load-cli/transform) docs.

‎docs/command-line-interface/ds-load-cli/plugins/cognito.mdx‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
sidebar_label: Cognito
3+
---
4+
5+
# Cognito plugin
6+
7+
## Usage
8+
9+
```sh
10+
Usage: ds-load-cognito <command>
11+
12+
Cognito directory loader
13+
14+
Commands:
15+
version version information
16+
fetch fetch cognito data
17+
transform transform cognito data
18+
export-transform export default transform template
19+
exec fetch and transform cognito data
20+
verify verify fetcher configuration and credentials
21+
22+
Flags:
23+
-h, --help Show context-sensitive help.
24+
-c, --config=CONFIG-FLAG Configuration file path
25+
-v, --verbosity=INT Use to increase output verbosity.
26+
27+
Run "ds-load-cognito <command> --help" for more information on a command.
28+
```
29+
30+
## Arguments
31+
32+
The Cognito plugin supports the following arguments:
33+
34+
```sh
35+
-k, --access-key=STRING AWS Access Key ($AWS_ACCESS_KEY)
36+
-s, --secret-key=STRING AWS Secret Key ($AWS_SECRET_KEY)
37+
-p, --user-pool-id=STRING AWS Cognito User Pool ID ($AWS_COGNITO_USER_POOL_ID)
38+
-r, --region=STRING AWS Region ($AWS_REGION)
39+
-g, --[no-]groups Retrieve Cognito groups ($AWS_COGNITO_GROUPS)
40+
```
41+
42+
## Transform
43+
44+
The Cognito plugin can retrieve both users and groups, and transform these into directory user objects, identity objects, group objects, and relationships between these.
45+
46+
To export the default transform, use:
47+
48+
`ds-load cognito export-transform`
49+
50+
You can use this as the basis for your own transform, which can be tweaked for a different mapping between Cognito and directory objects and relations.
51+
52+
To learn about the transformation language, refer to the [transform](/docs/command-line-interface/ds-load-cli/transform) docs.

‎docs/command-line-interface/ds-load-cli/plugins/custom.mdx‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
---
2+
sidebar_label: Custom Plugins
3+
---
4+
5+
# Writing a custom plugin
6+
7+
Plugins have a simple contract with the host `ds-load` command. They can be written in any language.
8+
9+
A set of go packages can be used as a "mini-SDK" for building custom plugins in go. This eliminates much of the "boilerplate" of writing a plugin, including commands like `export-transform`, providing help, and so on.
10+
11+
The plugins that come with `ds-load` can all be found [here](https://github.com/aserto-dev/ds-load/tree/main/plugins). They can be used as a guide for how to write a custom plugin for a new data source.
12+
13+
In addition, a standalone plugin for importing Hubspot data into the directory can be found [here](https://github.com/aserto-demo/ds-load-hubspot). This shows how to properly import the `ds-load` SDK in an external plugin.
14+
15+
## Retrieving data
16+
17+
Most of the "heavy lifting" of building a plugin involves retrieving data from the source. The Hubspot plugin uses the Hubspot REST API, and can easily be adapted to any other service that supports a REST interaction pattern.
18+
19+
## Transforming into objects and relations
20+
21+
The other main task is to author the default transform for the plugin. The [transformation](/docs/command-line-interface/ds-load-cli/transform.mdx) docs provide an annotated walkthrough of the Auth0 default transform, which can act as a guide.
22+
23+
The Hubspot sample plugin also has a default transform that shows how to create objects of different types, and relationships between them.
24+
25+
## Installation
26+
27+
To have `ds-load` pick up your custom plugin, simply call it `ds-load-<plugin-name>` and copy the binary to `~/.ds-load/plugins`.
28+
29+
```sh
30+
ls -l ~/.ds-load/plugins
31+
total 32000
32+
-rwxr-xr-x 1 ogazitt staff 16M Aug 3 10:50 ds-load-hubspot*```
33+
```
34+
35+
To verify that `ds-load` can load your plugin, use `ds-load exec --help`. Note that `hubspot` is in the list of available commands:
36+
37+
```sh
38+
Usage: ds-load exec <command> ...
39+
40+
import data in directory by running fetch, transform and publish
41+
42+
Arguments:
43+
<command> ... available commands are: hubspot|auth0|azuread|cognito|google|okta
44+
```
45+
46+
## Need help?
47+
48+
Please drop us a line in our [community slack](https://aserto.com/slack) if you need help writing a custom plugin! Also, we're happy to evaluate and accept new providers written by the community via a Pull Request in the [ds-load](https://github.com/aserto-dev/ds-load) repository.
49+
50+
Submitted plugins should follow the same design patterns as existing providers, and live in the [plugins](https://github.com/aserto-dev/ds-load/tree/main/plugins) directory.

‎docs/command-line-interface/ds-load-cli/plugins/google.mdx‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
sidebar_label: Google Workspace
3+
---
4+
5+
# Google Workspace plugin
6+
7+
## Usage
8+
9+
```sh
10+
Usage: ds-load-google <command>
11+
12+
Google Workspace directory loader
13+
14+
Commands:
15+
version version information
16+
fetch fetch google workspace data
17+
transform transform google workspace data
18+
export-transform export default transform template
19+
exec fetch and transform google workspace data
20+
get-refresh-token obtain a refresh token from GCP
21+
verify verify fetcher configuration and credentials
22+
23+
Flags:
24+
-h, --help Show context-sensitive help.
25+
-c, --config=CONFIG-FLAG Configuration file path
26+
-v, --verbosity=INT Use to increase output verbosity.
27+
28+
Run "ds-load-google <command> --help" for more information on a command.
29+
```
30+
31+
## Arguments
32+
33+
The Google Workspace plugin supports the following arguments:
34+
35+
```sh
36+
-i, --client-id=STRING Google Client ID ($GOOGLE_CLIENT_ID)
37+
-s, --client-secret=STRING Google Client Secret ($GOOGLE_CLIENT_SECRET)
38+
-r, --refresh-token=STRING Google Refresh Token ($GOOGLE_REFRESH_TOKEN)
39+
-g, --groups Retrieve Google groups ($GOOGLE_GROUPS)
40+
--customer="my_customer" Google Workspace Customer field ($GOOGLE_CUSTOMER)
41+
```
42+
43+
## Obtaining a refresh token
44+
45+
Once you've created an OAuth application in the Google Cloud Console, the Google Workspace plugin can generate a refresh token for you.
46+
47+
```sh
48+
Usage: ds-load-google get-refresh-token --client-id=STRING --client-secret=STRING
49+
50+
obtain a refresh token from GCP
51+
52+
Flags:
53+
-h, --help Show context-sensitive help.
54+
-c, --config=CONFIG-FLAG Configuration file path
55+
-v, --verbosity=INT Use to increase output verbosity.
56+
57+
-i, --client-id=STRING Google Client ID ($GOOGLE_CLIENT_ID)
58+
-s, --client-secret=STRING Google Client Secret ($GOOGLE_CLIENT_SECRET)
59+
-p, --port=8761 Port number to run callback server on ($GOOGLE_PORT)
60+
```
61+
62+
This refresh token can then be passed (along with the Client ID and Client Secret) to complete the configuration of the Google Workspace plugin.
63+
64+
## Transform
65+
66+
The Google Workspace plugin can retrieve both users and groups, and transform these into directory user objects, identity objects, group objects, and relationships between these.
67+
68+
To export the default transform, use:
69+
70+
`ds-load google export-transform`
71+
72+
You can use this as the basis for your own transform, which can be tweaked for a different mapping between Google Workspace and directory objects and relations.
73+
74+
To learn about the transformation language, refer to the [transform](/docs/command-line-interface/ds-load-cli/transform) docs.

‎docs/command-line-interface/ds-load-cli/plugins/ldap.mdx‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
sidebar_label: LDAP
3+
---
4+
5+
# LDAP plugin
6+
7+
## Usage
8+
```sh
9+
Usage: ds-load-ldap <command>
10+
11+
ldap directory loader
12+
13+
Commands:
14+
version version information
15+
fetch fetch ldap data
16+
transform transform ldap data
17+
export-transform export default transform template
18+
exec fetch and transform ldap data
19+
verify verify fetcher configuration and credentials
20+
21+
Flags:
22+
-h, --help Show context-sensitive help.
23+
-c, --config=CONFIG-FLAG Configuration file path
24+
-v, --verbosity=INT Use to increase output verbosity.
25+
26+
Run "ds-load-ldap <command> --help" for more information on a command.
27+
```
28+
29+
## Arguments
30+
31+
The LDAP plugin supports the following arguments:
32+
33+
```sh
34+
-h, --help Show context-sensitive help.
35+
-c, --config=CONFIG-FLAG Configuration file path
36+
-v, --verbosity=INT Use to increase output verbosity.
37+
38+
-u, --user=STRING LDAP user ($LDAP_USER)
39+
-p, --password=STRING LDAP password ($LDAP_PASSWORD)
40+
-s, --host=STRING LDAP host ($LDAP_HOST)
41+
-b, --base-dn="dc=example,dc=org" LDAP base DN ($LDAP_BASE_DN)
42+
-f, --user-filter="(&(objectClass=organizationalPerson))" LDAP user filter ($LDAP_USER_FILTER)
43+
-g, --group-filter="(&(objectClass=groupOfNames))" LDAP group filter ($LDAP_GROUP_FILTER)
44+
-i, --insecure Allow insecure LDAP connection ($LDAP_INSECURE)
45+
-U, --id-field="objectGUID" LDAP field to use as ID ($LDAP_ID_FIELD)
46+
-t, --template=STRING transformation template file path ($DS_TEMPLATE_FILE)
47+
```
48+
49+
## Transform
50+
51+
The LDAP plugin can retrieve both users and groups, and transform these into directory user objects, identity objects, groups, and relationships between these.
52+
53+
To export the default transform, use:
54+
55+
`ds-load ldap export-transform`
56+
57+
You can use this as the basis for your own transform, which can be tweaked for a different mapping between LDAP and directory objects and relations.
58+
59+
To learn about the transformation language, refer to the [transform](/docs/command-line-interface/ds-load-cli/transform) docs.
60+
61+
62+
## Config example
63+
```
64+
---
65+
host: "directory.prod.aserto.com:8443"
66+
api-key: "<your-directory-api-key>"
67+
tenant-id: "<your-tenant-id>"
68+
ldap:
69+
base-dn: "dc=aserto,dc=com"
70+
user: "CN=aserto,CN=Users,DC=aserto,DC=com"
71+
password: "<ldap-user-password>"
72+
host: "ldap://localhost:1389"
73+
user-filter: "(&(objectClass=organizationalPerson))"
74+
group-filter: "(&(objectClass=group))"
75+
id-field: "objectGUID"
76+
```

0 commit comments

Comments
 (0)