You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md
+129-1
Original file line number
Diff line number
Diff line change
@@ -4,8 +4,136 @@ sidebar_position: 80
4
4
unlisted: true
5
5
---
6
6
7
-
{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/customizing-configuration*/}
7
+
:::note
8
+
This guide is applicable to Dagster+.
9
+
:::
10
+
11
+
{/* /deployment/dagster-instance */}
12
+
The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/todo.md) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings.
For [Kubernetes agents](/todo.md) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent.
17
+
:::
18
+
19
+
## Enabling user code server TTL
20
+
21
+
User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven't served requests recently and will spin them back up the next time they're needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle.
22
+
23
+
To configure TTL:
24
+
```yaml
25
+
# dagster.yaml
26
+
instance_class:
27
+
module: dagster_cloud.instance
28
+
class: DagsterCloudAgentInstance
29
+
30
+
dagster_cloud_api:
31
+
agent_token:
32
+
env: DAGSTER_CLOUD_AGENT_TOKEN
33
+
deployment: prod
34
+
35
+
user_code_launcher:
36
+
module: dagster_cloud.workspace.docker
37
+
class: DockerUserCodeLauncher
38
+
config:
39
+
server_ttl:
40
+
enabled: true
41
+
ttl_seconds: 7200#2 hours
42
+
```
43
+
44
+
## Streaming compute logs
45
+
46
+
You can set up streaming compute logs by configuring the log upload interval (in seconds).
47
+
48
+
```yaml
49
+
# dagster.yaml
50
+
instance_class:
51
+
module: dagster_cloud.instance
52
+
class: DagsterCloudAgentInstance
53
+
54
+
dagster_cloud_api:
55
+
agent_token:
56
+
env: DAGSTER_CLOUD_AGENT_TOKEN
57
+
deployment: prod
58
+
59
+
user_code_launcher:
60
+
module: dagster_cloud.workspace.docker
61
+
class: DockerUserCodeLauncher
62
+
63
+
compute_logs:
64
+
module: dagster_cloud
65
+
class: CloudComputeLogManager
66
+
config:
67
+
upload_interval: 60
68
+
```
8
69
9
70
## Disabling compute logs
10
71
72
+
<<<<<<< HEAD
11
73
{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}
74
+
=======
75
+
You can disable forwarding compute logs to Dagster+ by configuring the `NoOpComputeLogManager` setting:
You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/todo.md) module.
101
+
102
+
You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.
103
+
104
+
Note: Dagster Labs will neither have nor use your AWS credentials. The Dagster+ UI will be able to show the URLs linking to the compute log files in your S3 bucket when you set the `show_url_only` parameter to `true`.
Refer to the [Managing deployments in Dagster+ guide](/todo.md) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.
18
+
19
+
## Settings schema
20
+
21
+
Settings are formatted using YAML. For example:
22
+
23
+
```yaml
24
+
run_queue:
25
+
max_concurrent_runs: 10
26
+
tag_concurrency_limits:
27
+
- key: "database"
28
+
value: "redshift"
29
+
limit: 5
30
+
31
+
run_monitoring:
32
+
start_timeout_seconds: 1200
33
+
cancel_timeout_seconds: 1200
34
+
max_runtime_seconds: 7200
35
+
36
+
run_retries:
37
+
max_retries: 0
38
+
39
+
sso_default_role: EDITOR
40
+
```
41
+
42
+
## Settings
43
+
44
+
For each deployment, you can configure settings for:
The `run_queue` settings allow you to specify how many runs can execute concurrently in the deployment.
55
+
56
+
```yaml
57
+
run_queue:
58
+
max_concurrent_runs: 10
59
+
tag_concurrency_limits:
60
+
- key: "database"
61
+
value: "redshift"
62
+
limit: 5
63
+
```
64
+
65
+
| Property | Description |
66
+
| --- | --- |
67
+
| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. <ul><li>**Default** - `10` (20 minutes)</li><li>**Maximum** - `500` <a href="/todo.md">(Hybrid)</a>, `50` <a href="/todo.md">(Serverless)</a></li></ul> |
68
+
| run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags. <ul><li>**Defaults** - `[]`</li></ul> Each list item may have the following properties: <ul><li>`key`</li><li>`value`</li><ul><li>If defined, the `limit` is applied only to the `key-value` pair.</li><li>If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.</li><li>If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.</li></ul><li>`limit`</li></ul> |
69
+
70
+
### Run monitoring (run_monitoring)
71
+
72
+
The `run_monitoring` settings allow you to define how long Dagster+ should wait for runs to start before making them as failed, or to terminate before marking them as canceled.
73
+
74
+
```yaml
75
+
run_monitoring:
76
+
start_timeout_seconds: 1200
77
+
cancel_timeout_seconds: 1200
78
+
max_runtime_seconds: 7200
79
+
```
80
+
81
+
| Property | Description |
82
+
| --- | --- |
83
+
| run_monitoring.start_timeout_seconds | The number of seconds that Dagster+ will wait after a run is launched for the process or container to start executing. After the timeout, the run will fail. This prevents runs from hanging in `STARTING` indefinitely when the process or container doesn't start. <ul><li>**Default** - `1200` (20 minutes)</li></ul> |
84
+
| run_monitoring.cancel_timeout_seconds | The number of seconds that Dagster+ will wait after a run termination is initiated for the process or container to terminate. After the timeout, the run will move into a `CANCELED` state. This prevents runs from hanging in `CANCELING` indefinitely when the process or container doesn't terminate cleanly. <ul><li>**Default** - `1200` (20 minutes)</li></ul> |
85
+
| run_monitoring.max_runtime_seconds | The number of seconds that Dagster+ will wait after a run is moved into a `STARTED` state for the run to complete. After the timeout, the run will be terminated and moved into a `FAILURE` state. This prevents runs from hanging in `STARTED` indefinitely if the process is hanging. <ul><li>**Default** - `No limit`</li></ul> |
86
+
87
+
### Run retries (run_retries)
88
+
89
+
The `run_retries` settings allow you to define how Dagster+ handles retrying failed runs in the deployment.
90
+
91
+
```yaml
92
+
run_retries:
93
+
max_retries: 0
94
+
```
95
+
96
+
| Property | Descripton |
97
+
| --- | --- |
98
+
| run_retries.max_retries | The maximum number of times Dagster+ should attempt to retry a failed run. Dagster+ will use the default if this setting is undefined. <ul><li>**Default** - `0`</li></ul> |
99
+
| run_retries.retry_on_asset_or_op_failure | Whether to retry runs that failed due to assets or ops in the run failing. Set this to false if you only want to retry failures that occur due to the run worker crashing or unexpectedly terminating, and instead rely on op or asset-level retry policies to retry assert or op failures. Setting this field to false will only change retry behavior for runs on dagster version 1.6.7 or greater. <ul><li>**Default** - `0`</li></ul> |
The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/todo.md).
105
+
106
+
```yaml
107
+
sso_default_role: EDITOR
108
+
```
109
+
110
+
| Property | Descripton |
111
+
| --- | --- |
112
+
| sso_default_role | If SAML SSO is enabled, this is the default role that will be assigned to Dagster+ users for this deployment. If SAML SSO is not enabled, this setting is ignored. <ul><li>**Default** - `Viewer`</li></ul> |
113
+
114
+
### Non-isolated runs
115
+
116
+
{/* /dagster-plus/deployment/serverless */}
117
+
Configure [non-isolated runs](/todo.md) on your deployment.
118
+
119
+
```yaml
120
+
non_isolated_runs:
121
+
enabled: True
122
+
max_concurrent_non_isolated_runs: 1
123
+
```
124
+
125
+
| Property | Descripton |
126
+
| --- | --- |
127
+
| enabled | If enabled, the `Isolate run environment` checkbox will appear in the Launchpad. <ul><li>**Default** - `true`</li></ul> |
128
+
| max_concurrent_non_isolated_runs | A limit for how many non-isolated runs to launch at once. Once this limit is reached, the checkbox will be greyed out and all runs will be isolated. This helps to avoid running out of RAM on the code location server. <ul><li>**Default** - `1`</li></ul> |
Copy file name to clipboardExpand all lines: docs/docs-beta/docs/guides/build/create-asset-pipelines/metadata.md
+13
Original file line number
Diff line number
Diff line change
@@ -120,6 +120,19 @@ import Experimental from '../../../partials/\_Experimental.md';
120
120
121
121
<Experimental />
122
122
123
+
import Preview from '../../../partials/\_Preview.md';
124
+
125
+
<Preview />
126
+
127
+
import Beta from '../../../partials/\_Beta.md';
128
+
129
+
<Beta />
130
+
131
+
import Superseded from '../../../partials/\_Superseded.md';
132
+
133
+
<Superseded />
134
+
135
+
123
136
To link assets with their source code, you can attach a **code reference**. Code references are a type of metadata that allow you to easily view those assets' source code from the Dagster UI, both in local development and in production.
This feature is considered in a preview stage and is under active development. It can change significantly, or be removed completely. It is not considered ready for production use.
0 commit comments