Skip to content

Commit

Permalink
[#436][FOLLOWUP] docs: Add documents for localStorageManagerClass and…
Browse files Browse the repository at this point in the history 
… MultiPartLocalStorageManager (#2266)

### What changes were proposed in this pull request?

Add config localStorageManagerClass description to introduce the default value and MultiPartLocalStorageManager.

### Why are the changes needed?

Add missing docs.

Fix: #436

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

No need.
  • Loading branch information
@maobaolong
maobaolong authored Dec 1, 2024
1 parent ac89c19 commit db611f4
Showing 1 changed file with 8 additions and 7 deletions.
15 changes: 8 additions & 7 deletions docs/server_guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,13 +124,14 @@ This document will introduce how to deploy Uniffle shuffle servers.
| rss.server.blockIdManagerClass | org.apache.uniffle.server.block.DefaultShuffleBlockIdManager | The block id manager class. It is used to manage block id. |

### Advanced Configurations
| Property Name | Default | Description |
|--------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| rss.server.storageMediaProvider.from.env.key | - | Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths |
| rss.server.decommission.check.interval | 60000 | The interval(ms) to check if all applications have finish when server is decommissioning |
| rss.server.decommission.shutdown | true | Whether shutdown the server after server is decommissioned |
| rss.server.health.checker.script.path | - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true. |
| rss.server.health.checker.script.execute.timeout | 5000 | Timeout for `HealthScriptChecker` execute health script.(ms) |
| Property Name | Default | Description |
|--------------------------------------------------|-------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| rss.server.storageMediaProvider.from.env.key | - | Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths |
| rss.server.decommission.check.interval | 60000 | The interval(ms) to check if all applications have finish when server is decommissioning |
| rss.server.decommission.shutdown | true | Whether shutdown the server after server is decommissioned |
| rss.server.health.checker.script.path | - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true. |
| rss.server.health.checker.script.execute.timeout | 5000 | Timeout for `HealthScriptChecker` execute health script.(ms) |
| rss.server.localStorageManagerClass | org.apache.uniffle.server.storage.LocalStorageManager | The class for implementing local storage management can be set to `org.apache.uniffle.server.storage.MultiPartLocalStorageManager`. This class enables the storage of each partition across multiple disks in multiple parts. Please note that this configuration requires coordination with the client. Therefore, ensure that the client version is equal to or newer than the this version before upgrading the server. |

### Huge Partition Optimization
A huge partition is a common problem for Spark/MR and so on, caused by data skew. And it can cause the shuffle server to become unstable. To solve this, we introduce some mechanisms to limit the writing of huge partitions to avoid affecting regular partitions, and introduce a hard limit config to reject extremely huge partition, more details can be found in [ISSUE-378](https://github.com/apache/incubator-uniffle/issues/378). The basic rules for limiting large partitions are memory usage limits and flushing individual buffers directly to persistent storage.
Expand Down

0 comments on commit db611f4

Please sign in to comment.