Add concurrency support on Minimap API #151
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
SakuraIsayeki
commented
Jul 29, 2023
SakuraIsayeki
commented
- Added concurrency support for the minimap API by delegating rendering to forked processes.
- Updated Dockerfile to optimize build
This commit adds support for parallel minimap rendering. It introduces changes to the `app.py` and `render.py` files. In `app.py`, a new function `on_job_finished()` is added to handle the completion of render jobs. Additionally, the work folder is ensured to exist, and if not, it is created. In `render.py`, several changes are made: - The function `on_job_finished()` now takes only the job ID as a parameter and removes both the "wowsreplay" and "mp4" files associated with the job. - A new function `get_filepath()` is introduced to generate file paths based on the job ID and file extension. - The existing function `start_render()` is modified to start a minimap render job on another process. It opens the replay file, retrieves replay information, initializes a renderer object, and starts rendering with specified settings. - The function `render_replay()` now reads the content of the uploaded replay file asynchronously. It saves the replay file to the work folder using binary mode. Then, it starts a separate process for rendering by calling `start_render()`. Finally, it adds a task for handling job completion using `on_job_finished()`. These changes enable parallel processing of minimap rendering jobs, improving performance and efficiency.
- Updated the Dockerfile.dev to use `python:3.10-alpine` as the base image for builder stage. - Added installation of `py3-pip`, `git`, `gcc`, `libc-dev`, `gfortran`, `g++`, `openblas-dev`, and `ffmpeg` packages in the app image. - Removed the creation of a user group and user in the app image. This commit switches the Minimap API Dockerfile image to an Alpine-based one, reducing its size and improving efficiency. It also installs necessary packages for building and running the application.
This commit adds a new query parameter `waitForCompletion` to the `RenderMinimap` method in the `ReplayController`. When set to true, it waits for the minimap rendering job to complete before returning a response. If set to false (default), it enqueues the job and returns a 202 status code. The `RenderMinimap` method now also includes an additional response code in its XML documentation, indicating that the minimap was rendered successfully (200). In addition, this commit modifies the LINQ query in the `ListPostIdsAsync` method of the `PostService` class. The post IDs are now ordered by their creation date before being returned as an async enumerable.
This commit updates the project version to 0.17.0 and removes the unused properties `DockerDefaultTargetOS`, `DockerComposeProjectPath`, `PlatformTarget` from the csproj file. These changes ensure consistency and improve code readability.
This commit adds a new Python script `trigger_rendering_minimaps.py` that allows triggering the rendering of minimaps for posts in an API. The script takes command-line arguments such as the API host, JWT auth token, concurrency level, and number of posts to skip. It uses a `MinimapRenderClient` class to fetch post IDs from the API, handle errors, and render minimaps by making PATCH requests to the API. The script also includes functions for validating URIs and JWT tokens. The main entry point function `main()` sets up argument parsing, logging configuration, and initializes an instance of `MinimapRenderClient`. It fetches all post IDs from the API, skips specified number of posts if needed, and then renders minimaps for each post using parallel requests with a configurable concurrency level.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.