Skip to content

Commit

Permalink
Merge pull request #88 from Squad-Wiki/pipeline-2.0
Browse files Browse the repository at this point in the history
Pipeline 2.0
  • Loading branch information
Shanomac99 committed Dec 20, 2023
2 parents a201352 + 6497640 commit a394124
Show file tree
Hide file tree
Showing 32 changed files with 2,710,589 additions and 651,220 deletions.
57 changes: 36 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
# Squad Wiki Pipeline: Map Data
<div align="center">
<img src="doc/images/Squad_Wiki_Logo.png" alt="Squad Wiki Pipeline Logo" width="500"/>

#### Squad Wiki Pipeline: Map Data

![GitHub release (with filter)](https://img.shields.io/github/v/release/Squad-Wiki/squad-wiki-pipeline-map-data)
![GitHub](https://img.shields.io/github/license/Squad-Wiki/squad-wiki-pipeline-map-data)
![GitHub issues](https://img.shields.io/github/issues-raw/Squad-Wiki/squad-wiki-pipeline-map-data)

![Discord](https://img.shields.io/discord/553984341143846912?style=flat&logo=discord&logoColor=FFC200)
![GitHub Repo stars](https://img.shields.io/github/stars/Squad-Wiki/squad-wiki-pipeline-map-data)

</div>

## About

The **Squad Wiki Pipeline** (in short: "pipeline") allows the extraction of game data for the tactical first-person-shooter [Squad](http://store.steampowered.com/app/393380/) by [Offworld Industries](https://www.offworldindustries.com/). The data is extracted from the [Squad SDK](https://squad.gamepedia.com/Squad_SDK) (not the game files!) and then uploaded to the [Squad Wiki](https://squad.gamepedia.com/Squad_Wiki). All of this is performed by the pipeline semi-automatically.

Expand All @@ -10,24 +24,22 @@ This pipeline currently only extracts **map layer** and **map vehicle assets** d

The pipeline consists of four components:

1. Export the game data from the [Squad SDK](https://squad.gamepedia.com/Squad_SDK)
2. Convert this raw export into [JSON format](https://en.wikipedia.org/wiki/JSON) for easier processing
3. Convert SDK labels into names used on the wiki
4. Upload the data to Data pages on the wiki

Most of these steps create interim output files which are used as input files for the following step.
1. Export the game data from the [Squad SDK](https://squad.gamepedia.com/Squad_SDK).
2. Convert SDK labels into names used on the wiki.
3. Upload the data to Data pages on the wiki.
4. Have the wiki display the data.

**Step 1** exports the raw data from the Squad SDK.
The last to steps create interim output files which are used as input files for the following step.

**Step 2** creates a file format that is easier to process the data in the following steps.
**Step 1** exports the data from the Squad SDK into a JSON format.

Before the SDK export can be uploaded to the wiki, naming convertions must be performed because the internal naming in the Squad SDK does often not match with the naming on the Squad Wiki - this is done in **step 3**.
Before the SDK export can be uploaded to the wiki, naming convertions must be performed because the internal naming in the Squad SDK does often not match with the naming on the Squad Wiki - this is done in **step 2**.

With **step 4** the data is uploaded to the wiki with the goal of making the data accessible in a [Cargo database](https://www.mediawiki.org/wiki/Extension:Cargo). However, there is no [MediaWiki API](https://www.mediawiki.org/wiki/API:Main_page) to access the Cargo tables directly, therefore we have to upload the data into interim Data pages - from there, using Cargo commands, the data goes finally into the Cargo tables.
With **step 3** the data is uploaded to the wiki with the goal of making the data accessible in a [Cargo database](https://www.mediawiki.org/wiki/Extension:Cargo). However, there is no [MediaWiki API](https://www.mediawiki.org/wiki/API:Main_page) to access the Cargo tables directly, therefore we have to upload the data into interim Data pages - from there, using Cargo commands, the data goes finally into the Cargo tables.

Cargo tables allow the automatic re-use of its data on many wiki pages. Therefore, we only have to update the Cargo data and all wiki pages are automatically updated with the latest data. The data is pulled from the Cargo tables using [Scribunto](https://www.mediawiki.org/wiki/Extension:Scribunto). Scribunto allows powerful programming (using the [programming language Lua](<https://en.wikipedia.org/wiki/Lua_(programming_language)>)) to manipulate and visualize the data onto wiki pages - everything done automatically.

If you want to learn more about the wiki-side programming and maintenance, see our [Squad Wiki Cargo Maintenence documentation](https://squad.gamepedia.com/Squad_Wiki_Editorial/Cargo_Maintenance). Below documentation focuses exclusively on the first four steps that make up the pipeline.
If you want to learn more about the wiki-side programming and maintenance, see our [Squad Wiki Cargo Maintenence documentation](https://squad.gamepedia.com/Squad_Wiki_Editorial/Cargo_Maintenance). Below documentation focuses exclusively on the first three steps that make up the pipeline.

## Latest output files

Expand All @@ -39,19 +51,23 @@ Documentation on how to install this pipeline can be read [here](doc/installatio

## Configuration

Documentation on how to configure this pipeline can be read [here](doc/configuration.md)
Documentation on how to configure this pipeline for the SDK can be read [here](doc/sdkConfiguration.md)

Documentation on how to configure this pipeline for the wiki can be read [here](doc/wikiConfiguration.md)

## Usage

Documentation on how to use this pipeline can be read [here](doc/usage.md)
Documentation on how to use this pipeline for the SDK data extraction can be read [here](doc/sdkUsage.md)

Documentation on how to use this pipeline for the wiki upload can be read [here](doc/wikiUsage.md)

Documentation about the output JSON file can be read [here](doc/json.md)

## More documentation
## Wiki Side Documentation

You can find more documentation for the usage of the pipeline [here](https://squad.gamepedia.com/Squad_Wiki_Editorial/Cargo_Maintenance) - this documentation covers the practical operation of the pipline on the wiki-side.

## Project status and roadmap
## Project Status and Roadmap

We have developed this pipeline in 2020 for the maintenance of the [Squad Wiki](https://squad.gamepedia.com/Squad_Wiki). We are utlizing the pipeline every time a new version of Squad is released. We are constantly improving the pipeline and make more and more use of the Cargo data on the wiki.

Expand All @@ -67,14 +83,13 @@ If you want to contribute to the pipeline for the [Squad Wiki](https://squad.gam

The pipeline was created and is being operated by the team in the [Squad Wiki Editorial](https://squad.gamepedia.com/Squad_Wiki_Editorial) for the [Squad Wiki](https://squad.gamepedia.com/Squad_Wiki). You can best reach us on our own [Discord](https://discord.gg/Y8vgeJ2).

## Authors and acknowledgment
## Authors and Acknowledgment

Listed by Discord user names:

- **Shanomac99**#9407 - programming and operation
- **Usgu**#2705 - concept, coordination and documentation
- **werewolfboy13**#0666 - contributor and documentation
- **[RIP] Rosarch**#1541 - contributor
- **shanomac99** - programming and operation
- **usgu** - concept, coordination and documentation
- **werewolfboy13** - contributor and documentation

This project would not have been possible without the invaluable help and support from [Offworld Industries](https://www.offworldindustries.com/), especially the developers **Virus.exe** and **FuzzHead**.

Expand Down
Binary file added SDK/LightingLayers.uasset
Binary file not shown.
Binary file added SDK/LightingLayersStructure.uasset
Binary file not shown.
Binary file added SDK/MapGrabAsset4_0.uasset
Binary file not shown.
Binary file removed SDK/MapGrabAssets.uasset
Binary file not shown.
Loading

0 comments on commit a394124

Please sign in to comment.