Merge branch 'master' into ddc-modernization-and-integration-14116118131799338522

Author: robertpelloni

PROJECT_ANALYSIS.md

@@ -0,0 +1,85 @@
+# Project Analysis and Feature Roadmap
+
+This document provides a comprehensive analysis of the `ddc-stepmania` project, its current state, and a detailed roadmap for future development.
+
+## Current State of the Project
+
+The project is currently in a foundational state. The core architecture is in place, the necessary data has been downloaded, and the codebase has been significantly refactored and modernized. However, the advanced features requested by the user are not yet implemented.
+
+### Key Accomplishments:
+
+*   **Data Acquisition**: All 12 official DDR data packs have been successfully downloaded and extracted into the `data/raw/` directory.
+*   **Project Structure**: The project has been organized into a proper Python package, `ddc_stepmania`, with a clear separation of concerns between data processing, model training, and inference.
+*   **Code Modernization**: The codebase has been migrated from Python 2/TensorFlow 1.x to Python 3/TensorFlow 2.x (using the `compat.v1` module).
+*   **Dependency Resolution**: Numerous bugs related to file paths, data loading, and missing libraries (`pydub`, `ffmpeg`) have been resolved.
+
+### Existing Components:
+
+*   **`ddc_stepmania` package**:
+    *   `dataset`: Scripts for processing raw `.sm` files into JSON and then pickled Python objects.
+    *   `learn`: The core machine learning package, containing the `OnsetNet` and `SymNet` models, as well as the training scripts (`onset_train.py`, `sym_train.py`).
+    *   `infer`: Scripts and a server for using the trained models to generate new stepcharts.
+*   **`ffr-difficulty-model`**: A pre-existing, integrated model for calculating the difficulty of a given stepchart.
+*   **`scripts/`**: A collection of shell scripts that orchestrate the data processing and training pipelines.
+
+## Feature Roadmap
+
+The following is a detailed roadmap for the implementation of the remaining features, in order of priority.
+
+### 1. Advanced Music Analysis
+
+This is the most critical set of features, as it forms the foundation for generating high-quality stepcharts.
+
+*   **BPM and Tempo Change Detection**:
+    *   **Goal**: Accurately determine the BPM of a song, including any variations or tempo changes.
+    *   **Implementation**: Utilize the `librosa` library, specifically the `librosa.beat.beat_track` function, to perform robust beat tracking and tempo estimation.
+    *   **Integration**: The detected BPM and beat information will be stored in the processed data files and used to inform the `Chart` and `BeatCalc` classes.
+
+*   **First and Last Beat Identification**:
+    *   **Goal**: Pinpoint the exact timestamps of the first and last significant beats in the audio.
+    *   **Implementation**: This information will be derived directly from the output of the beat tracking algorithm.
+    *   **Integration**: This will be used to define the time window within which arrows can be placed, preventing them from appearing in silent sections of the song.
+
+*   **Automatic Music Sectioning**:
+    *   **Goal**: Segment a song into meaningful sections (e.g., intro, verse, chorus, bridge, outro).
+    *   **Implementation**: Use a structural segmentation algorithm, such as `librosa.segment.agglomerative`, to identify the major sections of the song.
+    *   **Integration**: The section boundaries will be used to create more musically interesting and varied stepcharts, with different patterns and densities for each section.
+
+### 2. Refined Arrow Placement Logic
+
+This set of features will focus on improving the quality and playability of the generated stepcharts.
+
+*   **Beat-Aligned Arrow Placement**:
+    *   **Goal**: Ensure that all generated arrows are perfectly aligned with the beat grid of the song.
+    *   **Implementation**: Modify the `OnsetNet` to predict onsets that are quantized to the beat grid. This will prevent the creation of off-beat or awkwardly timed arrows.
+    *   **Integration**: The `gen_labels.py` script will be updated to generate beat-aligned labels, and the `OnsetNet` will be retrained on this new data.
+
+*   **Pattern-Based Arrow Selection**:
+    *   **Goal**: Generate arrow patterns that are idiomatic to DDR and avoid random or unplayable sequences.
+    *   **Implementation**: The `SymNet` model will be enhanced to learn common DDR patterns (e.g., crossovers, jacks, streams). This could involve using a more sophisticated model architecture or incorporating a pattern library.
+    *   **Integration**: The `sym_train.py` script will be updated to train the new `SymNet` model, and the inference scripts will be modified to use it.
+
+*   **Playability Enforcement**:
+    *   **Goal**: Implement a post-processing step to clean up the generated charts and enforce playability rules.
+    *   **Implementation**: This will involve creating a set of rules to prevent things like double-steps (placing arrows on both feet at the same time), overly dense patterns, and other unplayable sequences.
+    *   **Integration**: A new script will be added to the `infer` package to perform this post-processing step.
+
+### 3. Difficulty Generation and Rating
+
+This set of features will focus on the automatic generation of charts for all difficulty levels and the assignment of accurate difficulty ratings.
+
+*   **Multi-Difficulty Generation**:
+    *   **Goal**: Automatically generate charts for all standard DDR difficulties (Beginner, Basic, Difficult, Expert, Challenge).
+    *   **Implementation**: This will likely involve training separate models for each difficulty level, or creating a single, conditional model that takes the desired difficulty as an input.
+    *   **Integration**: The training and inference pipelines will be updated to support the generation of multiple difficulty levels.
+
+*   **Difficulty Rating Integration**:
+    *   **Goal**: Use the `ffr-difficulty-model` to analyze the generated charts and assign them an accurate difficulty rating.
+    *   **Implementation**: The API of the `ffr-difficulty-model` will be used to analyze the generated `.sm` files.
+    *   **Integration**: The inference pipeline will be updated to include a final step where the difficulty rating is calculated and added to the generated chart's metadata.
+
+### 4. Project Finalization
+
+*   **Packaging**: The entire project will be structured into a clean, distributable Python package, so that it can be easily installed and used by others.
+*   **Documentation**: Comprehensive documentation will be created, including an updated `README.md`, docstrings for all key functions and classes, and a `CONTRIBUTING.md` file.
+*   **Testing**: A suite of unit and integration tests will be created to ensure the correctness and robustness of the entire system.

convert_json_to_pickle.py

@@ -0,0 +1,50 @@
+import json
+import pickle
+import os
+import sys
+
+def convert_json_to_pickle(file_list_path):
+    """
+    Reads a file list (e.g., train.txt), loads each JSON file listed,
+    and saves it as a pickle file. Creates a new file list with .pkl paths.
+    """
+    if not os.path.exists(file_list_path):
+        print(f"Error: File list not found at {file_list_path}")
+        return
+
+    base_dir = os.path.dirname(file_list_path)
+    file_list_name = os.path.basename(file_list_path)
+    new_file_list_name = file_list_name.replace('.txt', '_pkl.txt')
+    new_file_list_path = os.path.join(base_dir, new_file_list_name)
+
+    print(f"Converting files listed in {file_list_path}...")
+    print(f"New file list will be saved to {new_file_list_path}")
+
+    with open(file_list_path, 'r') as f_in, open(new_file_list_path, 'w') as f_out:
+        for json_path_line in f_in:
+            json_path = json_path_line.strip()
+            if not json_path:
+                continue
+
+            pkl_path = os.path.splitext(json_path)[0] + '.pkl'
+
+            try:
+                with open(json_path, 'r') as f_json:
+                    data = json.load(f_json)
+
+                with open(pkl_path, 'wb') as f_pkl:
+                    pickle.dump(data, f_pkl)
+
+                f_out.write(pkl_path + '\n')
+            except Exception as e:
+                print(f"Could not process {json_path}: {e}")
+
+    print("Conversion complete.")
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print("Usage: python convert_json_to_pickle.py <path_to_file_list.txt>")
+        sys.exit(1)
+    
+    file_list = sys.argv[1]
+    convert_json_to_pickle(file_list)

create_data_splits.py

@@ -0,0 +1,46 @@
+import random
+import os
+
+def split_data(file_list_path, train_ratio=0.8, valid_ratio=0.1):
+    """
+    Splits a list of file paths into training, validation, and test sets.
+    """
+    if not os.path.exists(file_list_path):
+        print(f"Error: File list not found at {file_list_path}")
+        return
+
+    with open(file_list_path, 'r') as f:
+        all_files = [line.strip() for line in f if line.strip()]
+
+    if not all_files:
+        print("Error: The file list is empty.")
+        return
+
+    random.shuffle(all_files)
+
+    num_files = len(all_files)
+    num_train = int(num_files * train_ratio)
+    num_valid = int(num_files * valid_ratio)
+    
+    train_files = all_files[:num_train]
+    valid_files = all_files[num_train:num_train + num_valid]
+    test_files = all_files[num_train + num_valid:]
+
+    base_dir = os.path.dirname(file_list_path)
+
+    with open(os.path.join(base_dir, 'train_files.txt'), 'w') as f:
+        f.write('\n'.join(train_files))
+        
+    with open(os.path.join(base_dir, 'valid_files.txt'), 'w') as f:
+        f.write('\n'.join(valid_files))
+        
+    with open(os.path.join(base_dir, 'test_files.txt'), 'w') as f:
+        f.write('\n'.join(test_files))
+
+    print(f"Data split complete:")
+    print(f"  Training set: {len(train_files)} files")
+    print(f"  Validation set: {len(valid_files)} files")
+    print(f"  Test set: {len(test_files)} files")
+
+if __name__ == "__main__":
+    split_data('all_sm_files.txt')

create_splits.py

@@ -0,0 +1,35 @@
+import os
+import random
+
+def create_dataset_splits(data_dir, output_dir, train_split=0.8, valid_split=0.1):
+    """
+    Reads .pkl files from a directory, shuffles them, and splits them into
+    training, validation, and testing sets.
+    """
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
+
+    all_files = [os.path.join(data_dir, f) for f in os.listdir(data_dir) if f.endswith('.pkl')]
+    random.shuffle(all_files)
+
+    train_end = int(len(all_files) * train_split)
+    valid_end = int(len(all_files) * (train_split + valid_split))
+
+    train_files = all_files[:train_end]
+    valid_files = all_files[train_end:valid_end]
+    test_files = all_files[valid_end:]
+
+    with open(os.path.join(output_dir, 'train.txt'), 'w') as f:
+        for item in train_files:
+            f.write("%s\n" % item)
+
+    with open(os.path.join(output_dir, 'valid.txt'), 'w') as f:
+        for item in valid_files:
+            f.write("%s\n" % item)
+
+    with open(os.path.join(output_dir, 'test.txt'), 'w') as f:
+        for item in test_files:
+            f.write("%s\n" % item)
+
+if __name__ == '__main__':
+    create_dataset_splits('train.json', '.')

ddc_stepmania.egg-info/PKG-INFO

@@ -0,0 +1,5 @@
+Metadata-Version: 2.4
+Name: ddc_stepmania
+Version: 0.1
+License-File: LICENSE
+Dynamic: license-file

ddc_stepmania.egg-info/SOURCES.txt

@@ -0,0 +1,7 @@
+LICENSE
+README.md
+setup.py
+ddc_stepmania.egg-info/PKG-INFO
+ddc_stepmania.egg-info/SOURCES.txt
+ddc_stepmania.egg-info/dependency_links.txt
+ddc_stepmania.egg-info/top_level.txt

ddc_stepmania.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

ddc_stepmania.egg-info/top_level.txt

@@ -0,0 +1 @@
+

ddc_stepmania/dataset/analyze_json.py

@@ -0,0 +1,111 @@
+if __name__ == '__main__':
+    import argparse
+    from collections import Counter, defaultdict
+    import json
+    import os
+    from util import get_subdirs
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('dataset_fps', type=str, nargs='+', help='List of dataset filepaths to analyze')
+    parser.add_argument('--diff', type=str, help='If provided, only analyze charts of this difficulty')
+
+    parser.set_defaults(
+        diff='')
+
+    args = parser.parse_args()
+
+    json_fps = []
+    for dataset_fp in args.dataset_fps:
+        with open(dataset_fp, 'r') as f:
+            json_fps += f.read().splitlines()
+
+    chart_types = Counter()
+    chart_diff_coarse = Counter()
+    chart_feet = Counter()
+    chart_coarse_to_stream = {}
+    chart_freetexts = Counter()
+    beat_phases = Counter()
+    vocab = Counter()
+    songs_time_annotated = 0.0
+    charts_time_annotated = 0.0
+    stream_total = 0.0
+    feet_total = 0.0
+    arrows_total = 0
+    chart_coarse_to_superset = defaultdict(list)
+    for json_fp in json_fps:
+        with open(json_fp, 'r') as f:
+            song_meta = json.loads(f.read())
+
+        max_time_annotated = -1.0
+        coarse_to_beats = defaultdict(set)
+        for chart_meta in song_meta['charts']:
+            if args.diff and chart_meta['difficulty_coarse'] != args.diff:
+                continue
+
+            coarse = chart_meta['difficulty_coarse']
+            feet = chart_meta['difficulty_fine']
+            feet_total += feet
+
+            chart_types[chart_meta['type']] += 1
+            chart_diff_coarse[coarse] += 1
+            chart_feet[feet] += 1
+            chart_freetexts[chart_meta['desc_or_author']] += 1
+
+            num_arrows = 0
+            for _, beat, time, arrow in chart_meta['notes']:
+                beat_phase = beat - int(beat)
+                beat_phase = int(beat_phase * 100.0) / 100.0
+                beat_phases[beat_phase] += 1
+                vocab[arrow] += 1
+                if arrow != '0' * len(arrow):
+                    num_arrows += 1
+                coarse_to_beats[coarse].add(beat)
+            arrows_total += num_arrows
+
+            chart_time_annotated = chart_meta['notes'][-1][2] - chart_meta['notes'][0][2]
+            if chart_time_annotated > max_time_annotated:
+                max_time_annotated = chart_time_annotated
+            charts_time_annotated += chart_time_annotated
+
+            stream = num_arrows / chart_time_annotated
+            stream_total += stream
+            if feet not in chart_coarse_to_stream:
+                chart_coarse_to_stream[coarse] = []
+            chart_coarse_to_stream[coarse].append(stream)
+
+        songs_time_annotated += max_time_annotated
+
+        coarses = ['Beginner', 'Easy', 'Medium', 'Hard', 'Challenge']
+        for i, coarse in enumerate(coarses):
+            for coarse_next in coarses:
+                beats = coarse_to_beats[coarse]
+                beats_next = coarse_to_beats[coarse_next]
+                chart_coarse_to_superset[(coarse, coarse_next)].append(len(beats & beats_next) / float(len(beats)))
+
+    chart_coarse_to_stream = {k: sum(l) / len(l) for k, l in chart_coarse_to_stream.items()}
+    chart_coarse_to_superset = {k: (reduce(lambda x, y: x + y, l) / len(l)) for k, l in chart_coarse_to_superset.items()}
+
+    nsongs = len(json_fps)
+    ncharts = sum(chart_feet.values())
+    print ','.join(args.dataset_fps)
+    print 'Num songs: {}'.format(nsongs)
+    print 'Total music annotated (s): {}'.format(songs_time_annotated)
+    print 'Avg song length (s): {}'.format(songs_time_annotated / nsongs)
+
+    print 'Num charts: {}'.format(ncharts)
+    print 'Avg num charts per song: {}'.format(float(ncharts) / nsongs)
+    print 'Total chart time annotated (s): {}'.format(charts_time_annotated)
+    print 'Avg chart length (s): {}'.format(charts_time_annotated / ncharts)
+    print 'Avg chart length (steps): {}'.format(float(arrows_total) / ncharts)
+
+    print 'Chart types: {}'.format(chart_types)
+    print 'Chart coarse difficulties: {}'.format(chart_diff_coarse)
+    print 'Chart feet: {}'.format(chart_feet)
+    print 'Chart coarse avg arrows per second: {}'.format(chart_coarse_to_stream)
+    print 'Chart coarse avg superset: {}'.format(chart_coarse_to_superset)
+    print 'Chart freetext fields: {}'.format(chart_freetexts)
+    print 'Chart vocabulary (size={}): {}'.format(len(vocab), vocab)
+    print 'Beat phases: {}'.format(beat_phases)
+
+    print 'Avg feet: {}'.format(feet_total / ncharts)
+    print 'Avg arrows per second: {}'.format(stream_total / ncharts)