Skip to content

Latest commit

 

History

History
313 lines (231 loc) · 14.3 KB

README.md

File metadata and controls

313 lines (231 loc) · 14.3 KB
Raw

herofig_revised

Build status Netlify Status

HARP is a sample editor that allows for hosted, asynchronous, remote processing of audio with machine learning. HARP works by routing audio through Gradio endpoints. Since Gradio applications can be hosted locally or in the cloud (e.g. with HuggingFace Spaces), HARP lets users of Digital Audio Workstations (DAWs) capable of connecting with external sample editors (e.g. REAPER, Logic Pro X, or Ableton Live) access large state-of-the-art models using cloud-based services, without breaking the within-DAW workflow.

For a more formal introduction, please see our NeurIPS paper presenting an earlier version of HARP.

Table of Contents

Installation

HARP has been tested on the following operating systems:

OS macOS macOS Windows Ubuntu
Version(s) 13.0, 13.4, 14.2.1 10.15 10, 11 22.04

MacOS

  • Download the macOS ZIP file for HARP from the releases page.

  • Unzip and double click on the DMG file. This will open the window shown below.

HARP_DMG

  • Drag HARP.app to the Applications/ folder to install HARP.

  • Do NOT run harp from the installation window above. It will not run correctly if you do so.

Windows

  • Download the Windows ZIP file for HARP from the releases page.

  • Extract the contents of the ZIP file and move the directory containing HARP.exe to a location of your choice, e.g. C:\Program Files.

Linux

  • Download the Linux ZIP file for HARP from the releases page.

  • Extract the contents of the ZIP file and move the directory containing HARP to a location of your choice, e.g. /usr/local/bin/.

Setup

Standalone

To work as a standalone application, you just need to open HARP and start using it.

Opening HARP

MacOS

Run HARP.app to start the application.

Windows

Run HARP.exe to start the application.

Linux

Run HARP to start the application.

REAPER

To set up HARP for use from within Reaper, do the following.

Setting Up HARP

  • Choose REAPER > Preferences on the file menu for MacOS. OR hit keyboard command control+p on Windows

  • Scroll down to External Editors and click Add.

  • Click Browse to the right of the Primary Editor field.

  • Navigate to your HARP installation (e.g. HARP.app) and select "OK".

REAPER_Setup

Opening HARP

  • (Optional) Right click the audio for the track you want to process and select Render items as new take.

  • Right click the audio and select Open items in editor > Open items in 'HARP.app'.

REAPER_Opening

Logic Pro X

To set up HARP for use from within Logic Pro, do the following.

Setting Up HARP

  • Set HARP.app as an external sample editor following this guide.

Opening HARP

  • (Optional) Right click the audio for the track you want to process and select Bounce in Place.

  • Select any audio region and press Shift+W to open the corresponding audio file in HARP.

Acoustica Mixcraft

To set up HARP for use from within Mixcraft, do the following.

Setting Up HARP

  • Set HARP.exe as an external wave editor following "Configuring an External Wave Editor" on this manual page.

Opening HARP

  • Right click the audio for the track you want to process and select Edit In External Editor..., or via Sound > Edit In External Editor....

  • On the dialog that appears, choose Edit A Copy Of The Sound, and click Edit to open HARP.

  • After editing, close HARP and select Done in the Mixcraft dialog box to update the track.

Usage

HARP can be used to apply deep learning models to your audio either as a stand-alone or within any DAW (e.g. Logic Pro) that supports external sample editors.

If you use it stand-alone, just load a file, load a model and apply the model to the audio file. When you're happy with the result, save the output.

In a DAW, you select the exceprt you want to process, open it in HARP, process it, and select Save from the File menu in HARP. This will return the processed file back to the DAW.

Warning!

HARP is a destructive file editor. When you select Save, HARP overwrites the existing audio. After recording or loading audio into a track within your preferred DAW, it is recommended that you bounce-in-place (on Logic) or Render items as new take (on Reaper) the audio before processing it with HARP. In this way, you will avoid overwriting the original audio file and will be able to undo any changes introuced by HARP. Alternatively, overwriting can be circumvented by using the Save As functionality from the File menu in HARP.

Processing just a portion of a track

If you would like to process only an excerpt of a track, first trim the exceprt to the portion you want to process. Then, perform a bounce-in-place of the excerpt. This will make a new file that contains only the audio you want to process with HARP. Then, open the new file in HARP.

Models

While any algorithm or deep learning model can be deployed to HARP using the PyHARP API, at present, the following models have been made available:

Workflow

  • After opening HARP as an external sample editor or standalone application, the following window will appear.
Screenshot 2024-06-04 at 4 29 03 AM (1)

  • Select or type the Gradio endpoint of an available HARP-ready model (e.g. "hugggof/harmonic_percussive") in the field at the top of the application.

    • This will populate the window with controls for the model.

    • Loading can take some time if the HuggingFace Space is asleep.

Screenshot 2024-06-04 at 4 31 05 AM

  • Adjust the model controls to your liking and click process.

  • The resulting audio can be played by pressing the space bar or by clicking Play/Stop.

  • You can save the audio at any time using the Save and Save As features located in the File menu.

  • Any changes made in HARP will be automatically reflected in your DAW.

Contributing

To make a model that you can use in HARP, see Adding Models with PyHARP. To modify the HARP app, see Building Harp.

Version Compatibility

The current versions of HARP and PyHARP are shown below. They are fully compatible.

HARP PyHARP
2.0.0 0.2.0
1.3.0 0.1.1
1.2.0 0.1.0

Adding Models with PyHARP

ReadMe Card

We provide PyHARP, a lightweight API to build HARP-compatible Gradio apps with optional interactive controls. PyHARP allows machine learning researchers to create DAW-friendly user interfaces for virtually any audio processing code using a minimal Python wrapper.

Building HARP

HARP can be built from scratch with the following steps:

1. Clone Repository

git clone --recurse-submodules https://github.com/TEAMuP-dev/HARP

2. Enter Project

cd HARP/

3. Configure

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug

ARM vs. x86 MacOS

The OSX architecture for the build can be specified explicitly by setting CMAKE_OSX_ARCHITECTURES to either arm64 or x86_64:

cmake .. -DCMAKE_OSX_ARCHITECTURES=x86_64

Linux

Ensure your system satisfies all JUCE dependencies.

4. Build

MacOS/Linux

make -j <NUM_PROCESSORS>

Windows

cmake --build . --config Debug -j <NUM_PROCESSORS>

Debugging

To debug your HARP build in Visual Studio Code it is helpful to do the following.

Visual Studio Code

  1. Download Visual Studio Code.
  2. Install the C/C++ extension from Microsoft.
  3. Open the Run and Debug tab in VS Code and click create a launch.json file using CMake Debugger.
  4. Create a configuration to attach to the process (see the following example code to be placed in launch.json).
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(lldb) Standalone",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/build/HARP_artefacts/Debug/HARP.app", // macOS
            // "program": "${workspaceFolder}/build/HARP_artefacts/Debug/HARP.exe", // Windows
            // "program": "${workspaceFolder}/build/HARP_artefacts/Debug/HARP", // Linux
            "args": ["../test.wav"],
            "cwd": "${fileDirname}",
            "MIMode": "lldb" // macOS
        }
    ]
}
  1. Build the plugin using the flag -DCMAKE_BUILD_TYPE=Debug.
  2. Add break points and run the debugger.

Distribution

MacOS

Codesigning and packaging for distribution is done through the script located at packaging/package.sh. You'll need to set up a developer account with Apple and create a certificate in order to sign the plugin. For more information on codesigning and notarization for macOS, please refer to the pamplejuce template.

The script requires the following variables to be passed:

# Retrieve values from either environment variables or command-line arguments
DEV_ID_APPLICATION # Developer ID Application certificate
ARTIFACTS_PATH # should be packaging/dmg/HARP.app
PROJECT_NAME # "HARP"
PRODUCT_NAME # "HARP"
NOTARIZATION_USERNAME # Apple ID
NOTARIZATION_PASSWORD # App-specific password for notarization
TEAM_ID # Team ID for notarization

Usage:

./HARP/packaging/package.sh <DEV_ID_APPLICATION> <ARTIFACTS_PATH> <PROJECT_NAME> <PRODUCT_NAME> <NOTARIZATION_USERNAME> <NOTARIZATION_PASSWORD> <TEAM_ID>

After running package.sh, you should have a signed and notarized dmg file in the packaging/ directory.

Citing

If you use HARP in your research, please cite our NeurIPS paper:

@inproceedings{garcia2023harp,
    title     = {{HARP}: Bringing Deep Learning to the DAW with Hosted, Asynchronous, Remote Processing},
    author    = {Garcia, Hugo Flores and O’Reilly, Patrick and Aguilar, Aldo and Pardo, Bryan and Benetatos, Christodoulos and Duan, Zhiyao},
    year      = 2023,
    booktitle = {NeurIPS Workshop on Machine Learning for Creativity and Design}
}