Skip to content

Latest commit

 

History

History
1161 lines (865 loc) · 27 KB

README.md

File metadata and controls

1161 lines (865 loc) · 27 KB
Raw

goofi-pipe

Generative Organic Oscillation Feedback Isomorphism Pipeline

Installation

If you only want to run goofi-pipe and not edit any of the code, make sure you activated the desired Python environment with Python>=3.9 and run the following commands in your terminal:

pip install goofi # install goofi-pipe
goofi-pipe # start the application

Note

On some platforms (specifically Linux and Mac) it might be necessary to install the liblsl package for some of goofi-pipe's features (everything related to LSL streams). Follow the instructions provided here, or simply install it via

conda install -c conda-forge liblsl

Development

Follow these steps if you want to adapt the code of existing nodes, or create custom new nodes. In your terminal, make sure you activated the desired Python environment with Python>=3.9, and that you are in the directory where you want to install goofi-pipe. Then, run the following commands:

git clone [email protected]:PhilippThoelke/goofi-pipe.git # download the repository
cd goofi-pipe # navigate into the repository
pip install -e . # install goofi-pipe in development mode
goofi-pipe # start the application to make sure the installation was successful

Basic Usage

Accessing the Node Menu

To access the node menu, simply double-click anywhere within the application window or press the 'Tab' key. The node menu allows you to add various functionalities to your pipeline. Nodes are categorized for easy access, but if you're looking for something specific, the search bar at the top is a handy tool.

Common Parameters and Metadata

Common Parameters

Common Parameters: All nodes within goofi have a set of common parameters. These settings consistently dictate how the node operates within the pipeline.

  • AutoTrigger: This option, when enabled, allows the node to be triggered automatically. When disabled, the node is triggered when it receives input.

  • Max_Frequency: This denotes the maximum rate at which computations are set for the node.

Metadata

Metadata: This section conveys essential information passed between nodes. Each output node will be accompanied by its metadata, providing clarity and consistency throughout the workflow.

Here are some conventional components present in the metadata

  • Channel Dictionary: A conventional representation of EEG channels names.

  • Sampling Frequency: The rate at which data samples are measured. It's crucial for maintaining consistent data input and output across various nodes.

  • Shape of the Output: Details the format and structure of the node's output.

Playing with Pre-recorded EEG Signal using LslStream

This image showcases the process of utilizing a pre-recorded EEG signal through the LslStream node. It's crucial to ensure that the Stream Name in the LslStream node matches the stream name in the node receiving the data. This ensures data integrity and accurate signal processing in real-time.

Patch examples

Basic Signal Processing Patch

This patch provides a demonstration of basic EEG signal processing using goofi-pipe.

  1. EegRecording: This is the starting point where the EEG data originates.

  2. LslClient: The LslClient node retrieves the EEG data from EegRecording. Here, the visual representation of the EEG data being streamed in real-time is depicted. By default, the multiple lines in the plot correspond to the different EEG channels.

  3. Buffer: This node holds the buffered EEG data.

  4. Psd: Power Spectral Density (PSD) is a technique to measure a signal's power content versus frequency. In this node, the raw EEG data is transformed to exhibit its power distribution across distinct frequency bands.

  5. Math: This node is employed to execute mathematical operations on the data. In this context, it's rescaling the values to ensure a harmonious dynamic range between 0 and 1, which is ideal for image representation. The resultant data is then visualized as an image.

One of the user-friendly features of goofi-pipe is the capability to toggle between different visualizations. By 'Ctrl+clicking' on any plot within a node, you can effortlessly switch between a line plot and an image representation, offering flexibility in data analysis.

Sending Power Bands via Open Sound Control (OSC)

Expanding on the basic patch, the advanced additions include:

  • Select: Chooses specific EEG channels.
  • PowerBandEEG: Computes EEG signal power across various frequency bands.
  • ExtendedTable: Prepares data for transmission in a structured format.
  • OscOut: Sends data using the Open-Sound-Control (OSC) protocol.

These nodes elevate data processing and communication capabilities.

Real-Time Connectivity and Spectrogram

This patch highlights:

  • Connectivity: Analyzes relationships between EEG channels, offering selectable methods like wPLI, coherence, PLI, and more.

  • Spectrogram: Created using the PSD node followed by a Buffer, it provides a time-resolved view of the EEG signal's frequency content.

Principal Component Analysis (PCA)

PCA

Using PCA (Principal Component Analysis) allows us to reduce the dimensionality of raw EEG data, while retaining most of the variance. We use the first three components and visualize their trajectory, allowing us to identify patterns in the data over time. The topographical maps show the contrbution of each channel to the first four principal components (PCs).

Realtime Classification

leverage the multimodal framework of goofi, state-of-the-art machine learning classifiers can be built on-the-fly to predict behavior from an array of different sources. Here's a brief walkthrough of three distinct examples:

1. EEG Signal Classification

EEG Signal Classification This patch captures raw EEG signals using the EEGrecording and LslStreammodule. The classifier module allows to capture data from different states indicated by the user from n features, which in the present case are the 64 EEG channels. Some classifiers allow for visualization of feature importance. Here we show a topomap of the distribution of features importances on the scalp. The classifier outputs probability of being in each of the states in the training data. This prediction is smoothed using a buffer for less jiterry results.
Classifier parameters

2. Audio Input Classification

Audio Input Classification The audio input stream captures real-time sound data, which can also be passed through a classifier. Different sonic states can be predicted in realtime.

3. Video Input Classification

Video Input Classification image_classification2 In this example, video frames are extracted using the VideoStream module. Similarly, prediction of labelled visual states can be achieved in realtime. The images show how two states (being on the left or the right side of the image) can be detected using classification

These patches demonstrate the versatility of our framework in handling various types of real-time data streams for classification tasks.

Musical Features using Biotuner

This patch presents a pipeline for processing EEG data to extract musical features:

  • Data flows from the EEG recording through several preprocessing nodes and culminates in the Biotuner node, which specializes in deriving musical attributes from the EEG.

  • Biotuner Node: With its sophisticated algorithms, Biotuner pinpoints harmonic relationships, tension, peaks, and more, essential for music theory analysis.

Delving into the parameters of the Biotuner node:

  • N Peaks: The number of spectral peaks to consider.
  • F Min & F Max: Defines the frequency range for analysis.
  • Precision: Sets the precision in Hz for peak extraction.
  • Peaks Function: Method to compute the peaks, like EMD, fixed band, or harmonic recurrence.
  • N Harm Subharm & N Harm Extended: Configures number of harmonics used in different computations.
  • Delta Lim: Defines the maximal distance between two subharmonics to include in subharmonic tension computation.

For a deeper understanding and advanced configurations, consult the Biotuner repository.

Data Types

To simplify understanding, we've associated specific shapes with data types at the inputs and outputs of nodes:

  • Circles: Represent arrays.
  • Triangles: Represent strings.
  • Squares: Represent tables.

Node Categories

Analysis

Nodes that perform analysis on the data.

View Nodes
 AudioTagging
  • Inputs:
    • audioIn: ARRAY
  • Outputs:
    • tags: STRING
    • probabilities: ARRAY
    • embedding: ARRAY
 Avalanches
  • Inputs:
    • data: ARRAY
  • Outputs:
    • size: ARRAY
    • duration: ARRAY
 Binarize
  • Inputs:
    • data: ARRAY
  • Outputs:
    • bin_data: ARRAY
 Bioelements
  • Inputs:
    • data: ARRAY
  • Outputs:
    • elements: TABLE
 Bioplanets
  • Inputs:
    • peaks: ARRAY
  • Outputs:
    • planets: TABLE
    • top_planets: STRING
 Biorhythms
  • Inputs:
    • tuning: ARRAY
  • Outputs:
    • pulses: ARRAY
    • steps: ARRAY
    • offsets: ARRAY
 Biotuner
  • Inputs:
    • data: ARRAY
  • Outputs:
    • harmsim: ARRAY
    • tenney: ARRAY
    • subharm_tension: ARRAY
    • cons: ARRAY
    • peaks_ratios_tuning: ARRAY
    • harm_tuning: ARRAY
    • peaks: ARRAY
    • amps: ARRAY
    • extended_peaks: ARRAY
    • extended_amps: ARRAY
 CardiacRespiration
  • Inputs:
    • data: ARRAY
  • Outputs:
    • cardiac: ARRAY
 CardioRespiratoryVariability
  • Inputs:
    • data: ARRAY
  • Outputs:
    • MeanNN: ARRAY
    • SDNN: ARRAY
    • SDSD: ARRAY
    • RMSSD: ARRAY
    • pNN50: ARRAY
    • LF: ARRAY
    • HF: ARRAY
    • LF/HF: ARRAY
    • LZC: ARRAY
 Classifier
  • Inputs:
    • data: ARRAY
  • Outputs:
    • probs: ARRAY
    • feature_importances: ARRAY
 Clustering
  • Inputs:
    • matrix: ARRAY
  • Outputs:
    • cluster_labels: ARRAY
    • cluster_centers: ARRAY
 Compass
  • Inputs:
    • north: ARRAY
    • south: ARRAY
    • east: ARRAY
    • west: ARRAY
  • Outputs:
    • angle: ARRAY
 Connectivity
  • Inputs:
    • data: ARRAY
  • Outputs:
    • matrix: ARRAY
 Coord2loc
  • Inputs:
    • latitude: ARRAY
    • longitude: ARRAY
  • Outputs:
    • coord_info: TABLE
 Correlation
  • Inputs:
    • data1: ARRAY
    • data2: ARRAY
  • Outputs:
    • pearson: ARRAY
 DissonanceCurve
  • Inputs:
    • peaks: ARRAY
    • amps: ARRAY
  • Outputs:
    • dissonance_curve: ARRAY
    • tuning: ARRAY
    • avg_dissonance: ARRAY
 EigenDecomposition
  • Inputs:
    • matrix: ARRAY
  • Outputs:
    • eigenvalues: ARRAY
    • eigenvectors: ARRAY
 ERP
  • Inputs:
    • signal: ARRAY
    • trigger: ARRAY
  • Outputs:
    • erp: ARRAY
 FacialExpression
  • Inputs:
    • image: ARRAY
  • Outputs:
    • emotion_probabilities: ARRAY
    • action_units: ARRAY
    • main_emotion: STRING
 Fractality
  • Inputs:
    • data_input: ARRAY
  • Outputs:
    • fractal_dimension: ARRAY
 GraphMetrics
  • Inputs:
    • matrix: ARRAY
  • Outputs:
    • clustering_coefficient: ARRAY
    • characteristic_path_length: ARRAY
    • betweenness_centrality: ARRAY
    • degree_centrality: ARRAY
    • assortativity: ARRAY
    • transitivity: ARRAY
 HarmonicSpectrum
  • Inputs:
    • psd: ARRAY
  • Outputs:
    • harmonic_spectrum: ARRAY
    • max_harmonicity: ARRAY
    • avg_harmonicity: ARRAY
 Img2Txt
  • Inputs:
    • image: ARRAY
  • Outputs:
    • generated_text: STRING
 LempelZiv
  • Inputs:
    • data: ARRAY
  • Outputs:
    • lzc: ARRAY
 PCA
  • Inputs:
    • data: ARRAY
  • Outputs:
    • principal_components: ARRAY
 PoseEstimation
  • Inputs:
    • image: ARRAY
  • Outputs:
    • pose: ARRAY
 PowerBand
  • Inputs:
    • data: ARRAY
  • Outputs:
    • power: ARRAY
 PowerBandEEG
  • Inputs:
    • data: ARRAY
  • Outputs:
    • delta: ARRAY
    • theta: ARRAY
    • alpha: ARRAY
    • lowbeta: ARRAY
    • highbeta: ARRAY
    • gamma: ARRAY
 ProbabilityMatrix
  • Inputs:
    • input_data: ARRAY
  • Outputs:
    • data: ARRAY
 SpectroMorphology
  • Inputs:
    • data: ARRAY
  • Outputs:
    • spectro: ARRAY
 SpeechSynthesis
  • Inputs:
    • text: STRING
    • voice: ARRAY
  • Outputs:
    • speech: ARRAY
    • transcript: STRING
 TransitionalHarmony
  • Inputs:
    • data: ARRAY
  • Outputs:
    • trans_harm: ARRAY
    • melody: ARRAY
 TuningColors
  • Inputs:
    • data: ARRAY
  • Outputs:
    • hue: ARRAY
    • saturation: ARRAY
    • value: ARRAY
    • color_names: STRING
 TuningMatrix
  • Inputs:
    • tuning: ARRAY
  • Outputs:
    • matrix: ARRAY
    • metric_per_step: ARRAY
    • metric: ARRAY
 TuningReduction
  • Inputs:
    • tuning: ARRAY
  • Outputs:
    • reduced: ARRAY
 VAMP
  • Inputs:
    • data: ARRAY
  • Outputs:
    • comps: ARRAY
 VocalExpression
  • Inputs:
    • data: ARRAY
  • Outputs:
    • prosody_label: STRING
    • burst_label: STRING
    • prosody_score: ARRAY
    • burst_score: ARRAY

Array

Nodes implementing array operations.

View Nodes
 Clip
  • Inputs:
    • array: ARRAY
  • Outputs:
    • out: ARRAY
 Join
  • Inputs:
    • a: ARRAY
    • b: ARRAY
  • Outputs:
    • out: ARRAY
 Math
  • Inputs:
    • data: ARRAY
  • Outputs:
    • out: ARRAY
 Operation
  • Inputs:
    • a: ARRAY
    • b: ARRAY
  • Outputs:
    • out: ARRAY
 Reduce
  • Inputs:
    • array: ARRAY
  • Outputs:
    • out: ARRAY
 Reshape
  • Inputs:
    • array: ARRAY
  • Outputs:
    • out: ARRAY
 Select
  • Inputs:
    • data: ARRAY
  • Outputs:
    • out: ARRAY
 Transpose
  • Inputs:
    • array: ARRAY
  • Outputs:
    • out: ARRAY

Inputs

Nodes that provide data to the pipeline.

View Nodes
 Audiocraft
  • Inputs:
    • prompt: STRING
  • Outputs:
    • wav: ARRAY
 AudioStream
  • Inputs:
  • Outputs:
    • out: ARRAY
 ConstantArray
  • Inputs:
  • Outputs:
    • out: ARRAY
 ConstantString
  • Inputs:
  • Outputs:
    • out: STRING
 EEGRecording
  • Inputs:
  • Outputs:
 ExtendedTable
  • Inputs:
    • base: TABLE
    • array_input1: ARRAY
    • array_input2: ARRAY
    • array_input3: ARRAY
    • array_input4: ARRAY
    • array_input5: ARRAY
    • string_input1: STRING
    • string_input2: STRING
    • string_input3: STRING
    • string_input4: STRING
    • string_input5: STRING
  • Outputs:
    • table: TABLE
 FractalImage
  • Inputs:
    • complexity: ARRAY
  • Outputs:
    • image: ARRAY
 ImageGeneration
  • Inputs:
    • prompt: STRING
    • negative_prompt: STRING
    • base_image: ARRAY
  • Outputs:
    • img: ARRAY
 Kuramoto
  • Inputs:
    • initial_phases: ARRAY
  • Outputs:
    • phases: ARRAY
    • coupling: ARRAY
    • order_parameter: ARRAY
    • waveforms: ARRAY
 LoadFile
  • Inputs:
  • Outputs:
    • data_output: ARRAY
 LSLClient
  • Inputs:
  • Outputs:
    • out: ARRAY
 MeteoMedia
  • Inputs:
    • latitude: ARRAY
    • longitude: ARRAY
    • location_name: STRING
  • Outputs:
    • weather_data_table: TABLE
 OSCIn
  • Inputs:
  • Outputs:
    • message: TABLE
 PromptBook
  • Inputs:
    • input_prompt: STRING
  • Outputs:
    • out: STRING
 Reservoir
  • Inputs:
    • connectivity: ARRAY
  • Outputs:
    • data: ARRAY
 SerialStream
  • Inputs:
  • Outputs:
    • out: ARRAY
 Sine
  • Inputs:
  • Outputs:
    • out: ARRAY
 Table
  • Inputs:
    • base: TABLE
    • new_entry: ARRAY
  • Outputs:
    • table: TABLE
 TextGeneration
  • Inputs:
    • prompt: STRING
  • Outputs:
    • generated_text: STRING
 VideoStream
  • Inputs:
  • Outputs:
    • frame: ARRAY
 ZeroMQIn
  • Inputs:
  • Outputs:
    • data: ARRAY

Misc

Miscellaneous nodes that do not fit into other categories.

View Nodes
 AppendTables
  • Inputs:
    • table1: TABLE
    • table2: TABLE
  • Outputs:
    • output_table: TABLE
 ColorEnhancer
  • Inputs:
    • image: ARRAY
  • Outputs:
    • enhanced_image: ARRAY
 EdgeDetector
  • Inputs:
    • image: ARRAY
  • Outputs:
    • edges: ARRAY
 FormatString
  • Inputs:
    • input_string_1: STRING
    • input_string_2: STRING
    • input_string_3: STRING
    • input_string_4: STRING
    • input_string_5: STRING
    • input_string_6: STRING
    • input_string_7: STRING
    • input_string_8: STRING
    • input_string_9: STRING
    • input_string_10: STRING
  • Outputs:
    • output_string: STRING
 HSVtoRGB
  • Inputs:
    • hsv_image: ARRAY
  • Outputs:
    • rgb_image: ARRAY
 JoinString
  • Inputs:
    • string1: STRING
    • string2: STRING
    • string3: STRING
    • string4: STRING
    • string5: STRING
  • Outputs:
    • output: STRING
 RGBtoHSV
  • Inputs:
    • rgb_image: ARRAY
  • Outputs:
    • hsv_image: ARRAY
 SetMeta
  • Inputs:
    • array: ARRAY
  • Outputs:
    • out: ARRAY
 StringAwait
  • Inputs:
    • message: STRING
    • trigger: ARRAY
  • Outputs:
    • out: STRING
 TableSelectArray
  • Inputs:
    • input_table: TABLE
  • Outputs:
    • output_array: ARRAY
 TableSelectString
  • Inputs:
    • input_table: TABLE
  • Outputs:
    • output_string: STRING

Outputs

Nodes that send data to external systems.

View Nodes
 AudioOut
  • Inputs:
    • data: ARRAY
  • Outputs:
    • finished: ARRAY
 LSLOut
  • Inputs:
    • data: ARRAY
  • Outputs:
 MidiCCout
  • Inputs:
    • cc1: ARRAY
    • cc2: ARRAY
    • cc3: ARRAY
    • cc4: ARRAY
    • cc5: ARRAY
  • Outputs:
    • midi_status: STRING
 MidiOut
  • Inputs:
    • note: ARRAY
    • velocity: ARRAY
    • duration: ARRAY
  • Outputs:
    • midi_status: STRING
 OSCOut
  • Inputs:
    • data: TABLE
  • Outputs:
 SharedMemOut
  • Inputs:
    • data: ARRAY
  • Outputs:
 WriteCsv
  • Inputs:
    • table_input: TABLE
  • Outputs:
 ZeroMQOut
  • Inputs:
    • data: ARRAY
  • Outputs:

Signal

Nodes implementing signal processing operations.

View Nodes
 Buffer
  • Inputs:
    • val: ARRAY
  • Outputs:
    • out: ARRAY
 Cycle
  • Inputs:
    • signal: ARRAY
  • Outputs:
    • cycle: ARRAY
 EMD
  • Inputs:
    • data: ARRAY
  • Outputs:
    • IMFs: ARRAY
 FFT
  • Inputs:
    • data: ARRAY
  • Outputs:
    • mag: ARRAY
    • phase: ARRAY
 Filter
  • Inputs:
    • data: ARRAY
  • Outputs:
    • filtered_data: ARRAY
 FOOOFaperiodic
  • Inputs:
    • psd_data: ARRAY
  • Outputs:
    • offset: ARRAY
    • exponent: ARRAY
    • cf_peaks: ARRAY
    • cleaned_psd: ARRAY
 FrequencyShift
  • Inputs:
    • data: ARRAY
  • Outputs:
    • out: ARRAY
 Hilbert
  • Inputs:
    • data: ARRAY
  • Outputs:
    • inst_amplitude: ARRAY
    • inst_phase: ARRAY
    • inst_frequency: ARRAY
 IFFT
  • Inputs:
    • spectrum: ARRAY
    • phase: ARRAY
  • Outputs:
    • reconstructed: ARRAY
 PSD
  • Inputs:
    • data: ARRAY
  • Outputs:
    • psd: ARRAY
 Recurrence
  • Inputs:
    • input_array: ARRAY
  • Outputs:
    • recurrence_matrix: ARRAY
    • RR: ARRAY
    • DET: ARRAY
    • LAM: ARRAY
 Resample
  • Inputs:
    • data: ARRAY
  • Outputs:
    • out: ARRAY
 ResampleJoint
  • Inputs:
    • data1: ARRAY
    • data2: ARRAY
  • Outputs:
    • out1: ARRAY
    • out2: ARRAY
 Smooth
  • Inputs:
    • data: ARRAY
  • Outputs:
    • out: ARRAY
 StaticBaseline
  • Inputs:
    • data: ARRAY
  • Outputs:
    • normalized: ARRAY
 Threshold
  • Inputs:
    • data: ARRAY
  • Outputs:
    • thresholded: ARRAY
 TimeDelayEmbedding
  • Inputs:
    • input_array: ARRAY
  • Outputs:
    • embedded_array: ARRAY
 WelfordsZTransform
  • Inputs:
    • data: ARRAY
  • Outputs:
    • normalized: ARRAY