Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Productionize nf-polly plugin #1

Merged
merged 4 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
strategy:
fail-fast: false
matrix:
java_version: [11, 21]
java_version: [8, 11]

steps:
- name: Environment
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@
build
work
out

gradle.properties
102 changes: 35 additions & 67 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,114 +1,82 @@
# nf-hello plugin

This project contains a simple Nextflow plugin called `nf-hello` which provides examples of different plugin extensions:
# nf-polly plugin

- A custom trace observer that prints a message when the workflow starts and when the workflow completes
- A custom channel factory called `reverse`
- A custom operator called `goodbye`
- A custom function called `randomString`

NOTE: If you want to use this project as a starting point for a custom plugin, you must rename the `plugins/nf-hello` folder and update `settings.gradle` with your plugin name.

See the [Nextflow documentation](https://nextflow.io/docs/latest/plugins.html) for more information about developing plugins.
This project contains a Nextflow plugin called `nf-polly` which provides utilities for using the
Polly platform developed by [Elucidata](https://www.elucidata.io/).

## Plugin structure

- `settings.gradle`

Gradle project settings.

- `plugins/nf-hello`

The plugin implementation base directory.
- `settings.gradle` – Gradle project settings.

- `plugins/nf-hello/build.gradle`

Plugin Gradle build file. Project dependencies should be added here.
- `plugins/nf-polly` – The plugin implementation base directory.

- `plugins/nf-hello/src/resources/META-INF/MANIFEST.MF`

Manifest file defining the plugin attributes e.g. name, version, etc. The attribute `Plugin-Class` declares the plugin main class. This class should extend the base class `nextflow.plugin.BasePlugin` e.g. `nextflow.hello.HelloPlugin`.
- `plugins/nf-polly/build.gradle` – Plugin Gradle build file. Project dependencies should be added
here.

- `plugins/nf-hello/src/resources/META-INF/extensions.idx`

This file declares one or more extension classes provided by the plugin. Each line should contain the fully qualified name of a Java class that implements the `org.pf4j.ExtensionPoint` interface (or a sub-interface).
- `plugins/nf-polly/src/resources/META-INF/MANIFEST.MF` – Manifest file defining the plugin
attributes e.g. name, version, etc. The attribute `Plugin-Class` declares the plugin main class.
This class should extend the base class `nextflow.plugin.BasePlugin`
e.g. `nextflow.polly.PollyPlugin`.

- `plugins/nf-hello/src/main`
- `plugins/nf-polly/src/resources/META-INF/extensions.idx` – This file declares one or more
extension classes provided by the plugin. Each line should contain
the fully qualified name of a Java class that implements the `org.pf4j.ExtensionPoint` interface (
or a sub-interface).

The plugin implementation sources.
- `plugins/nf-polly/src/main` – The plugin implementation sources.

- `plugins/nf-hello/src/test`

The plugin unit tests.
- `plugins/nf-polly/src/test` – The plugin unit tests.

## Plugin classes

- `HelloConfig`: shows how to handle options from the Nextflow configuration

- `HelloExtension`: shows how to create custom channel factories, operators, and fuctions that can be included into pipeline scripts
- `PollyConfig` – to handle options from the Nextflow configuration

- `HelloFactory` and `HelloObserver`: shows how to react to workflow events with custom behavior
- `PollyExtension` – contains custom utilities (operators, functions, etc.) for use in the Polly
pipelines environment.

- `HelloPlugin`: the plugin entry point
- `PollyPlugin` – the plugin entry point

## Unit testing
## Unit testing

To run your unit tests, run the following command in the project root directory (ie. where the file `settings.gradle` is located):
To run your unit tests, run the following command in the project root directory (ie. where the file
`settings.gradle` is located):

```bash
./gradlew check
```

## Testing and debugging

To build and test the plugin during development, configure a local Nextflow build with the following steps:
To build and test the plugin during development, configure a local Nextflow build with the following
steps:

1. Clone the Nextflow repository in your computer into a sibling directory:
```bash
git clone --depth 1 https://github.com/nextflow-io/nextflow ../nextflow
```

2. Configure the plugin build to use the local Nextflow code:
```bash
echo "includeBuild('../nextflow')" >> settings.gradle
```

(Make sure to not add it more than once!)

3. Compile the plugin alongside the Nextflow code:
```bash
make assemble
```

4. Run Nextflow with the plugin, using `./launch.sh` as a drop-in replacement for the `nextflow` command, and adding the option `-plugins nf-hello` to load the plugin:
4. Run Nextflow with the plugin, using `./launch.sh` as a drop-in replacement for the `nextflow`
command, and adding the option `-plugins nf-polly` to load the plugin:
```bash
./launch.sh run nextflow-io/hello -plugins nf-hello
./launch.sh run my_org/my_pipeline -plugins nf-polly
```

## Testing without Nextflow build
## Publishing with a Nextflow installation

The plugin can be tested without using a local Nextflow build using the following steps:
This plugin can be made available to a local Nextflow installation using the following steps:

1. Build the plugin: `make buildPlugins`
2. Copy `build/plugins/<your-plugin>` to `$HOME/.nextflow/plugins`
3. Create a pipeline that uses your plugin and run it: `nextflow run ./my-pipeline-script.nf`

## Package, upload, and publish

The project should be hosted in a GitHub repository whose name matches the name of the plugin, that is the name of the directory in the `plugins` folder (e.g. `nf-hello`).

Follow these steps to package, upload and publish the plugin:

1. Create a file named `gradle.properties` in the project root containing the following attributes (this file should not be committed to Git):

* `github_organization`: the GitHub organisation where the plugin repository is hosted.
* `github_username`: The GitHub username granting access to the plugin repository.
* `github_access_token`: The GitHub access token required to upload and commit changes to the plugin repository.
* `github_commit_email`: The email address associated with your GitHub account.

2. Use the following command to package and create a release for your plugin on GitHub:
```bash
./gradlew :plugins:nf-hello:upload
```
2. Copy `build/plugins/nf-polly-1.2.3` to `$HOME/.nextflow/plugins`

3. Create a pull request against [nextflow-io/plugins](https://github.com/nextflow-io/plugins/blob/main/plugins.json) to make the plugin accessible to Nextflow.
To test, create a pipeline that uses the plugin and run it: `nextflow run ./my-pipeline-script.nf`
13 changes: 7 additions & 6 deletions plugins/nf-polly/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ sourceSets {
}

ext{
nextflowVersion = '24.01.0-edge'
nextflowVersion = '23.10.1'
}

dependencies {
Expand All @@ -68,15 +68,16 @@ dependencies {
implementation 'org.apache.logging.log4j:log4j-slf4j2-impl'
implementation 'org.apache.logging.log4j:log4j-1.2-api'
implementation 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.13.0'

// test configuration
testImplementation "org.apache.groovy:groovy:4.0.18"
testImplementation "org.apache.groovy:groovy-nio:4.0.18"
testImplementation "org.codehaus.groovy:groovy:3.0.17"
testImplementation "org.codehaus.groovy:groovy-nio:3.0.17"
testImplementation "io.nextflow:nextflow:$nextflowVersion"
testImplementation ("org.apache.groovy:groovy-test:4.0.18") { exclude group: 'org.apache.groovy' }
testImplementation ("org.codehaus.groovy:groovy-test:3.0.17") { exclude group: 'org.codehaus.groovy' }
testImplementation ("cglib:cglib-nodep:3.3.0")
testImplementation ("org.objenesis:objenesis:3.1")
testImplementation ("org.spockframework:spock-core:2.3-groovy-4.0") { exclude group: 'org.apache.groovy'; exclude group: 'net.bytebuddy' }
testImplementation ('org.spockframework:spock-junit4:2.3-groovy-4.0') { exclude group: 'org.apache.groovy'; exclude group: 'net.bytebuddy' }
testImplementation ("org.spockframework:spock-core:2.2-groovy-3.0") { exclude group: 'org.codehaus.groovy'; exclude group: 'net.bytebuddy' }
testImplementation ('org.spockframework:spock-junit4:2.2-groovy-3.0') { exclude group: 'org.codehaus.groovy'; exclude group: 'net.bytebuddy' }
testImplementation ('com.google.jimfs:jimfs:1.1')

testImplementation(testFixtures("io.nextflow:nextflow:$nextflowVersion"))
Expand Down
26 changes: 7 additions & 19 deletions plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy
Original file line number Diff line number Diff line change
@@ -1,35 +1,23 @@
package nextflow.hello
package nextflow.polly

import groovy.transform.PackageScope


/**
* This class allows model an specific configuration, extracting values from a map and converting
*
* In this plugin, the user can configure how the messages are prefixed with a String, i.e.
* due a nextflow.config
*
* hello {
* prefix = '>>'
* }
*
* when the plugin reverse a String it will append '>>' at the beginning instead the default 'Mr.'
*
* We anotate this class as @PackageScope to restrict the access of their methods only to class in the
* same package
*
* @author : jorge <[email protected]>
*
* We anotate this class as @PackageScope to restrict the access of their methods only to class in
* the same package
*/
@PackageScope
class PollyConfig {

final private String prefix
final private String metricsStreamName

PollyConfig(Map map){
PollyConfig(Map map) {
def config = map ?: Collections.emptyMap()
prefix = config.prefix ?: 'Mr.'
metricsStreamName = config.metricsStreamName ?: 'pravaah-dev-user-defined-metrics-events-stream-v1'
}

String getPrefix() { prefix }
String getMetricsStreamName() { metricsStreamName }
}
Loading
Loading