From d0bc721a11aa9ea9a1d13262764bae02b2458208 Mon Sep 17 00:00:00 2001 From: Saiful Khan Date: Fri, 12 Apr 2024 16:42:06 +0530 Subject: [PATCH 1/4] Cleanup unused code and rewrite README --- README.md | 102 ++++++------------ .../main/nextflow/polly/PollyConfig.groovy | 26 ++--- .../main/nextflow/polly/PollyExtension.groovy | 101 ++++------------- .../main/nextflow/polly/PollyFactory.groovy | 37 ------- .../main/nextflow/polly/PollyObserver.groovy | 42 -------- .../main/nextflow/polly/PollyPlugin.groovy | 6 +- .../src/resources/META-INF/MANIFEST.MF | 4 +- .../src/resources/META-INF/extensions.idx | 3 +- .../test/nextflow/polly/PollyDslTest.groovy | 42 +------- .../nextflow/polly/PollyFactoryTest.groovy | 36 ------- .../test/nextflow/polly/PollyHelper.groovy | 16 +-- .../test/nextflow/polly/PollyHelpers.groovy | 41 +++---- 12 files changed, 92 insertions(+), 364 deletions(-) delete mode 100644 plugins/nf-polly/src/main/nextflow/polly/PollyFactory.groovy delete mode 100644 plugins/nf-polly/src/main/nextflow/polly/PollyObserver.groovy delete mode 100644 plugins/nf-polly/src/test/nextflow/polly/PollyFactoryTest.groovy diff --git a/README.md b/README.md index dc8bc09..3f96b29 100644 --- a/README.md +++ b/README.md @@ -1,59 +1,44 @@ -# nf-hello plugin - -This project contains a simple Nextflow plugin called `nf-hello` which provides examples of different plugin extensions: +# nf-polly plugin -- A custom trace observer that prints a message when the workflow starts and when the workflow completes -- A custom channel factory called `reverse` -- A custom operator called `goodbye` -- A custom function called `randomString` - -NOTE: If you want to use this project as a starting point for a custom plugin, you must rename the `plugins/nf-hello` folder and update `settings.gradle` with your plugin name. - -See the [Nextflow documentation](https://nextflow.io/docs/latest/plugins.html) for more information about developing plugins. +This project contains a Nextflow plugin called `nf-polly` which provides utilities for using the +Polly platform developed by [Elucidata](https://www.elucidata.io/). ## Plugin structure - -- `settings.gradle` - - Gradle project settings. -- `plugins/nf-hello` - - The plugin implementation base directory. +- `settings.gradle` – Gradle project settings. -- `plugins/nf-hello/build.gradle` - - Plugin Gradle build file. Project dependencies should be added here. +- `plugins/nf-polly` – The plugin implementation base directory. -- `plugins/nf-hello/src/resources/META-INF/MANIFEST.MF` - - Manifest file defining the plugin attributes e.g. name, version, etc. The attribute `Plugin-Class` declares the plugin main class. This class should extend the base class `nextflow.plugin.BasePlugin` e.g. `nextflow.hello.HelloPlugin`. +- `plugins/nf-polly/build.gradle` – Plugin Gradle build file. Project dependencies should be added + here. -- `plugins/nf-hello/src/resources/META-INF/extensions.idx` - - This file declares one or more extension classes provided by the plugin. Each line should contain the fully qualified name of a Java class that implements the `org.pf4j.ExtensionPoint` interface (or a sub-interface). +- `plugins/nf-polly/src/resources/META-INF/MANIFEST.MF` – Manifest file defining the plugin + attributes e.g. name, version, etc. The attribute `Plugin-Class` declares the plugin main class. + This class should extend the base class `nextflow.plugin.BasePlugin` + e.g. `nextflow.polly.PollyPlugin`. -- `plugins/nf-hello/src/main` +- `plugins/nf-polly/src/resources/META-INF/extensions.idx` – This file declares one or more + extension classes provided by the plugin. Each line should contain + the fully qualified name of a Java class that implements the `org.pf4j.ExtensionPoint` interface ( + or a sub-interface). - The plugin implementation sources. +- `plugins/nf-polly/src/main` – The plugin implementation sources. -- `plugins/nf-hello/src/test` - - The plugin unit tests. +- `plugins/nf-polly/src/test` – The plugin unit tests. ## Plugin classes -- `HelloConfig`: shows how to handle options from the Nextflow configuration - -- `HelloExtension`: shows how to create custom channel factories, operators, and fuctions that can be included into pipeline scripts +- `PollyConfig` – to handle options from the Nextflow configuration -- `HelloFactory` and `HelloObserver`: shows how to react to workflow events with custom behavior +- `PollyExtension` – contains custom utilities (operators, functions, etc.) for use in the Polly + pipelines environment. -- `HelloPlugin`: the plugin entry point +- `PollyPlugin` – the plugin entry point -## Unit testing +## Unit testing -To run your unit tests, run the following command in the project root directory (ie. where the file `settings.gradle` is located): +To run your unit tests, run the following command in the project root directory (ie. where the file +`settings.gradle` is located): ```bash ./gradlew check @@ -61,18 +46,19 @@ To run your unit tests, run the following command in the project root directory ## Testing and debugging -To build and test the plugin during development, configure a local Nextflow build with the following steps: +To build and test the plugin during development, configure a local Nextflow build with the following +steps: 1. Clone the Nextflow repository in your computer into a sibling directory: ```bash git clone --depth 1 https://github.com/nextflow-io/nextflow ../nextflow ``` - + 2. Configure the plugin build to use the local Nextflow code: ```bash echo "includeBuild('../nextflow')" >> settings.gradle ``` - + (Make sure to not add it more than once!) 3. Compile the plugin alongside the Nextflow code: @@ -80,35 +66,17 @@ To build and test the plugin during development, configure a local Nextflow buil make assemble ``` -4. Run Nextflow with the plugin, using `./launch.sh` as a drop-in replacement for the `nextflow` command, and adding the option `-plugins nf-hello` to load the plugin: +4. Run Nextflow with the plugin, using `./launch.sh` as a drop-in replacement for the `nextflow` + command, and adding the option `-plugins nf-polly` to load the plugin: ```bash - ./launch.sh run nextflow-io/hello -plugins nf-hello + ./launch.sh run my_org/my_pipeline -plugins nf-polly ``` -## Testing without Nextflow build +## Publishing with a Nextflow installation -The plugin can be tested without using a local Nextflow build using the following steps: +This plugin can be made available to a local Nextflow installation using the following steps: 1. Build the plugin: `make buildPlugins` -2. Copy `build/plugins/` to `$HOME/.nextflow/plugins` -3. Create a pipeline that uses your plugin and run it: `nextflow run ./my-pipeline-script.nf` - -## Package, upload, and publish - -The project should be hosted in a GitHub repository whose name matches the name of the plugin, that is the name of the directory in the `plugins` folder (e.g. `nf-hello`). - -Follow these steps to package, upload and publish the plugin: - -1. Create a file named `gradle.properties` in the project root containing the following attributes (this file should not be committed to Git): - - * `github_organization`: the GitHub organisation where the plugin repository is hosted. - * `github_username`: The GitHub username granting access to the plugin repository. - * `github_access_token`: The GitHub access token required to upload and commit changes to the plugin repository. - * `github_commit_email`: The email address associated with your GitHub account. - -2. Use the following command to package and create a release for your plugin on GitHub: - ```bash - ./gradlew :plugins:nf-hello:upload - ``` +2. Copy `build/plugins/nf-polly-1.2.3` to `$HOME/.nextflow/plugins` -3. Create a pull request against [nextflow-io/plugins](https://github.com/nextflow-io/plugins/blob/main/plugins.json) to make the plugin accessible to Nextflow. +To test, create a pipeline that uses the plugin and run it: `nextflow run ./my-pipeline-script.nf` diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy index 617873d..b8e970e 100644 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy +++ b/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy @@ -1,4 +1,4 @@ -package nextflow.hello +package nextflow.polly import groovy.transform.PackageScope @@ -6,30 +6,18 @@ import groovy.transform.PackageScope /** * This class allows model an specific configuration, extracting values from a map and converting * - * In this plugin, the user can configure how the messages are prefixed with a String, i.e. - * due a nextflow.config - * - * hello { - * prefix = '>>' - * } - * - * when the plugin reverse a String it will append '>>' at the beginning instead the default 'Mr.' - * - * We anotate this class as @PackageScope to restrict the access of their methods only to class in the - * same package - * - * @author : jorge - * + * We anotate this class as @PackageScope to restrict the access of their methods only to class in + * the same package */ @PackageScope class PollyConfig { - final private String prefix + final private String metricsStreamName - PollyConfig(Map map){ + PollyConfig(Map map) { def config = map ?: Collections.emptyMap() - prefix = config.prefix ?: 'Mr.' + metricsStreamName = config.metricsStreamName ?: 'pravaah-dev-user-defined-metrics-events-v1' } - String getPrefix() { prefix } + String getMetricsStreamName() { metricsStreamName } } diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy index 88414ea..4f541ec 100644 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy +++ b/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy @@ -1,42 +1,28 @@ -package nextflow.hello - -import com.fasterxml.jackson.databind.ObjectMapper; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.slf4j.MDC; -import software.amazon.awssdk.core.SdkBytes; -import software.amazon.awssdk.services.kinesis.KinesisClient; -import software.amazon.awssdk.services.kinesis.model.KinesisException; -import software.amazon.awssdk.services.kinesis.model.PutRecordRequest; -import software.amazon.awssdk.services.kinesis.model.PutRecordResponse; -import java.util.Map; -import java.util.*; +package nextflow.polly +import com.fasterxml.jackson.databind.ObjectMapper import groovy.transform.CompileStatic import groovy.util.logging.Slf4j -import groovyx.gpars.dataflow.DataflowReadChannel -import groovyx.gpars.dataflow.DataflowWriteChannel -import nextflow.Channel import nextflow.Session -import nextflow.extension.CH -import nextflow.extension.DataflowHelper -import nextflow.plugin.extension.Factory import nextflow.plugin.extension.Function -import nextflow.plugin.extension.Operator import nextflow.plugin.extension.PluginExtensionPoint +import org.slf4j.Logger +import org.slf4j.LoggerFactory +import software.amazon.awssdk.core.SdkBytes +import software.amazon.awssdk.services.kinesis.KinesisClient +import software.amazon.awssdk.services.kinesis.model.PutRecordRequest +import software.amazon.awssdk.services.kinesis.model.PutRecordResponse /** * Example plugin extension showing how to implement a basic * channel factory method, a channel operator and a custom function. - * - * @author : jorge - * */ @Slf4j @CompileStatic class PollyExtension extends PluginExtensionPoint { - static final Logger logger = LoggerFactory.getLogger(PollyExtension.class); + static final Logger logger = LoggerFactory.getLogger(PollyExtension.class) + // static final ObjectMapper objMapper = new ObjectMapper(); /* * A session hold information about current execution of the script @@ -44,81 +30,30 @@ class PollyExtension extends PluginExtensionPoint { private Session session /* - * A Custom config extracted from nextflow.config under hello tag + * A Custom config extracted from nextflow.config under polly tag * nextflow.config * --------------- - * docker{ + * docker { * enabled = true * } * ... - * hello{ - * prefix = 'Mrs' + * polly { + * metricsStreamName = "my-kinesis-stream" * } */ - private PollyConfig config + private PollyConfig config /* * nf-core initializes the plugin once loaded and session is ready * @param session */ + @Override protected void init(Session session) { this.session = session - this.config = new PollyConfig(session.config.navigate('hello') as Map) - } - - /* - * {@code reverse} is a `producer` method and will be available to the script because: - * - * - it's public - * - it returns a DataflowWriteChannel - * - it's marked with the @Factory annotation - * - * The method can require arguments but it's not mandatory, it depends of the business logic of the method. - * - */ - @Factory - DataflowWriteChannel reverse(String message) { - final channel = CH.create() - session.addIgniter((action) -> reverseImpl(channel, message)) - return channel + this.config = new PollyConfig(session.config.navigate('polly') as Map) } - private void reverseImpl(DataflowWriteChannel channel, String message) { - channel.bind(message.reverse()); - channel.bind(Channel.STOP) - } - - /* - * {@code goodbye} is a *consumer* method as it receives values from a channel to perform some logic. - * - * Consumer methods are introspected by nextflow-core and include into the DSL if the method: - * - * - it's public - * - it returns a DataflowWriteChannel - * - it has only one arguments of DataflowReadChannel class - * - it's marked with the @Operator annotation - * - * a consumer method needs to proportionate 2 closures: - * - a closure to consume items (one by one) - * - a finalizer closure - * - * in this case `goodbye` will consume a message and will store it as an upper case - */ - @Operator - DataflowWriteChannel goodbye(DataflowReadChannel source) { - final target = CH.createBy(source) - final next = { target.bind("Goodbye $it".toString()) } - final done = { target.bind(Channel.STOP) } - DataflowHelper.subscribeImpl(source, [onNext: next, onComplete: done]) - return target - } - - /* - * Generate a random string - * - * Using @Function annotation we allow this function can be imported from the pipeline script - */ @Function void reportMetric(var key, var value) { // logger.info("Starting PutRecord Producer"); @@ -139,4 +74,4 @@ class PollyExtension extends PluginExtensionPoint { System.out.println("Failed to produce: " + e.getMessage()); } } -} \ No newline at end of file +} diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyFactory.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyFactory.groovy deleted file mode 100644 index 662855f..0000000 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyFactory.groovy +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright 2021, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package nextflow.hello - -import groovy.transform.CompileStatic -import nextflow.Session -import nextflow.trace.TraceObserver -import nextflow.trace.TraceObserverFactory -/** - * Implements the validation observer factory - * - * @author Paolo Di Tommaso - */ -@CompileStatic -class PollyFactory implements TraceObserverFactory { - - @Override - Collection create(Session session) { - final result = new ArrayList() - result.add( new PollyObserver() ) - return result - } -} diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyObserver.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyObserver.groovy deleted file mode 100644 index b3d0e79..0000000 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyObserver.groovy +++ /dev/null @@ -1,42 +0,0 @@ -/* - * Copyright 2021, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package nextflow.hello - -import groovy.transform.CompileStatic -import groovy.util.logging.Slf4j -import nextflow.Session -import nextflow.trace.TraceObserver - -/** - * Example workflow events observer - * - * @author Paolo Di Tommaso - */ -@Slf4j -@CompileStatic -class PollyObserver implements TraceObserver { - - @Override - void onFlowCreate(Session session) { - log.info "Pipeline is starting! 🚀" - } - - @Override - void onFlowComplete() { - log.info "Pipeline complete! 👋" - } -} diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyPlugin.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyPlugin.groovy index ea749e5..be05104 100644 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyPlugin.groovy +++ b/plugins/nf-polly/src/main/nextflow/polly/PollyPlugin.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.hello +package nextflow.polly import groovy.transform.CompileStatic import nextflow.plugin.BasePlugin @@ -22,9 +22,7 @@ import nextflow.plugin.Scoped import org.pf4j.PluginWrapper /** - * Implements the Hello plugins entry point - * - * @author Paolo Di Tommaso + * Implements the Polly plugins entry point */ @CompileStatic class PollyPlugin extends BasePlugin { diff --git a/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF b/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF index 84e7640..8ed85fa 100644 --- a/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF +++ b/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF @@ -1,6 +1,6 @@ Manifest-Version: 1.0 Plugin-Id: nf-polly -Plugin-Version: 0.5.0 -Plugin-Class: nextflow.hello.PollyPlugin +Plugin-Version: 0.1.0 +Plugin-Class: nextflow.polly.PollyPlugin Plugin-Provider: nextflow Plugin-Requires: >=24.01.0-edge diff --git a/plugins/nf-polly/src/resources/META-INF/extensions.idx b/plugins/nf-polly/src/resources/META-INF/extensions.idx index 43a5261..4ed52e5 100644 --- a/plugins/nf-polly/src/resources/META-INF/extensions.idx +++ b/plugins/nf-polly/src/resources/META-INF/extensions.idx @@ -1,2 +1 @@ -nextflow.hello.PollyFactory -nextflow.hello.PollyExtension \ No newline at end of file +nextflow.polly.PollyExtension \ No newline at end of file diff --git a/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy b/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy index a9a04c1..091aa3c 100644 --- a/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy +++ b/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy @@ -1,4 +1,4 @@ -package nextflow.hello +package nextflow.polly import java.nio.file.Files import java.util.jar.Manifest @@ -17,9 +17,7 @@ import java.nio.file.Path /** - * Unit test for Hello DSL - * - * @author : jorge + * Unit test for Polly DSL */ @Timeout(10) class PollyDslTest extends Dsl2Spec{ @@ -63,41 +61,11 @@ class PollyDslTest extends Dsl2Spec{ pluginsMode ? System.setProperty('pf4j.mode',pluginsMode) : System.clearProperty('pf4j.mode') } - def 'should perform a hi and create a channel' () { + def 'should return the key-value pair sent to the function' () { when: def SCRIPT = ''' - include {reverse} from 'plugin/nf-polly' - channel.reverse('hi!') - ''' - and: - def result = new MockScriptRunner([hello:[prefix:'>>']]).setScript(SCRIPT).execute() - then: - result.val == 'hi!'.reverse() - result.val == Channel.STOP - } - - def 'should store a goodbye' () { - when: - def SCRIPT = ''' - include {goodbye} from 'plugin/nf-polly' - channel - .of('folks') - .goodbye() - ''' - and: - def result = new MockScriptRunner([:]).setScript(SCRIPT).execute() - then: - result.val == 'Goodbye folks' - result.val == Channel.STOP - - } - - def 'can use an imported function' () { - when: - def SCRIPT = ''' - include {reportMetric} from 'plugin/nf-polly' - channel - .of( reportMetric("key","value") ) + include { reportMetric } from 'plugin/nf-polly' + reportMetric("key","value") ''' and: def result = new MockScriptRunner([:]).setScript(SCRIPT).execute() diff --git a/plugins/nf-polly/src/test/nextflow/polly/PollyFactoryTest.groovy b/plugins/nf-polly/src/test/nextflow/polly/PollyFactoryTest.groovy deleted file mode 100644 index 8a2c982..0000000 --- a/plugins/nf-polly/src/test/nextflow/polly/PollyFactoryTest.groovy +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2021, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package nextflow.hello - -import nextflow.Session -import spock.lang.Specification - -/** - * - * @author Paolo Di Tommaso - */ -class PollyFactoryTest extends Specification { - - def 'should return observer' () { - when: - def result = new PollyFactory().create(Mock(Session)) - then: - result.size()==1 - result[0] instanceof PollyObserver - } - -} diff --git a/plugins/nf-polly/src/test/nextflow/polly/PollyHelper.groovy b/plugins/nf-polly/src/test/nextflow/polly/PollyHelper.groovy index 5d05c0b..63f0eff 100644 --- a/plugins/nf-polly/src/test/nextflow/polly/PollyHelper.groovy +++ b/plugins/nf-polly/src/test/nextflow/polly/PollyHelper.groovy @@ -15,29 +15,23 @@ * limitations under the License. */ -package nextflow.hello +package nextflow.polly import com.google.common.jimfs.Configuration import com.google.common.jimfs.Jimfs -import groovy.transform.Memoized import java.nio.file.Files import java.nio.file.Path -import java.util.zip.GZIPInputStream -/** - * - * @author Paolo Di Tommaso - */ class PollyHelper { - static private fs = Jimfs.newFileSystem(Configuration.unix()); + static private fs = Jimfs.newFileSystem(Configuration.unix()) - static Path createInMemTempFile(String name='temp.file', String content=null) { - Path tmp = fs.getPath("/tmp"); + static Path createInMemTempFile(String name = 'temp.file', String content = null) { + Path tmp = fs.getPath("/tmp") tmp.mkdir() def result = Files.createTempDirectory(tmp, 'test').resolve(name) - if( content ) + if (content) result.text = content return result } diff --git a/plugins/nf-polly/src/test/nextflow/polly/PollyHelpers.groovy b/plugins/nf-polly/src/test/nextflow/polly/PollyHelpers.groovy index f6b315c..476ef4f 100644 --- a/plugins/nf-polly/src/test/nextflow/polly/PollyHelpers.groovy +++ b/plugins/nf-polly/src/test/nextflow/polly/PollyHelpers.groovy @@ -1,4 +1,4 @@ -package nextflow.hello +package nextflow.polly import groovy.util.logging.Slf4j import groovyx.gpars.dataflow.DataflowBroadcast @@ -41,30 +41,28 @@ class MockScriptRunner extends ScriptRunner { @Override def normalizeOutput(output) { - if( output instanceof ChannelOut ) { + if (output instanceof ChannelOut) { def list = new ArrayList(output.size()) - for( int i=0; i(output.size()) - for( def item : output ) { - ((List)result).add(read0(item)) + for (def item : output) { + ((List) result).add(read0(item)) } return result - } - - else { + } else { return read0(output) } } - private read0( obj ) { - if( obj instanceof DataflowBroadcast ) + private read0(obj) { + if (obj instanceof DataflowBroadcast) return obj.createReadChannel() return obj } @@ -100,14 +98,10 @@ class MockExecutorFactory extends ExecutorFactory { } } -/** - * - * @author Paolo Di Tommaso - */ class MockExecutor extends Executor { @Override - void signal() { } + void signal() {} protected TaskMonitor createTaskMonitor() { new MockMonitor() @@ -115,7 +109,7 @@ class MockExecutor extends Executor { @Override TaskHandler createTaskHandler(TaskRun task) { - return new MockTaskHandler(task) + return new MockTaskHandler(task) } } @@ -130,18 +124,18 @@ class MockMonitor implements TaskMonitor { * * @param handler A not null {@code TaskHandler} instance */ - boolean evict(TaskHandler handler) { } + boolean evict(TaskHandler handler) {} /** * Start the monitoring activity for the queued tasks * @return The instance itself, useful to chain methods invocation */ - TaskMonitor start() { } + TaskMonitor start() {} /** * Notify when a task terminates */ - void signal() { } + void signal() {} } @Slf4j @@ -154,12 +148,11 @@ class MockTaskHandler extends TaskHandler { @Override void submit() { log.info ">> launching mock task: ${task}" - if( task.type == ScriptType.SCRIPTLET ) { + if (task.type == ScriptType.SCRIPTLET) { task.workDir = Paths.get('.').complete() task.stdout = task.script task.exitStatus = 0 - } - else { + } else { task.code.call() } status = TaskStatus.COMPLETED @@ -177,6 +170,6 @@ class MockTaskHandler extends TaskHandler { } @Override - void kill() { } + void kill() {} } From 4762eaf4ecbe98c5207ce548b5255e1f907439b3 Mon Sep 17 00:00:00 2001 From: Saiful Khan Date: Fri, 12 Apr 2024 18:10:50 +0530 Subject: [PATCH 2/4] Fix reportMetric function --- .../main/nextflow/polly/PollyExtension.groovy | 59 +++++++++++++------ .../test/nextflow/polly/PollyDslTest.groovy | 30 +++++----- 2 files changed, 55 insertions(+), 34 deletions(-) diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy index 4f541ec..5508a76 100644 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy +++ b/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy @@ -1,6 +1,7 @@ package nextflow.polly -import com.fasterxml.jackson.databind.ObjectMapper + +import groovy.json.JsonOutput import groovy.transform.CompileStatic import groovy.util.logging.Slf4j import nextflow.Session @@ -54,24 +55,46 @@ class PollyExtension extends PluginExtensionPoint { this.config = new PollyConfig(session.config.navigate('polly') as Map) } + /** + * Report a single metric. This metric will be attached to the currently running job's ID. It + * can help provide domain-specific intelligence to pipeline job metrics. + * @param key The name of the metric + * @param value The value of the metric. Must be either a boolean, a number (int/float) or a + * string. + */ @Function void reportMetric(var key, var value) { - // logger.info("Starting PutRecord Producer"); - String streamName = "pravaah-dev-user-defined-metrics-events-v1"; - String partitionKey = "12345"; - Map keyValuePairs = Map.of(key, value); - try { - byte[] data = new ObjectMapper().writeValueAsBytes(Map.of("metricfromTest5", keyValuePairs)); - KinesisClient client = KinesisClient.builder().build(); - PutRecordRequest putRequest = PutRecordRequest.builder() - .partitionKey(partitionKey) - .streamName(streamName) - .data(SdkBytes.fromByteArray(data)) - .build(); - PutRecordResponse response = client.putRecord(putRequest); - System.out.println("Produced Record " + response.sequenceNumber() + " to Shard " + response.shardId() + " (line 145)"); - } catch (Exception e) { - System.out.println("Failed to produce: " + e.getMessage()); + logger.info(String.format("Putting record with key='%s' & value='%s'", key, value)) + String streamName = this.config.getMetricsStreamName() + String jobId = System.getenv("job_id") ?: "NA" + String partitionKey = key.toString() + try { + Map map = [job_id: jobId, key: key, value: value, type: getValueType(value)] + byte[] json = JsonOutput.toJson(map).getBytes() + KinesisClient client = KinesisClient.builder().build() + PutRecordRequest putRequest = PutRecordRequest.builder() + .partitionKey(partitionKey) + .streamName(streamName) + .data(SdkBytes.fromByteArray(json)) + .build() as PutRecordRequest + PutRecordResponse response = client.putRecord(putRequest) + logger.info( + "Submitted record %s to stream shard %s", + response.sequenceNumber(), + response.shardId() + ) + } catch (Exception e) { + logger.error("Failed to produce: " + e.getMessage()) + } + } + + private static String getValueType(var value) { + if (value instanceof Boolean) { + return "boolean" + } + if (value instanceof Number) { + return "number" + } + return "string" } -} } diff --git a/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy b/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy index 091aa3c..3fcdc3f 100644 --- a/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy +++ b/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy @@ -1,9 +1,5 @@ package nextflow.polly -import java.nio.file.Files -import java.util.jar.Manifest - -import nextflow.Channel import nextflow.plugin.Plugins import nextflow.plugin.TestPluginDescriptorFinder import nextflow.plugin.TestPluginManager @@ -13,16 +9,18 @@ import spock.lang.Shared import spock.lang.Timeout import test.Dsl2Spec +import java.nio.file.Files import java.nio.file.Path - +import java.util.jar.Manifest /** * Unit test for Polly DSL */ @Timeout(10) -class PollyDslTest extends Dsl2Spec{ +class PollyDslTest extends Dsl2Spec { - @Shared String pluginsMode + @Shared + String pluginsMode def setup() { // reset previous instances @@ -32,18 +30,18 @@ class PollyDslTest extends Dsl2Spec{ System.setProperty('pf4j.mode', 'dev') // the plugin root should def root = Path.of('.').toAbsolutePath().normalize() - def manager = new TestPluginManager(root){ + def manager = new TestPluginManager(root) { @Override protected PluginDescriptorFinder createPluginDescriptorFinder() { - return new TestPluginDescriptorFinder(){ + return new TestPluginDescriptorFinder() { @Override protected Manifest readManifestFromDirectory(Path pluginPath) { - if( !Files.isDirectory(pluginPath) ) + if (!Files.isDirectory(pluginPath)) return null final manifestPath = pluginPath.resolve('build/resources/main/META-INF/MANIFEST.MF') - if( !Files.exists(manifestPath) ) + if (!Files.exists(manifestPath)) return null final input = Files.newInputStream(manifestPath) @@ -58,20 +56,20 @@ class PollyDslTest extends Dsl2Spec{ def cleanup() { Plugins.stop() PluginExtensionProvider.reset() - pluginsMode ? System.setProperty('pf4j.mode',pluginsMode) : System.clearProperty('pf4j.mode') + pluginsMode ? System.setProperty('pf4j.mode', pluginsMode) : System.clearProperty('pf4j.mode') } - def 'should return the key-value pair sent to the function' () { + def 'should return the key-value pair sent to the function'() { when: def SCRIPT = ''' include { reportMetric } from 'plugin/nf-polly' - reportMetric("key","value") + reportMetric("key", "value") ''' and: def result = new MockScriptRunner([:]).setScript(SCRIPT).execute() then: - result.val.size() == 20 - result.val == Channel.STOP + // TODO: We need better tests that this! Need to come back to this later. + result != null } } From a37d3201ee4c2e382c0e31d4a368298121cc7527 Mon Sep 17 00:00:00 2001 From: Saiful Khan Date: Fri, 12 Apr 2024 23:39:11 +0530 Subject: [PATCH 3/4] Downgrade NF version requirement --- plugins/nf-polly/build.gradle | 2 +- plugins/nf-polly/src/resources/META-INF/MANIFEST.MF | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/nf-polly/build.gradle b/plugins/nf-polly/build.gradle index 690f261..5419f64 100644 --- a/plugins/nf-polly/build.gradle +++ b/plugins/nf-polly/build.gradle @@ -50,7 +50,7 @@ sourceSets { } ext{ - nextflowVersion = '24.01.0-edge' + nextflowVersion = '23.10.1' } dependencies { diff --git a/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF b/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF index 8ed85fa..5e449c7 100644 --- a/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF +++ b/plugins/nf-polly/src/resources/META-INF/MANIFEST.MF @@ -3,4 +3,4 @@ Plugin-Id: nf-polly Plugin-Version: 0.1.0 Plugin-Class: nextflow.polly.PollyPlugin Plugin-Provider: nextflow -Plugin-Requires: >=24.01.0-edge +Plugin-Requires: >=23.10.1 From 2f4ca46c488d9deea1aad8877297714d14548087 Mon Sep 17 00:00:00 2001 From: Saiful Khan Date: Mon, 15 Apr 2024 09:22:40 +0530 Subject: [PATCH 4/4] Fixed build configuration --- .github/workflows/build.yml | 2 +- .gitignore | 2 ++ plugins/nf-polly/build.gradle | 11 ++++++----- .../src/main/nextflow/polly/PollyConfig.groovy | 2 +- .../src/main/nextflow/polly/PollyExtension.groovy | 8 +++++--- .../src/test/nextflow/polly/PollyDslTest.groovy | 1 - 6 files changed, 15 insertions(+), 11 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 13eccbe..524221c 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -17,7 +17,7 @@ jobs: strategy: fail-fast: false matrix: - java_version: [11, 21] + java_version: [8, 11] steps: - name: Environment diff --git a/.gitignore b/.gitignore index a47b503..9142c16 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,5 @@ build work out + +gradle.properties diff --git a/plugins/nf-polly/build.gradle b/plugins/nf-polly/build.gradle index 5419f64..97a08f1 100644 --- a/plugins/nf-polly/build.gradle +++ b/plugins/nf-polly/build.gradle @@ -68,15 +68,16 @@ dependencies { implementation 'org.apache.logging.log4j:log4j-slf4j2-impl' implementation 'org.apache.logging.log4j:log4j-1.2-api' implementation 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.13.0' + // test configuration - testImplementation "org.apache.groovy:groovy:4.0.18" - testImplementation "org.apache.groovy:groovy-nio:4.0.18" + testImplementation "org.codehaus.groovy:groovy:3.0.17" + testImplementation "org.codehaus.groovy:groovy-nio:3.0.17" testImplementation "io.nextflow:nextflow:$nextflowVersion" - testImplementation ("org.apache.groovy:groovy-test:4.0.18") { exclude group: 'org.apache.groovy' } + testImplementation ("org.codehaus.groovy:groovy-test:3.0.17") { exclude group: 'org.codehaus.groovy' } testImplementation ("cglib:cglib-nodep:3.3.0") testImplementation ("org.objenesis:objenesis:3.1") - testImplementation ("org.spockframework:spock-core:2.3-groovy-4.0") { exclude group: 'org.apache.groovy'; exclude group: 'net.bytebuddy' } - testImplementation ('org.spockframework:spock-junit4:2.3-groovy-4.0') { exclude group: 'org.apache.groovy'; exclude group: 'net.bytebuddy' } + testImplementation ("org.spockframework:spock-core:2.2-groovy-3.0") { exclude group: 'org.codehaus.groovy'; exclude group: 'net.bytebuddy' } + testImplementation ('org.spockframework:spock-junit4:2.2-groovy-3.0') { exclude group: 'org.codehaus.groovy'; exclude group: 'net.bytebuddy' } testImplementation ('com.google.jimfs:jimfs:1.1') testImplementation(testFixtures("io.nextflow:nextflow:$nextflowVersion")) diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy index b8e970e..3d3d778 100644 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy +++ b/plugins/nf-polly/src/main/nextflow/polly/PollyConfig.groovy @@ -16,7 +16,7 @@ class PollyConfig { PollyConfig(Map map) { def config = map ?: Collections.emptyMap() - metricsStreamName = config.metricsStreamName ?: 'pravaah-dev-user-defined-metrics-events-v1' + metricsStreamName = config.metricsStreamName ?: 'pravaah-dev-user-defined-metrics-events-stream-v1' } String getMetricsStreamName() { metricsStreamName } diff --git a/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy b/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy index 5508a76..e61ad49 100644 --- a/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy +++ b/plugins/nf-polly/src/main/nextflow/polly/PollyExtension.groovy @@ -79,9 +79,11 @@ class PollyExtension extends PluginExtensionPoint { .build() as PutRecordRequest PutRecordResponse response = client.putRecord(putRequest) logger.info( - "Submitted record %s to stream shard %s", - response.sequenceNumber(), - response.shardId() + String.format( + "Submitted record %s to stream shard %s", + response.sequenceNumber(), + response.shardId() + ) ) } catch (Exception e) { logger.error("Failed to produce: " + e.getMessage()) diff --git a/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy b/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy index 3fcdc3f..de786d2 100644 --- a/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy +++ b/plugins/nf-polly/src/test/nextflow/polly/PollyDslTest.groovy @@ -35,7 +35,6 @@ class PollyDslTest extends Dsl2Spec { protected PluginDescriptorFinder createPluginDescriptorFinder() { return new TestPluginDescriptorFinder() { - @Override protected Manifest readManifestFromDirectory(Path pluginPath) { if (!Files.isDirectory(pluginPath)) return null