Refactored `tasks` package into `transfers` package. #162

jeff-cohere · 2025-10-14T23:33:43Z

The tasks package is kind of a mess--it has a single goroutine that updates the states of all the transfers in a very cross-cutting way, and then it has another goroutine that provides a "heartbeat" to poke it for the next update.

The transfers package that replaces it breaks up the work of file transfers into distinct goroutines with crisp areas of responsibility:

The dispatcher accepts transfer-related requests from a "clients" (a properly-routed request from the web service (the
prototype code in services/) and coordinates with the other goroutines
The store maintains the state of a transfer and makes metadata available to other goroutines
The stager handles file staging operations as needed
The mover handles the actual transfer of the payload
The manifestor generates the metadata manifest for a completed transfer, and transfers it into place with the rest of the payload.
A simple messaging service now publishes transfer status message to subscribers. Log messages and tests use this machinery, and we can easily use it to publish to external message queues.
Completed transfers (successful or not) are now recorded to the transfer journal.

I've used private package global variables for simple namespacing.

Closes #159

github-actions · 2025-10-14T23:34:15Z

PR Preview Action v1.6.2
🚀 View preview at https://kbase.github.io/dts/pr-preview/pr-162/
Built to branch `gh-pages` at 2025-11-06 22:57 UTC. Preview will be ready when the GitHub Pages deployment is complete.

jeff-cohere · 2025-10-28T22:57:37Z

We're almost there. I just need to do a bit more testing of the new orchestration logic within the service, and then propagate the config data structure.

codecov · 2025-10-28T23:45:55Z

Codecov Report

❌ Patch coverage is 17.94872% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.54%. Comparing base (43fed6f) to head (a86719d).

Files with missing lines	Patch %	Lines
dtstest/dtstest.go	0.00%	24 Missing ⚠️
databases/databases.go	0.00%	5 Missing ⚠️
services/prototype.go	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #162      +/-   ##
==========================================
+ Coverage   64.06%   66.54%   +2.48%     
==========================================
  Files          24       20       -4     
  Lines        2585     1949     -636     
==========================================
- Hits         1656     1297     -359     
+ Misses        754      542     -212     
+ Partials      175      110      -65

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jeff-cohere · 2025-10-29T16:39:37Z

I want to test this "in production" over the weekend before merging it.

mattldawson

Looks really nice! I just have a couple minor questions/comments, mostly about how goroutines interact.

What wasn't obvious to me at first is that successful staging triggers a move directly, and moves trigger manifests to be generated, and that all three directly set the status for the transfer. It all makes sense now, but I wonder if there is somewhere to put this info so it's easy for a newbie like me to get a picture of the overall workflow?

transfers/transfers.go

transfers/stager.go

transfers/mover.go

mattldawson · 2025-10-29T20:37:27Z

One thing I was curious about is why you set the channel buffers to 32? could there ever be more than one element in each channel?

jeff-cohere · 2025-10-29T20:51:39Z

One thing I was curious about is why you set the channel buffers to 32? could there ever be more than one element in each channel?

Good point. We should probably relate this to the number of allowed HTTP connections via a parameter. And I should go over the code and change the channel buffers to reflect the actual expected queue sizes.

The transfers package uses concurrent sequential programs to divide up the labor in transferring files.

…rs messages. This mechanism is useful for testing and can also be wired into a message queue down the line.

jeff-cohere · 2025-11-06T22:54:45Z

Okay, I think this is ready to go--I've tested it in Spin with dtspy and with Ken Chu's IMG -> KBase push service.

mattldawson

Looks great!

mattldawson · 2025-11-06T23:28:45Z

transfers/dispatcher.go

+
+type dispatcherChannels struct {
+	RequestTransfer  chan Specification // used by client to create a new transfer
+	ReturnTransferId chan uuid.UUID     // returns task ID to client


Suggested change

ReturnTransferId chan uuid.UUID // returns task ID to client

ReturnTransferId chan uuid.UUID // returns transfer ID to client

mattldawson · 2025-11-06T23:39:06Z

transfers/dispatcher.go

+	d.Channels.CancelTransfer <- transferId
+	err := <-d.Channels.Error
+	if err != nil {
+		slog.Error(fmt.Sprintf("Transfer %s: %s", transferId.String(), err.Error()))


Not a big deal, but if we log cancel errors, should we also log errors for other dispatcher functions?

mattldawson · 2025-11-07T00:07:57Z

transfers/transfers_test.go

+	/*
+		dispatcherTests := DispatcherTests{Test: t}
+		print("dispatcher start/stop\n")
+		dispatcherTests.TestStartAndStop()
+	*/


Should this be removed?

jeff-cohere added cleanup metadata labels Oct 14, 2025

jeff-cohere force-pushed the tasks-refactor branch from dc434a7 to 4562caf Compare October 16, 2025 23:06

jeff-cohere force-pushed the tasks-refactor branch 3 times, most recently from 1614b3e to 7c16405 Compare October 28, 2025 23:43

jeff-cohere marked this pull request as ready for review October 29, 2025 16:38

mattldawson approved these changes Oct 29, 2025

View reviewed changes

transfers/transfers.go Outdated Show resolved Hide resolved

transfers/stager.go Show resolved Hide resolved

transfers/mover.go Show resolved Hide resolved

jeff-cohere force-pushed the tasks-refactor branch from 50f6857 to c07c522 Compare October 30, 2025 22:07

jeff-cohere added 15 commits November 4, 2025 12:36

Starting transfer orchestrator refactor in earnest.

c0067c1

Breaking tasks logic up into concurrent sequential programs.

c189154

Replacing tasks package with transfers package.

552dcd0

The transfers package uses concurrent sequential programs to divide up the labor in transferring files.

Fixed some test failures.

88e529a

Roughing in some transfer tests.

e835116

Getting tests organized.

183c163

Adding a clock for polling other routines.

2475179

Debugging tests.

4287c9c

Fixed various issues with transfer tests.

64701f0

Fixed an issue with JSON descriptors.

831f634

Added transfer status callbacks useful for testing.

6cc3d8d

Replaced callbacks with a simple publish/subscribe system that delive…

ebc9d08

…rs messages. This mechanism is useful for testing and can also be wired into a message queue down the line.

Implemented saving and loading.

8376abe

Fixed transfer tests.

4d43a18

Test fixtures now mock file staging properly.

a0e1935

jeff-cohere added 8 commits November 4, 2025 12:37

Switched prototype from tasks to transfers package.

9d23b7f

Removing unused code.

775d53c

Partially propagated config instance into orchestration logic.

8025c64

Placating static analyzer.

8eda76b

Addressed PR comments aside from channel buffer sizes.

b136f15

Added orchestration README and diagram.

9ed81ef

Reduced channel buffering.

a82d1da

Rebased and patched up nmdc database constructor.

18a6882

jeff-cohere force-pushed the tasks-refactor branch from c07c522 to 18a6882 Compare November 4, 2025 20:37

jeff-cohere added 3 commits November 5, 2025 15:21

Changes from testing DTS in Spin.

2453c9c

Fixed a staging issue.

cf1217c

Fixing last batch of defects.

0ff128a

jeff-cohere requested a review from mattldawson November 6, 2025 22:54

Removed debugging statements.

a86719d

mattldawson approved these changes Nov 7, 2025

View reviewed changes

	ReturnTransferId chan uuid.UUID // returns task ID to client
	ReturnTransferId chan uuid.UUID // returns transfer ID to client

Refactored tasks package into transfers package. #162

Are you sure you want to change the base?

Refactored tasks package into transfers package. #162

Conversation

jeff-cohere commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Built to branch gh-pages at 2025-11-06 22:57 UTC. Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

jeff-cohere commented Oct 28, 2025

Uh oh!

codecov bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jeff-cohere commented Oct 29, 2025

Uh oh!

mattldawson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattldawson commented Oct 29, 2025

Uh oh!

jeff-cohere commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeff-cohere commented Nov 6, 2025

Uh oh!

mattldawson left a comment

Choose a reason for hiding this comment

Uh oh!

mattldawson Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

mattldawson Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

mattldawson Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Refactored `tasks` package into `transfers` package. #162

Refactored `tasks` package into `transfers` package. #162

jeff-cohere commented Oct 14, 2025 •

edited

Loading

github-actions bot commented Oct 14, 2025 •

edited

Loading

Built to branch `gh-pages` at 2025-11-06 22:57 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

codecov bot commented Oct 28, 2025 •

edited

Loading

jeff-cohere commented Oct 29, 2025 •

edited

Loading