Skip to content
This repository was archived by the owner on Jun 30, 2025. It is now read-only.

Commit 53d4912

Browse files
bajtosjuliangruber
andauthored
feat: Zinnia Runtime and Peer Checker module (#92)
* feat: install Zinnia runtime Add another post-install step to download `zinniad` for running Zinnia modules. Signed-off-by: Miroslav Bajtoš <[email protected]> * feat: install mod-peer-checker Signed-off-by: Miroslav Bajtoš <[email protected]> * feat: run peer checker module via Zinnia Signed-off-by: Miroslav Bajtoš <[email protected]> * fix formatting Signed-off-by: Miroslav Bajtoš <[email protected]> * fixup! windows zinniad extension Signed-off-by: Miroslav Bajtoš <[email protected]> * test: fix never ending test Signed-off-by: Miroslav Bajtoš <[email protected]> * upgrade zinniad to v0.7.0 * Update commands/station.js Co-authored-by: Julian Gruber <[email protected]> * update tests Signed-off-by: Miroslav Bajtoš <[email protected]> * ci: execute Docker image and test activity Signed-off-by: Miroslav Bajtoš <[email protected]> * fixup! fix failing tests Signed-off-by: Miroslav Bajtoš <[email protected]> * fixup! docker CI workflow Signed-off-by: Miroslav Bajtoš <[email protected]> * troubleshoot docker CI not running Signed-off-by: Miroslav Bajtoš <[email protected]> * enable Docker workflow for all pushes * Revert "troubleshoot docker CI not running" This reverts commit 11a56e7. * more docker tweaks Signed-off-by: Miroslav Bajtoš <[email protected]> * feat: switch base Docker image to full `node:18` We cannot use `node:18-alpine` because Zinnia does not support that yet. I tried to use `node:18-slim`, but Saturn L2 was not able to start in that system. The Node.js project recommends to use `node:18`, so we should be fine. See https://github.com/nodejs/docker-node#image-variants Signed-off-by: Miroslav Bajtoš <[email protected]> * fixup! standard coding style Signed-off-by: Miroslav Bajtoš <[email protected]> * fix zip assets on windows * fixup! remove github from zinnia module repo name Signed-off-by: Miroslav Bajtoš <[email protected]> * bump zinnia to 0.8.0 Signed-off-by: Miroslav Bajtoš <[email protected]> * rename Module Runtime to Zinnia Signed-off-by: Miroslav Bajtoš <[email protected]> * move mocha config to config file and add --exit Signed-off-by: Miroslav Bajtoš <[email protected]> * make test/station easier to troubleshoot Signed-off-by: Miroslav Bajtoš <[email protected]> * fix expected Saturn messages Signed-off-by: Miroslav Bajtoš <[email protected]> --------- Signed-off-by: Miroslav Bajtoš <[email protected]> Co-authored-by: Julian Gruber <[email protected]>
1 parent a2a27f2 commit 53d4912

File tree

12 files changed

+262
-38
lines changed

12 files changed

+262
-38
lines changed

.github/workflows/ci.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,3 +93,6 @@ jobs:
9393
9494
- name: Check | Saturn started
9595
run: docker exec station bin/station.js activity | grep "Saturn module started"
96+
97+
- name: Check | Zinnia started
98+
run: docker exec station bin/station.js activity | grep "Zinnia started"

.mocharc.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
timeout: 15000
2+
exit: true

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM node:18-alpine
1+
FROM node:18
22
LABEL org.opencontainers.image.source https://github.com/filecoin-station/core
33
USER node
44
WORKDIR /usr/src/app

bin/station.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Sentry.init({
2121

2222
const core = new Core(getDefaultRootDirs())
2323
const modules = [
24+
'zinnia',
2425
'saturn-L2-node',
2526
'bacalhau'
2627
]

commands/station.js

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { join } from 'node:path'
22
import * as saturnNode from '../lib/saturn-node.js'
3+
import * as zinniaRuntime from '../lib/zinnia.js'
34
import { formatActivityObject } from '../lib/activity.js'
45
import lockfile from 'proper-lockfile'
56
import { maybeCreateFile } from '../lib/util.js'
@@ -36,6 +37,16 @@ export const station = async ({ core, json, experimental }) => {
3637
logStream: core.logs.createWriteStream(
3738
join(core.paths.moduleLogs, 'saturn-L2-node.log')
3839
)
40+
}),
41+
zinniaRuntime.start({
42+
FIL_WALLET_ADDRESS,
43+
STATE_ROOT: join(core.paths.moduleState, 'zinnia'),
44+
CACHE_ROOT: join(core.paths.moduleCache, 'zinnia'),
45+
metricsStream: await core.metrics.createWriteStream('zinnia'),
46+
activityStream: core.activity.createWriteStream('Zinnia'),
47+
logStream: core.logs.createWriteStream(
48+
join(core.paths.moduleLogs, 'zinnia.log')
49+
)
3950
})
4051
]
4152

lib/modules.js

Lines changed: 40 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,14 @@ export const getBinaryModuleExecutable = ({
2121
return join(
2222
moduleBinaries,
2323
module,
24-
`${executable}${os.platform() === 'win32' ? '.exe' : ''}`
24+
getExecutableFileName(executable)
2525
)
2626
}
2727

28+
const getExecutableFileName = executable => {
29+
return `${executable}${os.platform() === 'win32' ? '.exe' : ''}`
30+
}
31+
2832
export const installBinaryModule = async ({
2933
module,
3034
repo,
@@ -79,8 +83,9 @@ export const installBinaryModule = async ({
7983
const [entry] =
8084
/** @type {[UnzipStreamEntry]} */
8185
(await once(parser, 'entry'))
82-
if (entry.path === executable) {
83-
const outPath = join(moduleBinaries, module, executable)
86+
const executableFileName = getExecutableFileName(executable)
87+
if (entry.path === executableFileName) {
88+
const outPath = join(moduleBinaries, module, executableFileName)
8489
await pipeline(entry, createWriteStream(outPath))
8590
await chmod(outPath, 0o755)
8691
return
@@ -92,3 +97,35 @@ export const installBinaryModule = async ({
9297
}
9398
console.log(`[${module}] ✓ ${outFile}`)
9499
}
100+
101+
export async function downloadSourceFiles ({ module, repo, distTag }) {
102+
await mkdir(moduleBinaries, { recursive: true })
103+
const outDir = join(moduleBinaries, module)
104+
105+
console.log(`[${module}] ⇣ downloading source files`)
106+
107+
const url = `https://github.com/${repo}/archive/refs/tags/${distTag}.tar.gz`
108+
const res = await fetch(url, {
109+
headers: {
110+
...(authorization ? { authorization } : {})
111+
},
112+
redirect: 'follow'
113+
})
114+
115+
if (res.status >= 300) {
116+
throw new Error(
117+
`[${module}] Cannot fetch ${module} archive for tag ${distTag}: ${res.status}\n` +
118+
await res.text()
119+
)
120+
}
121+
122+
if (!res.body) {
123+
throw new Error(
124+
`[${module}] Cannot fetch ${module} archive for tag ${distTag}: no response body`
125+
)
126+
}
127+
128+
// `{ strip: 1}` tells tar to remove the top-level directory (e.g. `mod-peer-checker-v1.0.0`)
129+
await pipeline(res.body, gunzip(), tar.extract(outDir, { strip: 1 }))
130+
console.log(`[${module}] ✓ ${outDir}`)
131+
}

lib/zinnia.js

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import timers from 'node:timers/promises'
2+
import { execa } from 'execa'
3+
import * as Sentry from '@sentry/node'
4+
import { installBinaryModule, downloadSourceFiles, getBinaryModuleExecutable } from './modules.js'
5+
import { moduleBinaries } from './paths.js'
6+
7+
const ZINNIA_DIST_TAG = 'v0.8.0'
8+
const ZINNIA_MODULES = [
9+
{
10+
module: 'peer-checker',
11+
repo: 'filecoin-station/mod-peer-checker',
12+
distTag: 'v1.0.0'
13+
}
14+
]
15+
16+
export async function install () {
17+
await Promise.all([
18+
installBinaryModule({
19+
module: 'zinnia',
20+
repo: 'filecoin-station/zinnia',
21+
distTag: ZINNIA_DIST_TAG,
22+
executable: 'zinniad',
23+
targets: [
24+
{ platform: 'darwin', arch: 'arm64', asset: 'zinniad-macos-arm64.zip' },
25+
{ platform: 'darwin', arch: 'x64', asset: 'zinniad-macos-x64.zip' },
26+
{ platform: 'linux', arch: 'arm64', asset: 'zinniad-linux-arm64.tar.gz' },
27+
{ platform: 'linux', arch: 'x64', asset: 'zinniad-linux-x64.tar.gz' },
28+
{ platform: 'win32', arch: 'x64', asset: 'zinniad-windows-x64.zip' }
29+
]
30+
}),
31+
32+
...Object.values(ZINNIA_MODULES).map(downloadSourceFiles)
33+
])
34+
}
35+
36+
async function start ({
37+
FIL_WALLET_ADDRESS,
38+
STATE_ROOT,
39+
CACHE_ROOT,
40+
metricsStream,
41+
activityStream,
42+
logStream
43+
}) {
44+
logStream.write('Starting Zinnia')
45+
46+
const zinniadExe = getBinaryModuleExecutable({ module: 'zinnia', executable: 'zinniad' })
47+
const modules = [
48+
// all paths are relative to `moduleBinaries`
49+
'peer-checker/peer-checker.js'
50+
]
51+
const childProcess = execa(zinniadExe, modules, {
52+
cwd: moduleBinaries,
53+
env: {
54+
FIL_WALLET_ADDRESS,
55+
STATE_ROOT,
56+
CACHE_ROOT
57+
}
58+
})
59+
60+
const readyPromise = new Promise((resolve, reject) => {
61+
childProcess.stdout.setEncoding('utf-8')
62+
childProcess.stdout.on('data', data => {
63+
logStream.write(data)
64+
handleEvents({ activityStream, metricsStream }, data)
65+
})
66+
67+
childProcess.stderr.setEncoding('utf-8')
68+
childProcess.stderr.on('data', data => {
69+
logStream.write(data)
70+
})
71+
72+
childProcess.stdout.once('data', _data => {
73+
// This is based on an implicit assumption that zinniad reports an info activity
74+
// after it starts
75+
resolve()
76+
})
77+
childProcess.catch(reject)
78+
})
79+
80+
childProcess.on('close', code => {
81+
logStream.write(`Zinnia closed all stdio with code ${code ?? '<no code>'}`)
82+
childProcess.stderr.removeAllListeners()
83+
childProcess.stdout.removeAllListeners()
84+
Sentry.captureException('Zinnia exited')
85+
})
86+
87+
childProcess.on('exit', (code, signal) => {
88+
const reason = signal ? `via signal ${signal}` : `with code: ${code}`
89+
const msg = `Zinnia exited ${reason}`
90+
logStream.write(msg)
91+
activityStream.write({ type: 'info', message: msg })
92+
})
93+
94+
try {
95+
await Promise.race([
96+
readyPromise,
97+
timers.setTimeout(500)
98+
])
99+
} catch (err) {
100+
const errorMsg = err instanceof Error ? err.message : '' + err
101+
const message = `Cannot start Zinnia: ${errorMsg}`
102+
logStream.write(message)
103+
activityStream.write({ type: 'error', message })
104+
}
105+
}
106+
107+
function handleEvents ({ activityStream, metricsStream }, text) {
108+
text
109+
.trimEnd()
110+
.split(/\n/g)
111+
.forEach(line => {
112+
try {
113+
const event = JSON.parse(line)
114+
switch (event.type) {
115+
case 'activity:info':
116+
activityStream.write({
117+
type: 'info',
118+
message: event.message.replace(/Module Runtime/, 'Zinnia'),
119+
source: event.module ?? 'Zinnia'
120+
})
121+
break
122+
123+
case 'activity:error':
124+
activityStream.write({
125+
type: 'error',
126+
message: event.message.replace(/Module Runtime/, 'Zinnia'),
127+
source: event.module ?? 'Zinnia'
128+
})
129+
break
130+
131+
case 'jobs-completed':
132+
metricsStream.write({ totalJobsCompleted: event.total, totalEarnings: '0' })
133+
break
134+
135+
default:
136+
console.error('Ignoring Zinnia event of unknown type:', event)
137+
}
138+
} catch (err) {
139+
console.error('Ignoring malformed Zinnia event:', line)
140+
}
141+
})
142+
}
143+
144+
export { start }

package-lock.json

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,10 @@
1515
"scripts": {
1616
"build": "tsc",
1717
"format": "prettier --write .",
18+
"start": "node ./bin/station.js",
1819
"test": "npm run build && npm run test:lint && npm run test:unit",
1920
"test:lint": "prettier --check . && standard",
20-
"test:unit": "mocha --timeout 15000",
21+
"test:unit": "mocha",
2122
"version": "npm run build && node ./scripts/version.js",
2223
"postinstall": "node ./scripts/post-install.js",
2324
"postpublish": "node ./scripts/post-publish.js",
@@ -29,6 +30,7 @@
2930
]
3031
},
3132
"devDependencies": {
33+
"get-stream": "^6.0.1",
3234
"mocha": "^10.2.0",
3335
"np": "^7.6.3",
3436
"prettier": "^2.8.4",

scripts/post-install.js

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,10 @@
22

33
import { install as installSaturn } from '../lib/saturn-node.js'
44
import { install as installBacalhau } from '../lib/bacalhau.js'
5+
import { install as installZinnia } from '../lib/zinnia.js'
56

67
await Promise.all([
78
installSaturn(),
8-
installBacalhau()
9+
installBacalhau(),
10+
installZinnia()
911
])

0 commit comments

Comments
 (0)