Background jobs?

[fritzblue]

First of all, lots of cool ideas here, thank you for sharing it with the world.

One common use case I'm not seeing a solution for in the roadmap yet (though I may have missed it): Are there plans to support cron jobs (scheduled tasks), background jobs, etc?

Things like: "At 5am, compute and send this daily report to all subscribers"

This would help performance by offloading computationally intensive tasks. It would also be useful for common scenarios from automating routine maintenance tasks to queueing up and performing long-running work in the background.

Any plans for supporting this?

[kentcdodds]

Definitely

[kentcdodds]

As an update on this, I'm actively working on something that I think is gonna be very awesome for people.

Anyone looking to have something right now should look into using inngest. In the future, we'll have something simple and self-hostable (side-car process) for simpler use cases which is very similar to inngest.

[bvkimball]

How far into the future do you think... Roughly?

[kentcdodds]

Two or three months I think.

[kiliman]

You can spin up Quirrel as a Docker image on Fly.io. Here's an article. He spins up his own Redis instance, but you can also connect to Fly's Upstash Redis for free.

https://dev.to/remixtape/self-hosting-quirrel-5af7

[kiliman]

I don't think it's much different. You simply set the REDIS_URL environment variable to Redis Upstash. Make sure you include family=6 to enable IPv6 otherwise, it won't connect properly.

REDIS_URL=redis://default:***@your-app.upstash.io?family=6

[jacobparis]

@kiliman how exactly does the process differ when using redis upstash?

Notably, BullMQ does not support upstash (I don't know why) so if that's what you want to use for your queues you'll need a different type of Redis

I just pushed a regular redis image to fly and use that one for all my personal projects

[PhilDL]

I have to test it again to be sure but upstash/issues#18 (comment) was about the compatibility between BullMQ and Upstash, it seems that it's resolved now.

[thefrana]

unless you store them in a DB

I'm planning on keeping the jobs in a separate SQLite database managed by LiteFS. So not any extra service necessary.

I had exactly the same idea. There are solutions that runs on top of Redis, but using just SQLite database is awesome approach. Do you have any progress or new thoughts on this?

[kentcdodds]

What I'm collaborating on right now uses this approach. I don't want to reveal too much because it's not just me working on it and I'm not sure how much my partners want me to share. Stay tuned!

[phifa]

In a monorepo setup, the cron jobs could just be a separate app with its own process, triggering functions or remix endpoints. Monorepo Discussion --> #57

[chessydk]

How about using supercronic in a fly process (https://fly.io/docs/app-guides/supercronic/ ) and use getInstanceInfo from 'litefs-js' to only run on the primary (the latter is already in use in the project - e.g. in entry.server.tsx) ?

[jacobparis]

I use BullMQ to handle my background jobs. It does require spinning up a redis instance, but that's a one line deploy on Fly and all my app regions can talk to the same redis instance since latency stops being a factor in background jobs

On the developer's side, usage is simple enough: import the queue and add new docs to it

import { emailQueue } from "./queue.server"
await emailQueue.add("send-welcome-email", {
  to: "[email protected]",
  subject: "Welcome to our app!",
  text: "Thanks for signing up!",
})

import { renderQueue } from "./queue.server"
const job = await renderQueue.getJob("123")
console.log(job.progress) // 0.5
console.log(job.returnvalue) // undefined

const TIMEOUT_MILLISECONDS = 30 * 1000
await job.waitUntilFinished(
  renderQueue.events,
  TIMEOUT_MILLISECONDS,
)
console.log(job.progress) // 1
console.log(job.returnvalue) // { url: "123.png" }

https://www.jacobparis.com/content/bullmq-integration-guide

[kentcdodds]

I understand these constraints. I'm trying to weigh this with simplicity. Background jobs aren't always so intensive. There's prior art in other frameworks like Laravel, .NET, and Rails which is I understand correctly do support running within the app instance and people use that.

I think starting people with in-app-instance background jobs + having it easy to swap with extracting it to a worker is pretty reasonable.

[sergiodxa]

In Rails, people use that locally, in prod you always use something else.

[kentcdodds]

Yep, pretty sure this is where we'll land:

• starter allows you to locally use something custom that runs within the process
• this in-process worker works in production as well and that's how it's configured in the starter
• it's adaptable so you can easily switch to a worker instead
• we have docs on why, when, and how to switch to a worker

[stephen776]

@kentcdodds Can you describe a bit how the "switch to worker" option would work in a scenario where we need to communicate with the main SQLite instance? It was my understanding that we could only connect to the DB through the main app process that's running SQLite/LiteFS.

[kentcdodds]

I'm hoping this will become a thing before too long: superfly/litefs#326

Right now we work around this limitation for the cache by having our own route for instance-to-instance communication:

epic-stack/app/routes/admin+/cache_.sqlite.tsx

Line 48 in d7f6891

return fetch(`${domain}/admin/cache/sqlite`, {

If LiteFS doesn't get the HTTP Query API by the time I get around to adding support for this, then that's what I'd recommend.

[ChrisSummers]

I looked at bree, toad-scheduler, and croner. Out of those I think croner is a good starting point that would fit the stated goals.

I'm currently leaning epic-stack + croner + a db table to track executions and which server instance owns each job.

I'm new to everything, what file would start the jobs? In my testing I've been using ./server/index.ts

[kentcdodds]

Hi @ChrisSummers! That looks great to me. I think ./server/index.ts is a reasonable place to kick things off. It has the benefit of not having to worry about spawning a separate process. I'd need to play around it myself a bit. Would you be interested in making an example of this? That would go a long way to help me get this built into the starter.

[ChrisSummers]

For something very simple croner is great. I'm actually leaning resource routes with Amazon EventBridge Scheduler or trigger.dev. There's so much work to getting distributed scheduled jobs with monitoring and recovery working that I don't feel like it's worth continuing to build it into my stack.

[scefali]

@kentcdodds Are there any updates on this? I want to add cron jobs to my Epic Stack derived project.

[moishinetzer]

Can GitHub actions be used for scheduled non-repeating tasks?

i.e. 1 day after signing up, send an email to a user.

[kentcdodds]

You could do this by making a table to keep track of the tasks and then the cron can just trigger the tasks to be run. But I would hesitate to go too deep on this sort of thing a reality. Much past this point you'll start getting into relatively complicated things at which point it may make sense to use: https://www.inngest.com/

[kentcdodds]

Another thing I'm thinking is that if GitHub actions are sufficient for your use case then it may even be better to just have the cron running directly on the primary machine. Like some script that gets kicked off as part of the CMD in the dockerfile.

I do still need to evaluate. I do like to avoid unnecessary services, so I need to evaluate how wide a net of use cases a cron on the primary VM satisfies and whether it's wide enough to justify having the simpler version or if we should just go straight to inngest/trigger.

[WillSmithTE]

Just to add my 2c, I've used GitHub actions with an api like this in the past, and in terms of simplicity, cost and maintenance it's hard to beat.

The main issue was how unreliable the workflows were scheduled. For things like a daily schedule with a bit of buffer it was fine, but when you get down closer to the quickest cadence (a workflow every 5 minutes), you regularly get huge gaps. There were some days where my 5-10 cadence minute jobs would only run a few times per hour. Also definitely didn't work when I needed for example a notification sent out at exactly 10:00 to everyone.

I'd still use it again, because for a lot of cases that doesn't matter too much, but something to be aware of. They recommend scheduling things for "off-hours" when their servers will have less load, but on a schedule like 5-10 min I didn't find any difference between e.g. starting at 1:00 vs 1:03

[kentcdodds]

Thanks for sharing that experience @WillSmithTE! That's very helpful!

[wladpaiva]

If anyone is looking for a more robust solution, trigger.dev has been my go to recently

[scefali]

@wladpaiva How would that work with a LiteFS backed DB? Feels like a lot of these SaaS solutions don't work well with how Epic Stack is built.

[wladpaiva]

Well... Trigger.dev will only make a request to your server and your server will deal with the data layer so it doesn't actually matter how you access your data from the trigger.dev perspective

[jacobparis]

There's an open issue here for Trigger's long running server support triggerdotdev/trigger.dev#430

[andrecasal]

If you need something simple:

• Create a new Model in the database that can hold queued items
• Add items to be processed to the queue
• Install node-cron to run a function every X seconds/minutes and use it to check your queue

Works pretty well for me.

Example, if you want to send an email sequence:
prisma.schema:

model EmailSchedule {
   id          String   @id @default(cuid())
   createdAt   DateTime @default(now())
   to          String
   sequence    Int
   scheduledAt DateTime
   sent        Boolean  @default(false)

   @@unique([to, sequence])
}

On some action, call scheduleEmailSequence(email):

async function scheduleEmailSequence(email: string) {
	const now = new Date()
	const in1Day = new Date(now.getTime() + 1 * 24 * 3600 * 1000)
	const in2Days = new Date(now.getTime() + 2 * 24 * 3600 * 1000)
	const in3Days = new Date(now.getTime() + 3 * 24 * 3600 * 1000)
	const in4Days = new Date(now.getTime() + 4 * 24 * 3600 * 1000)
	const scheduleDates = [in1Day, in2Days, in3Days, in4Days]
	for (let i = 0; i < scheduleDates.length; i++) {
		const scheduledAt = scheduleDates[i]
		const sequence = i + 1
		await prisma.emailSchedule.upsert({
			where: {
				to_sequence: {
					to: email,
					sequence,
				},
			},
			update: {
				scheduledAt: scheduledAt,
			},
			create: {
				to: email,
				sequence,
				scheduledAt: scheduledAt,
			},
		})
	}
}

root.tsx:

new CronJob(
	'* * * * *', // cronTime
	async function () {
		// Check if there are emails in the queue
		const emailsToSend = await prisma.emailSchedule.findMany()
		// etc...
	}, // onTick
	null, // onComplete
	true, // start
	'America/Los_Angeles', // timeZone
) 

[kentcdodds]

Not everything needs complicated stuff and while this has lots of problems and limitations for some use cases, it's simple and functional for simpler use cases!

[ovistoica]

Can you detail the problems and limitations that could come from this approach?

From this thread, this looks like a reasonable starting point for most background jobs people would require for the epic stack

[thefrana]

In my experience, a robust job processing system requires several key features: retry mechanism for failed jobs, timeouts for stuck or long-running jobs, priority management for task scheduling, offloading of intensive tasks to separate processes, distributed locking to prevent duplicate job execution across multiple processes

[stephen776]

Has anyone had any experience with https://worker.graphile.org/ ?