Closes #675 Implemented Webhooks Engine - #691
Open
p3ris0n wants to merge 7 commits into
Open
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| * @returns {array} 200 - Array of webhook deliveries | ||
| * @returns {object} 404 - Webhook not found | ||
| */ | ||
| router.get('/webhooks/:id/deliveries', authenticate, asyncHandler(async (req, res) => { |
Comment on lines
+104
to
+124
| router.get('/webhooks/:id/deliveries', authenticate, asyncHandler(async (req, res) => { | ||
| const webhook = await Webhook.findOne({ | ||
| _id: req.params.id, | ||
| ownerPublicKey: req.user.publicKey, | ||
| }); | ||
|
|
||
| if (!webhook) throw new AppError('Webhook not found', 404, 'NOT_FOUND'); | ||
|
|
||
| const { status, limit = 50, skip = 0 } = req.query; | ||
| const query = { webhookId: webhook._id }; | ||
| if (status) { | ||
| query.status = status; | ||
| } | ||
|
|
||
| const deliveries = await WebhookDelivery.find(query) | ||
| .sort({ createdAt: -1 }) | ||
| .skip(Number(skip)) | ||
| .limit(Number(limit)); | ||
|
|
||
| res.json({ success: true, data: deliveries }); | ||
| })); |
| query.status = status; | ||
| } | ||
|
|
||
| const deliveries = await WebhookDelivery.find(query) |
| * @returns {object} 200 - Success confirmation | ||
| * @returns {object} 404 - Delivery not found or not in DLQ | ||
| */ | ||
| router.post('/webhooks/deliveries/:deliveryId/retry', authenticate, asyncHandler(async (req, res) => { |
Comment on lines
+137
to
+161
| router.post('/webhooks/deliveries/:deliveryId/retry', authenticate, asyncHandler(async (req, res) => { | ||
| const delivery = await WebhookDelivery.findById(req.params.deliveryId).populate('webhookId'); | ||
| if (!delivery) throw new AppError('WebhookDelivery not found', 404, 'NOT_FOUND'); | ||
|
|
||
| if (delivery.webhookId.ownerPublicKey !== req.user.publicKey) { | ||
| throw new AppError('Unauthorized access to WebhookDelivery', 403, 'FORBIDDEN'); | ||
| } | ||
|
|
||
| if (delivery.status !== 'DLQ' && delivery.status !== 'FAILED') { | ||
| throw new AppError('Only FAILED or DLQ deliveries can be manually retried', 400, 'BAD_REQUEST'); | ||
| } | ||
|
|
||
| const { webhookQueue } = require('../services/webhook-queue'); | ||
|
|
||
| // Reset status to PENDING and attempts to 0 for a fresh retry schedule | ||
| delivery.status = 'PENDING'; | ||
| delivery.attempts = 0; | ||
| await delivery.save(); | ||
|
|
||
| await webhookQueue.add('webhookDelivery', { | ||
| deliveryId: String(delivery._id), | ||
| }); | ||
|
|
||
| res.json({ success: true, message: 'Delivery queued for retry' }); | ||
| })); |
kei, see wetin them do this codebase
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature: Webhook Engine with BullMQ and Exponential Backoff
Closes #675
Summary
Refactored the webhook dispatch system to use a robust, queue-based architecture powered by BullMQ. This update ensures reliable webhook delivery by implementing exponential backoff for failed requests and introducing a Dead-Letter Queue (DLQ) for dispatches that exhaust their retry attempts.
Key Changes
Database & Models
WebhookDelivery Model: Added a new schema to track individual webhook dispatches, including their payload, number of attempts, status (PENDING, SUCCESS, FAILED, DLQ), and detailed error logs.
Webhook Model: Fixed a schema bug caused by duplicate enum declarations in the events field.
Architecture & Queueing
BullMQ Integration (server/services/webhook-queue.js):
Initialized webhookQueue connected to the existing Redis instance.
Built a dedicated worker that fetches pending WebhookDelivery records, executes the HTTP request, and updates the delivery status.
Configured automatic exponential backoff (up to 5 attempts, starting with a 5-minute interval).
Added listeners to catch fully exhausted jobs and update their delivery status to DLQ.
Service Refactoring (server/services/webhook-service.js):
Replaced the naive synchronous retry loop in the dispatch function. It now asynchronously creates a WebhookDelivery record and pushes a job to BullMQ, ensuring the server doesn't block on network latency.
Server Initialization (server/index.js): Added a hook to initialize the worker on application startup.
Dashboard Features
Added GET /api/webhooks/:id/deliveries to fetch a webhook's delivery history.
Added POST /api/webhooks/deliveries/:deliveryId/retry to allow manual requeueing of FAILED or DLQ deliveries.
Testing
Added server/tests/webhook-engine.test.js to simulate webhook dispatches, verify database record creation, queue insertions, and test the worker's success and failure routines.
Type of Change
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Testing Instructions
Ensure your local .env has a valid REDIS_URL configured.
Run npm install in the server directory.
Run npx jest server/tests/webhook-engine.test.js to verify functionality.
Trigger a webhook locally and observe the background worker processing the job and updating the MongoDB document.