Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
439158a
chore(release): backmerge main into develop [skip ci]
lerian-studio Jun 23, 2026
bac6a4e
chore(changelog): backmerge main into develop [skip ci]
lerian-studio Jun 23, 2026
3643e66
fix(ci): upgrade shared-workflows to v1.37.0
bedatty Jun 24, 2026
5dcad9a
fix(ci): upgrade shared-workflows to v1.37.0 (#290)
bedatty Jun 24, 2026
1b6d7ed
chore(release): 2.0.3-beta.1
lerian-studio Jun 24, 2026
230154c
fix(ci): upgrade shared-workflows to v1.40.3
bedatty Jun 24, 2026
b50e0a2
fix(ci): upgrade shared-workflows to v1.39.0 (#299)
bedatty Jun 25, 2026
a204bf1
chore(release): 2.0.3-beta.2
lerian-studio Jun 25, 2026
e94ecd6
fix(pkg): correct plugin_crm datasource mapping
brunobls Jul 3, 2026
0fd07f5
fix(worker): always decrypt plugin_crm PII fields
brunobls Jul 3, 2026
16d1e48
feat(pkg): add per-service tenant-manager API keys
brunobls Jul 3, 2026
1f65755
test(tests): allow insecure TLS in e2e app env
brunobls Jul 3, 2026
904a9bb
feat: forward port maintenance to develop (#305)
brunobls Jul 6, 2026
48a8fff
chore(release): 2.1.0-beta.1
lerian-studio Jul 6, 2026
4766104
chore(deps): bump the go-dependencies group across 1 directory with 8…
dependabot[bot] Jul 6, 2026
149d842
chore(deps): bump the go-dependencies group across 1 directory with 8…
brunobls Jul 6, 2026
465fb1b
chore(release): 2.1.0-beta.2
lerian-studio Jul 6, 2026
5831e98
fix(ci): restrict PR source branches to hotfix and release-candidate
bedatty Jul 14, 2026
de58f69
fix(ci): restrict PR source branches to hotfix and release-candidate …
bedatty Jul 14, 2026
ed92b21
feat(manager)!: adopt lib-commons Huma wrapper
fredcamaral Jul 14, 2026
af7fdee
docs(docs): document Huma OpenAPI workflow
fredcamaral Jul 14, 2026
ce84443
fix(manager): document OpenAPI artifact permissions
fredcamaral Jul 14, 2026
e7c8a5e
fix(worker): convert Mongo pool size safely
fredcamaral Jul 14, 2026
24b44bd
fix(pkg): document controlled security inputs
fredcamaral Jul 14, 2026
71b2ae8
fix(tests): bound fuzz byte mutation
fredcamaral Jul 14, 2026
be8e7ec
fix(manager): address CodeRabbit review findings
fredcamaral Jul 14, 2026
c5b007f
build(infra): update Go builders to 1.26.5
fredcamaral Jul 14, 2026
571800d
build(deps): update lib-streaming to v1.8.0
fredcamaral Jul 14, 2026
4fccc37
feat(manager)!: adopt lib-commons Huma wrapper (#314)
fredcamaral Jul 14, 2026
b807691
chore(release): 3.0.0-beta.1
lerian-studio Jul 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/pr-validation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ jobs:
validate:
uses: LerianStudio/github-actions-shared-workflows/.github/workflows/go-pr-validation.yml@v1.36.5
with:
enforce_source_branches: true
allowed_source_branches: 'hotfix/*|release-candidate'
target_branches_for_source_check: 'main'
pr_title_scopes: |
manager
worker
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ permissions:

jobs:
pipeline:
uses: LerianStudio/github-actions-shared-workflows/.github/workflows/go-release.yml@v1.36.5
uses: LerianStudio/github-actions-shared-workflows/.github/workflows/go-release.yml@v1.40.3
with:
enable_changelog: ${{ github.ref == 'refs/heads/main' }}
release_single_app: true
Expand Down
15 changes: 8 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Lerian Fetcher is an enterprise data extraction platform that unifies access to
```bash
make set-env # Copy .env.example to .env for all components
make generate-master-key # Generate APP_ENC_KEY (REQUIRED before services start)
make dev-setup # Install tools (golangci-lint, swag, mockgen, gosec) + tidy
make dev-setup # Install tools (golangci-lint, mockgen, gosec) + tidy
```

### Build & Run
Expand Down Expand Up @@ -44,7 +44,8 @@ make lint # golangci-lint with --fix (28 linters, max cyclomatic compl
make format # go fmt
make tidy # go mod tidy
make sec # gosec security analysis
make generate-docs # Regenerate Swagger docs (swag init)
make generate-docs # Regenerate the canonical Huma OpenAPI 3.1 specification
make check-openapi # Fail when the committed OpenAPI specification is stale
```

## Architecture
Expand Down Expand Up @@ -115,7 +116,7 @@ Async RabbitMQ consumer. Does NOT follow CQRS - uses a single `UseCase` struct i

**CQRS services (Manager only):** Struct with `Execute()` method. Constructor `NewXxx(deps...)`. Dependencies are port interfaces, never concrete implementations.

**HTTP handlers:** Thin handlers in `internal/adapters/http/in/` with Swagger annotations (swaggo). Extract org ID via `httpUtils.GetOrganizationID(c)`, product name from `X-Product-Name` header.
**HTTP operations:** Typed Huma callbacks in `internal/adapters/http/in/huma_*.go`, registered through the single runtime/spec assembly. Preserve the request context installed by auth/tenant middleware and read the product name from the `X-Product-Name` header when required. Return errors through the lib-commons RFC 9457 problem model.

**Error handling:** Use `pkg.ValidateBusinessError(constant.ErrXxx, "entityType", args...)` for business errors. Custom types in `pkg/errors.go` map to HTTP status codes.

Expand Down Expand Up @@ -165,19 +166,19 @@ Async RabbitMQ consumer. Does NOT follow CQRS - uses a single `UseCase` struct i
| `components/manager/README.md` | Manager service (HTTP control plane, runs over the engine) |
| `components/worker/README.md` | Worker service (async data plane, runs over the engine) |
| `tests/e2e/README.md` | E2E test conventions, patterns, test catalog |
| `components/manager/api/swagger.yaml` | OpenAPI specification |
| `components/manager/api/openapi.yaml` | Canonical OpenAPI 3.1 specification |
| `components/manager/api/requests.http` | API request examples for manual testing |
| `.golangci.yml` | Full linter configuration (28 linters) |

## Common Workflows

### Adding a new API endpoint

1. Create handler in `components/manager/internal/adapters/http/in/` with Swagger annotations
2. Register route in the routes file
1. Add the typed callback and operation metadata in the appropriate `components/manager/internal/adapters/http/in/huma_*.go` file
2. Register the operation through the shared Huma assembly in `huma_operations.go`
3. Create command or query service in `internal/services/command/` or `internal/services/query/`
4. Wire dependencies in `internal/bootstrap/`
5. Run `make generate-docs`
5. Run `make generate-docs` and `make check-openapi`

### Adding a new database adapter

Expand Down
44 changes: 20 additions & 24 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,8 @@ help:
@echo ""
@echo ""
@echo "Documentation Commands:"
@echo " make generate-docs - Generate Swagger documentation"
@echo " make generate-docs - Generate the canonical Huma OpenAPI 3.1 specification"
@echo " make check-openapi - Fail when the committed OpenAPI specification is stale"
@echo ""
@echo ""
@echo "Test Suite Aliases:"
Expand Down Expand Up @@ -230,8 +231,8 @@ set-env:
# Deterministic, non-live quality gate. Mirrors the Lerian CI pattern
# (cf. plugin-br-bank-transfer) restricted to steps that need no Docker
# daemon, no live infrastructure, and no GITHUB_TOKEN — so it runs the
# same locally and in CI. Docker-bound steps (generate-docs, test-e2e,
# test-chaos) stay available standalone and run in their own CI stages.
# same locally and in CI. Docker-bound steps (test-e2e, test-chaos) stay
# available standalone and run in their own CI stages.
.PHONY: ci
ci:
$(call print_title,Running full CI pipeline)
Expand All @@ -240,6 +241,7 @@ ci:
@$(MAKE) vet
@$(MAKE) lint
@$(MAKE) sec
@$(MAKE) check-openapi
@$(MAKE) test-unit
@echo "[ok] CI pipeline passed"

Expand Down Expand Up @@ -795,20 +797,21 @@ ps:

.PHONY: generate-docs
generate-docs:
$(call print_title,Generating Swagger API documentation)
@if ! command -v swag >/dev/null 2>&1; then \
echo "Installing swag..."; \
go install github.com/swaggo/swag/cmd/swag@latest; \
fi
@swag init -g ./components/manager/cmd/app/main.go -d ./ -o ./components/manager/api --parseDependency --parseInternal
@docker run --rm -v $(ROOT_DIR):/local --user $(shell id -u):$(shell id -g) openapitools/openapi-generator-cli:v5.1.1 generate -i /local/components/manager/api/swagger.json -g openapi-yaml -o /local/components/manager/api
@mv ./components/manager/api/openapi/openapi.yaml ./components/manager/api/openapi.yaml
@rm -rf ./components/manager/api/README.md ./components/manager/api/.openapi-generator* ./components/manager/api/openapi
@if [ -f "$(ROOT_DIR)/scripts/package.json" ]; then \
echo "Installing npm dependencies for validation..."; \
cd $(ROOT_DIR)/scripts && npm install > /dev/null; \
fi
@echo "[ok] Swagger API documentation generated successfully"
$(call print_title,Generating canonical Huma OpenAPI specification)
@go run ./components/manager/cmd/huma-spec -output ./components/manager/api/openapi.yaml
@echo "[ok] OpenAPI 3.1 specification generated successfully"

.PHONY: check-openapi
check-openapi:
$(call print_title,Checking committed OpenAPI specification)
@tmp=$$(mktemp); \
trap 'rm -f "$$tmp"' EXIT; \
go run ./components/manager/cmd/huma-spec -output "$$tmp" || { echo "OpenAPI specification generation failed"; exit 1; }; \
cmp -s "$$tmp" ./components/manager/api/openapi.yaml || { \
echo "OpenAPI specification is stale; run 'make generate-docs'"; \
exit 1; \
}
@echo "[ok] Committed OpenAPI specification is current"

#-------------------------------------------------------
# Developer Helper Commands
Expand All @@ -822,10 +825,6 @@ dev-setup:
echo "Installing golangci-lint..."; \
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest; \
fi
@if ! command -v swag >/dev/null 2>&1; then \
echo "Installing swag..."; \
go install github.com/swaggo/swag/cmd/swag@latest; \
fi
@if ! command -v mockgen >/dev/null 2>&1; then \
echo "Installing mockgen..."; \
go install go.uber.org/mock/mockgen@latest; \
Expand Down Expand Up @@ -1020,6 +1019,3 @@ generate-master-key: ## Generate a new cryptographically secure master key
echo "" && \
echo "IMPORTANT: Store this key securely. It cannot be recovered if lost." && \
echo "Add to your environment as APP_ENC_KEY."



7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,14 +232,15 @@ Lerian Fetcher is built as a cloud-native platform following Hexagonal Architect
| `GET` | `/health` | Liveness check |
| `GET` | `/readyz` | Readiness check — parallel dependency probes (MongoDB, RabbitMQ, Redis; S3 on Worker), returns 503 while draining on SIGTERM |
| `GET` | `/version` | Version info |
| `GET` | `/swagger/*` | Swagger UI |
| `GET` | `/swagger/docs` | Scalar API reference (`SWAGGER_ENABLED=true`) |
| `GET` | `/swagger/openapi.{json,yaml}` | OpenAPI 3.1 contract (`SWAGGER_ENABLED=true`) |

### API Reference & Testing

For hands-on API exploration and testing scenarios, the following resources are available:

- **[`components/manager/api/requests.http`](components/manager/api/requests.http)**: Ready-to-use HTTP request examples covering all API endpoints — useful for quick testing with VS Code REST Client, IntelliJ, or similar tools.
- **[`components/manager/api/swagger.yaml`](components/manager/api/swagger.yaml)**: Full OpenAPI specification for the Manager API, which can be imported into Postman, Insomnia, or any OpenAPI-compatible tool.
- **[`components/manager/api/openapi.yaml`](components/manager/api/openapi.yaml)**: Canonical OpenAPI 3.1 specification for the Manager API, generated from the same Huma assembly used at runtime.
- **[`tests/e2e/`](tests/e2e/)**: End-to-end test suite covering connection management, data extraction across all supported databases, filtering, multi-datasource/multi-schema scenarios, schema validation, and error handling. These tests serve as practical usage examples and can be referenced to understand expected behaviors and edge cases.

### Technical Highlights
Expand Down Expand Up @@ -315,7 +316,7 @@ Single-tenant deployments emit with stable tenant ID `single-tenant`; multi-tena

5. **Access the API:**
- REST API: `http://localhost:4006`
- Swagger UI: `http://localhost:4006/swagger/index.html`
- Scalar API reference (when enabled): `http://localhost:4006/swagger/docs`
- RabbitMQ Management: `http://localhost:3008`

### Security
Expand Down
16 changes: 8 additions & 8 deletions components/manager/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,7 @@ RABBITMQ_HEALTH_CHECK_URL=http://${RABBITMQ_HOST}:${RABBITMQ_PORT_HOST}
#RABBITMQ_TLS=false

# SWAGGER
SWAGGER_TITLE='Fetcher'
SWAGGER_DESCRIPTION='Documentation for fetcher'
SWAGGER_VERSION=${VERSION}
SWAGGER_HOST=${SERVER_ADDRESS}
SWAGGER_BASE_PATH=/
SWAGGER_SCHEMES=http
SWAGGER_LEFT_DELIM={{
SWAGGER_RIGHT_DELIM=}}
SWAGGER_ENABLED=false

# AUTHORIZATION
PLUGIN_AUTH_ENABLED=false
Expand Down Expand Up @@ -87,6 +80,13 @@ MULTI_TENANT_ENABLED=false
#MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD=5
#MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC=30
#MULTI_TENANT_SERVICE_API_KEY=
# Per-service tenant-manager API keys (optional). Scheme:
# MULTI_TENANT_SERVICE_API_KEY_<SERVICE> where <SERVICE> = the service name, hyphens->underscores, upper-cased.
# When set, the resolver authenticates that service's tenant-config lookups with the matching key;
# falls back to MULTI_TENANT_SERVICE_API_KEY when a per-service key is absent.
# Adding a new service is env-only (no code change). Two env vars normalizing to the same token fail fast at startup.
#MULTI_TENANT_SERVICE_API_KEY_LEDGER=
#MULTI_TENANT_SERVICE_API_KEY_PLUGIN_CRM=
#MULTI_TENANT_CACHE_TTL_SEC=120
#MULTI_TENANT_TIMEOUT=30
#MULTI_TENANT_ALLOW_INSECURE_HTTP=false
Expand Down
2 changes: 1 addition & 1 deletion components/manager/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# syntax=docker/dockerfile:1.4

FROM golang:1.26.4-alpine AS builder
FROM golang:1.26.5-alpine AS builder

# Install required packages: git is needed for private modules
RUN apk add --no-cache git
Expand Down
14 changes: 8 additions & 6 deletions components/manager/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ info:
@echo " "
@echo "## App commands:"
@echo " "
@echo " COMMAND=\"generate-docs\" Generates Swagger API documentation and an OpenAPI Specification."
@echo " COMMAND=\"generate-docs\" Generates the canonical Huma OpenAPI 3.1 specification."
@echo " COMMAND=\"check-openapi\" Fails when the committed OpenAPI specification is stale."
@echo " "
@echo " "

Expand Down Expand Up @@ -85,10 +86,11 @@ ps:

.PHONY: generate-docs
generate-docs:
@swag init -g ./cmd/app/main.go -d ./ -o ./api --parseDependency --parseInternal
@docker run --rm -v $(pwd):/local --user $(shell id -u):$(shell id -g) openapitools/openapi-generator-cli:v5.1.1 generate -i ./manager/api/swagger.json -g openapi-yaml -o ./manager/api
@mv ./api/openapi/openapi.yaml ./api/openapi.yaml
@rm -rf ./api/README.md ./api/.openapi-generator* ./api/openapi
@$(MAKE) -C ../.. generate-docs

.PHONY: check-openapi
check-openapi:
@$(MAKE) -C ../.. check-openapi

.PHONY: setup-git-hooks
setup-git-hooks:
Expand Down Expand Up @@ -149,4 +151,4 @@ test:
set-env:
@echo "$(BLUE)Setting up environment files...$(NC)"
cp -r ./.env.example ./.env
@echo "$(BLUE)Environment files created successfully$(NC)"
@echo "$(BLUE)Environment files created successfully$(NC)"
12 changes: 7 additions & 5 deletions components/manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The Manager is the Fetcher HTTP API server. It runs the synchronous control plane: connection CRUD, schema discovery and validation, connection testing, idempotency, rate limiting, license enforcement, and job dispatch onto RabbitMQ. The Worker does the actual extraction asynchronously.

The Manager is built on [Fiber](https://gofiber.io) and serves on port `4006`. It follows Hexagonal Architecture with CQRS: commands mutate state, queries read it, and both depend only on port interfaces.
The Manager is built on [Fiber](https://gofiber.io) with typed [Huma](https://huma.rocks) operations and serves on port `4006`. The same assembly drives runtime routing and the canonical OpenAPI 3.1 contract. It follows Hexagonal Architecture with CQRS: commands mutate state, queries read it, and both depend only on port interfaces.

## Table of Contents

Expand Down Expand Up @@ -37,7 +37,7 @@ The Manager wires **two** engine instances, each a focused authority:

```mermaid
graph TB
subgraph Manager["Manager (Fiber, port 4006)"]
subgraph Manager["Manager (Fiber + Huma, port 4006)"]
Handlers["HTTP handlers\ninternal/adapters/http/in"]
CmdQ["CQRS commands & queries\ninternal/services"]
ConnEng["connection engine\nbootstrap/connection_engine.go"]
Expand Down Expand Up @@ -121,7 +121,7 @@ graph LR
Qry -. delegates rules .-> Engine
```

- **Inbound adapters** (`internal/adapters/http/in/`) are thin Fiber handlers with Swagger annotations. They extract the organization ID via `httpUtils.GetOrganizationID(c)` and the product name from the `X-Product-Name` header, then call a service.
- **Inbound adapters** (`internal/adapters/http/in/`) expose typed Huma callbacks over Fiber. Operation metadata, request/response schemas, and RFC 9457 errors come from the same assembly used to generate OpenAPI 3.1. The callbacks preserve the auth/tenant request context, read the `X-Product-Name` header when required, then call a service.
- **Commands** (`internal/services/command/`) handle connection Create, Update, and Delete, plus job creation (`CreateFetcherJob`) and connection assignment (`AssignConnection`).
- **Queries** (`internal/services/query/`) handle Get, List, Test, and Validate.
- **Bootstrap** (`internal/bootstrap/`) wires dependencies, including the two engines, and holds configuration.
Expand Down Expand Up @@ -191,14 +191,14 @@ License enforcement is **fail-closed**: when `DEPLOYMENT_MODE` is not `local`, t
| Path | Contents |
|------|----------|
| `cmd/app/main.go` | Entry point |
| `internal/adapters/http/in/` | Fiber HTTP handlers (with Swagger annotations) |
| `internal/adapters/http/in/` | Typed Huma operations and Fiber integration |
| `internal/services/command/` | CQRS commands (Create, Update, Delete) |
| `internal/services/query/` | CQRS queries (Get, List, Test, Validate) |
| `internal/adapters/cache/` | Redis schema cache adapter |
| `internal/bootstrap/` | Dependency injection, config, and engine wiring |
| `internal/bootstrap/connection_engine.go` | Connection engine wiring |
| `internal/bootstrap/schema_engine.go` | Schema engine wiring |
| `api/swagger.yaml` | OpenAPI specification |
| `api/openapi.yaml` | Canonical OpenAPI 3.1 specification |
| `api/requests.http` | Manual API request examples |

## Configuration
Expand All @@ -218,3 +218,5 @@ make generate-master-key
| other non-`local` | Enforced (fail-closed) | — |

See `internal/bootstrap/config.go` for the full configuration surface and `../../README.md` for the project-wide quick start.

Set `SWAGGER_ENABLED=true` to expose the Scalar reference at `/swagger/docs` and the OpenAPI 3.1 contract at `/swagger/openapi.json` or `/swagger/openapi.yaml`. The routes are absent when the flag is disabled.
Loading
Loading