-
Notifications
You must be signed in to change notification settings - Fork 1
arc42 Documentation
Philip Jake edited this page Mar 5, 2025
·
1 revision
The Veganify API serves as the backend system for the Veganify application ecosystem, which helps users identify vegan and non-vegan products and ingredients. The primary functional requirements include:
- Product Information: Retrieve vegan status and details about products via barcode scanning
- Ingredient Analysis: Determine if individual ingredients or ingredient lists are vegan
- Cruelty-Free Verification: Check if brands/companies are on PETA's cruelty-free list
- Translation Services: Process ingredient lists in multiple languages
The API primarily serves client applications like the Veganify web interface, providing consistent and reliable data about product ingredients and their vegan status.
| Priority | Quality Goal | Scenario |
|---|---|---|
| 1 | Reliability | The API must provide consistent and accurate information about the vegan status of products and ingredients |
| 2 | Performance | API requests should be processed within reasonable time frames (< 1s for most operations) |
| 3 | Maintainability | The system should be modular, allowing for easy updates and extensions |
| 4 | Scalability | The API should handle growth in user base and request volume without degradation |
| 5 | Security | The system should protect against abuse through rate limiting and other measures |
| Role | Contact | Expectations |
|---|---|---|
| End Users | Accurate and quick information about products' vegan status | |
| Frontend Developers | FrontendNetwork | Stable API with clear documentation |
| API Maintainers | FrontendNetwork, Veganify Contributors | Maintainable codebase, CI/CD pipeline |
| Project Manager | [email protected] | System stability, feature implementation progress |
| External Data Providers | OpenFoodFacts, PETA, etc. | Proper usage of their data |
| Constraint | Description |
|---|---|
| NestJS Framework | The API is built using NestJS, a progressive Node.js framework |
| TypeScript | The codebase must use TypeScript for type safety |
| Containerization | The application must be deployable as Docker containers |
| Kubernetes | Deployment is managed through Kubernetes orchestration |
| Constraint | Description |
|---|---|
| Open Source | The project follows an open source model under MIT license |
| CI/CD Pipeline | All changes must pass through GitHub Actions workflow |
| Review Process | Pull requests require review from maintainers |
| Security Requirements | Must adhere to defined security policies in SECURITY.md |
The Veganify API operates within an ecosystem of services related to vegan product verification:
flowchart TD
User[End Users] -->|Uses| WebApp(Veganify Mobile App)
WebApp -->|API Requests| VeganifyAPI
VeganifyAPI -->|Product Data Lookup| OpenFoodFacts[OpenFoodFacts API]
VeganifyAPI -->|Ingredient Translation| DeepL[DeepL Translation API]
VeganifyAPI -->|Cruelty-Free Verification| PETA[PETA Database]
VeganifyAPI -->|Product Grade Lookup| GradesDB[Grades Database]
VeganifyAPI -->|EAN Lookup| OpenEANDB[OpenEAN Database]
The Veganify API serves as the central integration point, taking requests from client applications and coordinating with external data sources to provide comprehensive vegan product information.
flowchart TD
subgraph "Client Applications"
WebApp[Veganify Web App]
ThirdParty[Third-party Integrations]
end
subgraph "Veganify API Infrastructure"
HTTPS[HTTPS Endpoints]
K8s[Kubernetes Cluster]
APIService[API Service]
APIDocumentation[Swagger Documentation]
end
subgraph "External Services"
DeepLAPI[DeepL API]
OpenFoodFactsAPI[OpenFoodFacts REST API]
PushoverAPI[Pushover Notification Service]
PETADatabase[PETA Database]
OpenEAN[OpenEAN Database]
end
WebApp -->|HTTPS| HTTPS
ThirdParty -->|HTTPS| HTTPS
HTTPS --> K8s
K8s --> APIService
K8s --> APIDocumentation
APIService -->|HTTPS| DeepLAPI
APIService -->|HTTPS| OpenFoodFactsAPI
APIService -->|HTTPS| PushoverAPI
APIService -->|HTTPS| PETADatabase
APIService -->|HTTPS| OpenEAN
| Connection | Protocol | Description |
|---|---|---|
| Client → API | HTTPS | RESTful API requests over HTTPS |
| API → DeepL | HTTPS | Translation requests with authentication |
| API → OpenFoodFacts | HTTPS | Product database queries |
| API → Pushover | HTTPS | Notification for missing products |
| API → PETA Database | Local File | Cached PETA cruelty-free data (updated daily) |
| API → OpenEAN | HTTPS | Fallback product database |
The Veganify API employs the following key strategies to address its requirements and quality goals:
| Aspect | Strategy |
|---|---|
| Architecture Pattern | The system follows a modular architecture pattern using NestJS, with clear separation of controllers, services, and DTOs |
| Technology Stack | Node.js with TypeScript, NestJS framework, and Express |
| Key Quality Goals | - Reliability: Multiple data sources with fallback mechanisms - Performance: Efficient data processing and caching - Maintainability: Modular design with clear responsibilities - Scalability: Containerization and Kubernetes deployment |
| Organizational Decisions | - Continuous Integration/Deployment through GitHub Actions - Open source development model - Automated testing and linting - Daily updates for PETA data |
The core architectural approach separates the system into endpoints (controllers) that handle specific domains of the API, with dedicated services handling business logic and external integrations. This ensures that the system can evolve and scale independently in each area.
The Veganify API system consists of several interconnected building blocks organized as NestJS modules:
classDiagram
class AppModule {
+configure()
}
class RateLimiterMiddleware {
-limiter
+use()
}
class HealthController {
+check()
}
class HealthService {
+check()
}
class GradesController {
+checkBarcode()
}
class GradesService {
+checkBarcode()
+notifyMissingBarcode()
}
class ProductController {
+getProductDetails()
}
class ProductService {
+fetchProductDetails()
}
class IngredientsController {
-isNotVegan[]
-isVegan[]
+onModuleInit()
+getIngredients()
-loadAndPreprocessList()
-sophisticatedMatch()
-parseIngredients()
-sendResponse()
}
class IngredientsV1Controller {
-isNotVegan[]
-isVegan[]
-isMaybeNotVegan[]
+onModuleInit()
+getIngredients()
-loadAndPreprocessList()
-runWorker()
-parseIngredients()
-sendResponse()
}
class TranslationService {
+translateText()
}
class PetaController {
+getPetaByType()
+getPetaWithoutType()
}
class ErrorsController {
+getOpenApi()
+getSecurityTxt()
+handlePostWildcard()
+handleGetWildcard()
+handleMethodNotAllowed()
+handleOptions()
-handleWildcard()
}
AppModule --> RateLimiterMiddleware
AppModule --> HealthController
AppModule --> GradesController
AppModule --> ProductController
AppModule --> IngredientsController
AppModule --> IngredientsV1Controller
AppModule --> PetaController
AppModule --> ErrorsController
HealthController --> HealthService
GradesController --> GradesService
ProductController --> ProductService
IngredientsController --> TranslationService
IngredientsV1Controller --> TranslationService
| Building Block | Responsibility |
|---|---|
| AppModule | Main application module that configures and connects all components |
| RateLimiterMiddleware | Enforces API usage limits to prevent abuse |
| HealthController/Service | Monitors and reports on system health |
| GradesController/Service | Manages product grade information |
| ProductController/Service | Handles product lookups across multiple databases |
| IngredientsController | Processes ingredient lists to determine vegan status (v0) |
| IngredientsV1Controller | Enhanced ingredient processing with more detailed categorization (v1) |
| TranslationService | Interfaces with DeepL API for translating ingredients |
| PetaController | Provides access to PETA cruelty-free lists |
| ErrorsController | Manages API errors and fallback responses |
flowchart TD
subgraph "Ingredients Processing System"
Controller[IngredientsV1Controller]
Parser[Ingredients Parser]
TranslationService[Translation Service]
Worker[Worker Thread]
VeganDB[(Vegan\nIngredient List)]
NonVeganDB[(Non-Vegan\nIngredient List)]
MaybeNonVeganDB[(Maybe-Not-Vegan\nIngredient List)]
end
Client[Client Application] -->|Request| Controller
Controller --> Parser
Controller --> TranslationService
Controller --> Worker
Parser -->|Normalize| InputIngredients[Input Ingredients]
TranslationService -->|Translate| DeepL[DeepL API]
Worker -->|Load| VeganDB
Worker -->|Load| NonVeganDB
Worker -->|Load| MaybeNonVeganDB
Worker -->|Process| ProcessedIngredients[Processed Ingredients]
Controller -->|Format Response| Client
The Ingredients Processing System is one of the core components of the Veganify API. It:
- Parses and normalizes ingredient lists submitted by clients
- Optionally translates non-English ingredients to English for processing
- Utilizes a worker thread for efficient ingredient categorization
- Categorizes ingredients as vegan, non-vegan, maybe-not-vegan, or unknown
- Optionally translates results back to the original language
- Returns structured results to the client
flowchart TD
subgraph "Product Information System"
Controller[ProductController]
Service[ProductService]
subgraph "External Product Databases"
OpenFoodFacts[OpenFoodFacts API]
OpenEANDB[OpenEAN Database]
GradesAPI[Grades API]
PETAList[PETA Cruelty-Free]
end
end
Client[Client Application] -->|Barcode Request| Controller
Controller -->|Forward Request| Service
Service -->|Primary Lookup| OpenFoodFacts
Service -->|Fallback Lookup| OpenEANDB
Service -->|Grade Lookup| GradesAPI
Service -->|Cruelty-Free Check| PETAList
Service -->|Consolidate Results| Results[Product Results]
Results -->|Format Response| Client
The Product Information System is responsible for:
- Accepting product barcode queries
- Retrieving product information from multiple sources
- Determining vegan status based on product labels and ingredients
- Checking if the product's brand is cruelty-free
- Consolidating data from multiple sources into a comprehensive response
sequenceDiagram
participant Client
participant Controller as IngredientsController
participant TranslationSvc as TranslationService
participant DeepL as DeepL API
participant Worker as Ingredient Worker
Client->>Controller: GET /v1/ingredients/{ingredients}?translate=true
Controller->>Controller: Parse ingredients
alt translate is true
Controller->>TranslationSvc: translateText(ingredients, "EN", timeout)
TranslationSvc->>DeepL: API Request
DeepL-->>TranslationSvc: Translated text
TranslationSvc-->>Controller: Translated ingredients
Controller->>Worker: Process ingredients (translated)
Worker-->>Controller: Categorized ingredients
Controller->>TranslationSvc: Back-translate results to original language
TranslationSvc->>DeepL: API Request
DeepL-->>TranslationSvc: Translated text
TranslationSvc-->>Controller: Original language results
else translate is false
Controller->>Worker: Process ingredients (original)
Worker-->>Controller: Categorized ingredients
end
Controller->>Controller: Format response
Controller-->>Client: JSON Response with categorized ingredients
alt Translation service unavailable
Controller-->>Client: 503 Service Unavailable
end
sequenceDiagram
participant Client
participant ProdCtrl as ProductController
participant ProdSvc as ProductService
participant GradeSvc as GradesService
participant FoodDB as OpenFoodFacts
participant EANDB as OpenEANDB
participant PETA as PETA Database
Client->>ProdCtrl: POST /v0/product/{barcode}
ProdCtrl->>ProdSvc: fetchProductDetails(barcode)
ProdSvc->>GradeSvc: Check for product grade
GradeSvc-->>ProdSvc: Grade information (if available)
ProdSvc->>FoodDB: Query product data
alt Product found in OpenFoodFacts
FoodDB-->>ProdSvc: Product data
ProdSvc->>ProdSvc: Extract vegan/vegetarian status
ProdSvc->>PETA: Check if brand is cruelty-free
PETA-->>ProdSvc: Cruelty-free status
ProdSvc-->>ProdCtrl: Consolidated product data
ProdCtrl-->>Client: JSON Response with product details
else Product not found in OpenFoodFacts
FoodDB-->>ProdSvc: Not found
ProdSvc->>EANDB: Fallback query
alt Product found in OpenEANDB
EANDB-->>ProdSvc: Basic product data
ProdSvc->>ProdSvc: Determine vegan status from EAN category
ProdSvc-->>ProdCtrl: Consolidated product data
ProdCtrl-->>Client: JSON Response with product details
else Product not found anywhere
EANDB-->>ProdSvc: Not found
ProdSvc-->>ProdCtrl: 404 Not Found
ProdCtrl-->>Client: 404 Not Found
ProdSvc->>GradeSvc: notifyMissingBarcode()
GradeSvc->>GradeSvc: Send notification
end
end
sequenceDiagram
participant GitHubActions
participant PETAScript
participant PETAAPI as PETA Website
participant FileSystem
participant Git
GitHubActions->>PETAScript: Execute daily (cron job)
PETAScript->>PETAAPI: Fetch crueltyfree list
PETAAPI-->>PETAScript: HTML data
PETAScript->>PETAScript: Extract company names
PETAScript->>PETAScript: Process data (remove markers)
PETAScript->>FileSystem: Write to peta_cruelty_free.json
PETAScript->>Git: Commit changes
Git->>Git: Push to main branch
flowchart TD
subgraph "Client Side"
WebApp[Web App]
end
subgraph "CI/CD Pipeline"
GitHub[GitHub Repository]
GitHubActions[GitHub Actions]
GitHubPackages[(GitHub Container Registry)]
end
subgraph "Kubernetes Cluster"
subgraph "Veganify Namespace"
APIService[Veganify Backend Deployment]
APIService2[Veganify Backend Pod 2]
APIService3[Veganify Backend Pod 3]
end
subgraph "Veganify-Staging Namespace"
StagingService[Veganify Staging Backend]
end
end
subgraph "External Services"
DeepL[DeepL API]
OpenFoodFacts[OpenFoodFacts API]
Pushover[Pushover Notification]
GradesAPI[Grades API]
end
WebApp -->|HTTPS Requests| APIService
GitHub -->|Push to main/release| GitHubActions
GitHubActions -->|Build & Push| GitHubPackages
GitHubPackages -->|Deploy| APIService
GitHubPackages -->|Deploy to Staging| StagingService
APIService -->|Translation Requests| DeepL
APIService -->|Product Data| OpenFoodFacts
APIService -->|Notifications| Pushover
APIService -->|Product Grades| GradesAPI
StagingService -.->|Same Dependencies| DeepL
StagingService -.->|Same Dependencies| OpenFoodFacts
StagingService -.->|Same Dependencies| Pushover
StagingService -.->|Same Dependencies| GradesAPI
The Veganify API is deployed to a Kubernetes cluster with the following characteristics:
-
Production Environment: Multiple replicas of the API service are deployed in the
veganifynamespace for redundancy and load balancing -
Staging Environment: A separate deployment in the
veganify-stagingnamespace for testing new features - CI/CD Automation: GitHub Actions workflow builds Docker images and deploys them to the appropriate environment
flowchart TD
subgraph "Kubernetes Deployment"
subgraph "Veganify Backend Pod"
NestApp[NestJS Application]
ConfigMap[ConfigMap]
Secrets[Secrets]
end
Service[Kubernetes Service]
Ingress[Ingress Controller]
end
Client[Client Applications] -->|HTTPS| Ingress
Ingress -->|Route| Service
Service -->|Load Balance| NestApp
ConfigMap -->|Environment Config| NestApp
Secrets -->|API Keys| NestApp
NestApp -->|HTTP Requests| ExternalAPIs[External APIs]
Each Veganify API pod contains:
- The NestJS application container built from the Dockerfile
- Configuration mounted from ConfigMaps
- Secrets for API keys mounted from Kubernetes Secrets
The Kubernetes Service provides stable internal networking, while the Ingress Controller manages external access with TLS termination.
The Veganify API implements URL path-based versioning (e.g., /v0/ and /v1/) to maintain backward compatibility while allowing evolution of the API. This enables:
- Older client applications to continue using existing endpoints
- New features and improvements to be introduced in newer versions
- Gradual migration of clients to newer API versions
A consistent error handling approach is implemented across the API:
- HTTP status codes indicate the type of error
- Error responses follow a standardized JSON format
- Detailed error messages for developers
- Fallbacks for handling unavailable external services
To protect the API from abuse and ensure fair resource allocation:
- Requests are limited to 350 per minute per client
- Rate limit headers indicate usage and limits
- Clear error messages when limits are exceeded
The system uses a sophisticated algorithm for ingredient classification:
- Normalization of ingredient names (lowercase, whitespace removal)
- Matching against known vegan/non-vegan/maybe-not-vegan lists
- Word boundary checking to avoid partial matches
- Worker thread processing for efficient categorization
For multi-language support:
- Original ingredients are translated to English for processing
- Results are translated back to the original language
- Timeout mechanism prevents hanging on translation service issues
- Fallback to non-translated processing when translation is unavailable
Decision: Use NestJS as the primary framework for the API.
Rationale:
- Provides a structured, modular architecture based on decorators and dependency injection
- TypeScript support for type safety
- Built-in support for middleware, pipes, and guards
- Well-documented and maintained
Decision: Package the application as Docker containers.
Rationale:
- Consistent deployment across environments
- Isolation of dependencies
- Scalability through Kubernetes orchestration
- Simplified CI/CD pipeline
Decision: Use Node.js Worker Threads for ingredient classification.
Rationale:
- Prevents blocking the main event loop during intensive processing
- Improves performance for large ingredient lists
- Better utilization of multi-core systems
Decision: Integrate with multiple product databases (OpenFoodFacts, OpenEAN).
Rationale:
- Increased product coverage
- Fallback options when primary source lacks data
- Different databases have different strengths (food vs. cosmetics)
flowchart TD
Quality((Quality))
Reliability[Reliability]
Performance[Performance]
Maintainability[Maintainability]
Security[Security]
Usability[Usability]
Quality --- Reliability
Quality --- Performance
Quality --- Maintainability
Quality --- Security
Quality --- Usability
Reliability --- DataAccuracy[Data Accuracy]
Reliability --- Availability[Availability]
Reliability --- FaultTolerance[Fault Tolerance]
Performance --- ResponseTime[Response Time]
Performance --- Throughput[Throughput]
Performance --- Scalability[Scalability]
Maintainability --- Modularity[Modularity]
Maintainability --- Testability[Testability]
Maintainability --- Documentation[Documentation]
Security --- RateLimiting[Rate Limiting]
Security --- InputValidation[Input Validation]
Security --- SecureComms[Secure Communications]
Usability --- APIConsistency[API Consistency]
Usability --- Documentation[Clear Documentation]
Usability --- ErrorMessages[Helpful Error Messages]
| Quality Aspect | Scenario | Expected Response |
|---|---|---|
| Performance | The system receives a request to analyze 50 ingredients | Response is provided within 1 second |
| Reliability | An external API (OpenFoodFacts) is temporarily unavailable | System uses fallback data source (OpenEAN) and continues functioning |
| Scalability | Traffic increases by 300% during peak hours | System scales horizontally through Kubernetes to handle the increased load |
| Security | A client exceeds the rate limit | 429 status code is returned with clear information about limits |
| Maintainability | A new ingredient category needs to be added | The modular design allows for adding new categories without modifying existing code |
| Availability | One pod crashes due to an unexpected error | Kubernetes automatically restarts the pod, with minimal service interruption |
| Risk/Debt | Description | Mitigation |
|---|---|---|
| External API Dependencies | The system relies on several external APIs which may experience downtime | Implement caching, fallback mechanisms, and circuit breakers |
| Data Accuracy | Ingredient classification depends on complete and accurate lists | Regular updates to vegan/non-vegan lists and community contributions |
| Translation Limitations | DeepL may not correctly translate specialized ingredient terms | Maintain multi-language ingredient lists for common ingredients |
| Kubernetes Complexity | Kubernetes deployment adds operational complexity | Comprehensive documentation and automated deployment scripts |
| Rate Limit Bypass | Distributed clients could bypass rate limits | Consider adding API keys for authenticated access |
| Term | Definition |
|---|---|
| API | Application Programming Interface - endpoints that allow systems to communicate |
| Barcode | Unique product identifier (e.g., EAN, UPC) used to look up products |
| CI/CD | Continuous Integration/Continuous Deployment - automated build and deployment pipeline |
| Controller | NestJS component that handles HTTP requests |
| DeepL | Translation service used to process non-English ingredients |
| DTOs | Data Transfer Objects - structures defining data formats |
| Ingredients | Individual components or substances that make up a product |
| Kubernetes | Container orchestration system for automating deployment and scaling |
| NestJS | Node.js framework used to build the API |
| OpenFoodFacts | Open database of food products |
| PETA | People for the Ethical Treatment of Animals - source of cruelty-free data |
| Service | NestJS component containing business logic |
| Vegan | Free from animal products and not tested on animals |