feature(examples): Add JBoard connector

[fivertran-karunveluru]

Jboard Connector

Created: 2025-01-31

Business Owner: Talent Acquisition & Recruitment Operations Team

Technical Owner: Data Engineering Team

Last Updated: 2025-01-31

Business Context

• Data Source: Jboard API for comprehensive job board, employer, and recruitment data
• Business Criticality: High - supports talent acquisition, market intelligence, and recruitment analytics
• Data Consumers: Talent acquisition teams, recruitment specialists, HR analytics teams, executive leadership
• Business SLAs: Data must be fresh within 6 hours for active job searches, 24 hours for market intelligence reporting
• Compliance Requirements: GDPR compliance for candidate data, data privacy regulations for alert subscriptions
• Budget Constraints: Jboard API access based on subscription tier, rate limits based on plan level

Technical Context

• API Documentation: https://app.jboard.io/api/documentation
• Authentication Method: Bearer token authentication with API key
• Rate Limits: Varies by Jboard plan, typically 1000-5000 requests/hour
• Data Volume:
  • Employers: 1,000-100,000+ employer profiles per integration
  • Categories: 50-500+ job categories with hierarchical structure
  • Alert Subscriptions: 500-50,000+ active user subscriptions
• Data Velocity: Employer data updated on changes, categories updated weekly/monthly, alert subscriptions updated in real-time
• Data Quality: Structured JSON with consistent schema, some fields may be null for incomplete records
• Network Considerations: HTTPS only, RESTful API with standard reliability

Operational Context

• Deployment Environment: Development (sandbox), staging, and production environments
• Monitoring Requirements: Alert on >2% error rate, >3 hour sync time, missing employer or category data
• Maintenance Windows: Weekends for non-critical updates, immediate deployment for recruitment-critical fixes
• Team Structure: Data Engineering team, Talent Acquisition Operations, Recruitment Analytics, HR teams
• Escalation Path: Data Engineer → Team Lead → Talent Acquisition Director → CHRO

API-Specific Details

• Base Endpoint: https://app.jboard.io/api/v1 (production)
• Authentication: Bearer token in Authorization header (API key)
• Pagination: page and per_page parameters (max 100 per page)
• Date Format: ISO 8601 (e.g., 2024-01-15T10:30:00Z)
• Response Format: JSON with nested objects and arrays
• Key Endpoints:
  • /employers - Employer profiles, company information, and job posting metadata
  • /categories - Job categories with hierarchical structure and organization
  • /alert_subscriptions - User job alert subscriptions with search criteria and preferences

Data Schema Overview

• employers: Employer profiles with company details, logos, featured status, and job posting history
• categories: Job categories with hierarchical parent-child relationships and sorting metadata
• alert_subscriptions: User alert subscriptions with search queries, location filters, job type preferences, and category filters

Data Replication Expectations

• Initial Sync: Last 90 days of employer, category, and subscription data for baseline analysis
• Incremental Sync: Data since last successful sync timestamp using created_at_from filters
• Sync Frequency:
  • Production: Every 6 hours for employer data, daily for categories and subscriptions
  • Development: Daily for all data types
• Data Retention: 2 years of historical employer and subscription data for trend analysis
• Backfill Capability: Full historical data available based on Jboard retention policies
• Data Consistency: Near real-time with 6-hour maximum lag for active recruitment operations

Operational Requirements

• Uptime SLA: 99.5% availability during business hours
• Performance SLA:
  • Initial sync: <4 hours for 90 days of data
  • Incremental sync: <45 minutes for daily updates
• Error Handling:
  • Automatic retry with exponential backoff and jitter
  • Dead letter queue for failed employer or subscription records
  • Alert on consecutive sync failures during peak recruitment periods
• Monitoring:
  • API response times and error rates
  • Employer count trends and anomaly detection
  • Subscription activity and engagement metrics
  • Category hierarchy completeness validation
• Security:
  • API keys rotated every 90 days
  • Access logs maintained for 2 years
  • User email privacy handling per data regulations

Rate Limiting Strategy

• Starter Plan: 1,000 requests/hour, 10,000 requests/day
• Professional Plan: 2,500 requests/hour, 25,000 requests/day
• Enterprise Plan: 5,000 requests/hour, 50,000 requests/day
• Recommended: Implement exponential backoff with jitter for 429 responses
• Error Handling: 429 status code indicates rate limit exceeded, respect Retry-After header
• Monitoring: Track rate limit utilization and plan for subscription upgrades

Data Quality Considerations

• Required Fields: employer_id, category_id, subscription_id, name, email (for subscriptions)
• Optional Fields: description, website, logo_url, parent_id (categories), tags, location
• Data Validation:
  • Employer IDs must be unique
  • Category IDs must be unique within hierarchy
  • Email addresses must be valid format for alert subscriptions
  • Website URLs must be valid format
  • Tags arrays properly serialized to JSON strings
• Data Completeness:
  • Employers: 95%+ have basic profile information
  • Categories: 100% have name and structure data
  • Alert Subscriptions: 90%+ have search criteria defined
• Duplicate Handling: Primary key constraints prevent duplicate records

Integration Points

• Fivetran Destinations: Snowflake, BigQuery, Redshift, PostgreSQL
• Downstream Systems:
  • Applicant tracking systems (ATS)
  • Recruitment analytics platforms
  • Talent intelligence systems
  • HR information systems (HRIS)
  • Market intelligence dashboards
• Data Dependencies: None - standalone job board data source
• External Dependencies: Jboard API availability, job posting update frequency

Disaster Recovery

• Backup Strategy: Daily snapshots of all employer, category, and subscription tables
• Recovery Time Objective: 6 hours for full data recovery
• Recovery Point Objective: 4 hours maximum data loss for recruitment-critical data
• Failover: Automatic failover to backup API credentials
• Testing: Monthly disaster recovery drills with Talent Acquisition team validation

Compliance & Security

• Data Classification: User email addresses - sensitive PII, employer data - business sensitive
• Retention Policy: 2 years for recruitment analytics, 1 year for operational data
• Access Controls: Strict role-based access with principle of least privilege
• Audit Trail: All data access logged and monitored for compliance audits
• Encryption: Data encrypted in transit and at rest with enterprise-grade security
• Privacy: GDPR compliance for EU users, CCPA compliance for CA users, email privacy protection

Performance Optimization

• Parallel Processing: Multiple API calls for different data types (employers, categories, subscriptions)
• Caching: Category hierarchy cached for 24 hours
• Indexing: Employer ID, category ID, subscription ID, email, and date columns indexed
• Partitioning: Subscription data partitioned by creation date for efficient querying
• Compression: Historical subscription data compressed for storage efficiency
• Streaming: Memory-efficient generator patterns prevent data accumulation

Troubleshooting Guide

• Common Issues:
  • Rate limit exceeded: Reduce sync frequency or upgrade Jboard plan
  • API key expired: Verify key validity and permissions
  • Missing employer data: Check API key permissions and employer active status
  • Timeout errors: Increase timeout values or reduce batch size
  • Category hierarchy issues: Validate parent_id relationships and sort_order
  • Subscription email format errors: Verify email validation and privacy handling
• Debug Mode: Enable detailed logging for employer and subscription data troubleshooting
• Support Contacts:
  • Technical: Data Engineering team
  • Business: Talent Acquisition Operations team
  • Vendor: Jboard support (for API and account issues)
  • Compliance: Legal/Compliance team (for privacy and regulatory issues)

Checklist

Some tips and links to help validate your PR:

• Tested the connector with fivetran debug command.
• Added/Updated example specific README.md file, refer here for template.
• Followed Python Coding Standards, refer here

[github-actions]

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.