feature(examples): Add Rotamaster connector

[fivertran-karunveluru]

Rotamaster API Connector

Created: 2025-10-31

Business Owner: HR & Workforce Operations Team

Technical Owner: Data Engineering Team

Last Updated: 2025-10-31

Business Context

• Data Source: Rotamaster API for employee, team, and role data management
• Business Criticality: High - supports workforce management, organizational structure tracking, and employee data analytics
• Data Consumers: HR teams, workforce managers, organizational planners, analytics teams, executive leadership
• Business SLAs: Data must be fresh within 6 hours for workforce planning, 24 hours for organizational reporting
• Compliance Requirements: GDPR compliance for employee data, data retention requirements for employment records
• Budget Constraints: Rotamaster API access based on subscription tier, rate limits vary by plan

Technical Context

• API Documentation: Rotamaster Data API (https://data-api.rotamasterweb.co.uk)
• Authentication Method: Bearer token (API key) authentication
• Rate Limits: Varies by Rotamaster plan, typically 500-5000 requests/hour
• Data Volume:
  • People: 50-10,000+ employees per organization
  • Teams: 5-500+ teams per organization
  • Roles: 10-200+ roles per organization
• Data Velocity: Employee data updated on changes, teams/roles updated less frequently (weekly/monthly)
• Data Quality: Structured JSON with consistent schema, some fields may be null for incomplete records
• Network Considerations: HTTPS only, RESTful API with standard reliability, UK-based infrastructure

Operational Context

• Deployment Environment: Development (sandbox), staging, and production environments
• Monitoring Requirements: Alert on >3% error rate, >3 hour sync time, employee data discrepancies
• Maintenance Windows: Weekends for non-critical updates, immediate deployment for workforce-critical fixes
• Team Structure: Data Engineering team, HR Operations, Workforce managers, Organizational planners
• Escalation Path: Data Engineer → Team Lead → HR Director → CHRO

API-Specific Details

• Base Endpoint: https://data-api.rotamasterweb.co.uk
• Authentication: Bearer token in Authorization header (API key)
• Pagination: numberOfRecords parameter (1-1000 per page, default 100)
• Date Format: ISO 8601 (e.g., 2024-01-15T10:30:00Z)
• Response Format: JSON with arrays of records
• Key Endpoints:
  • /api/People - Employee personal information, employment details, and status
  • /api/Team - Team and organizational group information
  • /api/Role - Role definitions and organizational hierarchy

Data Schema Overview

• people: Employee personal information, employment details, contact information, and employment status flags
• teams: Team and organizational group information with external references
• roles: Role definitions with nominal codes for organizational hierarchy

Data Replication Expectations

• Initial Sync: Last 90 days of employee data for baseline workforce snapshot (configurable up to 365 days)
• Incremental Sync: Data since last successful sync timestamp
• Sync Frequency:
  • Production: Every 6 hours for employee data, daily for teams/roles
  • Development: Daily for all data types
• Data Retention: 7 years of historical employee data for compliance requirements
• Backfill Capability: Full historical data available based on Rotamaster retention policies
• Data Consistency: Near real-time with 6-hour maximum lag for workforce operations

Operational Requirements

• Uptime SLA: 99.5% availability during business hours
• Performance SLA:
  • Initial sync: <4 hours for 90 days of employee data
  • Incremental sync: <1 hour for daily updates
• Error Handling:
  • Automatic retry with exponential backoff (3 attempts default, configurable)
  • Dead letter queue for failed employee records
  • Alert on consecutive sync failures during critical workforce periods
• Monitoring:
  • API response times and error rates
  • Employee count trends and anomaly detection
  • Team and role data completeness validation
• Security:
  • API keys rotated every 90 days
  • Access logs maintained for 3 years (compliance)
  • Employee PII handling per privacy regulations

Rate Limiting Strategy

• Standard Plan: 500 requests/hour, 5,000 requests/day
• Professional Plan: 2,000 requests/hour, 20,000 requests/day
• Enterprise Plan: 5,000 requests/hour, 50,000 requests/day
• Recommended: Implement exponential backoff with jitter for 429 responses
• Error Handling: 429 status code indicates rate limit exceeded, respect Retry-After header
• Monitoring: Track rate limit utilization and plan for subscription upgrades

Data Quality Considerations

• Required Fields: id, first_name, last_name, email
• Optional Fields: external_id, date_of_birth, gender, mobile_number, landline_number, employment_start_date, employment_end_date
• Data Validation:
  • Employee IDs must be unique
  • Email addresses must be valid format
  • Employment dates must be valid and not future dates (for start dates)
  • Status flags (is_active, is_archived, is_suspended, is_deleted) must be boolean
• Data Completeness:
  • People: 95%+ have basic demographic data (name, email)
  • Teams: 100% have name and ID
  • Roles: 100% have name and ID
• Duplicate Handling: Primary key constraints prevent duplicate employee/team/role records

Integration Points

• Fivetran Destinations: Snowflake, BigQuery, Redshift, PostgreSQL
• Downstream Systems:
  • HR information systems (HRIS)
  • Workforce planning and analytics platforms
  • Organizational charting and visualization tools
  • Time and attendance tracking systems
  • Employee engagement platforms
• Data Dependencies: None - standalone workforce management data source
• External Dependencies: Rotamaster API availability, organizational structure updates

Disaster Recovery

• Backup Strategy: Daily snapshots of all employee, team, and role tables
• Recovery Time Objective: 6 hours for full data recovery
• Recovery Point Objective: 3 hours maximum data loss for workforce-critical data
• Failover: Automatic failover to backup API credentials
• Testing: Monthly disaster recovery drills with HR team validation

Compliance & Security

• Data Classification: Employee PII - highly sensitive, organizational data - moderate sensitivity
• Retention Policy: 7 years for employment records (compliance), 3 years for operational HR data
• Access Controls: Strict role-based access with principle of least privilege
• Audit Trail: All data access logged and monitored for compliance audits
• Encryption: Data encrypted in transit and at rest with enterprise-grade security
• Privacy: GDPR compliance for EU employees, CCPA compliance for CA employees, UK GDPR compliance

Performance Optimization

• Parallel Processing: Multiple API calls for different data types (people, teams, roles) where supported
• Caching: Team and role data cached for 24 hours (change infrequently)
• Indexing: Employee ID, team ID, role ID, and date columns indexed
• Partitioning: Employee data partitioned by employment_start_date and employment status for efficient querying
• Compression: Historical employee data compressed for storage efficiency
• Memory Efficiency: Generator-based streaming prevents memory accumulation for large employee datasets

Configuration Options

• initial_sync_days: Days of historical data for initial sync (default: 90, range: 1-365)
• max_records_per_page: Records per API request (default: 100, range: 1-1000)
• request_timeout_seconds: API request timeout in seconds (default: 30)
• retry_attempts: Failed request retry attempts (default: 3)
• enable_people_sync: Enable employee data sync (default: true)
• enable_teams_sync: Enable team data sync (default: true)
• enable_roles_sync: Enable role data sync (default: true)
• include_active_people: Include active employees (default: true)
• include_archived_people: Include archived employees (default: false)
• include_suspended_people: Include suspended employees (default: false)
• include_deleted_people: Include deleted employees (default: false)

Troubleshooting Guide

• Common Issues:
  • Rate limit exceeded: Reduce sync frequency, reduce max_records_per_page, or upgrade Rotamaster plan
  • API key expired: Verify API key validity and permissions for People, Team, and Role endpoints
  • Missing employee data: Check include_active/archived/suspended/deleted configuration flags
  • Timeout errors: Increase request_timeout_seconds or reduce max_records_per_page
  • Authentication errors: Verify API key format and ensure Bearer token prefix is included
  • Partial data sync: Check enable_people_sync, enable_teams_sync, enable_roles_sync flags
• Debug Mode: Enable detailed logging via enable_debug_logging configuration parameter
• Support Contacts:
  • Technical: Data Engineering team
  • Business: HR Operations team
  • Vendor: Rotamaster support (for API and account issues)
  • Compliance: Legal/Compliance team (for privacy and regulatory issues)

Checklist

Some tips and links to help validate your PR:

• Tested the connector with fivetran debug command.
• Added/Updated example specific README.md file, refer here for template.
• Followed Python Coding Standards, refer here

[github-actions]

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.