feature(examples): Add Xurrent connector

[fivertran-karunveluru]

Xurrent Connector

Created: 2025-10-31

Business Owner: IT Operations & Project Management Team

Technical Owner: Data Engineering Team

Last Updated: 2025-10-31

Business Context

• Data Source: Xurrent API for organizations, products, and projects data
• Business Criticality: High - supports IT asset management, project tracking, and organizational reporting
• Data Consumers: IT teams, project managers, product managers, executives, finance teams
• Business SLAs: Data must be fresh within 6 hours for operational reporting, 24 hours for analytics
• Compliance Requirements: Data governance for organizational structure, project data retention policies
• Budget Constraints: Xurrent API access based on subscription tier, rate limits vary by plan

Technical Context

• API Documentation: Xurrent Developer Portal (https://developer.xurrent.com/)
• Authentication Method: OAuth2 with Bearer token and account-level access via X-Xurrent-Account header
• Rate Limits: Varies by Xurrent plan, typically 1000-5000 requests/hour
• Data Volume:
  • Organizations: 10-10,000+ organizations per account (hierarchical structures)
  • Products: 100-50,000+ products per account
  • Projects: 50-5,000+ active projects per account
• Data Velocity: Organizational data updated on structural changes, products updated when catalog changes, projects updated in real-time during active work
• Data Quality: Structured JSON with consistent schema, nested objects flattened to normalized tables, some fields may be null
• Network Considerations: HTTPS only, RESTful API with standard reliability

Operational Context

• Deployment Environment: Development (sandbox), staging, and production environments
• Monitoring Requirements: Alert on >2% error rate, >4 hour sync time, data completeness issues
• Maintenance Windows: Weekends for non-critical updates, immediate deployment for data accuracy fixes
• Team Structure: Data Engineering team, IT Operations, Project Management Office (PMO), Product Management
• Escalation Path: Data Engineer → Team Lead → IT Director → CIO

API-Specific Details

• Base Endpoint: https://api.xurrent.com/v1
• Authentication: Bearer token in Authorization header (OAuth2), account ID in X-Xurrent-Account header
• Pagination: page and per_page parameters (max 100 per page, default 25)
• Date Format: ISO 8601 (e.g., 2024-01-15T10:30:00Z)
• Response Format: JSON with nested objects and arrays
• Key Endpoints:
  • /organizations - Organization hierarchy, parent-child relationships, and management structure
  • /products - Product catalog with categories, brands, services, and support teams
  • /projects - Project management data with status, timelines, and customer relationships

Data Schema Overview

• organizations: Organization hierarchy, parent-child relationships, manager assignments, business unit associations, region and remarks
• products: Product catalog with brand, category, model, service relationships, support team assignments, supplier information, depreciation details
• projects: Project management data with subject, category, status, program, justification, completion targets, customer and manager relationships

Data Replication Expectations

• Initial Sync: Last 90 days of organizational, product, and project data (configurable 1-365 days)
• Incremental Sync: Data since last successful sync timestamp using updated_at filtering
• Sync Frequency:
  • Production: Every 6 hours for operational data freshness
  • Development: Daily for all data types
• Data Retention: 3 years of historical organizational and project data for audit purposes
• Backfill Capability: Full historical data available based on Xurrent data retention policies
• Data Consistency: Near real-time with 6-hour maximum lag for operational reporting

Operational Requirements

• Uptime SLA: 99.5% availability during business hours
• Performance SLA:
  • Initial sync: <4 hours for 90 days of data across all tables
  • Incremental sync: <60 minutes for 6-hour updates
• Error Handling:
  • Automatic retry with exponential backoff and jitter (default 3 attempts)
  • Respect Retry-After header for rate limit responses
  • Alert on consecutive sync failures
  • Checkpointing every page/batch for large dataset recovery
• Monitoring:
  • API response times and error rates
  • Organization and product count trends
  • Project status distribution and anomaly detection
  • Data completeness validation
• Security:
  • OAuth tokens handled securely, never logged
  • Access logs maintained for 1 year (compliance)
  • Account ID authentication via header

Rate Limiting Strategy

• Starter Plan: 1,000 requests/hour, 10,000 requests/day
• Professional Plan: 3,000 requests/hour, 30,000 requests/day
• Enterprise Plan: 5,000 requests/hour, 50,000 requests/day
• Recommended: Implement exponential backoff with jitter for 429 responses
• Error Handling: 429 status code indicates rate limit exceeded, respect Retry-After header (up to 60 seconds max wait)
• Monitoring: Track rate limit utilization and plan for subscription upgrades

Data Quality Considerations

• Required Fields: id, name/subject (for projects), created_at, updated_at
• Optional Fields: source_id, parent_id, manager_id, category, status, remarks, region
• Data Validation:
  • IDs must be unique within each table
  • Organization IDs must be valid for parent relationships
  • Product and project relationships (service_id, customer_id, manager_id) must reference valid entities
  • Timestamps must be valid ISO 8601 format
• Data Completeness:
  • Organizations: 100% have basic identification data (id, name)
  • Products: 95%+ have category and brand information
  • Projects: 90%+ have status and customer information
• Duplicate Handling: Primary key constraints prevent duplicate records, upsert operations handle updates

Integration Points

• Fivetran Destinations: Snowflake, BigQuery, Redshift, PostgreSQL
• Downstream Systems:
  • IT asset management systems (ITAM)
  • Project portfolio management (PPM) platforms
  • Product information management (PIM) systems
  • Organizational chart and HR systems
  • Business intelligence and analytics platforms
  • Financial reporting systems
• Data Dependencies: None - standalone organizational/product/project data source
• External Dependencies: Xurrent API availability, organizational structure changes, project lifecycle events

Disaster Recovery

• Backup Strategy: Daily snapshots of all organization, product, and project tables
• Recovery Time Objective: 6 hours for full data recovery
• Recovery Point Objective: 6 hours maximum data loss for operational data
• Failover: Automatic failover to backup OAuth credentials and account IDs
• Testing: Quarterly disaster recovery drills with IT Operations team validation

Compliance & Security

• Data Classification: Organizational structure - operational sensitive, project data - business sensitive, product catalog - public/internal
• Retention Policy: 3 years for organizational and project data (audit compliance), 1 year for operational product data
• Access Controls: Role-based access with principle of least privilege, account-level access via X-Xurrent-Account header
• Audit Trail: All data access logged and monitored for compliance audits
• Encryption: Data encrypted in transit (HTTPS) and at rest with enterprise-grade security
• Privacy: Data governance policies for organizational hierarchy, project confidentiality requirements

Performance Optimization

• Parallel Processing: Sequential processing of organizations, products, and projects with memory-efficient streaming
• Caching: Organization hierarchy cached during sync processing, service and customer relationships cached
• Indexing: Organization ID, product ID, project ID, parent_id, manager_id, customer_id, and date columns indexed
• Partitioning: Projects partitioned by status and date for efficient querying, products partitioned by category
• Memory Efficiency: Generator-based streaming prevents memory accumulation for large datasets, checkpointing every page/batch

Troubleshooting Guide

• Common Issues:
  • Rate limit exceeded: Reduce sync frequency, increase per_page size, or upgrade Xurrent plan
  • OAuth token expired: Verify token refresh mechanism and permissions in Developer Portal
  • Missing organizational hierarchy: Check parent-child relationships and account access
  • Timeout errors: Increase timeout values (configurable 10-300 seconds) or reduce batch size
  • Project status inconsistencies: Validate status field values and project lifecycle handling
  • Nested object flattening issues: Verify mapping functions for organizations, products, and projects
• Debug Mode: Enable detailed logging with enable_debug_logging configuration flag
• Support Contacts:
  • Technical: Data Engineering team
  • Business: IT Operations team
  • Vendor: Xurrent support (for API and account issues)
  • Integration: Project Management Office (for project data validation)

Checklist

Some tips and links to help validate your PR:

• Tested the connector with fivetran debug command.
• Added/Updated example specific README.md file, refer here for template.
• Followed Python Coding Standards, refer here

[github-actions]

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.

[Copilot]

Pull Request Overview

This PR adds a new Xurrent API connector example to the Fivetran Connector SDK repository. The connector demonstrates memory-efficient streaming patterns for syncing organizations, products, and projects data from the Xurrent API.

Key changes:

• Implements a complete connector with OAuth2 authentication, pagination, and incremental sync capabilities
• Includes comprehensive error handling with exponential backoff and rate limiting support
• Provides configuration template and documentation following SDK best practices

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
connectors/xurrent/connector.py	Main connector implementation with API integration, data mapping, schema definition, and sync logic
connectors/xurrent/configuration.json	Configuration template with placeholders for API credentials and sync parameters
connectors/xurrent/README.md	Comprehensive documentation covering setup, authentication, features, and data handling
README.md	Added xurrent connector entry to the main connectors list

[fivetran-dejantucakov]

@fivertran-karunveluru Docs approved with changes