feature(examples): Add Omnisend connector

[fivertran-karunveluru]

Omnisend Connector

Created: 2025-10-31

Business Owner: Marketing Operations Team

Technical Owner: Data Engineering Team

Last Updated: 2025-10-31

Business Context

• Data Source: Omnisend API for marketing automation, e-commerce, and customer engagement data
• Business Criticality: High - supports marketing campaign performance analysis, customer segmentation, and product catalog management
• Data Consumers: Marketing teams, e-commerce managers, data analysts, campaign managers, customer success teams
• Business SLAs: Data must be fresh within 4-6 hours for campaign reporting, 24 hours for customer segmentation
• Compliance Requirements: GDPR compliance for contact data, CAN-SPAM compliance for email marketing, data privacy regulations
• Budget Constraints: Omnisend API access included with subscription, rate limits based on plan tier

Technical Context

• API Documentation: https://api-docs.omnisend.com/reference/overview
• Authentication Method: API key (X-API-KEY header) for most endpoints, Bearer token for campaigns endpoint
• Rate Limits: Varies by Omnisend plan, typically 500-5000 requests/hour
• Data Volume:
  • Products: 100-100,000+ products per account
  • Contacts: 1,000-1,000,000+ contacts per account
  • Campaigns: 50-5,000+ campaigns per year per account
  • Campaign Performance: Updates real-time as campaigns are sent
• Data Velocity: Product data updated on catalog changes, contacts updated on customer activity, campaigns updated on send/performance events
• Data Quality: Structured JSON with consistent schema, some nested fields may be null for incomplete records
• Network Considerations: HTTPS only, RESTful API with standard reliability, global infrastructure

Operational Context

• Deployment Environment: Development (sandbox), staging, and production environments
• Monitoring Requirements: Alert on >2% error rate, >2 hour sync time, campaign data discrepancies
• Maintenance Windows: Off-peak hours for non-critical updates, immediate deployment for campaign-critical fixes
• Team Structure: Data Engineering team, Marketing Operations, Campaign Managers, E-commerce specialists
• Escalation Path: Data Engineer → Team Lead → Marketing Director → CMO

API-Specific Details

• Base Endpoint: https://api.omnisend.com/v5
• Authentication: X-API-KEY header (primary), Bearer token (campaigns endpoint fallback)
• Pagination: offset and limit parameters (max 100 per page)
• Date Format: ISO 8601 (e.g., 2024-01-15T10:30:00Z)
• Response Format: JSON with nested objects and arrays
• Key Endpoints:
  • /products - Product catalog with variants, images, and categories
  • /contacts - Customer contact information and channel preferences
  • /campaigns - Email campaign details and performance statistics
• Incremental Sync: Timestamp-based using updatedAtFrom parameter for products and contacts

Data Schema Overview

• products: Product catalog with variants, images, categories, vendor information
• contacts: Customer contact information, demographics, channel preferences, custom properties
• campaigns: Email campaign metadata, scheduling, performance metrics (sent, delivered, opened, clicked)

Data Replication Expectations

• Initial Sync: Last 90 days of product, contact, and campaign data (configurable)
• Incremental Sync: Data since last successful sync timestamp using updatedAtFrom parameter
• Sync Frequency:
  • Production: Every 4-6 hours for contact/campaign data, daily for product catalog
  • Development: Daily for all data types
• Data Retention: Historical data available based on Omnisend account retention policies
• Backfill Capability: Full historical data available based on Omnisend retention policies and account limits
• Data Consistency: Near real-time with 4-6 hour maximum lag for marketing operations

Operational Requirements

• Uptime SLA: 99.5% availability during business hours (campaign reporting critical)
• Performance SLA:
  • Initial sync: <4 hours for 90 days of data
  • Incremental sync: <45 minutes for daily updates
• Error Handling:
  • Automatic retry with exponential backoff and jitter
  • Rate limit handling with Retry-After header support
  • Dead letter queue for failed product/contact/campaign records
  • Alert on consecutive sync failures during campaign periods
• Monitoring:
  • API response times and error rates
  • Contact count trends and anomaly detection
  • Campaign performance data completeness validation
  • Product catalog sync accuracy
• Security:
  • API keys stored securely and never logged
  • Access logs maintained for compliance
  • Contact PII handling per privacy regulations (GDPR, CAN-SPAM)

Rate Limiting Strategy

• Free Plan: 500 requests/hour, 5,000 requests/day
• Standard Plan: 1,500 requests/hour, 15,000 requests/day
• Pro Plan: 3,000 requests/hour, 30,000 requests/day
• Enterprise Plan: 5,000 requests/hour, 50,000 requests/day
• Recommended: Implement exponential backoff with jitter for 429 responses
• Error Handling: 429 status code indicates rate limit exceeded, respect Retry-After header
• Monitoring: Track rate limit utilization and plan for subscription upgrades if approaching limits

Data Quality Considerations

• Required Fields:
  • Products: id, title
  • Contacts: contact_id, email
  • Campaigns: id, name
• Optional Fields:
  • Products: description, vendor, variants, images, tags, categories
  • Contacts: first_name, last_name, phone, address, custom_properties, tags, channel preferences
  • Campaigns: subject, scheduled_date, sent_date, performance statistics
• Data Validation:
  • Product IDs must be unique
  • Contact IDs must be unique
  • Email addresses should be valid format (where present)
  • Campaign IDs must be unique
  • Timestamps must be valid ISO 8601 format
• Data Completeness:
  • Products: 95%+ have title and basic information
  • Contacts: 90%+ have email address
  • Campaigns: 100% have name and status
• Duplicate Handling: Primary key constraints prevent duplicate records (id for products/campaigns, contact_id for contacts)

Integration Points

• Fivetran Destinations: Snowflake, BigQuery, Redshift, PostgreSQL
• Downstream Systems:
  • Marketing automation platforms
  • Customer data platforms (CDPs)
  • Business intelligence and analytics tools
  • E-commerce platforms
  • CRM systems
  • Email service providers
• Data Dependencies: None - standalone marketing/e-commerce data source
• External Dependencies: Omnisend API availability, campaign processing schedules

Disaster Recovery

• Backup Strategy: Daily snapshots of all marketing and e-commerce tables
• Recovery Time Objective: 6 hours for full data recovery
• Recovery Point Objective: 4 hours maximum data loss for campaign-critical data
• Failover: Automatic failover to backup API credentials if primary fails
• Testing: Monthly disaster recovery drills with marketing team validation

Compliance & Security

• Data Classification: Contact PII - highly sensitive, campaign data - marketing sensitive, product data - business sensitive
• Retention Policy: Based on Omnisend account retention and compliance requirements, typically 2-7 years
• Access Controls: Strict role-based access with principle of least privilege
• Audit Trail: All data access logged and monitored for compliance audits
• Encryption: Data encrypted in transit and at rest with enterprise-grade security
• Privacy: GDPR compliance for EU contacts, CAN-SPAM compliance for email marketing, CCPA compliance for CA contacts

Performance Optimization

• Parallel Processing: Sequential processing with generator-based streaming prevents memory accumulation
• Caching: Product and campaign metadata can be cached for short periods (not implemented in current version)
• Indexing: Product ID, contact ID, campaign ID, and date columns indexed
• Partitioning: Campaign data partitioned by date for efficient querying
• Streaming: Memory-efficient generator pattern processes records individually to prevent memory issues with large datasets

Configuration Parameters

• api_key: Omnisend API key for authentication (required)
• sync_frequency_hours: How often to run sync in hours (default: 4)
• initial_sync_days: Number of days to sync initially (default: 90, max: 365)
• max_records_per_page: Maximum records per API request (default: 100, max: 1000)
• request_timeout_seconds: API request timeout in seconds (default: 30)
• retry_attempts: Number of retry attempts for failed requests (default: 3)
• enable_incremental_sync: Enable timestamp-based incremental sync (default: true)
• enable_debug_logging: Enable detailed debug logging (default: false)
• sync_products: Enable product data synchronization (default: true)
• sync_contacts: Enable contact data synchronization (default: true)
• sync_campaigns: Enable campaign data synchronization (default: true)

Troubleshooting Guide

• Common Issues:
  • Rate limit exceeded: Reduce sync frequency or upgrade Omnisend plan
  • API key expired: Verify API key validity and permissions
  • Missing contact data: Check contact active status and segmentation filters
  • Timeout errors: Increase timeout values or reduce batch size (max_records_per_page)
  • Campaign authentication failures: Verify Bearer token fallback mechanism for campaigns endpoint
  • Duplicate product records: Verify primary key constraints and product ID uniqueness
  • Contact email validation issues: Verify email format and contact status
• Debug Mode: Enable detailed debug logging via enable_debug_logging configuration parameter
• Support Contacts:
  • Technical: Data Engineering team
  • Business: Marketing Operations team
  • Vendor: Omnisend support (for API and account issues)
  • Compliance: Legal/Compliance team (for privacy and regulatory issues)

Checklist

Some tips and links to help validate your PR:

• Tested the connector with fivetran debug command.
• Added/Updated example specific README.md file, refer here for template.
• Followed Python Coding Standards, refer here

[github-actions]

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.