Chain.io Custom Processor Examples

A comprehensive collection of custom processor examples and practical guide for business analysts and JavaScript developers to create custom data transformations in Chain.io flows.

📁 Repository Contents

This repository contains:

• Real-world examples - Production-ready custom processors (see .js files in this repo)
• Complete documentation - Everything you need to get started
• Best practices - Proven patterns from the Chain.io community
• EXECUTION_SEARCH.md - Detailed guide for the executionSearchByPartner() function
• XML_LIBRARY.md - XML parsing and manipulation reference

Example Files in This Repository

• alphabetically_sort_xml.js - Sort XML elements alphabetically to support testing
• excel_to_csv.js - Convert Excel files to CSV format
• filter_shipments_with_update_or_delete_action_type.js - Filter shipments by action type
• overwrite_output_field_with_mapping.js - Map field values using lookup tables
• error_edi_810_cancel_files.js - Handle EDI cancellation files
• nonstandard_edi_value_replace.js - Replace non-standard EDI values
• port_of_discharge_to_port_of_destination.js - Port mapping logic
• log_another_flow_with_search.js - Search for previous flow executions using executionSearchByPartner

Table of Contents

• Quick Start
• What Are Custom Processors?
• When to Use Custom Processors
• Setting Up Your First Custom Processor
• Available Tools and Libraries
• Common Use Cases with Examples
• Best Practices
• Troubleshooting
• Advanced Examples
• Using Async Operations
• File Compatibility
• Limitations to Remember
• Getting Help
• Contributing

Quick Start

What you need to know:

• Basic JavaScript knowledge (variables, functions, arrays, objects)
• Understanding of JSON and CSV data formats
• 5 minutes to read this guide

What you can accomplish:

• Transform data between different formats
• Add calculated fields to your data
• Filter and validate data
• Enrich data with custom business logic

What Are Custom Processors?

Custom processors are JavaScript scripts that run within your Chain.io flows to transform data. Think of them as your personal data transformation assistants that can:

• Pre-processors: Modify data before it goes through Chain.io's main processing
• Post-processors: Modify data after Chain.io processes it but before final delivery

Key Benefits

• ✅ No server setup required - runs in Chain.io's cloud
• ✅ Secure sandboxed environment
• ✅ Built-in logging and error handling
• ✅ Access to powerful libraries (Excel, XML, date handling)

When to Use Custom Processors

Perfect for:

• Data Enrichment: Adding calculated fields, lookups, or business rules
• Format Conversion: Converting between JSON, CSV, XML formats
• Data Validation: Checking data quality and completeness
• Field Mapping: Renaming or restructuring data fields
• Conditional Logic: Processing data based on business rules

Not suitable for:

• ❌ External API calls (not allowed)
• ❌ Database connections (not supported)
• ❌ File system operations (sandboxed environment)
• ❌ Long-running processes (60-second timeout)

Setting Up Your First Custom Processor

Step 1: Access the Custom Processor Section

1. Open your Chain.io flow
2. Navigate to the "Edit Flow" screen
3. Find the "Custom Processors" section
4. Choose either "Pre-processor" or "Post-processor"

Step 2: Write Your First Script

Here's a simple example that adds a timestamp to every record:

// Add current timestamp to each file
const processedFiles = sourceFiles.map(file => {
  // Parse the JSON data
  const data = JSON.parse(file.body)
  
  // Add timestamp
  data.processed_at = DateTime.now().toISO()
  
  // Return the modified file
  return {
    ...file,
    body: JSON.stringify(data)
  }
})

// Log what we did
userLog.info(`Added timestamps to ${processedFiles.length} files`)

// Return the processed files
returnSuccess(processedFiles)

Step 3: Test and Deploy

1. Save your script
2. Deploy your flow
3. Monitor the execution logs for any issues

Available Tools and Libraries

Your custom processors have access to these powerful tools:

Core Variables

• sourceFiles (pre-processors): Array of incoming files
• destinationFiles (post-processors): Array of processed files
• userLog: For logging messages (userLog.info(), userLog.warning(), userLog.error())

File Object Structure

Every file in Chain.io custom processors is represented as a JavaScript object with the following properties:

{
  uuid: "550e8400-e29b-41d4-a716-446655440000",    // Unique identifier for the file
  type: "file",                                    // Always "file" for file objects
  file_name: "example.json",                       // Original filename with extension
  format: "json",                                  // File format (json, csv, xml, etc.)
  mime_type: "application/json",                   // MIME type of the file
  body: "{ \"key\": \"value\" }"                   // File content as string
}

Required Properties

• body (string): The actual file content. Always a string regardless of original format
• file_name (string): The filename including extension (e.g., "orders.json", "data.csv")
• type (string): Always set to "file"
• uuid (string): Unique identifier. Generate with uuid() when creating new files

Optional Properties

• format (string): File format identifier (json, csv, xml, xlsx, etc.)
• mime_type (string): Standard MIME type (application/json, text/csv, text/xml, etc.)

Important Notes About File Content

Binary Files (Excel, Images, etc.)

// Excel files come as base64-encoded strings
const workbook = XLSX.read(file.body, { type: 'base64' })

Text Files (JSON, CSV, XML)

// Text files are plain strings
const data = JSON.parse(file.body)  // For JSON
const lines = file.body.split('\n') // For CSV

Creating New File Objects

When creating new files in your processor, use this structure:

const newFile = {
  uuid: uuid(),                    // Generate unique ID
  type: 'file',                   // Always "file"
  file_name: 'output.json',       // Set appropriate filename
  format: 'json',                 // Set format
  mime_type: 'application/json',  // Set MIME type
  body: JSON.stringify(data)      // Convert data to string
}

Modifying Existing Files

When modifying files, preserve the original structure:

const modifiedFile = {
  ...originalFile,                // Keep all original properties
  body: newContent,              // Update only the content
  file_name: newFileName         // Optionally update filename
}

Built-in Libraries

• lodash (v4.17.21): Utility functions for arrays and objects
• DateTime (Luxon v3.3.0): Date and time manipulation
• uuid(): Generate unique identifiers (Node.js built-in)
• XLSX (SheetJS v0.20.3): Read and write Excel files
• xml (Chain.io v0.3.0): XML parsing and manipulation - see XML_LIBRARY.md for detailed documentation
• xmldom (v0.8.8) & xpath (v0.0.32): Advanced XML processing

Return Functions

• returnSuccess(files): Process completed successfully
• returnError(files): Process failed with error
• returnSkipped(files): Skip this execution

Advanced Functions

executionSearchByPartner(partnerUUID, args)

Search for flow execution records by trading partner. This function provides programmatic access to the same execution data you see in the Flow Execution Search screen in the Chain.io portal.

Quick Example:

(async () => {
  // Search for recent executions from a partner
  const results = await executionSearchByPartner('partner-uuid-here', {
    startDateAfter: '2024-01-01T00:00:00Z',
    dataTag: 'ORDER_BATCH_123'
  })
  
  userLog.info(`Found ${results.data.length} executions`)
  
  // Use the results in your processing
  const processedFiles = sourceFiles.map(file => {
    // Your logic here
    return file
  })
  
  return returnSuccess(processedFiles)
})()

Key Points:

• ⚠️ Requires async wrapper: Must wrap entire script in (async () => { ... })()
• ⚠️ Rate limited: Maximum 10 searches per execution
• ⚠️ Returns Promise: Use await and return returnSuccess()
• 📖 Complete Documentation: See detailed guide for all parameters, options, and use cases

Common Use Cases:

• Check for duplicate processing
• Enrich data with execution history
• Track batch processing status
• Analyze execution patterns

💡 See also:

• EXECUTION_SEARCH.md - Complete documentation with all parameters and examples
• log_another_flow_with_search.js - Working code example

Common Use Cases with Examples

1. Adding Calculated Fields

Scenario: Add total price calculation to order data

const processedFiles = sourceFiles.map(file => {
  const orders = JSON.parse(file.body)
  
  // Add total calculation to each order
  orders.forEach(order => {
    order.total = order.quantity * order.unit_price
    order.total_with_tax = order.total * 1.08 // 8% tax
  })
  
  return {
    ...file,
    body: JSON.stringify(orders)
  }
})

userLog.info(`Calculated totals for ${processedFiles.length} order files`)
returnSuccess(processedFiles)

2. Data Validation and Filtering

Scenario: Only process orders above a certain value

💡 See also: filter_shipments_with_update_or_delete_action_type.js for filtering shipments by action type

const processedFiles = sourceFiles.map(file => {
  const orders = JSON.parse(file.body)
  
  // Filter orders above $100
  const validOrders = orders.filter(order => {
    const total = order.quantity * order.unit_price
    if (total < 100) {
      userLog.warning(`Skipping order ${order.id}: total $${total} below minimum`)
      return false
    }
    return true
  })
  
  return {
    ...file,
    body: JSON.stringify(validOrders)
  }
})

userLog.info(`Filtered to high-value orders only`)
returnSuccess(processedFiles)

3. Format Conversion (CSV to JSON)

Scenario: Convert CSV data to structured JSON

const processedFiles = sourceFiles.map(file => {
  // Assuming CSV content in file.body
  const lines = file.body.split('\n')
  const headers = lines[0].split(',')
  
  const jsonData = lines.slice(1).map(line => {
    const values = line.split(',')
    const record = {}
    
    headers.forEach((header, index) => {
      record[header.trim()] = values[index]?.trim()
    })
    
    return record
  })
  
  return {
    ...file,
    body: JSON.stringify(jsonData),
    file_name: file.file_name.replace('.csv', '.json')
  }
})

userLog.info(`Converted ${processedFiles.length} CSV files to JSON`)
returnSuccess(processedFiles)

4. Working with Excel Files

Scenario: Extract data from Excel spreadsheet

💡 See also: excel_to_csv.js in this repository for a production example

const processedFiles = sourceFiles.map(file => {
  // Skip non-Excel files (like the example in this repo)
  if (!file.file_name?.match(/\.xls[xbm]$/)) return null
  
  // Parse Excel file - note: use 'base64' type for binary data
  const workbook = XLSX.read(file.body, { type: 'base64' })
  const sheetName = workbook.SheetNames[0]
  const worksheet = workbook.Sheets[sheetName]
  
  // Convert to JSON
  const jsonData = XLSX.utils.sheet_to_json(worksheet)
  
  // Add processing metadata
  jsonData.forEach(row => {
    row.source_sheet = sheetName
    row.processed_date = DateTime.now().toISODate()
  })
  
  return {
    uuid: uuid(),
    type: 'file',
    body: JSON.stringify(jsonData),
    file_name: file.file_name.replace(/\.xls[xbm]$/, '.json'),
    format: 'json',
    mime_type: 'application/json'
  }
}).filter(x => x) // Remove null entries

userLog.info(`Processed ${processedFiles.length} Excel files`)
returnSuccess(processedFiles)

5. XML Data Processing

Scenario: Extract specific data from XML files

const processedFiles = sourceFiles.map(file => {
  try {
    // Parse XML
    const xmlDoc = xml.XmlParser.parseFromString(file.body)
    
    // Extract order information
    const orders = xml.elements(xmlDoc, '//order').map(orderElement => ({
      id: xml.text(orderElement, 'id'),
      customer: xml.text(orderElement, 'customer'),
      amount: parseFloat(xml.text(orderElement, 'amount')),
      date: xml.text(orderElement, 'date')
    }))
    
    return {
      ...file,
      body: JSON.stringify(orders),
      file_name: file.file_name.replace('.xml', '.json')
    }
  } catch (error) {
    userLog.error(`Failed to process XML file ${file.file_name}: ${error.message}`)
    return file // Return original file if processing fails
  }
})

userLog.info(`Processed ${processedFiles.length} XML files`)
returnSuccess(processedFiles)

6. Data Enrichment with Lookups

Scenario: Add customer information based on customer ID

💡 See also: overwrite_output_field_with_mapping.js for XML field mapping and port_of_discharge_to_port_of_destination.js for port lookups

// Sample customer lookup data
const customerLookup = {
  'CUST001': { name: 'Acme Corp', tier: 'Premium' },
  'CUST002': { name: 'Beta Inc', tier: 'Standard' },
  'CUST003': { name: 'Gamma LLC', tier: 'Premium' }
}

const processedFiles = sourceFiles.map(file => {
  const orders = JSON.parse(file.body)
  
  // Enrich each order with customer data
  orders.forEach(order => {
    const customer = customerLookup[order.customer_id]
    if (customer) {
      order.customer_name = customer.name
      order.customer_tier = customer.tier
      
      // Apply tier-based discount
      if (customer.tier === 'Premium') {
        order.discount_percent = 10
        order.discounted_total = order.total * 0.9
      }
    } else {
      userLog.warning(`Unknown customer ID: ${order.customer_id}`)
    }
  })
  
  return {
    ...file,
    body: JSON.stringify(orders)
  }
})

userLog.info(`Enriched orders with customer data`)
returnSuccess(processedFiles)

Best Practices

1. Always Handle Errors

try {
  // Your processing logic here
  const data = JSON.parse(file.body)
  // ... process data
} catch (error) {
  userLog.error(`Processing failed: ${error.message}`)
  return file // Return original file or handle appropriately
}

2. Use Descriptive Logging

userLog.info(`Starting to process ${sourceFiles.length} files`)
userLog.info(`Successfully processed ${validRecords.length} records`)
userLog.warning(`Skipped ${skippedRecords.length} invalid records`)

3. Validate Your Data

// Check if required fields exist
if (!order.customer_id || !order.amount) {
  userLog.warning(`Invalid order missing required fields: ${JSON.stringify(order)}`)
  return null // Skip this record
}

4. Keep It Simple

• Break complex logic into smaller functions
• Use meaningful variable names
• Comment your code for future reference

5. Test with Small Data Sets

• Start with a few records to test your logic
• Gradually increase data volume once working

Troubleshooting

Common Issues and Solutions

Issue: "Invalid processor output"

// ❌ Wrong - not returning files properly
return data

// ✅ Correct - always return file objects
returnSuccess(processedFiles)

Issue: "Timeout after 60 seconds"

// ❌ Avoid complex loops on large datasets
sourceFiles.forEach(file => {
  // Expensive operation on every record
})

// ✅ Use efficient processing
const processedFiles = sourceFiles.map(file => {
  // Quick transformation
  return transformFile(file)
})

Issue: "Cannot parse JSON"

// ❌ Assuming data is always valid JSON
const data = JSON.parse(file.body)

// ✅ Handle parsing errors
try {
  const data = JSON.parse(file.body)
  // Process data
} catch (error) {
  userLog.error(`Invalid JSON in file ${file.file_name}`)
  return file // Return original or skip
}

Debugging Tips

1. Use console logging: userLog.info(JSON.stringify(data, null, 2))
2. Check file structure: Log the file object to understand its properties
3. Validate step by step: Add logging at each major step
4. Test with minimal data: Use simple test cases first

Advanced Examples

Working with Multiple File Types

const processedFiles = sourceFiles.map(file => {
  // Handle different file types
  if (file.file_name.endsWith('.json')) {
    return processJsonFile(file)
  } else if (file.file_name.endsWith('.csv')) {
    return processCsvFile(file)
  } else if (file.file_name.endsWith('.xml')) {
    return processXmlFile(file)
  } else {
    userLog.warning(`Unsupported file type: ${file.file_name}`)
    return file
  }
})

function processJsonFile(file) {
  const data = JSON.parse(file.body)
  // JSON-specific processing
  return { ...file, body: JSON.stringify(data) }
}

function processCsvFile(file) {
  // CSV processing logic
  return file
}

function processXmlFile(file) {
  // XML processing logic
  return file
}

returnSuccess(processedFiles)

Publishing Custom Data Tags

Data tags appear in the Flow Execution Screen for tracking and monitoring:

// Process your data
const processedFiles = sourceFiles.map(file => {
  const orders = JSON.parse(file.body)
  
  // Calculate summary statistics
  const totalOrders = orders.length
  const totalValue = orders.reduce((sum, order) => sum + order.total, 0)
  
  // Publish data tags for monitoring
  publishDataTags([
    { label: 'Total Orders', value: totalOrders.toString() },
    { label: 'Total Value', value: `$${totalValue.toFixed(2)}` },
    { label: 'Processing Date', value: DateTime.now().toISODate() }
  ])
  
  return { ...file, body: JSON.stringify(orders) }
})

returnSuccess(processedFiles)

Note: Data tag labels and values are automatically truncated to 255 bytes each to ensure compatibility with the Chain.io platform. If your data contains labels or values longer than 255 bytes, they will be truncated when published. (A unicode character is 1-4 bytes. If you need to see how many bytes a particular string is, there are online tools available. Here is an example: UTF-8 String Length & Byte Counter )

Conditional Flow Control

const processedFiles = sourceFiles.map(file => {
  const data = JSON.parse(file.body)
  
  // Check if data meets processing criteria
  if (data.length === 0) {
    userLog.warning('No data to process')
    return null
  }
  
  if (data.some(record => !record.required_field)) {
    userLog.error('Data missing required fields')
    return null
  }
  
  // Process valid data
  return { ...file, body: JSON.stringify(data) }
}).filter(file => file !== null) // Remove null entries

if (processedFiles.length === 0) {
  userLog.warning('No valid files to process')
  returnSkipped([])
} else {
  returnSuccess(processedFiles)
}

Using Async Operations

Custom processors support asynchronous operations, but they require a specific wrapper pattern.

When You Need Async

You need to use the async wrapper when:

• Using executionSearchByPartner() to search for previous executions
• Using await with any Promise-based operation
• Calling any function that returns a Promise

The Async Wrapper Pattern

❌ This will NOT work:

// Without async wrapper - will cause errors
const results = await executionSearchByPartner('partner-uuid')
returnSuccess(sourceFiles)

✅ This WILL work:

// With async wrapper - correct pattern
(async () => {
  const results = await executionSearchByPartner('partner-uuid')
  return returnSuccess(sourceFiles)  // Note: use "return"
})()

Key Rules for Async Code

1. Wrap everything in (async () => { ... })()
2. Use return before your returnSuccess(), returnError(), or returnSkipped() calls
3. Keep it simple - the wrapper goes around your entire script
4. Remember the timeout - all async operations must complete within the 60 second total runtime alloted to custom processors

Complete Async Example

(async () => {
  // You can now use await anywhere in your code
  const results = await executionSearchByPartner('partner-uuid-here', {
    startDateAfter: '2024-01-01T00:00:00Z'
  })
  
  userLog.info(`Found ${results.data.length} previous executions`)
  
  // Process your files normally
  const processedFiles = sourceFiles.map(file => {
    const data = JSON.parse(file.body)
    
    // Use the search results to enrich your data
    data.previousExecutionCount = results.data.length
    
    return {
      ...file,
      body: JSON.stringify(data)
    }
  })
  
  // Don't forget to use "return" before returnSuccess
  return returnSuccess(processedFiles)
})()

Async with Conditional Logic

(async () => {
  // Search for previous executions
  const results = await executionSearchByPartner('partner-uuid')
  
  // Use conditional logic as normal
  if (results.data.length === 0) {
    userLog.warning('No previous executions found')
    return returnSkipped([])
  }
  
  return returnSuccess(processedFiles)
})()

File Compatibility

Supported Formats

• ✅ JSON: Full read/write support
• ✅ CSV: Parse and generate CSV data
• ✅ XML: Parse, query, and generate XML
• ✅ Excel: Read .xlsx files, extract data

EDI Files

• ❌ Cannot be processed in custom processors (though you can make minor edits to individual fields like changing charge codes)
• ✅ Use Chain.io's Custom Mapping Tool instead
• ✅ Process EDI after conversion to JSON

Limitations to Remember

• No external libraries: Cannot require() or import additional packages beyond the built-in libraries listed above
• 60-second timeout: Total execution time (including any async operations) must complete within 60 seconds
• 10,000 character limit: Per processor (pre and post can each be 10,000 characters)
• No Symbol object access: Security restriction
• Async operations require wrapper: If using await or async functions like executionSearchByPartner, you must wrap your entire script in (async () => { ... })() and use return statements

Getting Help

When You're Stuck

1. Check the execution logs in Chain.io for error messages
2. Start simple - test with minimal data first
3. Use logging extensively to understand data flow
4. Review the examples in this guide for similar use cases

Common Questions

Q: Can I call external APIs? A: No, custom processors run in a sandboxed environment without external network access. However, you can use executionSearchByPartner() to search for previous execution data within Chain.io.

Q: How do I use async/await in my processor? A: Wrap your entire script in (async () => { ... })() and use return before your returnSuccess(), returnError(), or returnSkipped() calls. See the "Advanced Functions" section above for complete examples.

Q: Why do I get "await is only valid in async function" error? A: You need to wrap your entire script in the async wrapper: (async () => { /* your code here */ })(). Don't forget to add return before your return function calls.

Q: How do I handle large files? A: Process data in chunks and use efficient algorithms. Consider splitting large files before processing.

Q: Can I save state between executions? A: No, each execution is independent. However, you can use executionSearchByPartner() to look up data from previous executions.

Q: What happens if my code has errors? A: The flow will fail with an error status, and details will appear in the execution logs.

Contributing

We welcome contributions from the Chain.io community!

How to Contribute

1. Fork this repository - Click the "Fork" button at the top of this page
2. Add your example - Create a new .js file with a descriptive name
3. Follow the pattern - Look at existing examples for structure and commenting style
4. Test your code - Make sure it works in your Chain.io environment
5. Submit a pull request - Create a pull request with your changes

Example Contribution Guidelines

• Use descriptive filenames - convert_xml_to_json.js instead of processor.js
• Add comments - Explain what your processor does and any assumptions
• Include error handling - Show how to handle edge cases
• Keep it focused - One clear use case per example
• Test with real data - Ensure your example works in production

Need Help?

• Review existing examples in this repository
• Ask questions in your pull request - the community is here to help!

---

This repository contains practical examples and comprehensive documentation for Chain.io custom processors. For specific business requirements or complex scenarios, consider consulting with your development team or Chain.io support.