vlocity-deploy

@vlocode/vlocity-deploy a hyper fast 🚀 Vlocity Datapack deployment Library

A fast pure-JS library for deploying any Vlocity Datapack to Salesforce.

This library is built from scratch to provide a fast and reliable way to deploy Datapack definitions to Salesforce not having to rely on APEX REST APIs and DataRaptors.

Key differences with vlocityinc/vlocity_build and Vlocity DX

• 🚀 Vlocode is significantly faster up to 10x-20x compared to Vlocity DX depending on the use case
• 💻 Vlocode does all the heavy lifting, dependency resolution, converting of datapacks client side
• 🌈 Vlocode supports true delta check that is both fast and reliable as it can detect changes made in your org and restore them without relying on git
• 💪 Vlocode battle tested on production deployments and covered by Jest unit tests
• ❤️ Vlocode is fully written in TypeScript with extensive code-level documentation
• ✏️ Vlocode deployment logic can easily be extended using Spec classes and Hook functions
• 📃: Vlocode is fully documented

Also...

• Vlocode does client-side OmniScript to LWC compilation and deployment
• Vlocode does client-side OmniScript Activation avoiding SOQL limit exceptions when activating big scripts

What does it not do...

• Vlocode vlocity-deploy library is meant for deploying datapacks not for exporting/extracting them from an org
• Activation of flex cards
• Provide you with a (fancy) UI

Why is Vlocode faster?

Vlocode is significantly faster then the regular Vlocity/OmniStudio Datapack deployment because it takes a fundamental different approach to deployment of Datapacks. Instead of relying on server side Apex REST APIs and DataRaptors Vlocode does all the heavy lifting client side. This means that Vlocode does not need to make any round trips to the server to resolve dependencies that it just deployed, convert Datapacks to Salesforce records and deploy them to Salesforce.

Especially when deploying larger projects with many dependencies Vlocode can be up to 10x-20x faster then the regular Vlocity/OmniStudio deployment tools. The following table shows the results of a benchmark test that was run on a project with 1000+ datapacks and 100+ dependencies.

Tool	Time
Vlocode	7m 54s
Vlocity/OmniStudio	92m 12s

How does it work?

1. First Vlocode converts all the datapacks to Salesforce records the process takes about 5 seconds per 1000 datapacks and depends primarily on the performance of the hardware on which Vlocode is running. Vlocode will also validate the records and report any errors that it encounters during the conversion process such as none-existing fields or invalid field values.

2. The second step in the deployment process is building a virtual dependency graph. Vlocode will determine which record to deploy first and which records need to wait for their dependencies to be deployed. It considered all to be deployed records individually outside of their datapack context, this allows Vlocode to deploy records in the most optimal order. The dependency resolution process takes about 1 second per 1000 records.

   Some datapacks require special handling, for example OmniScripts. To allow for this Vlocode uses a deployment spec/hook registry that allows to registering custom functions for specific datapacks. Specs can change the datapack prior to deployment, run pre- and post-deployment activation logic and can also be used to manipulate the record data of individual records before they are deployed. Vlocode comes with a set of default specs for standard datapacks such as OmniScripts and Product2 but custom specs can be registered to extend the deployment process.

3. The third step in the deployment process is deploying the records to Salesforce. Vlocode uses the Salesforce Collections API or Bulk API to deploy records directly to Salesforce. After a record is deployed Vlocode validates field integrity of `*GlobalKey__c`` fields as these are overwritten by Vlocity when a new record is inserted. If these do not matching an extra update is done with all triggers disabled to correct the field value.

4. The final step in the deployment process is activating the deployed records. Vlocode uses the Salesforce Tooling API to activate OmniScripts and LWCs. The activation process is done in parallel and is significantly faster then the regular Vlocity/OmniStudio activation process.

How to use

The Vlocode vlocity-deploy is a library that you can use inside your own deployment toolkit. It depends on other vlocode packages for communication with Salesforce and for providing utility functions. You shouldn't need to depend on on any other @vlocode libraries to use @vlocode/vlocity-deploy but if you do it is advisable to keep all @vlocode packages aligned to each other to avoid loading multiple version of each library.

A convenience function is exported from @vlocode/vlocity-deploy which starts a new deployment. In order to connect to Salesforce the deploy functions needs either an SFDX username -or- an JSForce compatible connection object.

You can also change how @vlocode/vlocity-deploy logs messages; by default informational messages are printed through the console interface.

Option	Description
connection	JSForce compatible connection object; vlocode will not directly use the connection passed if it wasn't created through the @Vlocode/salesforce library. In such cases Vlocode will create a new vlocode.salesforceConnection. This is required to work around several bugs in jsforce which would otherwise break the deployment.
sfdxUser	When an SFDX user name is passed Vlocode will create a new SFDX connection (vlocode.salesforceConnection) and use that for interacting with Salesforce.
logger	Custom logging interface to which the deployment writes all messages, see Use a Custom logger to read how to implement a custom logger.

The deploy function returns a DatapackDeployment object which contains extensive information about the deployment including the deployed records and warnings/error messages (see getMessages() in DatapackDeployment).

Example snippet (JS)

import { deploy } from '@vlocode/vlocity-deploy';
import { connection } from 'jsforce';

const deployment = await deploy('./vlocity/datapacks/', {
  connection: new connection(...) /* JS-Force connection */
  sfdxUser: '[email protected]' /* SFDX User Name */
  // logger: { ... }
});

for (const message of deployment.getMessages()) {
  // Each message also contains a `record` object that caused the message
  console.log(`${type}: ${message}`);
}

Vlocode deploy options

Below the list of options that can be passed to the deploy function

Option	Default	Description
disableTriggers	false	Disable all Vlocity Triggers before starting the deployment; triggers are automatically re-enabled after the deployment completes.
maxRetries	1	Number of times to retry the update or insert operation when it fails; defaults to 1 when not set.
retryChunkSize	5	Size of the chunk when retrying deployment of records that previously failed. The deployment limits the number of records inserted or updated to avoid running into limits due to active triggers. Do not change unless you have govern limit errors
lookupFailedDependencies	false	Attempt to lookup dependencies that are part of the deployment but failed to deploy. By setting this to true when part of a datapack fails to deploy the deployment will attempt to lookup an existing record that also matches the lookup requirements. This can help resolve deployment issues whe deploying datapacks from which the parent record cannot be updated, but it does introduce a risk of incorrectly linking records.
purgeMatchingDependencies	false	Purge dependent records after deploying any record. This setting controls whether or not the deployment will delete direct dependencies linked through a matching (not lookup) dependency. This is especially useful to delete for example PCI records and ensure that old relationships are deleted.
purgeLookupOptimization	true	This setting controls how embedded datapacks without matching keys are deleted from the target org when enabled purging of existing records happens in bulk, this is more efficient but in this mode it is not possible to related errors that deleting records to a particular datapack.
deltaCheck	false	When enabled the deployment compares the records that it intends to deploy to the current state of the record in the target org and will only update the fields that are not matching the datapack.
continueOnError	false	Continue the deployment when a fatal error occurs, note that continuing the deployment on fatal errors will result in an incomplete deployment. This setting affects fatal errors such as unable to convert a datapack to valid Salesforce records and should not be enabled on production deployments.
strictOrder	false	When strict order is enabled the deployment will wait for all records in a datapack to complete before proceeding with deploying any dependent datapacks. By default Vlocode determines deployment order based on record level dependencies this allows for optimal chunking improving the overall speed of the deployment. By setting strictOrder to true Vlocode also enforces that any datapack that is dependent on another datapack is deployed after the datapack it depends on. Enable this if you are seeing errors during deployment that are caused by records not being deployed in the correct order. Note: enabling this impacts deployment performance as the deployment will be split in smaller chunks increasing the number of API calls required to deploy all datapacks.
allowUnresolvedDependencies	false	When enabled the deployment will not fail when a dependency cannot be resolved. If a record has a dependency that cannot be resolved the record will normally be skipped and marked as failed. Deploying a record with unresolved dependencies can cause errors during deployment or cause the record to be deployed incorrectly. Only enable this if you are sure that records can be deployed without all dependencies resolved
skipLwcActivation	false	When enabled LWC enabled OmniScripts will not get compiled into native LWC components and be deployed to the target org during deployment. Use this if you want to manually compile OmniScripts into LWC or have a batch process ot activate OmniScript LWCs in bulk.
useMetadataApi	false	When set LWC components are deployed using the metadata API instead of the tooling API. The tooling API is usually faster and thus the preferred way to compiled deploy LWC components. Disable this if you need to use the metadata API to deploy LWC components.
remoteScriptActivation	false	When enabled the deployment will activate OmniScripts in the target org using Anonyms Apex.

Custom logging

When using a custom logger it should be passed as an object that has 1 method called write which can either be a static member or instance member. Vlocode will pass all log entries to the logger which can then decide how to log them.

The write -function can be async in which case the returned promise is awaited.

export interface LogWriter {
    /**
     * Write a entry to the log
     * @param entry Entry to write
     */
    write(entry: LogEntry): void | Promise<void>;
}

Log entires passed to the logger are described as follows:

export interface LogEntry {
    level: LogLevel;
    time: Date;
    category: string;
    message: string;
}

Matching keys

Matching keys are used to match datapacks tp records in the target org. When determine whether to insert a new record or update an existing record the fields deified in matching key configuration object are used to query for the existing record in the target org. By default the matching keys are loaded from the DRMatchingKey__mdt object defined in the target org. The MatchingKeyFields__c contains the comma separated list of fields that uniquely records and vlocity_cmt__ObjectAPIName__c contains the API name of the SObject for which the matching key is defined.

All primary Datapack record types should have a matching key. If for any primary datapack record no matching key is defined the deployment will always insert new records and never update existing records.

When no matching key is defined for an embedded record type (not primary) the deployment will delete all existing records of that type and insert the new records as defined in the datapack. For example when you delete the matching key for ProductChildItemc the deployment will delete all existing ProductChildItemc records before deploying the new records. This is useful for child records that cannot be uniquely defined or for which you want to make sure that records deleted from the datapack are also removed from the target org.

If you define a none-unique matching key the deployment will report a warning when and will not attempt to lookup records to update for that datapack. For example when you define a matching key for Product2 with only the ProductFamily field the deployment will report a warning and will instead create new Product2 records instead of updating existing ones.

Modifying deployment process

Vlocode uses deployment specs to add specific logic per datapack or SObject type. A deployment spec contains Datapack specific logic and allows hooking into the deployment process to manipulate what will be deployed and executes post- and pre-deployment activation logic.

All deployment specs are maintained in a deployment-spec registry object that contains the spec classes and spec functions that are currently registered. To register a new spec function or object register it in the DatapackDeploymentSpecRegistry before starting the deployment.

See the datapack deployment spec interface for the supported hook functions.

Specs for standard datapacks is implemented under the ./deploymentSpecs folder.

Customization sample

import { DatapackDeploymentSpecRegistry } from '@vlocode/vlocity-deploy';

// This spec updates all product Names before deployment replacing
// the word `Apple` by `Pear` - does not change the actual datapack
DatapackDeploymentSpecRegistry.register(
    // A filter can either be a string, or a filter object that filters based on record type
    // or based on datapack type
    { recordFilter: /^Product2$/i },
    {
        // Preproceses the datapack prior to the deployment; matches all datapacks that have
        // Product2 as object and updates the name of a datapack property
        preprocess(datapack) {
            // Update the product names prior to deployment
            datapack.Name = datapack.Name.replace('Apple', 'Pear');
        },
    }
);

DatapackDeploymentSpecRegistry.register(
    { recordFilter: /^Product2$/i },
    {
        afterRecordConversion(records) {
            // Validate records and add a warning message when they do not
            // match the expectations - this can also be used to make adjustments
            // to the records or add dependencies; see the `DatapackDeploymentRecord` documentation
            // for all the operations that can be performed on the record object
            for (record of records) {
                if (record.Name?.includes('Apple')) {
                    record.addWarning('you should not deploy apples');
                }
            }
        },
    }
);

// or register single functions that will run on a specific hook
DatapackDeploymentSpecRegistry.register('Product2', 'afterDeploy', event => {
    for (record of event.records) {
        console.log(`Record ${record.recordId}: ${record.status} - time: ${record.deployTime}`);
    }
});

Overview of hook functions

Hook	Params	Description
preprocess	VlocityDatapack	This function executes before a datapack is converted into a set of DatapackDeploymentRecord's and allow changing the datapack data, correct faulty field values that would otherwise cause errors during record conversion or remove certain fields that should not be deployed. This hook can also be used to override certain fields with autogenerated values; i.e. setting the element order and level field of OmniScript elements.
afterRecordConversion	readonly DatapackDeploymentRecord[]	This hook is similar to the preprocess hook except that it executes after the datapack to record conversion step. This hook receives list of records that contains all the records generated for a single datapack; it is not possible to remove/drop any record but individual records can be manipulated. Note At this point dependencies are not yet resolved
beforeDeploy	DatapackDeploymentEvent	This hook is called before deploying the first record in a datapack, and is only called once very datapack in the deployment. Use this hook to execute and pre-deployment actions. This action is comparable with the pre-step job from the Vlocity tools library.
afterDeploy	DatapackDeploymentEvent	This hook is called after all records of a datapack are deployed and will only be called once for every datapack in the deployment. Use this hook to execute datapack activation logic; it is comparable to post-step in Vlocity tools and should also execute any activation logic if required.
beforeDeployRecord	readonly DatapackDeploymentRecord[]	Executed just before the records are being deployed; the event contains a readonly array of records. At this stage values for the records can still be manipulated. This hook point should only be used to run logic that depends on all dependencies on individual record level are resolved.
afterDeployRecord	readonly DatapackDeploymentRecord[]	Executes after a group of records is deployed, at this stage it is not possible to change any field value anymore. The hook point can be used to run post-deployment validations or post-deployment activation processes.

Datapack deployment spec interface

export type DatapackFilter =
    | { recordFilter?: RegExp | string; datapackFilter: RegExp | string }
    | { recordFilter: RegExp | string; datapackFilter?: RegExp | string };

interface DatapackDeploymentSpec {
    preprocess?(datapack: VlocityDatapack): Promise<any> | any;
    afterRecordConversion?(records: readonly DatapackDeploymentRecord[]): Promise<any> | any;
    beforeDeploy?(event: DatapackDeploymentEvent): Promise<any> | any;
    afterDeploy?(event: DatapackDeploymentEvent): Promise<any> | any;
    beforeDeployRecord?(event: readonly DatapackDeploymentRecord[]): Promise<any> | any;
    afterDeployRecord?(event: readonly DatapackDeploymentRecord[]): Promise<any> | any;
}