TECHNICAL_REFERENCE.adoc

Skyflow connector Module Connector Reference

Core class of the connector. Here all the connector general constrain and usages are defined.

Configurations

---

Config

Default configuration

Parameters

Name	Type	Description	Default Value	Required
Name	String	The name for this configuration. Connectors reference the configuration with this name.		x
Connection	Credentials File   JWT connection	The connection types that can be provided to this configuration.		x
Name	String	The identifier of this element used to reference it in other components		x
Expiration Policy	Expiration Policy	Configures the minimum amount of time that a dynamic configuration instance can remain idle before the runtime considers it eligible for expiration. This does not mean that the platform will expire the instance at the exact moment that it becomes eligible. The runtime will actually purge the instances when it sees it fit.

Connection Types

Credentials File

Parameters

Name	Type	Description	Default Value	Required
Base URL	String	Base URL where the Skyflow instance lies.		x
Response Buffer Size	Buffer Size	Defines the size of the buffer in bytes used to store the HTTP response.
Proxy Config	Proxy Config
Credentials file path	String	Path to the credentials.json file required to authenticate the service account.		x
Token Expiration Time	Token Expiration Time	Expiration time of the bearer token. Default value is 30 minutes.
Response Timeout	Timeout
Client Timeout	Timeout	This sets the SO_TIMEOUT value on client sockets.
Connection Timeout	Timeout	Time to wait until an outbound connection to a remote server is successfully created.
Connection Idle Timeout	Timeout	Defines the amount of time that a connection can remain idle before being closed.
Reconnection	Reconnection	When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

JWT connection

Parameters

Name	Type	Description	Default Value	Required
Base URL	String	Base URL where the Skyflow instance lies.		x
Response Buffer Size	Buffer Size	Defines the size of the buffer in bytes used to store the HTTP response.
Proxy Config	Proxy Config
Client Id	String	The ID of the client required to authenticate the service account.		x
Key ID	String	The key ID required to authenticate the service account.		x
Token URI	String	The Token URI	https://manage.skyflowapis.com/v1/auth/sa/oauth/token	x
Private Key	String	The private key used to encrypt the JWT authentication request.		x
Token Expiration Time	Token Expiration Time	Expiration time of the bearer token. Default value is 30 minutes.
Response Timeout	Timeout
Client Timeout	Timeout	This sets the SO_TIMEOUT value on client sockets.
Connection Timeout	Timeout	Time to wait until an outbound connection to a remote server is successfully created.
Connection Idle Timeout	Timeout	Defines the amount of time that a connection can remain idle before being closed.
Reconnection	Reconnection	When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Associated Operations

• Detokenize  

• Tokenize  

Operations

Detokenize

<skyflow:detokenize>

Detokenizes a group of fields identified in a particular integration by invoking the Skyflow Gateway.

Parameters

Name	Type	Description	Default Value	Required
Configuration	String	The name of the configuration to use.		x
Fields	Any	A Map of fields with its values.	#[payload]
Config Ref	ConfigurationProvider	The name of the configuration to be used to execute this component		x
Streaming Strategy	Repeatable In Memory Stream Repeatable File Store Stream non-repeatable-stream	Configure if repeatable streams should be used and their behaviour
Connection ID	String	The Connection ID.		x
Relative Path	String	The path of the operation.		x
Target Variable	String	The name of a variable on which the operation’s output will be placed
Target Value	String	An expression that will be evaluated against the operation’s output and the outcome of that expression will be stored in the target variable	#[payload]
Reconnection Strategy	Reconnect Reconnect Forever	A retry strategy in case of connectivity errors

Output

Type

Any

For Configurations

• Config  

Throws

• SKYFLOW:CONNECTIVITY  

• SKYFLOW:INVALID_CONNECTION_ID  

• SKYFLOW:INVALID_FIELDS  

• SKYFLOW:INVALID_RELATIVE_PATH  

• SKYFLOW:NOT_FOUND  

• SKYFLOW:RETRY_EXHAUSTED  

• SKYFLOW:TIMEOUT  

• SKYFLOW:TRANSFORMATION  

Tokenize

<skyflow:tokenize>

Tokenizes a group of fields identified in a particular integration by invoking the Skyflow Gateway.

Parameters

Name	Type	Description	Default Value	Required
Configuration	String	The name of the configuration to use.		x
Fields	Any	A Map of fields with its values.	#[payload]
Config Ref	ConfigurationProvider	The name of the configuration to be used to execute this component		x
Streaming Strategy	Repeatable In Memory Stream Repeatable File Store Stream non-repeatable-stream	Configure if repeatable streams should be used and their behaviour
Connection ID	String	The Connection ID.		x
Relative Path	String	The path of the operation.		x
Target Variable	String	The name of a variable on which the operation’s output will be placed
Target Value	String	An expression that will be evaluated against the operation’s output and the outcome of that expression will be stored in the target variable	#[payload]
Reconnection Strategy	Reconnect Reconnect Forever	A retry strategy in case of connectivity errors

Output

Type

Any

For Configurations

• Config  

Throws

• SKYFLOW:CONNECTIVITY  

• SKYFLOW:INVALID_CONNECTION_ID  

• SKYFLOW:INVALID_FIELDS  

• SKYFLOW:INVALID_RELATIVE_PATH  

• SKYFLOW:NOT_FOUND  

• SKYFLOW:RETRY_EXHAUSTED  

• SKYFLOW:TIMEOUT  

• SKYFLOW:TRANSFORMATION  

Types

Buffer Size

Field	Type	Description	Default Value	Required
Size	Number	Amount of DataUnits that compose this buffer.	10
Data Unit	Enumeration, one of: BYTE KB MB GB	Data unit of the buffer.	KB

Proxy Config

Field	Type	Description	Default Value	Required
Host	String	The host of the proxy.		x
Port	Number	The port of the proxy.
Non Proxy Hosts	Array of String	HTTP proxy configuration for making HTTP requests through an NTLM authenticated proxy.
Password	String	Password of the HTTP proxy.
Username	String	Username of the HTTP proxy.

Token Expiration Time

Field	Type	Description	Default Value	Required
Amount	Number	Amount of TimeUnits that compose this duration.	30
Time Unit	Enumeration, one of: NANOSECONDS MICROSECONDS MILLISECONDS SECONDS MINUTES HOURS DAYS	Time unit of the duration.	MINUTES

Timeout

Field	Type	Description	Default Value	Required
Amount	Number	Amount of TimeUnits that compose this duration.	30
Time Unit	Enumeration, one of: NANOSECONDS MICROSECONDS MILLISECONDS SECONDS MINUTES HOURS DAYS	Time unit of the duration.	SECONDS

Reconnection

Field	Type	Description	Default Value	Required
Fails Deployment	Boolean	When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy
Reconnection Strategy	Reconnect Reconnect Forever	The reconnection strategy to use

Reconnect

Field	Type	Description	Default Value	Required
Frequency	Number	How often (in ms) to reconnect
Blocking	Boolean	If false, the reconnection strategy will run in a separate, non-blocking thread
Count	Number	How many reconnection attempts to make

Reconnect Forever

Field	Type	Description	Default Value	Required
Frequency	Number	How often (in ms) to reconnect
Blocking	Boolean	If false, the reconnection strategy will run in a separate, non-blocking thread

Expiration Policy

Field	Type	Description	Default Value	Required
Max Idle Time	Number	A scalar time value for the maximum amount of time a dynamic configuration instance should be allowed to be idle before it’s considered eligible for expiration
Time Unit	Enumeration, one of: NANOSECONDS MICROSECONDS MILLISECONDS SECONDS MINUTES HOURS DAYS	A time unit that qualifies the maxIdleTime attribute

Repeatable In Memory Stream

Field	Type	Description	Default Value	Required
Initial Buffer Size	Number	This is the amount of memory that will be allocated in order to consume the stream and provide random access to it. If the stream contains more data than can be fit into this buffer, then it will be expanded by according to the bufferSizeIncrement attribute, with an upper limit of maxInMemorySize.
Buffer Size Increment	Number	This is by how much will be buffer size by expanded if it exceeds its initial size. Setting a value of zero or lower will mean that the buffer should not expand, meaning that a STREAM_MAXIMUM_SIZE_EXCEEDED error will be raised when the buffer gets full.
Max Buffer Size	Number	This is the maximum amount of memory that will be used. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error will be raised. A value lower or equal to zero means no limit.
Buffer Unit	Enumeration, one of: BYTE KB MB GB	The unit in which all these attributes are expressed

Repeatable File Store Stream

Field	Type	Description	Default Value	Required
In Memory Size	Number	Defines the maximum memory that the stream should use to keep data in memory. If more than that is consumed then it will start to buffer the content on disk.
Buffer Unit	Enumeration, one of: BYTE KB MB GB	The unit in which maxInMemorySize is expressed