Platform Integration of Omni Core
Last Updated Aug 27 2018
The following article aims to provide a recommend guideline for the integration of the Omni Core client into a platform for purposes of providing basic wallet functionality, balance information, transaction information, and/or transaction sending/receiving capabilities.
In some environments, use of a different wallet software will be required/preferred. During these integrations a more advanced setup will require the use of Omni Core's raw transaction creation rpc calls instead of the inbuilt omni_send or omni_sendall rpc calls in order to allow proper interfacing with other wallet software. Example documentation of using the raw transaction API to create a Simple Send transaction
- Requirements
- Installation
- Configuration and initial launch
- What is Omni
- Uniqueness and Asset Naming / Property ID
- Asset Balances/Ownership/UTXOs
- Client Versioning
- JSON-RPC-API
- Transaction Structure and Network Fees
- Transaction Types
- Integrating Omni Core to receive payments
- Block reorganizations
- Check token balances
- Send or Sweep tokens
- Best Practices
It is expected that you have already downloaded and installed the Omni Core client. Additionally it needs to be running and fully synced with the current blockchain before you can proceed with Mainnet Integrations/operations.
Note: The Omni Protocol is built on and uses the Bitcoin blockchain. As such, the Omni Core client is built over the Bitcoin Core code base and running Omni Core requires adequate space for the Bitcoin blockchain.
Please refer to the current installation information in the project repo
Omni Core is based on Bitcoin Core and utilizes the same configuration setup:
- The default configuration directory defaults to
~/.bitcoin/directory. - The default configuration file defaults to
bitcoin.conf
While most normal bitcoin core config options can be utilized there are a few critical pieces integrators will need to enable, including txindex and server options.
A sample config shown below:
rpcthreads=64 #adjust based on your system resources/need
server=1
txindex=1
rpcuser=<unique username>
rpcpassword=<unique password>
For a full list of configuration and launch options please see this wiki article.
Note: during initial launch the client will have to synchronize the current Bitcoin blockchain. If you notice this taking an unusually amount of time you may want to add the command line option -dbcache=xxxx where xxxx = some number of megabytes less than your system RAN. The default option is 300 MB, however for systems with multiple gigabytes of ram (more than 3 GB) setting this to 2000 or higher can help increase initial sync speeds. This value should never exceed the system ram amount as it can cause stability issues.
Note: Larger integrations which perform an excessive number of transactions from a Hot wallet may want to disable the feature which spends unconfirmed change transactions: spendzeroconfchange=0 . This feature can cause transaction delays or issues on the larger platforms when the outbound transaction volume is consistent and substantial.
Omni is a platform for creating and trading custom digital assets and currencies. It is a software layer built on top of the most popular, most audited, most secure blockchain -- Bitcoin. Omni transactions are Bitcoin transactions that enable next-generation features on the Bitcoin Blockchain. Our reference implementation, Omni Core is an enhanced Bitcoin Core that provides all the features of Bitcoin as well as advanced Omni Layer features.
Within the Omni Protocol, anyone can create their own new digital asset at anytime. When the protocol was first developed we decided that the individual uniqueness of a token would be represented by the Property ID and not by the Asset Name. The Property ID is a uniquely generated integer which is automatically assigned to new assets at the time the asset creation transaction is confirmed on the blockchain. This unique Property ID cannot be requested or duplicated and is only assigned by the protocol itself. Any transfer of funds from one address to another will reference the Property ID of asset being transferred. When it comes to identifying a digital asset, all integrators should check/verify using the Property ID. Asset names are designed for easy recognition but should be treated as secondary to the Property ID.
Unlike the Bitcoin network, balances on the Omni Protocol are not linked/tied to the specific bitcoin transactions that created them. The Omni Protocol treats all Asset Balances as a large state machine/ledger. Transactions that send/move asset balances are actually instructions that the protocol interprets and validates before updating the internal balance ledger respectively. In this way, all tokens are 100% owned/controlled by the address they are currently on. And only a transaction that is both valid on the Bitcoin network as well as validated by the Omni Protocol can affect a tokens balance. UTXOs are not linked to Omni Protocol balances and can be used as needed.
Because the Omni Protocol is governed by a set of rules, it is Critically Important that all integrators are using the same rules. Omni Core is our reference client and is the ultimate truth when it comes to interpreting these rules. We routinely release updates to the client, sometimes these updates are minor bug fixes to the RPC-API, other times there are critical consensus changes that will be activated at a later date. When critical consensus changes are released, It is crucial that all integrators update to the same version. We maintain an announcement/mailing list that all integrators should subscribe to:https://omnifoundation.page.link/announcements. It is used to inform users of the new client software when available.
As a safeguard, built into the client is automatic shutdown code which will cause older versioned clients to shutdown and prevent them from restarting when new consensus critical changes are activated. However this should not be relied upon by integrators as the method for notification for upgrade.
Being built on Bitcoin Core means that Omni Core inherits all the native Bitcoin Core RPCs. In addition to these commands there is also a list of Omni Protocol Specific RPC calls. These are available by calling the ‘omnicore-cli help’ command and also documented here
As Omni is built on and uses the Bitcoin network, all Omni Protocol transactions are, at their base, Bitcoin transactions. The Omni Protocol Data is typically encoded into an OP_RETURN output which is appended to the transaction. In order for the Omni Protocol data instructions to be validated, this transaction will need to be broadcast to the bitcoin network. This means it will need to be a valid Bitcoin transaction, with some BTC used for Inputs and Outputs and the bitcoin network mining fee.
The average size of an Omni Protocol transaction with 1 input is around 256 bytes. The native Bitcoin Core client has a built in fee estimation feature (see this post for some additional information). At the time of this writing the native Bitcoin Core code base is 0.13 and the fee estimation function is not the most reliable. It is therefore recommend that integrators utilize the config file option paytxfee and the RPC settxfee to adjust the fee per kilobyte used by the client to an appropriate value for the current network conditions.
Sites like Bitcoin Fees provide API information that can be periodically queried to determine appropriate current network fee rates.
The most common type of transaction an integration is likely to see is a Simple Send (protocol type_int 0). However in some cases it is likely that you may see other transaction types, Your setup should be able to identify the difference in types of transactions and know how to handle the ones relevant to your platform. The Omni Protocol spec (undergoing updates) can be referenced for a list of current transaction types.
Special attention should be made to identify a token freeze transaction (protocol type_int 185). This special transaction allows the Issuer of a Managed Token to Freeze all of their Property Tokens on a specific address and prevent those tokens from being sent/moved. Any integration that supports a managed token (example: Tether US #31) should take care to monitor for this transaction type and if received immediately freeze and corresponding users account pending further contact from the Asset Issuer.
The generally accepted practice for integrating Omni Core involves:
- Generating deposit addresses for your users
- Monitoring those addresses for Valid incoming transactions/deposits and then crediting the corresponding balance to your internal database balance for the user.
- Verifying balance information on your addresses and sweeping deposited funds to a central hot wallet
- Processing withdrawals for users from your central hot wallet
To generate a new destination to receive payments, the getnewaddress default RPC can be used. You should generate a unique destination address for each user if not for every incoming payment.
With the omni_gettransaction RPC information about a specific Omni Layer transactions can be retrieved. When a user sends tokens, it is usually done with a Simple Send transaction, which transfers tokens from the sendingaddress to the referenceaddress. An example output is given in the referenced documentation. It is crucial to check, whether the transaction is valid! This is indicated by the valid field.
When checking for received payments/transactions, the omni_listtransactions rpc call omni_listtransactions “” <count> can be used to return the most recent <count> transactions the wallet has received along with the corresponding transaction details information (same information returned by omni_gettransaction). Integrators should periodically check for and compare the receiving transactions in this list against the user deposit addresses they have generated/issued to check on which account the transaction may be referencing. Please take great care to read the next section on Validating Payments. Failure to do so could open your platform up to withdrawal exploitation
While unconfirmed Omni Layer transactions can be listed with the RPC omni_listpendingtransactions command, integrations should not take any ‘actionable’ events based on this information beyond information display about pending actions. Unconfirmed transactions are not valid and can not be validated until they are confirmed.
Advanced transaction Information Integrations that do not use the native omnicore wallet will need utilize a more complex transaction filtering setup. To list all Omni Layer transactions of a block, the omni_listblocktransactions RPC can be used. Using this call integrations can checking any/all transactions in a block against addresses they control to determine actionable events.
There is a ZMQ interface to receive push notifications about new blocks. The configuration option blocknotify may also be used to indicate, whether a new block was found. To get information about the length of the chain, the RPCs getblockcount and getbestblockhash can be used.
The Omni Core client will parse and decode any transaction sent on the Omni Protocol to allow for a full audit trail of any/all attempted actions on the protocol. The validity of a transaction can only be determined once the transaction has been confirmed on the blockchain AND processed by the Omni Core client.
When checking transactions with omni_gettransaction, omni_listtransactions or similar it is necessary to confirm, whether the transaction is indeed valid, which is indicated by the valid field. If a transaction is invalid, then there is an additional field invalidreason, which explicitly provides information about why the transaction is not considered valid, e.g. due to too low balance.
It is also highly recommended to confirm the actual balance of an user with the balance related RPCs described below.
Omni Layer transactions are depending on the order of transactions in blocks. In case of a block reorganization, the state can change. If this happens, previous information should be discarded and new information should be gathered. It is especially important to verify a user has enough balance, which can be done with the balance related RPCs as described below.
There are five RPCs to check balances: omni_getbalance, omni_getallbalancesforid, omni_getallbalancesforaddress, omni_getwalletbalances, and omni_getwalletaddressbalances. Because the global state of the Omni Layer depends on the order of transactions in blocks, relying on unconfirmed balances should never be done.
Using the native bitcoin core listreceivedbyaddress 1 true you can get a list of addresses in your wallet. There are a few options for generating the balances of your address
- Option 1: Iterate through your list of addresses and call omni_getallbalancesforaddress
- Option 2: If your platform only deals with a few specific Omni Assets you can use the omni_getallbalancesforid call to return a list of balance information for a specific property and check the returned list for any presence of you addresses. This has the benefit of only requiring 2 rpc calls instead of multiples
- Option 3: (Preferred) As of v0.3.1 there are native wallet balance rpc calls which will return either the total balance of all tokens in your wallet omni_getwalletbalances or the balances per address omni_getwalletaddressbalances.
Note: An address may have 3 different types of balances values when queries : balance, reserved, frozen.
-
balance: the current available balance that can be used/transferred in any valid transaction -
reserved: the current balance currently reserved/tied to an order on one of the Omni Protocol Decentralized Exchanges -
frozen: the total balance held by this address that has been frozen by the token issuer. Frozen funds can not be moved/transferred until unfrozen by token issuer.
For processing withdrawals or sweeping funds to a central hot wallet, a specified amount of tokens can be sent with the omni_send RPC. Note: all token transfers with simple send require the specification of the assets unique Property ID. Please ensure you are specifying the correct asset Property ID when sending.
It is also possible to sweep all available tokens of from a sending address with the omni_sendall RPC, which is recommended, because it ensures no tokens are left behind and it saves fees due to it's slightly lower payload. Note: Great care should be take to ensure that no ‘mix up’ occurs between your call of the omni_sendall vs omni_send when processing withdrawals. Just like Bitcoin transactions, once an Omni protocol transaction is broadcast and confirmed there is no ‘undo’ option. The only recovery mechanism is with the direct involvement of the recipient.
Broadcasting any transaction will require an appropriate amount of bitcoins on the sending address to cover transaction fees (see the previous section on Transaction Structure and Network Fees for more details). In the event the address you are trying to send/sweep funds from does not have enough BTC to cover the network fees (RPC interface returns an error Error with selected inputs for the send transaction) you will need to ‘top it up’ and send some BTC to it from an alternative address first.
Starting with v0.3.1 there are now two new funded calls: omni_funded_send and omni_funded_sendall. Similar to the original calls these will still send/sweep from one address to another. However the added benefit is that you can now specify a fee address that additional BTC can be taken/used from when the client creates the transaction, thus eliminating the need to ‘top up’ the original sending address before processing. As of v0.3.1 this is the recommended and preferred integration path.
The following is a list of recommended best practices and operations for integrators.
- You should subscribe to the Omni Core mailing list and update your client promptly when new consensus critical changes are released: https://omnifoundation.page.link/announcements
- Current network mining fees should be checked and (if needed) your client updated every 20-30 min.
- Transaction validity should always be confirmed before crediting a users balance
- You should check/watch for any potential block reorganizations and ensure deposits for users are credited after at least 3-6 confirmations. In the event of a reorganization you should immediately recheck any previous deposits from the past X blocks the block reorganization affected.
- When processing deposits/sweeps, you are reminded that you want to exclude your actual hot wallet address from any lists that you are trying to sweep from.
- To minimize redundant sweeps and maximize the effectiveness of a sweep for users doing multiple deposits daily, you may consider only running your sweep process periodically. Every 1-3 hours, to allow time for additional deposits / transfers to confirm and minimize duplicate sweeping efforts.
- Balance information does not include pending transactions, if you have already swept from an address you should take care to ensure the previous sweep is confirmed/processed and the balance information updated before trying to sweep again.
- You should ensure your hot wallet has the appropriate balance to process outgoing withdrawals. Take into account that the balance returned by omni_getbalance does NOT include any recently sent/pending withdrawals. It is your responsibility to ensure proper tracking of your pending balance.
- Outgoing withdrawals should be checked to ensure they have been broadcast properly and confirmed ‘valid’. In the event you send a withdrawal that turns out to be invalid, you should automatically refund your user and check why the tx was invalid to address any issues.
- Users are not infallible, they will screw up and try to send assets to addresses that may not support them. Once it leaves your platform you may not be able to do much more but you should take great care to ensure users are aware of what they are sending and where they ‘should be sending it’.
- If you distinguish deposit address in your platform for different assets, it is recommend to add an automatic check in your withdrawal system that ensures if a user tries to withdrawal Token X to their own deposit address for Token Y it is not processed (i.e. withdrawing USDt to a BTC deposit address in the same account. It happens, frequently. )
- Deposit addresses should be clearly marked with the token they support and users made aware of what is accepted and what is not.