EOSIO-Taurus: Blockchain for Enterprises

Over the past two years Bullish has been developing, testing, and implementing a next generation version of EOSIO designed for Enterprise grade performance. We are now ready to open source these advancements and share them with the public. Meet EOSIO-Taurus.

https://github.com/EOSIO/taurus-node

Overview

Bullish is one of the top six cryptocurrency exchanges in the world in terms of total daily traded spot volume (Coin Metrics) and safeguarding customers’ assets without sacrificing performance has been our primary priority. We deliberately architected the Bullish Custody and Exchange systems from the ground up to provide:

• Cryptographic security for customers’ funds (all transactions require private keys).
• Non-repudiation for transactions (no super user and only customers can transact).
• Immutability & auditability of exchange operations for regulator inspections.
• The capacity to enforce that customer balances square up after every transaction.

We did an extensive survey of all existing blockchains and found that each provided unique subsets of capabilities but not all the required features. We therefore decided to start with the EOSIO codebase and forked it to build the unique feature set needed to satisfy our single-producer private blockchain business requirements. With these features being battle tested for the last two years by Bullish, and supporting $200+ billion trading volume, we hope other companies embarking on building applications leveraging single-producer private blockchains in a variety of domains will find them useful.

Further we are excited that this initiative helps Bullish contribute back to the open source ecosystem and strengthen the community.

The unique features we have built into EOSIO-Taurus

Some of the key requirements at Bullish were:

• The ability to handle a large number of transactions in a secure manner.
• Support for producer resilience, automatic failover and disaster recovery.
• Low latency.
• Event publishing from the blockchain to external systems.
• High performance queries of the blockchain data.
• Preserving the order of transactions .
• Providing debugging tools for developers to single step through smart contract execution.
• And more.

In this section we cover some of the features that make the EOSIO-Taurus blockchain uniquely suited for enterprises considering building applications on blockchains:

Support for producer high availability

To ensure high availability (HA) for enterprise blockchain deployments with 24×7 availability requirements, the producer_ha_plugin provides a block producer (BP) HA solution based on the RAFT consensus protocol by running multiple producer nodes to reach consensus through the Raft protocol for block production. Further, the solution also enables the blockchain to survive and recover from disasters (DR) in extreme cases such as when a whole datacenter is lost.

The producer_ha_plugin can provide:

• If any producing BP within the RAFT quorum goes down or block production stops, another BP will automatically take over as the producing BP to continue producing blocks, if it can do this safely. The delay is relatively short (within seconds).
• If there are conflicting blocks, one and only one will be broadcast and visible to the blockchain network.
• Only after a newly produced block has been broadcast to and committed by the quorum of BPs, the trace for the transactions in that block will be sent back to the client as execution results and as the confirmation of acceptance of the transaction.

To support DR, one or more independent producer HA groups can be deployed as standby groups in disaster independent regions/datacenters, so that the loss of a whole region/datacenter along with the producer nodes and hardware keys will not lead to significant data loss or down time of the blockchain.

Support for preserving the input order of transactions for enterprise use cases

The amqp_trx_plugin enables the consumption of transactions from an AMQP queue such as RabbitMQ, which is widely used in enterprise applications. The transactions are processed in a first-in first-out (FIFO) order, even when the producer (nodeos) switches during auto failover by the producer_ha_plugin.

This feature can make it easier to integrate the blockchain with enterprise applications that use queues widely. This is especially important in exchange and banking industries, with use cases where the order of transactions matter, for example, a deposit must occur before a withdrawal transaction can be executed.

Support for high performance query to the blockchain and streaming from the blockchain

Blockchains are notorious for being relatively easy to store data into the chain but provide very little tooling to extract data from the chain and act on that data in real-time. Here we provide two important innovations to address this shortcoming – a streaming interface and a high performance query interface.

Support for streaming from smart contracts to external systems

The event_streamer_plugin enables streaming of messages from smart contracts. The smart contracts can call the push_event intrinsic provided to send a message to an AMQP queue. The streaming support gives the ability to smart contracts to proactively update off-chain services in real time as the state changes are recorded on chain. For example, if a customer’s account is now authorized after AML checks have been completed on the chain then an event message pushed can notify the backend system to update that status without the backend system having to constantly poll the chain. Any nodeos in a blockchain cluster can be configured to push messages, and a cluster can be configured to have one or more dedicated nodeos instances for streaming certain messages so that the streaming solution can scale up horizontally.

High performance multithreaded queries of the blockchain state

To support queries of the blockchain state from large scale enterprise applications, the rodeos_plugin provides a high performance multithreaded query engine and interface to run concurrent read-only queries against the blockchain state. The plugin provides a series of RPC endpoints to support queries of data concurrently, enabling queries from many micro services for a large scale system.

The plugin incorporates all the functionality formerly provided by the rodeos binary and obviates the need for running a separate state_history_plugin to source the requisite data. At startup the plugin resyncs with the latest state from nodeos’ chainbase (the nodeos blockchain state database) through memory copies. The rodeos_plugin makes use of in-memory transfers of blockchain state from chainbase to the plugin at the end of production or replay of every block. Hence, the plugin itself does not need to maintain a durable copy of the latest state on disk between restarts, and the queries are handled from data in memory directly for higher performance.

Ability to debug and single step through smart contract execution

Debugging plays a critical role in the software engineering process to ensure the correctness of complex applications. This is especially useful in building financial systems where identifying and debugging edge cases is of paramount importance to ensure that the system operates as designed under all conditions. 

Smart contracts are compiled to WASM code to be run on nodeos. This carries both benefits and drawbacks. One drawback is that traditional debugging is not well supported for WASM code and it is especially difficult to debug within the nodeos virtual machine used for executing the WASM binaries. For this reason, EOSIO-Taurus provides a solution consisting of generating x86 native code files for contracts and a tester tool to execute and debug the native code files on the local machine, along with support in nodeos to load the native code file when the corresponding contract code is invoked. Since the blockchain execution is deterministic, the WASM execution and x86 execution are identical. The benefit provided by the x86 instruction set is that it easily allows for debugging via standard tooling like gdb and lldb, and it allows the contract developer to single step through the contract execution at run time when it is invoked in nodeos to enable easy debugging. With this support, contract developers can use a common debugger and single step through the smart contract execution, dramatically improving efficiency of the smart contract development.

Protocol Buffers support for contract action and blockchain data

Protocol Buffers is a portable data serialization/deserialization format used widely in enterprise applications. EOSIO-Taurus supports using Protocol Buffers as the data structure encoding format for blockchain and transaction data, including the action data, table data, and return values. Protocol Buffers has certain advantages for blockchain:

• ID based field encoding. Because the on-chain data history is immutable, we must make sure the formats are strictly controlled with the enforced ID based encoding/decoding. The field IDs ensure on-chain data and interface stability.
• Backwards compatibility support. Protocol Buffers makes it easy to upgrade the message data structure, like removing/adding fields. Manual code review is no longer heavily relied on to avoid corrupting on-chain data for on-chain data upgrading.
• Language-neutral message format, and extensive high quality libraries for various languages. With these libraries, there will be less code needed, and it will be faster to evolve the system. Microservices don’t have to struggle with the sometimes hardcoded serialization.
• Fast serialization/deserialization and compact binary message encoding. Further compiler provided optimizations can help make this code more efficient.

With the Protocol Buffers support, the same message format can be used among microservices and blockchain, making the integration easier, the on-chain data more stable, and the smart contract development more efficient. This is in contrast to the current ABI based encoding with a customized ABIEOS library which are then integrated in applications in other languages such as by JNI for access from Java which can make integrations error prone and complex.

Support for advanced signature provider and signature algorithms

Enterprise applications and deployments may have requirements on the signature providers and signature algorithms for higher security requirements and easier integrations. To meet such requirements, EOSIO-Taurus provides support for advanced signature providers and extends the signature algorithms and key formats supported.

TPM support for signatures providing higher security

To meet security requirements for enterprise deployments where non-extractable keys in hardware devices are preferred or required, EOSIO-Taurus includes a new TPM signature provider which allows nodeos and cleos to sign transactions and/or blocks with non-extractable keys from TPM devices.

Standard ECDSA key support in contracts for enterprise application integration

Standard ECDSA key formats are widely used by enterprise applications. EOSIO-Taurus adds support to the standard ECDSA key formats within contracts for easier integrations. The ECDSA public key follows the Standards for Efficient Cryptography.

RSA signature support in contracts for enterprise application integration

To make it easier to integrate enterprise applications that use the RSA algorithms, EOSIO-Taurus adds support to the RSA signature verification within contracts through performant intrinsics.

Ability to use snapshots for state persistence for stability and reliability

Previously, the blockchain state was persisted and stored as a shared memory map file on disk. The shared memory map file solution has two major drawbacks causing issues in enterprise deployments: the shared memory file is sensitive to changes in compiler, libc, and boost versions, and changes of compiler or library version will make an existing shared memory file incompatible; the shared memory file is not fault tolerant, and if the process crashes, the shared memory file left is likely in the “Dirty DB” state and it can not be used to reload the blockchain state.

To get rid of these limitations, EOSIO-Taurus persists the state in the portable snapshot format and also makes sure the state file creation is fault tolerant and crash safe. To support persisting the blockchain state as a snapshot, EOSIO-Taurus:

• Creates a snapshot during shutdown, and regularly spawns a background process with a clone of the process state by making use of the efficient copy-on-write memory cloning, to create a snapshot, and atomically store the snapshot file on the disk using the atomic file system APIs.
• Loads its state from the snapshot during restarts next time.
• Makes the EOSIO-Taurus VM Optimized Compiler (OC) cache in-memory, and makes the fork db crash safe.

With such mechanisms, the state persistent file as a snapshot is stable to successfully reload the nodeos. It could be only a little bit old if nodeos crashes. The stable snapshot-based blockchain state makes the blockchain system more reliable when running in unpredictable cloud environments where reboots may be initiated by the vendor without warning.

Ability to edit snapshots and import into local dev machines for debugging

One of the fundamental issues in application development is the discovery of bugs that only occur in production deployments and are not reproducible in developer’s testing environment. The usage of hardware keys in the production blockchain deployment also makes it impossible to continue a blockchain from the production environment in a testing or local machine as is. The EOSIO-Taurus blockchain provides some special debug function support that enables developers to download a state snapshot from the production machines and make changes to it (including replacing block signing keys and account keys) and continue production on the local machine. This enables microservices developers to test their local instantiation of the service to explore in-depth to pinpoint the source of the issue.

Support for long running time transactions for large scale contracts

Smart contracts implementing enterprise application logic may need to run on top of large scale data entries because of the complexity of the business logic and the scale of the blockchain state. For supporting such requirements, the EOSIO-Taurus producer supports long running time transactions for large contract actions by allowing the transaction execution time to exceed the block time, controlled by configuration parameters. Once the long running transaction is executed successfully, the producer can be reconfigured to respect the max transaction time appropriate for the network.

Asynchronous block signing for improving block production performance

Asynchronous block signing allows EOSIO-Taurus to ensure the block producing performance, and sign blocks using more secure yet slower signature providers to enhance the security.

Those more secure signature providers, such as a TPM device or an HSM hardware based signing service may need more time to sign a block. For example, signing blocks using a TPM device may need time in the approximate range from 30 to 60 milliseconds. As a high performance blockchain, EOSIO-Taurus can not be blocked by these devices for signing the blocks.

Previously, block signing was integrated into the block building process. In EOSIO-Taurus, block signing and block construction occur in separate threads. Block signing takes place after the completion of block construction, thus it can effectively utilize a TPM signature provider without slowing down block production.

Results and Summary

The EOSIO-Taurus blockchain has now been used by Bullish for over two years with more than 99.9% uptime and has supported well over $200B+ trading volume. It has incorporated newer innovations. For example, the zero downtime upgrades of the EOSIO-Taurus nodes including the active block producer node makes it possible for the exchange to upgrade components without any downtime to trading or custody operations. This makes it ideal, in our opinion, for enterprise scale deployments of the platform. In addition, the EOSIO-Taurus blockchain has proven itself to be a highly performant blockchain with more than 90% head room for future growth even under the heaviest load conditions on days of extreme volatility in the crypto market making it ideal for workload heavy applications. It should be noted that the EOSIO-Taurus blockchain, as deployed at Bullish, is a single producer private blockchain with unique requirements and as such other users, especially those dependent on multi-producer public blockchains, may need to perform additional due diligence before adopting these configurations verbatim.

We would also like to mention that EOSIO-Taurus continues to mature as an enterprise grade blockchain for use in the fin-tech industry and we are actively developing sophisticated features such as:

• Synchronous calls between contracts to radically simplify the call-back based inter-contract communication.
• On-chain atomic migrations to ensure that the contract code update and the migration of  its data will be completed atomically.
• Ability to create arbitrary secondary indexes on-chain to simplify evolution of KV table entries without requiring data migrations.

We look forward to communicating these and other exciting features that we are working on with the community in the near future.
For more details, please check the EOSIO-Taurus repository (https://github.com/EOSIO/taurus-node) and the documents included in the repository.

All repositories and other materials are provided subject to the terms of this IMPORTANT notice and you must familiarize yourself with its terms. The notice contains important information, limitations and restrictions relating to our software, publications, trademarks, third-party resources, and forward-looking statements. By accessing any of our repositories and other materials, you accept and agree to the terms of the notice.