EOSIO™ Alpha Release: History Tools

As blockchains grow with each authenticated transaction, querying and reading history and state data proves to be a growing challenge. In order to tackle this challenge, we have been working on a series of History Tools designed to make data more accessible for EOSIO based blockchains. You can find more technical details and documentation in the History Tools repository on GitHub. Documentation for History Tools can be found here.

We have previously released the State History Plugin that makes accessing blockchain data easier. The state history plugin saves blockchain state data to a new flat-file format and exposes a websocket interface to read the block/state data. With the alpha release of the History-Tools solution, we take this one step further and introduce database fillers to read the state data and populate PostgreSQL and LMDB databases that can be queried with the specialized client and server WASMs.

Rich user experiences and reliable applications built on EOSIO require tools that allow applications to efficiently access and query data from the blockchain and smart contracts. By nature, blockchain systems store data in a sequential manner that doesn’t easily lend itself to scalable random access query and retrieval systems. 

As this is an alpha release, we expect to iterate changes throughout the development process. It is our hope that feedback from members of the community will help to provide us additional guidance as we roll out these tools.

WASM-Query Language

WASM-Query Language (WASM-QL) implements a client-server architecture to design and execute queries in WASM. This implementation pattern allows contract authors to design and code queries using the same tooling used to create contracts and leverage the client-server query architecture. It also minimizes the back-and-forth communication required between client and server optimizing speed and efficiency. Once enabled, this system would make it possible to inspect historical states at any given block height. In addition, since replication in this system is relegated away from nodeos towards low memory-intensive query processes, it is more conducive to scalability.

This offers a more streamlined approach for the general-purpose query engines that enable applications to gather data from the EOSIO smart contracts. In addition, authors of contracts and applications can design their own queries with the same toolset that contracts are designed with, providing overall greater synergy.

There are two types of queries that wasm-ql-pg supports:

• Binary queries are the most flexible. A client-side WASM packs the query into a binary format, a server-side WASM running in WASM-QL executes the query then produces a result in binary format, then the client-side WASM converts the binary to JSON. The toolset helps authors create both WASMs.
• An emulation of the /v1/ RPC API.

In time, we may choose to drop the client-side WASMs and switch the format of the first type of query to JSON RPC, Graph QL, or another format. We appreciate community feedback as we make these decisions, so please comment in the repository with the recommendation you support.

Fill-pg

Fill-pg is designed to offer applications a lightweight solution to gather all pertinent data they would normally receive from monitoring the main chain. The information gathered includes:

• Header information from each block
• Transaction and action traces, including inline actions and deferred transactions
• Contract table history, at the block level
• Tables which track the history of chain state, including
  • Accounts, including permissions and linkauths
  • Account resource limits and usage
  • Contract code
  • Contract ABIs
  • Consensus parameters
  • Activated consensus upgrades

Fill-pg cuts dead weight by excluding blocks from the equation since they don’t include the inline actions and deferred transaction data that applications need. It supports both full and partial history so users can decide if they want to query the entire chain or a subsection. 

Fill-lmdb, wasm-ql-lmdb

This pair operates identically to fill-pg and wasm-pg, however, data is stored using lmdb as opposed to postgresql; as an embedded database lmdb might provide administration advantages.

Making Blockchain Data Accessible

In order to deliver seamless user experiences for more complex applications that maintain the integrity of blockchain data, developers need robust access to deterministic databases that can be queried like their non-blockchain counterparts. Blockchain nodes have a narrow scope of query options which causes problems to arise when complex applications try to store and retrieve indexed data. Traditionally, this left developers with two options; either processing the data within the application or storing sorted data on the blockchain itself which can be inefficient and expensive.

By reducing the back-and-forth between application databases and the EOSIO blockchain we’re following through on providing superior tools for Data Access Scalability, outlined in the EOSIO Strategic Vision. As we continue to receive feedback, we’ll continue to optimize methods that can reduce resource loads and help developers create more efficient application architectures.

Stay Connected

We appreciate the efforts of those who continue to help test and provide input on these tools as they evolve to best serve the EOSIO community. If you would like to offer feedback and work more closely with our team to improve EOSIO for developers, you can send our developer relations team an email at [email protected].

Remember, you can also keep up-to-date with future announcements by subscribing to our mailing list on the new EOSIO website. We are excited to be regularly improving the usability of the software for EOSIO developers as we continue to lay a foundation for the mass adoption of blockchain technology.

. . .

Important Note: All material is provided subject to this important notice and you must familiarize yourself with its terms. The notice contains important information, limitations, and restrictions relating to our software, publications, trademarks, third-party resources and forward-looking statements. By accessing any of our material, you accept and agree to the terms of the notice.