Searching and Processing Blockchain Data with dfuse

We spoke with Marc-Antoine Ross, CEO of dfuse, about how blockchain developers need methods to efficiently process large amounts of data.

Could you introduce dfuse for us?

dfuse is a massively scalable open-source platform for searching and processing blockchain data. dfuse empowers developers with the capabilities they need to build modern blockchain applications with fast, fluid interfaces that deliver exceptional user experiences.

dfuse began its journey by seizing an opportunity to support the launch of the most performant blockchain software available, EOSIO, and providing open-source tooling. This involvement gave us first-hand knowledge of both the capabilities and the limitations of the latest generations of blockchain technology.

After quickly positioning ourselves as one of the technical leaders within the EOSIO software community, we were sought out by hundreds of developers who shared their challenges and obstacles when building on blockchain. This experience quickly identified the missing piece of the puzzle to put blockchain within reach of every developer: powerful APIs for reading and accessing blockchain data.

We quickly rolled out a query layer for EOSIO, bringing true searchability to blockchain for the first time ever. In October of 2018, we launched the free public beta of dfuse to the global community of EOSIO developers.

What does dfuse offer today?

dfuse’s primary goal was to make interacting with blockchain data as simple as interacting with standard databases. This meant building out custom APIs that delivered the performance, reliability, and high availability that developers were used to. Armed with a team of big data engineers, security experts, and top dev ops engineers, we knew that we could deliver this to the blockchain community in a way that had never been seen before.

Since that time we have been working to further deepen our platform capabilities. Today, our blockchain data stack is composed of 4 products, offered over REST, Websocket, GraphQL and gRPC APIs:

• dfuse Search: Real-time, historical and fork-aware search engine that enables you to search the entire transaction history of the blockchain using a simple but powerful query language, with sub-second response times.
• dfuse State: Offers snapshot views of the blockchain state, at any block height historically, in real time and is fully fork-aware.
• dfuse Lifecycle: Track the lifecycle of a transaction as it changes state: pending, included in a block, forked, replaced, failed, expired, etc.
• dfuse Push Guarantee: Submit transactions to the blockchain and receive guaranteed irreversibility in a single call, eliminating the need to write complex code to handle and recover from intricate failure cases.

We built dfuse from first principles, asking ourselves how we would want to read blockchain data if we weren’t constrained by the standard JSON-RPC interfaces. We provide higher-level blockchain APIs, which do more, with less effort, than the native APIs provided by blockchain nodes.

Why did you decide to use blockchain technology, and specifically EOSIO?

Simply put, our background in delivering solutions in other high-throughput environments led us to the blockchain platform that had the highest throughput: EOSIO.

Capable of thousands of actions per second, it was clear that EOSIO would need a great solution for reading that data. Coupled with fee-less transactions, it was clear that data (and not just value transfers) was going to be a big part of the future of EOSIO blockchains.

Developers love data, and need to be able to manipulate it. Whereas with other blockchains that were simply stores of value, there was yet to be a pressing need for a product that delivers the level of detail that dfuse does.

What stage is dfuse at?

dfuse is currently deployed on many blockchains based on EOSIO and Ethereum technologies (including public networks, consortium chains, and private chains). Other blockchain protocols are under active development. 

dfuse provides the massively scalable infrastructure for hundreds of public and private deployments, including exchanges, trading operations, interactive games, social media platforms, supply chain solutions, and high-throughput regulated environments. From hobbyist to multinational enterprise, dfuse can keep up with your needs.

What are the challenges you have had to overcome, and how did you do it?

The EOS Public Blockchain is unlike any other platform out there today. Its high transaction throughput and subsecond block times mean that the amount of blockchain data which must be stored by history providers can increase quickly.

Thanks to proper planning and foresight, we have developed our infrastructure to scale with the demands of the blockchain, enabling us to keep our users’ applications fed with the data they need to service their users. By initially architecting dfuse to handle the millions of transactions per second that Block.one strives to deliver with EOSIO, we have been able to scale our offering to provide reliability.

What are your plans for scaling?

Every day, we speak to more Enterprise users who are looking to integrate blockchain technology. Their needs are unique, but as we deliver solutions, those improvements trickle down to our global service. In turn, this creates a more robust and feature-rich service, creating a positive feedback loop. Our plan is to continually listen to those who seek out blockchain data, and figure out how to best deliver it to them. 

We are also developing solutions across different blockchain platforms, listening to the needs of developers across the entire blockchain ecosystem. We believe that in the future, developers will be multi-chain enabled. Rather than having to learn the nuances of interacting and reading from each blockchain, dfuse can be a middle layer that obfuscates complexity, while offering the benefits found from each platform.

What EOSIO community initiatives have you engaged in and how has the community responded to dfuse?

Most proudly, as the dfuse Platform, we open sourced our blockchain data stack under the Apache 2.0 license with Block.one embracing our solution. Open sourcing the dfuse data stack empowers developers to build better blockchain applications with free access to the most performant and scalable platform available.

It helps accelerate development efforts to integrate dfuse on additional blockchain platforms, as well as to develop extensions such as low-latency syncs, specialized indexers, big-data analytics, and block explorers, and help drive the discovery of new applications and use cases.

In addition, dfuse has contributed many open-source developments to the EOSIO ecosystem:

• dfuse Community Edition – Launched and hosted by EOS Nation, dfuse Community Edition is a freely available set of dfuse APIs. The mission is to provide greater access to data for all developers seeking to integrate with the EOS Public Blockchain.
• eosq – The most in-depth and high-precision block explorer available for an EOSIO network. A GUI for dfuse, it has access to all of the unique information that dfuse exposes.
• eosc – The most flexible and powerful command line tool for any developer to interact with an EOSIO chain.
• eosio.forum – A smart contract to handle asynchronous voting and messaging by account holders on an EOSIO network. Adopted to handle the needs of a referendum system for the EOS Public Blockchain. Also utilized to back a Reddit-style threaded forum, with all messages placed on chain.
• EOS BIOS Boot – We spent countless days and nights onboarding Block Producer candidates to EOSIO through the first open-source tool to launch a blockchain in a decentralized fashion.
• eosio.unregd – A smart contract containing the information of all Ethereum addresses of users who did not successfully register their accounts to claim their EOS tokens upon launch of the EOS Public Blockchain.

Can you introduce your team and tell us what makes them/it special?

The founding team is made up of serial entrepreneurs, with deep expertise in big data, cybersecurity, dev ops and blockchain. Combined, we have built and sold 6 companies, with the last one being sold to Intel. We have a passion for blockchain technology, and our goal is to help developers build performant applications by organizing the world’s decentralized data.

Marc-Antoine Ross has founded and operated several companies, and advised and invested in others. He has authored two patents, participated in many company acquisitions and was acquired twice, the second by Intel in 2014. Marc turned the rockstar team he assembled over the years into a high impact project within Intel. Marc wanted to build a great team again, and launched his latest venture in 2018 with dfuse.

Richard is a serial entrepreneur and company builder, currently active as an investor, advisor, and Board member with a number of high-growth companies at the intersection of blockchain, cybersecurity, cloud, and IoT, the same spaces in which he has built and successfully exited several companies, and in which he holds numerous patents.  

Alexandre loves to tackle difficult data problems. He wrote his first botnet at 12, later graduated in classical piano, and went on to a prolific career in software engineering, with notable open-source contributions. Alexandre co-founded two startups, including Bitcredits (a bitcoin payments processor). He then helped PasswordBox (acquired by Intel) craft their data stack and ended up as a lead Data Scientist in the Intel Security Consumer division. Today, Alexandre is very active in the blockchain space, advising several early stage companies.

Mathieu is a former Vice President in investment banking at National Bank Financial. His expertise is in both buy- and sell-side M&A transactions, as well as public and private equity issuances across several industries, with a focus on the technology and biotechnology sectors. He has been following the blockchain space for years, drawn in by its ability to radically transform financial structures for the better.

What other projects have you partnered with?

Having worked with and provided support for hundreds of active blockchain projects there is a long list of incredible teams building on EOSIO with whom we’ve had the pleasure of partnering.

Our deepest partnerships are currently with:

• Blockchain Service Network (BSN) – Led by the State Information Center of China, BSN provides infrastructure as a service to blockchain developers in an effort to simplify the onboarding process. Developers will be able to connect to any of the BSN-supported chains as simply as connecting to a cloud service today. BSN will offer dfuse endpoints for developers on both the EOS Public Blockchain and Ethereum mainnet, ensuring industry-leading data access to the developers they empower.
• EOS Nation – A Block Producer on the EOS Public Blockchain, EOS Nation is a strong technical team with whom we have collaborated on multiple projects within the EOSIO ecosystem. We are currently working with them in their efforts to host the dfuse Community Edition.
• MYKEY – Cross-chain and multi-currency wallet, enabling simple and secure access to blockchain assets. Based in China, MYKEY provides help and support to developers looking to integrate dfuse into their tech stack in one of the most blockchain-advanced countries in the world.
• Rewired.one – Providing enterprise support and strategic consulting services for blockchain engineers, Rewired.one has helped enterprises to adopt and integrate EOSIO technology to enable their vision. Based mainly in Australia, Rewired.one keeps us connected to a forward-looking continent filled with innovation.
• Ultra – With a mission to completely revolutionize the gaming industry, from initial development and funding, to in-game marketplaces and secondhand digital purchases, Ultra seeks to redistribute the power that gaming companies have taken from developers. Through this collaboration, Ultra has selected dfuse as their API solution for accessing historical data on the Ultra blockchain.
• WAX – Demystifying blockchain integration for the game developer as well as the end user, WAX is making ownership of in-game assets simple, quick, and most importantly, easily accessible. In an effort to greatly help developer adoption, dfuse endpoints on WAX have allowed multiple developers to port their applications to WAX simply by changing from one dfuse endpoint to another. This brought the barrier to entry to almost nothing for a developer when deciding on a multi-chain deployment.

Where do you see dfuse in the future?

We foresee dfuse as being the Elasticsearch of the new decade: an open-source platform for streaming and manipulating blockchain-based databases and data flows. 

We are building a solution to empower developers with better queries and streaming semantics than blockchain nodes can ever natively provide (because nodes are optimized for a different role). 

dfuse aims to enable massively parallel operations on ever-growing log-based blockchain databases and to offer performance that scales beyond what is currently being worked on by the most ambitious protocol teams.

In just one sentence, how do you want readers to remember dfuse? 

dfuse is a massively scalable open-source platform for searching and processing blockchain data.

. . .

Building on EOSIO?

Our #BuiltOnEOSIO series showcases some of the amazing projects leveraging EOSIO technology to build a more secure and connected world. If you would like to suggest a project for us to feature please send an email to [email protected] for our Developer Relations team to review.

–Block.one Developer Relations team

. . .

Important Note: All material is provided subject to this important notice and you must familiarize yourself with its terms. The notice contains important information, limitations and restrictions relating to our software, publications, trademarks, third-party resources and forward-looking statements. By accessing any of our material, you accept and agree to the terms of the notice.