From Data Source to Intelligent Analysis: The Evolution of Blockchain Data Indexing Technology

1. Introduction

Since the first batch of decentralized applications ( dApp ) was launched in 2017, dApps on the Blockchain have flourished, covering various fields such as finance, gaming, and social networking. In this process, the on-chain data sources that dApps rely on have become a topic worth discussing.

In 2024, artificial intelligence and Web3 became hot topics. In the field of AI, data is the foundation for the system to "learn" and "think", just like the importance of sunlight and moisture for plants. Without sufficient data support, no matter how advanced the AI algorithms are, they cannot realize their potential.

This article will delve into the development of blockchain data accessibility, analyze the evolution of data indexing technologies, and compare traditional indexing protocols such as The Graph with emerging data service protocols like Chainbase and Space and Time that integrate AI technologies, discussing their similarities and differences in data services and product architecture.

2. The evolution of data indexing: from nodes to full-chain databases

2.1 Data Source: Blockchain Node

Blockchain nodes are the foundation of the entire network, responsible for recording, storing, and disseminating all on-chain transaction data. Although in theory anyone can run a node, in practice, most users prefer to rely on third-party services.

To address this issue, RPC node providers have emerged. They bear the costs and management of the nodes and provide data access services through RPC endpoints. While public RPC endpoints are free, they have rate limits; private RPC endpoints perform better, but are not efficient for complex queries and are difficult to scale.

2.2 Data Parsing: From Raw Data to Usable Data

The raw data provided by blockchain nodes is usually encrypted and encoded, and directly using this data requires a lot of technical knowledge and computing resources. The data parsing process converts complex raw data into a more understandable and operable format, which is a key link in the entire data indexing process.

2.3 Evolution of Data Indexers

As the amount of Blockchain data increases, the demand for data indexers is growing. Indexers organize on-chain data and provide a unified query interface, greatly simplifying the data retrieval process. Different types of indexers include full node indexers, lightweight indexers, specialized indexers, and aggregate indexers, each optimized for different scenarios.

Compared to traditional RPC endpoints, the indexer provides more efficient data indexing and query capabilities, supporting complex queries and multi-chain data aggregation, which enhances the efficiency and reliability of data retrieval.

2.4 Full-Chain Database: Aligning to Stream Priority

As application demands become more complex, traditional indexers gradually struggle to meet diverse query needs. The "stream-first" approach in modern data pipeline architecture provides new ideas for solving this problem, enabling real-time data processing and analysis.

Blockchain data service providers are also moving towards building real-time data streams. Products like The Graph's Substreams, Goldsky's Mirror, and real-time data lake services such as Chainbase and SubSquid aim to provide more comprehensive real-time data parsing and querying capabilities.

This new type of data pipeline allows us to view the management, storage, and application potential of on-chain data from a new perspective, providing customized high-performance data sets for various business scenarios.

3. AI + Database: In-depth comparison of The Graph, Chainbase, and Space and Time

3.1 The Graph

The Graph provides multi-chain data indexing and query services through a decentralized network of nodes. Its core products include a data query execution market and a data indexing cache market, catering to users' data query needs.

The network consists of four roles: indexers, curators, delegators, and developers, ensuring the system operates through an economic incentive mechanism. The Graph is also rapidly developing in the wave of AI, with tools developed by Semiotic Labs such as AutoAgora, Allocation Optimizer, and AgentC enhancing the system's intelligence and user-friendliness.

3.2 Chainbase

Chainbase is a full-chain data network that integrates all Blockchain data into one platform. Its features include:

• Real-time Data Lake
• Dual-chain architecture based on Eigenlayer AVS
• Innovative "manuscripts" data format standard
• Theia, a cryptocurrency world model combined with AI technology

Chainbase empowers data services through AI, providing not only data access but also proactive analysis, offering users broader data insights.

3.3 Space and Time

Space and Time (SxT) is dedicated to creating a verifiable computing layer that extends zero-knowledge proofs to decentralized data warehouses. Its core technology, Proof of SQL, implements an efficient data validation method, changing the resource consumption model under traditional consensus mechanisms.

SxT collaborates with Microsoft's AI Innovation Lab to develop generative AI tools that enable users to process blockchain data through natural language.

Conclusion and Outlook

Blockchain data indexing technology has evolved from the initial node data source, through the development of data parsing and indexers, to the AI-enabled full-chain data services, undergoing a process of gradual improvement. These technological advancements have not only enhanced the efficiency and accuracy of data access but also brought about an intelligent user experience.

In the future, with the development of new technologies such as AI and zero-knowledge proofs, Blockchain data services will further become intelligent and secure, continuing to drive industry innovation and progress as infrastructure.