reth-indexer

reth-indexer reads directly from the reth db and indexes the data into traditional and alternative databases / datastores (postgres, GCP bigquery, parquet, etc) all decoded with a simple config file and no extra setup alongside exposing a API ready to query the data.

<img src="./assets/demo.gif" />

Disclaimer

This is an R&D project and most likely has missing features and bugs. Also most likely plenty of optimistations we can do in rust land. PRs more then welcome to make this even faster and better.

Why

If you want to get data from the chain you tend to have to use a provider like infura or alchemy, it can get expensive with their usage plans if you are trying to just get event data. On top of that pulling huge amount of data fast is not possible freely. Over the wire JSONRPC calls adds a lot of overhead and are slow. You have the TLS handshake, you may be calling a API which is located the other side of the world to you, it adds TCP connections to your backend and scaling this is not easy mainly because of how over the wire JSONRPC calls work and with that your bill increases with your provider.

If you wish to build a big data lake or even just fetch dynamic events from the chain this task is near impossible without a third party paid tool or something like thegraph hoster services and most do not let you pull in millions of rows at a time fast. This data should be able to be fetched for free, blazing fast and customisable to your needs. This is what reth-indexer does.

This project aims to solve this by reading directly from the reth node db and indexing the data into traditional databases that you can then query fast with indexes already applied for you. You can also scale this easily by running multiple instances of this program on different boxes and pointing them at the same underlying database (we should build that in the tool directly).

This tool is perfect for all kinds of people from developers, to data anaylsis, to ML developers to anyone who wants to just get a snapshot of event data and use it in a production app or just for their own reports.

Features

• Creates database tables for you automatically (postgres, gcp biquery, parquet, or others)
• Creates indexes on the tables for you automatically to allow you to query the data fast
• Can write index to multiple data stores (i.e. postgres, gcp bigquery, parquet, etc)
• Indexes any events from the reth node db
• Supports indexing any events from multiple contracts or all contracts at the same time
• Supports filtering even on the single input types so allowing you to filter on every element of the event
• Snapshot between from and to block numbers
• No code required it is all driven by a json config file that is easy to edit and understand
• Created on your own infrastructure so you can scale it as you wish
• Exposes a ready to go API for you to query the data (postgres implementation only at this time)
• Abstractions to allow for extension of this indexer to different databases / datastores

Benchmarks

Very hard to benchmark as it is all down to the block range and how often your event is emitted but roughly: (most likely could be speed up with more some optimisations). Note this applies only to the postgres database write.

• indexes around 30,000 events a second (depending on how far away the events are in each block)
• scans around 10,000 blocks which have no events within 400ms using blooms

Head to head

Only compared what I can right now but happy for others to do head to head Note: this applies to reth-indexer indexing to local postgres database

providers:

• The Graph Hosted (Substreams) - note we are not comparing legacy as it is 100x slower then substream so you can do the maths on legacy

| Task | The Graph Hosted (Substreams) | reth Indexer | reth % faster | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------ | ------------- | | indexing reth (rocket pool eth) transfer/approval events both starting on block 11446767 and finishing on block 17576926<br/>config to run yourself here | 19.5 hours <br/> https://thegraph.com/hosted-service/subgraph/data-nexus/reth-substreams-mainnet | 15.9 minutes | 73.5x faster |

Note on benchmarking

We should compare this tool to other resync tools which go from block N not one which is already resynced. If you have the resynced information already then the results will always be faster as the data is already indexed. This goes block by block scanning each blockm for information so the bigger block ranges you have of course the longer it takes to process. How fast it is depeneds on how many events are present in the blocks.

Indexes

This right now it is just focusing on indexing events it does not currently index ethereum transactions or blocks, that said it would not be hard to add this functionality on top of the base logic now built - PR welcome.

• [] Indexes blocks
• [] Indexes transactions
• [] Indexes eth transfers
• [x] Indexes and decodes event logs

Requirements

• This must be ran on the same box and the reth node is running
• You must have a postgres database running on the box (note: this applies only if writing to the postgres database)

How it works

reth-indexer goes block by block using reth db directly searching for any events that match the event mappings you have supplied in the config file. It then writes the data to a csv file and then bulk copies the data into the postgres database (or other databases, as configured). It uses blooms to disregard blocks it does not need to care about. It uses CSVs and the postgres COPY syntax as that can write thousands of records a second bypassing some of the processing and logging overhead associated with individual INSERT statements, when you are dealing with big data this is a really nice optimisation.

The database write optimizations (i.e. CSV copy) apply mainly to the postgres database (and potentially other locally-running databases). Other databases / datasources, and in particular remote / cloud databases may not have this optimization built-in.

How to use

Syncing

• git clone this repo on your box - git clone https://github.com/joshstevens19/reth-indexer.git
• create a reth config file example structure is in reth-indexer-config-example.json, you can use cp reth-indexer-config-example.json reth-indexer-config.json to create the file with the template. Note you can map this config file anywhere but by default it will look for reth-indexer-config.json in the root of the project, this can be overwritten with the CONFIG flag.
• map your config file (we going through what all property means below)
• run RUSTFLAGS="-C target-cpu=native" CONFIG="./reth-indexer-config.json" cargo run --profile maxperf --features jemalloc to run the indexer
  • <strong>notes CONFIG can be replaced with any location you wish as long as it points to the config file</strong>
• see all the data get synced to your postgres database

TIP: You can use the handy CONFIG flag to allow you to run many indexers at the same time side by side in its own process.

Advise

reth-indexer goes block by block this means if you put block 0 to an end block it will have to check all the blocks - it does use blooms so its very fast at knowing if a block have nothing we need, but if the contract was not deployed till block x then its pointless use of resources, put in the block number the contract was deployed at as the from block number if you wanted all the events for that contract. Of course you should use the from and to block number as you wish but this is just a tip.

API

You can also run an basic API alongside this which exposes a REST API for you to query the data. This is not meant to be a full blown API but just a simple way to query the data if you wish. This is not required to run the syncing logic. Alongside you can resync data and then load the API up to query the data.

• you need the same mapping as what you synced as that is the source of truth
• run RUSTFLAGS="-C target-cpu=native" API=true CONFIG="./reth-indexer-config.json" cargo run --profile maxperf --features jemalloc to run the api
  • <strong>notes CONFIG can be replaced with any location you wish as long as it points to the config file</strong>
• it will expose an endpoints on localhost:3030/api/
  • The rest structure is the name of ABI event name you are calling so:

  {
    "anonymous": false,
    "inputs": [
      {
        "indexed": true,
        "internalType": "address",
        "name": "from",
        "type": "address"
      },
      {
        "indexed": true,
        "internalType": "address",
        "name": "to",
        "type": "address"
      },
      {
        "indexed": false,
        "internalType": "uint256",
        "name": "value",
        "type": "uint256"
      }
    ],
    "name": "Transfer",
    "type": "event"
  }
  

  • if you wanted data from this event you query localhost:3030/api/transfer and it will return all the data for that event.
• you can use limit to define the amount you want brought back - localhost3030/api/transfer?limit=100
• you can use offset to page the results - localhost:3030/api/transfer?limit=100&offset=100
• you can use latest to order the results in ASC(false) or DESC(true) order - `localhost:3030/ap