Warning I've stopped maintaining ette.

ette

EVM-based Blockchain Indexer, with historical data query & real-time notification support 😎

Deploy your ette instance today

Table of Contents

• Why did you build ette ?
• What do I need to have to use it ?
• How to install it ?
• What are possible use cases of ette ?
• How do I generate APIKey(s) ?
• How to use it ?
  • Historical Data
    • Custom REST
      • Query historical block data
      • Query historical transaction data
      • Query historical event data
    • GraphQL ( Recommended )
      • Query historical block data
      • Query historical transaction data
      • Query historical event data
  • Real-time Data
    • Real-time block mining notification
    • Real-time transaction notification ( 🤩 Filters Added )
    • Real-time log event notification ( 🤩 Filters Added )
  • Snapshotting
    • Take snapshot
    • Restore from snapshot

Inspiration 🤔

I was looking for one tool which will be able to keep itself in sync with latest happenings on EVM based blockchain i.e. index blockchain data, while exposing REST & GraphQL API for querying blockchain data with various filters. That tool will also expose real time notification functionalities over websocket, when subscribed to topics.

It's not that I was unable find any solution, but wasn't fully satisfied with those, so I decided to write ette, which will do following

• Sync upto latest state of blockchain

• Listen for all happenings on EVM based blockchain

• Persist all happenings in local database

• Expose REST & GraphQL API for querying 👇, while also setting block range/ time range for filtering results. Allow querying latest X entries for events emitted by contracts.

  • Block data
  • Transaction data
  • Event data

• Expose websocket based real time notification mechanism for

  • Blocks being mined
  • Transactions being sent from address and/ or transactions being received at address
  • Events being emitted by contract, with indexed fields i.e. topics

• All historical data query requests must carry authentication header as APIKey, which can be generated by users, using webUI, packed with ette.

• All real-time event subscription & unsubscription requests must carry apiKey in their payload.

• It has very minimalistic webUI for creating & managing APIKey(s).

• It has capability to process blocks in delayed fashion, if asked to do so. To address chain reorganization issue, this is very effective. All you need to do, specify how many block confirmations you require before considering that block to be finalized in .env file. Now ette will do everything with block ( if real-time subscription mode is enabled, it'll publish data to clients who're interested i.e. subscribed ) expect putting it in persistent data store. Rather block identifier to be put in waiting queue, from where it'll be eventually picked up by workers to finally persist it in DB. Only downside of using this feature is you might not get data back in response of query for certain block number, which just got mined but not finalized as per your set up i.e. BlockConfirmations environment variable's value. You can always skip it, default value will be 0.

• ette can help you in taking snapshot of whole database, it's relying on, into a single binary file, where block data is serialized into Protocol Buffer format, efficient for deserialization also i.e. while restoring back from snapshot.

  • EtteMode = 4, attempts to take a snapshot of whole database.

• Restoring from snapshoted data file, can be attempted by ette when EtteMode = 5. Make sure you've cleaned backing data store before so & recreated database. [ Table migration to be automatically taken care of ]

• For snapshotting purposes, you can always set sink/ source data file in SnapshotFile in .env.

• 👆 snapshotting feature is helpful, if you're willing migrate ette to different machine or setting up new instance of ette. If you want to avoid a lengthy whole chain data syncing, you must take snapshot from existing instance of ette & attempt to restore from binary snapshot file in new ette instance.

And that's ette

Prerequisite 👍

• Make sure you've Go ( >= 1.15 ) installed
• You need to also install & set up PostgreSQL. I found this guide helpful.

Make sure you've pgcrypto extension enabled on PostgreSQL Database.

Check existing extensions using : \dx

Create extension using : create extension pgcrypto;

• Redis needs to be installed too. Consider following this guide.

Note : Redis v6.0.6 is required

Note : Setting password in Redis instance has been made optional from now on, though it's recommended.

• Blockchain Node's both HTTP & Websocket connection URL required, because we'll be querying block, transaction, event log related data using HTTP interface & listening for block mining events in real time over Websocket.

Installation 🛠

• First fork this repository & clone it, some where out side of GOPATH.

git clone git@github.com:username/ette.git

• Now get inside ette

cd ette

• Create a .env file in this directory.

  • Make sure PostgreSQL has md5 authentication mechanism enabled.
  • Please enable password based authentication in Redis Server
  • Skipping RedisPassword is absolutely fine, if you don't want to use any password in Redis instance. [ Not recommended ]
  • Replace Domain with your domain name i.e. ette.company.com
  • Set Production to yes before running it in production; otherwise you can simply skip it
  • ette can be run in any of 👇 5 possible modes, which can be set by EtteMode

  ---

  EtteMode | Interpretation --- | --- 1 | Only Historical Data Query Allowed 2 | Only Real-time Subscription Allowed 3 | Both Historical Data Query & Real-time Subscription Allowed 4 | Attempt to take snapshot from data in backing DB 5 | Attempt to restore data from snapshot file

  ---

  • For testing historical data query using browser based GraphQL Playground in ette, you can set EtteGraphQLPlayGround to yes in config file
  • For processing block(s)/ tx(s) concurrently, it'll create ConcurrencyFactor * #-of CPUs on machine workers, who will pick up jobs submitted to them.
  • If nothing is specified, it defaults to 1 & assuming you're running ette on machine with 4 CPUs, it'll spawn worker pool of size 4. But more number of jobs can be submitted, only 4 can be running at max.
  • 👆 being done for controlling concurrency level, by putting more control on user's hand.
  • If you want to persist blocks in delayed fashion, you might consider setting BlockConfirmations to some number > 0.
  • That will make ette think you're asking it 80 is latest block, which can be persisted in final data store, when latest mined block number is 100 & BlockConfirmations is set to 20.
  • This option is recommended to be used, at least in production, to address chain reorganization issue.
  • For range based queries BlockRange can be set to limit how many blocks can be queried by client in a single go. Default value 100.
  • For time span based queries TimeRange can be set to put limit on max time span ( in terms of second ), can be used by clients. Default value 3600 i.e. 1 hour.
  • If you're attempting to take snapshot/ restore from binary snapshot file, you can set SnapshotFile in .env file, to set sink/ source file name, respectively. Default file name echo $(echo $(pwd)/snapshot.bin) in i.e. from where ette gets invoked. Consider setting EtteMode correctly, depending upon what you want to attain.

RPCUrl=https://<domain-name>
WebsocketUrl=wss://<domain-name>
PORT=7000
DB_USER=user
DB_PASSWORD=password
DB_HOST=x.x.x.x
DB_PORT=5432
DB_NAME=ette
RedisConnection=tcp
RedisAddress=x.x.x.x:6379
RedisPassword=password
Domain=localhost
Production=yes
EtteMode=3
EtteGraphQLPlayGround=yes
ConcurrencyFactor=5
BlockConfirmations=200
BlockRange=1000
TimeRange=21600
SnapshotFile=snapshot.bin

• Create another file in same directory, named .plans.json, whose content will look like 👇.

  • This file holds subscription plans for clients, allowed by this ette instance.
  • Each plan is denoted by one unique name & deliveryCount, where deliveryCount denotes number of times data to be delivered to client application in 24 hours of time span.
  • Because each request must be accompanied with APIKey, ette knows which user is requesting for resources & how many were delivered successfully in last 24 hours of time span.
  • If one user crosses allowed request limit in 24 hours, no new request will be taken under consideration & any existing connection will stop delivering data to client.

Quick Tip : Setting deliveryCount is fully upto you. Please consider VM specifications before doing so.

{
    "plans": [
        {
            "name": "TIER 1