Anton

This project is an open-source tool that extracts and organizes data from the TON blockchain, efficiently storing it in PostgreSQL and ClickHouse databases.

If you have any questions, you can ask them in the Telegram group.

Overview

Before you start, take a look at the official docs.

Consider an arbitrary contract. It has a state that is updated with any transaction on the contract's account. Each state has the contract code and data. The contract data can be complex, but developers typically provide get-methods in the contract, which can be executed to retrieve the necessary data. The TON has standard contracts (such as TEP-62, TEP-74), and they have predefined get-method names. Therefore, you can attempt to match accounts found in the network to these standards by checking the presence of the get-methods. Contract standards also specify TL-B constructor tags (or operation ids) for each acceptable message to contract, defined as the first 32 bits of the parsed message payload cell. So you if you know standard of a given contract, you can determine the type of message to it (for example, NFT item transfer) by parsing the first 32 bits of message body.

Anton allows you to define the contract interface in just one JSON schema. Format of every schema is described in detail in abi/README.md. Every schema comprises contract get-methods, as well as incoming and outgoing message schemas for the contract. Once contract interfaces are defined and stored in the database, Anton begins scanning new blocks on the network. The tool stores every account state, transaction, and message in the database. For get-methods without arguments in the contract interface, Anton emulates these methods and saves the returned values to the database. When a message is sent to a known contract interface, Anton attempts to match the message to a known schema by comparing the parsed operation ID. If the message is successfully parsed using the identified schema, Anton also stores the parsed data.

To explore contract interfaces known to this project, visit the abi/known directory. This will provide you with an understanding of the various contract interfaces already supported and serve as examples for adding your own.

Currently, Anton offers a REST API for retrieving filtered and aggregated data from the databases. To see example queries, refer to the API.md file.

To explore how Anton stores data, visit the migrations' directory.

Project structure

| Folder | Description | |--------------|---------------------------------------------------| | abi | get-methods and tlb cell parsing | | abi/known | contract interfaces known to this project | | api/http | JSON API Swagger documentation | | docs | only API query examples for now | | config | custom postgresql configuration | | migrations | database migrations | | internal | database repositories and services implementation |

Internal directory structure

| Folder | Description | |-------------------|----------------------------------------------------------------------------------| | core | contains project domain | | core/rndm | generates random domain structures | | core/filter | describes filters | | core/aggregate | describes aggregation metrics | | core/repository | implements database repositories with filters and aggregation | | app | contains all services interfaces and their configs | | app/parser | service determines contract interfaces, parse contract data and message payloads | | app/fetcher | service concurrently fetches data from blockchain | | app/indexer | service scans blocks and save parsed data to databases | | app/rescan | service parses data by updated contract description | | app/query | service aggregates database repositories | | api/http | implements the REST API |

Starting it up

Cloning repository

git clone https://github.com/tonindexer/anton
cd anton

Running tests

Run tests on abi package:

go test -p 1 $(go list ./... | grep /abi) -covermode=count

To run the tests you might need to provide a path to the emulator library. For example:

CGO_LDFLAGS="-L /Users/user/go/src/github.com/tonkeeper/tongo/lib/darwin/ -Wl,-rpath,/Users/user/go/src/github.com/tonkeeper/tongo/lib/darwin/ -l emulator" \
  go test -p 1 $(go list ./... | grep /abi) -covermode=count

Run repositories tests:

# start databases up
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d postgres clickhouse

go test -p 1 $(go list ./... | grep /internal/core) -covermode=count

Running linter

Firstly, install golangci-lint.

golangci-lint run

Configuration

Installation requires some environment variables.

cp .env.example .env
nano .env

| Name | Description | Default | Example | |-----------------------|------------------------------------|---------|--------------------------------------------------------------------| | DB_NAME | Database name | | idx | | DB_USERNAME | Database username | | user | | DB_PASSWORD | Database password | | pass | | DB_CH_URL | Clickhouse URL to connect to | | clickhouse://clickhouse:9000/db_name?sslmode=disable | | DB_PG_URL | PostgreSQL URL to connect to | | postgres://username:password@postgres:5432/db_name?sslmode=disable | | FROM_BLOCK | Master chain seq_no to start from | 1 | 23532000 | | WORKERS | Number of indexer workers | 4 | 8 | | RESCAN_WORKERS | Number of rescan workers | 4 | 8 | | RESCAN_SELECT_LIMIT | Number of rows to fetch for rescan | 3000 | 1000 | | LITESERVERS | Lite servers to connect to | | 135.181.177.59:53312 aF91CuUHuuOv9rm2W5+O/4h38M3sRm40DtSdRxQhmtQ= | | DEBUG_LOGS | Debug logs enabled | false | true |

Building

# building it locally
go build -o anton .

# build local docker container via docker cli
docker build -t anton:latest .
# or via compose
docker compose -f docker-compose.yml -f docker-compose.dev.yml build

# pull public images
docker compose pull

Running

We have several options for compose run via override files:

• base (docker-compose.yml) - allows to run services with near default configuration;
• dev (docker-compose.dev.yml) - allows to rebuild Anton image locally and exposes databases ports;
• prod (docker-compose.prod.yml) - allows to configure and backup databases, requires at least 128GB RAM.

You can combine it by your own. Also, there are optional profiles:

• migrate - runs optional migrations service.

Take a look at the following run examples:

# run base compose
docker compose up -d

# run dev compose (build docker image locally)
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d

# run prod compose
# WARNING: requires at least 128GB RAM
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

To run Anton, you need at least one defined contract interface. You can find some known interfaces in the abi/known directory. These can be added during the initial startup of the indexer:

# If you are running the indexer without Docker, 
# you need to specify the directory containing the known contracts:
anton indexer --contract-dir /root/anton/abi/known

# If you are running it using Docker, 
# there is no need to specify the contracts directory,
# as they are already included in the container:
docker compose up -d indexer

Database schema migration

# run migrations service on running compose
docker compose run migrations

Reading logs

docker compose logs -f

Taking a backup

# starting up databases and API service
docker compose                      \
    -f docker-com