Nebula

Nebula is an agnostic network crawler. It starts with a set of bootstrap peers, asks them for other peers in the network and recursively repeats the process until all peers in the network have been contacted.

Currently, Nebula supports the following networks:

• IPFS - Amino DHT
• Bitcoin | Litecoin | Dogecoin (alpha)
• Ethereum - Consensus Layer (discv5) | Execution Layer (discv4)
• Optimism compatible chains
• Portal - (alpha - wire protocol not implemented)
• Filecoin
• Polkadot - Kusama | Rococo | Westend
• Avail - Mainnet | Turing | <small>Light Client + Full Node versions</small>
• Celestia - Mainnet | Mocha | Arabica
• Pactus
• Dria
• Gnosis
• Monero
• ... your network? Get in touch team@probelab.io.

You can run nebula networks to get a list of all supported networks.

Nebula supports the JSON, Postgres and ClickHouse storage backends. It powers many graphs on probelab.io for most of the supported networks above. Check out the "Topology" sections for any of the networks to get an idea. Gnosis uses Nebula to source its Network metrics here.

Commercial Offering

In addition to the open-source version, ProbeLab maintains a commercial offering of Nebula which includes:

• GossipSub Topic Tracking — detailed tracking of GossipSub topic subscriptions across the network.
• Ethereum Handshake Analysis — in-depth Ethereum protocol handshake data collection.
• Monero Network Crawling — specialized peer discovery for the Monero network.
• Algorand Network Crawling — specialized peer discovery for the Algorand network.

Going forward, new feature development will continue exclusively as part of the commercial offering. The open-source version will remain available and maintained for existing functionality, but all new capabilities will be part of the commercial offering.

For pricing and access, please reach out to team@probelab.io.

Awards

The crawler was:

• 🏆 awarded a prize in the DI2F Workshop hackathon. 🏆
• 🎓 used for the ACM SigCOMM'22 paper Design and Evaluation of IPFS: A Storage Layer for the Decentralized Web 🎓

You can find a demo on YouTube: Nebula: A Network Agnostic DHT Crawler 📺

<small>Grafana Dashboard is not part of this repository</small>

Table of Contents

• Table of Contents
• Project Status
• Usage
• Install
  • From source
• How does it work?
  • crawl
  • monitor
  • resolve
• Development
  • Database
  • Tests
• Report
• Related Efforts
• Demo
• Maintainers
• Contributing
• Support
• Other Projects
• License

Project Status

The crawler is powering critical IPFS Amino DHT KPIs, used for Weekly IPFS Reports as well as for many metrics on probelab.io. The main branch will contain the latest changes and should not be considered stable. The latest stable release that is production ready is version 2.4.0.

Install

Precompile Binaries

Head over to the release section and download binaries from the latest stable release.

From source

git clone https://github.com/dennis-tra/nebula
cd nebula
just build

Now you should find the nebula executable in the dist subfolder.

Usage

Nebula is a command line tool and provides the crawl sub-command.

Dry-Run

To simply crawl the IPFS Amino DHT network run:

nebula --dry-run crawl

[!NOTE] For backwards compatibility reasons IPFS is the default if no network is specified

The crawler can store its results as JSON documents, in a Postgres, or in a Clickhouse database - the --dry-run flag prevents it from doing any of it. Nebula will just print a summary of the crawl at the end instead. For the IPFS network, a crawl takes ~5-10 min depending on your internet connection. You can also specify the network you want to crawl by appending, e.g., --network FILECOIN and limit the number of peers to crawl by providing the --limit flag with the value of, e.g., 1000. Example:

nebula --dry-run crawl --network FILECOIN --limit 1000

To find out which other network values are supported, you can run:

nebula networks

JSON Output

To store crawl results as JSON files provide the --json-out command line flag like so:

nebula --json-out ./results/ crawl

After the crawl has finished, you will find the JSON files in the ./results/ subdirectory.

When providing only the --json-out command line flag you will see that the *_neighbors.json document is empty. This document would contain the full routing table information of each peer in the network which is quite a bit of data (~250MB for the Amino DHT as of April '23) and is therefore disabled by default

Track Routing Table Information

To populate the document, you'll need to pass the --neighbors flag to the crawl subcommand.

nebula --json-out ./results/ crawl --neighbors

The routing table information forms a graph and graph visualization tools often operate with adjacency lists. To convert the *_neighbors.json document to an adjacency list, you can use jq and the following command:

jq -r '.NeighborIDs[] as $neighbor | [.PeerID, $neighbor] | @csv' ./results/2025-02-16T14:32_neighbors.json > ./results/2025-02-16T14:32_neighbors.csv

Postgres

If you want to store the information in a proper database, you could run just start-postgres to start a local postgres instance via docker in the background and run Nebula like:

nebula --db-user nebula_local --db-name nebula_local crawl --neighbors

At this point, you can also start Nebula's monitoring process, which would periodically probe the discovered peers to track their uptime. Run in another terminal:

nebula --db-user nebula_local --db-name nebula_local monitor

When Nebula is configured to store its results in a postgres database, then it also tracks session information of remote peers. A session is one continuous streak of uptime (see below).

However, this is not implemented for all supported networks. The ProbeLab team is using the monitoring feature for the IPFS, Celestia, Filecoin, and Avail networks. Most notably, the Ethereum discv4/discv5 and Bitcoin monitoring implementations still need work.

---

There are a few more command line flags that are documented when you runnebula --help and nebula crawl --help:

How does it work?

crawl

The crawl sub-command starts by connecting to a set of bootstrap nodes and then requesting the information of other peers in the network using the network-native discovery protocol. For most supported networks these are several Kademlia FIND_NODE RPCs. For Bitcoin-related networks it's a getaddr RPC.

For Kademlia-based networks Nebula constructs the routing tables (kademlia k-buckets) of the remote peer based on its [PeerID](https://docs.libp2p.io/conc