Bigset

<p align="center"> <img src="assets/logo-dark.png" alt="BigSet" width="280" /> </p> <p align="center"> <strong>Build and maintain any dataset from the live web, that refreshes regularly</strong> </p> <p align="center"> <a href="https://github.com/tinyfish-io/bigset/stargazers"><img src="https://img.shields.io/github/stars/tinyfish-io/bigset?style=flat" alt="GitHub Stars" /></a> <a href="https://github.com/tinyfish-io/bigset/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue" alt="License" /></a> <a href="https://github.com/tinyfish-io/bigset/issues"><img src="https://img.shields.io/github/issues/tinyfish-io/bigset" alt="Issues" /></a> <a href="https://x.com/Tiny_Fish"><img src="https://img.shields.io/twitter/follow/Tiny_Fish?style=flat" alt="Follow TinyFish" /></a> </p>

---

⚠️ BigSet is experimental. It works, sometimes surprisingly well, but expect rough edges. We're building in the open and shipping fast. Things will break, improve, and change. Issues and feedback are very welcome.

---

What Is BigSet?

You type a sentence:

"YC companies that are currently hiring engineers, with their funding stage, location, and number of open roles."

BigSet infers the schema automatically, sends autonomous agents to research it on the live web, verifies what they find against real sources, deduplicates, and hands you a structured dataset. Download as CSV or XLSX.

You can even set a refresh cadence (30 min, 6 hours, 12 hours, daily, weekly) and the agents re-run on schedule, pulling fresh data so the dataset never goes stale.

Built on TinyFish APIs.

Quick set up guide (~3 mins)

<p align="center"> <a href="https://youtu.be/ixuOBI9qUnQ"> <img src="assets/demo-thumb.jpg" alt="Watch the demo" width="100%" /> </a> </p>

✨ Why BigSet?

At the end of the day, every interaction with the web, whether it's you or your AI agent, ultimately comes down to data. Prices, companies, jobs, research, availability, inventory. The web has all of it, scattered across millions of pages.

There are great tools out there for parts of this problem. Scraping frameworks that extract content from URLs you point them at. Search APIs that return ranked results. Pre-built actors for specific sites. Lead gen platforms that produce verified lists of people and companies. They work, and they work well for what they do.

But the moment you need something that cuts across those categories, or something none of them cover, you're back to square one. Stitching together search, extraction, schema design, deduplication, verification, and a cron job to keep it fresh. For every dataset. Every time. The data is right there on the web. Getting it into a table you can use is still a project.

BigSet closes that gap. One sentence in, verified structured data out, refreshed on whatever cadence you set. Your agents get live data to reason over; you get a table you can actually use.

Any dataset. Any source. Always fresh. That's the idea.

How It Works

1. You describe the dataset in plain English, as vague or specific as you like
2. AI infers the schema: column names, types, primary keys, where to look on the web
3. An orchestrator agent discovers entities via web search
4. Sub-agents fan out in parallel: each one investigates a single entity, fetches real data, and inserts a verified row
5. You get a structured table: browse it in the UI, export CSV or XLSX
6. Set a refresh cadence and the agents re-run on schedule, keeping the dataset current automatically

Things to Know Before You Start

• It's experimental. Expect rough edges; schema inference isn't always perfect, and some topics work better than others.
• Dataset generation takes 2-5 minutes. The agents are doing real web research: searching, fetching pages, verifying data. It's not instant, but the output is real.
• It works best for topics with publicly available web data. If the information exists on public web pages, BigSet can probably find it. Data behind logins or paywalls is out of reach for now.
• Scheduled refresh keeps datasets current. Set a cadence (30 min to weekly) and the agents re-run automatically. No manual re-runs.
• Datasets are downloadable, not queryable. You can browse in the UI and export CSV/XLSX. SQL query support is on the roadmap.

---

🚀 Quick Start

Prerequisites: Node.js 22+ with npm. No Docker needed.

npm install --global @adamexu/bigset
bigset

That's it. The bigset command downloads the current local BigSet release, starts Convex, the backend, the frontend, and the local credential bridge, then prints the app URL. Open 127.0.0.1:3500 in your web browser to use it.

The first run caches release files under ~/.bigset; after that, starting BigSet is designed to take only a few seconds.

On first launch, BigSet sends you to setup. You'll connect two services:

| Service | What it's for | Get your key | |---------|--------------|-------------| | TinyFish | Web search + page fetching | tinyfish.ai/api-keys | | OpenRouter | LLM calls (schema inference + agents) | openrouter.ai/settings/keys |

Local API keys are stored in your OS keychain.

For a one-off run without installing globally:

npx @adamexu/bigset

Useful local options:

| Command | What it does | |---------|-------------| | bigset --force | Redownload the latest cached release | | bigset --app-port 4500 --backend-port 4501 | Use alternate app/backend ports | | bigset --home ~/.bigset-dev | Use a separate local cache directory |

CLI dataset commands

Once bigset is running and setup is complete, you can create and export datasets directly from your terminal.

Create a dataset:

bigset create "fintech startups in the bay area"

Create a dataset, wait for population to finish, and export CSV:

bigset create "fintech startups in the bay area" --rows 10 --wait --csv fintech.csv

List datasets:

bigset list

Inspect an existing dataset:

bigset status <datasetId>
bigset rows <datasetId> --json

Export an existing dataset:

bigset export <datasetId> --csv out.csv

Manage a run:

bigset populate <datasetId>
bigset stop <datasetId>

Useful create flags:

| Flag | What it does | |------|-------------| | --rows <n> | Maximum rows to collect | | --wait | Keep the command open until population finishes | | --csv <path> | Write populated rows to a CSV file | | --skip-populate | Create the dataset without starting population | | --cadence manual\|30m\|6h\|12h\|daily\|weekly | Set refresh cadence | | --backend-port <port> | Target a non-default local backend port |

Agents such as Codex or Claude Code can use the same commands. A typical agent workflow is:

bigset create "AI infrastructure startups hiring backend engineers" --rows 30 --wait --csv ai-infra-hiring.csv

The agent can then inspect ai-infra-hiring.csv and summarize or transform the results.

---

Developing From Source

Use this path when you're changing BigSet itself. The supported development workflow is still make dev.

Prerequisites: Node.js 22+ with npm, Docker, and Make.

Step 1: Clone the repo

git clone https://github.com/tinyfish-io/bigset.git
cd bigset

Step 2: Start everything

make dev

make dev creates a local .env if needed, installs dependencies, builds and starts all Docker services (Postgres, Convex, frontend, backend, Mastra), and deploys the Convex schema. On first run, it automatically generates the Convex admin key. See How make dev Works for the full breakdown.

Once everything is ready, you'll see:

| Service | URL | |---------|-----| | BigSet app | localhost:3500 | | Convex dashboard | localhost:6791 | | Mastra Studio (workflow inspector) | localhost:4111 |

Open localhost:3500. The setup screen will ask for TinyFish and OpenRouter credentials and save them to your OS keychain for this workspace.

Step 3: Connect TinyFish and OpenRouter

TinyFish powers web search and page fetching. OpenRouter routes LLM calls to the models BigSet uses for schema inference and agents.

1. Create a TinyFish key at agent.tinyfish.ai/api-keys
2. Create an OpenRouter key at openrouter.ai/settings/keys
3. Paste both into BigSet's setup screen

OpenRouter is pay-as-you-go; $5-10 is plenty to start.

Note: root .env is the only local env file. If you edit Convex functions in frontend/convex/, run make convex-push to deploy the changes.

Free tier: cloud signed-in accounts get 2,500 row operations per calendar month (resets on the 1st, UTC). Local mode bypasses the cloud quota and uses your TinyFish/OpenRouter accounts directly.

Step 4 (optional): Load curated datasets

BigSet includes 9 curated public datasets (AI companies hiring, GPU prices, model pricing, etc.) that show on the landing page:

make seed-public-datasets

This is idempotent; safe to run multiple times.

---

How make dev Works

make dev is designed to handle everything — first run, subsequent runs, and recovery from bad state. You should never need to run any other setup command. Here's what it does, in order: