GoScrapy: Web Scraping Framework in Go

GoScrapy aims to be a powerful web scraping framework in Go, inspired by Python's Scrapy framework. It offers an easy-to-use Scrapy-like experience for extracting data from websites, making it an ideal tool for various data collection and analysis tasks, especially for those coming from Python and wanting to try scraping in Golang..

Features

🚀 Blazing Fast — Built on Go's concurrency model for high-throughput parallel scraping
🐍 Scrapy-inspired — Familiar architecture for anyone coming from Python's Scrapy
🛠️ CLI Scaffolding — Generate project structure instantly with goscrapy startproject
🔁 Smart Retry — Automatic retries with exponential back-off on failures
🍪 Cookie Management — Maintains separate cookie sessions per scraping target
🔍 CSS & XPath Selectors — Flexible HTML parsing with chainable selectors
📦 Built-in Pipelines — Export scraped data to CSV, JSON, MongoDB, Google Sheets, and Firebase out of the box
🧩 Built-in Middleware Support — Plug in built-in middlewares.
🔌 Extensible by Design — Almost every layer of the framework — pipelines, middlewares, HTTP client, and selectors — is built to be swapped or extended without touching the core

Getting Started

Goscrapy requires Go version 1.22 or higher to run.

1: Project Initialization

go mod init books_to_scrape

2. Install goscrapy cli

go install github.com/tech-engine/goscrapy@latest

Note: make sure to always keep your goscrapy cli updated.

3. Verify Installation

goscrapy -v

4. Create a New Project

goscrapy startproject books_to_scrape

This will create a new project directory with all the files necessary to begin working with GoScrapy.

\iyuioy\go\go-test-scrapy> goscrapy startproject books_to_scrape

🚀 GoScrapy generating project files. Please wait!

✔️  books_to_scrape\constants.go
✔️  books_to_scrape\errors.go
✔️  books_to_scrape\job.go
✔️  main.go
✔️  books_to_scrape\record.go
✔️  books_to_scrape\spider.go

✨ Congrates. books_to_scrape created successfully.

spider.go

In your spider.go file, set up and execute your spider.

For detailed code, please refer to the sample code here.

package scrapejsp

import (
	"context"
	"encoding/json"
	"fmt"
	"log"

	"github.com/tech-engine/goscrapy/cmd/gos"
	"github.com/tech-engine/goscrapy/pkg/core"
)

type Spider struct {
	gos.ICoreSpider[*Record]
}

func NewSpider(ctx context.Context) (*Spider, <-chan error) {

	// use proxies
	// proxies := core.WithProxies("proxy_url1", "proxy_url2", ...)
	// core := gos.New[*Record]().WithClient(
	// 	gos.DefaultClient(proxies),
	// )

	core := gos.New[*Record]()

	// Add middlewares
	core.MiddlewareManager.Add(MIDDLEWARES...)
	// Add pipelines
	core.PipelineManager.Add(PIPELINES...)

	errCh := make(chan error)

	go func() {
		errCh <- core.Start(ctx)
	}()

	return &Spider{
		core,
	}, errCh
}

// This is the entrypoint to the spider
func (s *Spider) StartRequest(ctx context.Context, job *Job) {

	req := s.NewRequest()
	// req.Meta("JOB", job)
	req.Url("https://jsonplaceholder.typicode.com/todos/1")

	s.Request(req, s.parse)
}

func (s *Spider) Close(ctx context.Context) {
}

func (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {
	fmt.Printf("status: %d", resp.StatusCode())

	var data Record
	err := json.Unmarshal(resp.Bytes(), &data)
	if err != nil {
		log.Fatalln(err)
	}

	// to push to pipelines
	s.Yield(&data)
}

Wiki

Please follow the wiki docs for details.

Note

GoScrapy is not stable, so its API may change drastically. Please exercise caution when using it in production.

License

GoScrapy is available under the BSL with an additional usage grant that allows free internal use. Please ensure that you agree with the license before contributing to GoScrapy, as by contributing to the GoScrapy project, you agree to the terms of the license.

Roadmap

~~Cookie management~~
~~Builtin & Custom Middlewares support~~
~~Css & Xpath Selectors~~
Logging
Tests(work in progress)

Partners

Get in touch

Discord

AI generated doc

deepwiki

Note: accuracy of the ai generated doc hasn't been verified. Follow the Github wiki for accurate doc.

Goscrapy

Install / Use

README