Crawler
Distributed crawler written in PHP
Install / Use
/learn @teamtnt/CrawlerREADME
About
A distributed crawler
Requirements
Installation
Via Composer:
composer require teamtnt/crawler
Configuration
Each instance needs to have an identifier. This can be added in .env
NODE_NAME="Instance 1"
The domain feeder needs to start with a seed domain. After that, running
php artisan crawler
For scraping a single url
php artisan url:frontier www.example.com/something
Crawler Topology
Domain Feeder
Single Instance
URL Frontier
Related Skills
node-connect
348.5kDiagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps
frontend-design
109.1kCreate distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.
openai-whisper-api
348.5kTranscribe audio via OpenAI Audio Transcriptions API (Whisper).
qqbot-media
348.5kQQBot 富媒体收发能力。使用 <qqmedia> 标签，系统根据文件扩展名自动识别类型（图片/语音/视频/文件）。
