21 skills found
hisorange / Browser DetectBrowser Detection for Laravel by hisorange!
scrapy-plugins / Scrapy Zyte SmartproxyZyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
JayBizzle / Laravel Crawler DetectA Laravel wrapper for CrawlerDetect - the web crawler detection library
Xyntax / FileSensorDynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具
loadkpi / Crawler DetectRuby gem to detect bots and crawlers via the user agent
ninoseki / UzenWebsite crawler with YARA detection
x-way / CrawlerdetectGolang module to detect bots and crawlers via the user agent
isonic1 / Appium Native CrawlerAppium Native Crawler CLI - Features include: Screenshots, Performance, Accessibility Detection, Google Translate, Applitools, Monkey Tester
mohammadhashemii / Web Crawler DetectionWeb Crawler Detection using Unsupervised Algorithms
jocialiang / Gender ClassifierDeep learning, Face detection, CNN, Tensorflow, Keras, OpenCV, Python crawler
tn3w / Flask HumanifyA strong bot protection system for Flask with many features: rate limiting, special rules for users, web crawler detection, and automatic bot detection.
stephenafamo / IsbotDetect bots/crawlers/spiders using the user agent string.
BryanMorgan / IsbotRust library to detect bots using a user-agent string
zivdar001matin / Web Crawler DetectionFinal project of Rahnema College internship. Crawler detection using unsupervised learning methods.
xfifix / SEO REPOKeywords enrichment by autocompletion (AWS, PM, RDC, CDS, ...), google suggestion scraping Heavy multithreaded semantic corpus crawler Similarity metrics implementing TF/IDF, duplicate content detection SERP inspector with multiple IPs and squid proxy Emulating geolocalisation with uule parameter Sitemap/List ultra fast crawler GWT API connection In-house Nutch plugin to parse ZTD, attributes completion
yangsijie666 / Xiaohongshu Crawler小红书数据采集工具 — 基于 Playwright 的隐匿爬虫，支持搜索、笔记详情、评论采集，双层反检测，导出 JSON/Excel，内置 MCP 协议服务 | Xiaohongshu (RedNote) stealth crawler with anti-detection
JerryZhi / AI Crawler DetectorTool for Fast Detection of Website/Server AI Crawler Blocking Policies（Not robots.txt）
arunprasanna / PopStar And PopRankMalicious Website Detection using Site Agility and Social Reputation. a web crawler to capture source code, links and take screenshot of a webpage. Also developed code for histogram analysis to compute similarity of images using Opencv. Used a machine learning classifier to classify websites into benign or malicious and also ranked search engine results such that water holing websites rank lower.
amirhossein-izadi / Web Crawler DetectionWeb Crawler Detection using Unsupervised Algorithms
Anajrajeev / DeepHarvestDeepHarvest is a Python web crawler with JavaScript rendering, distributed crawling, ML-based trap detection, and multilingual support. Extracts content from HTML, PDFs, Office docs, images, and media.
