SkillAgentSearch skills...

web-scraper-api

Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting

Install / Use

npx skills add oxylabs/agent-skills --skill web-scraper-api

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

88/100

Category

Automation

Supported Platforms

Universal

Our assessment of web-scraper-api

web-scraper-api scores 88/100 on our quality scale, 1568th of 2,790 Automation skills we index.

Its SKILL.md is 5.7 KB long, well organised into 12 sections with 6 code examples: a solid amount of guidance for an agent.

It has 875 GitHub stars, a meaningful sign that others use it.

Substance
26/30
Structure
20/20
Description
15/15
Adoption
13/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so web-scraper-api is actively maintained.
  • Our last check on 2026-11-01 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

web-scraper-api compared with similar skills

All 4 of these similar skills score higher than web-scraper-api; compare them before choosing.

SkillScoreStarsUpdatedFormat
web-scraper-api (this skill)by oxylabs8887559d agoSKILL.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
rufloby ruvnet10081.9ktodayCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md
Claude-Code-Game-Studiosby Donchitos10039.7k7d agoCLAUDE.md

Frequently asked questions

How do I install web-scraper-api?
Run npx skills add oxylabs/agent-skills --skill web-scraper-api. The install tabs above show the steps for each supported agent.
Which AI agents does web-scraper-api work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is web-scraper-api safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is web-scraper-api still maintained?
The repository was last updated about 2 months ago, so web-scraper-api is actively maintained.

name: web-scraper-api description: Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting. Use when the user needs to scrape web pages, extract product data, get search results, or collect structured data from supported e-commerce and search platforms without worrying about getting blocked and when geo targeting is required.

Oxylabs Web Scraper API

Authentication

Requires HTTP Basic Auth with credentials from environment variables:

curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...

Endpoint

POST https://realtime.oxylabs.io/v1/queries   # immediate response
POST https://data.oxylabs.io/v1/queries       # Push-Pull jobs, callbacks, storage
Content-Type: application/json

Core Parameters

| Parameter | Required | Description | |-----------|----------|-------------| | source | Yes | Target scraper (e.g., universal, amazon_product, google_search) | | url | Conditional | URL to scrape (for universal and *_url sources) | | query | Conditional | Search query or product ID (for *_search and *_product sources) | | parse | No | Enable structured data parsing (recommended for supported sources) | | render | No | JavaScript rendering: html or png | | geo_location | No | Geographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported | | session_id | No | Reuse the same proxy IP across multiple jobs | | content_encoding | No | Set to base64 when downloading image files via Realtime or Push-Pull | | user_agent_type | No | Device/browser preset, e.g., desktop_chrome, mobile_ios, tablet_android | | locale | No | Interface language / Accept-Language, e.g., de-DE | | callback_url | No | Push-Pull callback endpoint | | storage_type, storage_url | No | Push-Pull cloud upload target (gcs, s3, tos, s3_compatible) | | markdown, xhr | No | Enable markdown or captured XHR result types | | browser_instructions | No | Rendered browser actions; requires render: "html" | | parsing_instructions, parser_preset | No | Custom parser rules or saved preset; pair with parse: true | | client_notes | No | Client-side job tag saved with the job metadata | | domain, subdomain, start_page, pages, limit, store_id, delivery_zip, fulfillment_type | Source-specific | Marketplace/search/store localization and pagination fields |

user_agent_type values: desktop, desktop_chrome, desktop_edge, desktop_firefox, desktop_opera, desktop_safari, mobile, mobile_android, mobile_ios, tablet, tablet_android, tablet_ios.

Context Parameters

Add these as { "key": "...", "value": ... } objects in context:

| Key | Use | |-----|-----| | force_headers, headers | Merge custom headers with managed headers | | force_cookies, cookies | Merge custom cookies with managed cookies | | http_method, content | Use post with Base64-encoded body content | | follow_redirects | Follow 3xx redirect chains | | successful_status_codes | Treat specific non-standard HTTP codes as successful |

For multi-format output, enable types in the payload (parse, markdown, xhr, render: "png") and request them with ?type=raw,parsed,png,markdown,xhr.

For batch Push-Pull jobs, use POST /v1/queries/batch with arrays only for query or url; keep all other parameters singular. Maximum batch size is 5,000 values.

Quick Start

Scrape any URL:

curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "universal", "url": "https://example.com"}'

Google search with parsing:

curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "google_search", "query": "best laptops", "parse": true}'

Amazon product by ASIN:

curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'

Choosing the Right Source

  1. Use specific sources when available (amazon_product, google_search) - better parsing and reliability
  2. Use universal for unsupported sites - works with any URL
  3. Enable parse: true for structured JSON output on supported sources

Response Structure

{
  "results": [{
    "content": "...",
    "status_code": 200,
    "url": "https://..."
  }]
}

With parse: true, content contains structured data (title, price, reviews, etc.) instead of raw HTML.

Available Sources

For the complete list of 40+ supported sources organized by category, see sources.md.

More Examples

For detailed request/response examples including geo-location, JavaScript rendering, and custom headers, see examples.md.

Error Handling

| Code | Meaning | |------|---------| | 200 | Success | | 400 | Invalid parameters | | 401 | Authentication failed | | 403 | Access denied | | 429 | Rate limit exceeded |

Key Guidelines

  • Always set parse: true for supported sources to get structured data
  • Use ZIP codes for US e-commerce geo-location (e.g., "90210")
  • Use country/state format for search engines (e.g., "California,United States")
  • Add render: "html" for JavaScript-heavy pages
  • Use render: "" only to disable automatic forced rendering for force-rendered pages; set client timeouts near 180 seconds for rendered Realtime or Proxy Endpoint requests
  • Add content_encoding: "base64" when scraping image URLs, then decode results[0].content before saving the file

Related Skills

View on GitHub
GitHub Stars875
CategoryAutomation
Updated1mo ago
Forks2

Languages

JavaScript

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions