W3lib
Python library of web-related functions
Install / Use
/learn @scrapy/W3libREADME
===== w3lib
.. image:: https://github.com/scrapy/w3lib/actions/workflows/tests-ubuntu.yml/badge.svg :target: https://github.com/scrapy/w3lib/actions/workflows/tests-ubuntu.yml
.. image:: https://img.shields.io/codecov/c/github/scrapy/w3lib/master.svg :target: http://codecov.io/github/scrapy/w3lib?branch=master :alt: Coverage report
Overview
This is a Python library of web-related functions, such as:
- remove comments, or tags from HTML snippets
- extract base url from HTML snippets
- translate entites on HTML strings
- convert raw HTTP headers to dicts and vice-versa
- construct HTTP auth header
- converting HTML pages to unicode
- sanitize urls (like browsers do)
- extract arguments from urls
Requirements
Python 3.10+
Install
pip install w3lib
Documentation
See http://w3lib.readthedocs.org/
License
The w3lib library is licensed under the BSD license.
Related Skills
node-connect
352.0kDiagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps
claude-opus-4-5-migration
111.1kMigrate prompts and code from Claude Sonnet 4.0, Sonnet 4.5, or Opus 4.1 to Opus 4.5
frontend-design
111.1kCreate distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.
model-usage
352.0kUse CodexBar CLI local cost usage to summarize per-model usage for Codex or Claude, including the current (most recent) model or a full model breakdown. Trigger when asked for model-level usage/cost data from codexbar, or when you need a scriptable per-model summary from codexbar cost JSON.
