<p align="center"> <img src="img/webstor.png" width="95%"> </p><p>&nbsp;</p>

WebStor

Fast Identification of Vulnerable Web Technologies in your Organization

WebStor is a tool implemented in Python under the MIT license for quickly enumerating all websites across all of your organization's networks, storing their responses, and querying for known web technologies and versions, such as those with zero-day vulnerabilities. It is intended, in particular, to solve the unique problem presented in mid to large sized organizations with decentralized administration, wherein it can be almost impossible to track all of the web technologies deployed by various administrators distributed across different units and networks.

WebStor achieves its goal by performing the following actions:

1. Performs DNS zone transfers to collect an organization's A and CNAME records.
2. Uses Masscan to scan for open HTTP/HTTPS ports on an organization's net ranges, as well as any IP addresses outside those ranges that were present in the organization's A and CNAME records.
3. Uses the Python requests library to collect all responses and store in a MariaDB database. All DNS names corresponding to an IP with open HTTP/HTTPS ports will be included in requests in addition to the IP address, so that sites using different headers will not cause a website to be missed.
4. Downloads Wappalyzer web technologies database and stores in MariaDB database, enabling users to query the location(s) of a common web technology by name.
5. Allows users to query the location(s) where custom regexes are contained within stored responses.

Supported platforms

WebStor presently will run on Linux systems. As it is written in Python, conversion to support Windows would be trivial and is likely to happen in the future.

Prerequisites

Applications

• Masscan
  • If you will be using a cron job to update the database (typical), it is critical that you configure sudo nopasswd for any user executing Masscan scanning via WebStor.
• MariaDB 10.0.5 or later
  • The default credentials tried by WebStor will be root and a blank password. See the "Secure options" section for configuring WebStor to use other usernames and passwords to connect to the database.

Python libraries

• pip3 install dnspython
• pip3 install beautifulsoup4
• pip3 install mysql-connector-python
• pip3 install js-regex
• pip3 install gevent
• pip3 install requests

Availability via PyPI

• If you are simply looking to run WebStor and not edit it, you may install the prerequisite applications and then use 'sudo pip3 install webstor'.
• After installing WebStor via PyPI, webstor will be in the path and can be run with at the command line regardless of working directory with 'webstor' instead of 'webstor.py', e.g. 'webstor -g'.

Basic usage

       webstor.py [-h] [--ADD-HTTP-PORT HTTPPORTTOADD] [--CLEAR-HTTP]
                  [--ADD-HTTPS-PORT HTTPSPORTTOADD] [--CLEAR-HTTPS]
                  [--ADD-CUSTOM-FINGERPRINT FINGERPRINT]
                  [--DELETE-CUSTOM-FINGERPRINT FINGERPRINTNAMETODELETE]
                  [--IMPORT-CUSTOM-FINGERPRINT IMPORTFINGERPRINTFILE]
                  [--CLEAR-CUSTOM-FINGERPRINTS] [--SHOW-CONFIG]
                  [--SHOW-CONFIG-FULL] [--RUN-MASSCAN]
                  [--SET-MASSCAN-RANGES SETSCANRANGES]
                  [--ADD-RANGE RANGETOADD] [--DELETE-RANGE RANGETODELETE]
                  [--IMPORT-MASSCAN-RANGES IMPORTSCANRANGES]
                  [--ADD-PATH PATHTOADD] [--DELETE-PATH PATHTODELETE]
                  [--CLEAR-PATHS] [--REFRESH-RESPONSES]
                  [--RESPONSES-ADD-FOR-PATH RESPONSESADDFORPATH]
                  [--SEARCH-PATTERN SEARCHPATTERN]
                  [--SEARCH-CUSTOM-FINGERPRINT SEARCHFINGERPRINT]
                  [--SEARCH-WAPPALYZER SEARCHWAPPALYZER] [--NO-TSIG-KEY]
                  [--TSIG-KEY-IMPORT IMPORTTSIGFILE]
                  [--TSIG-KEY-REPLACE REPLACEMENTTSIGFILE]
                  [--DELETE-TSIG TSIGTODELETE]
                  [--USE-TSIG-FILE-ONLY USETSIGFILEONLY]
                  [--DOWNLOAD-NEW-WAPPALYZER] [--LIST-WAPPALYZER-TECH-NAMES]
                  [--ZONE-XFER] [--ADD-DOMAIN DOMAINDETAILS]
                  [--DELETE-DOMAIN DOMAINTODELETE]
                  [--IMPORT-ZONE-FILE IMPORTZONEFILE] [--CLEAR-DOMAINS]
                  [--LIST-DOMAINS] [--LIST-OUTSIDE] [--SQL-CREDS SQLCREDSFILE]

optional arguments:
  -h, --help            show this help message and exit
  --ADD-HTTP-PORT HTTPPORTTOADD, -a HTTPPORTTOADD
                        Add a custom HTTP port.
  --CLEAR-HTTP, -aC     Clear any custom HTTP ports and revert to default of
                        80.
  --ADD-HTTPS-PORT HTTPSPORTTOADD, -b HTTPSPORTTOADD
                        Add a custom HTTPS port.
  --CLEAR-HTTPS, -bC    Clear any custom HTTPS ports and revert to default of
                        443.
  --ADD-CUSTOM-FINGERPRINT FINGERPRINT, -c FINGERPRINT
                        Add a custom fingerprint in the form <Name>,<RegEx>.
  --DELETE-CUSTOM-FINGERPRINT FINGERPRINTNAMETODELETE, -cD FINGERPRINTNAMETODELETE
                        Delete a custom fingerprint by name.
  --IMPORT-CUSTOM-FINGERPRINT IMPORTFINGERPRINTFILE, -cI IMPORTFINGERPRINTFILE
                        Import a custom fingerprint file with the path
                        specified.
  --CLEAR-CUSTOM-FINGERPRINTS, -cC
                        Clears all custom fingerprints stored in DB.
  --SHOW-CONFIG, -g     Show current WebStor configuration (brief).
  --SHOW-CONFIG-FULL, -gF
                        Show current WebStor configuration (full).
  --RUN-MASSCAN, -m     Runs a new port scan with Masscan on all configured
                        TCP ports for HTTP and HTTPS, against all configured
                        ranges and any IP addresses from DNS records that are
                        outside those ranges.
  --SET-MASSCAN-RANGES SETSCANRANGES, -mR SETSCANRANGES
                        Scan range or ranges, replaces existing ranges in DB,
                        comma separated, such as: -s
                        10.10.0.0/16,10.13.0.0/16,192.168.1.0/24
  --IMPORT-MASSCAN-RANGES IMPORTSCANRANGES, -mI IMPORTSCANRANGES
                        Import scan ranges (CIDR blocks) from a specified
                        file.
  --DELETE-RANGE RANGETODELETE, -mD RANGETODELETE
                        Delete scan range.
  --ADD-PATH PATHTOADD, -p PATHTOADD
                        Add paths for which to request and store responses
                        besides '/'.
  --DELETE-PATH PATHTODELETE, -pD PATHTODELETE
                        Delete paths for which to request and store responses
                        besides '/'.
  --CLEAR-PATHS, -pC    Clear any custom URL request paths and revert to
                        default of '/'.
  --REFRESH-RESPONSES, -r
                        Refresh URL responses in DB.
  --RESPONSES-ADD-FOR-PATH RESPONSESADDFORPATH, -rP RESPONSESADDFORPATH
                        Add URL responses for a one-time path in with the
                        current responses in the DB.
  --SEARCH-PATTERN SEARCHPATTERN, -sP SEARCHPATTERN
                        Search for string or regular expression in WebStor
                        database.
  --SEARCH-CUSTOM-FINGERPRINT SEARCHFINGERPRINT, -sC SEARCHFINGERPRINT
                        Search for technology by name of user-provided custom
                        fingerprint.
  --SEARCH-WAPPALYZER SEARCHWAPPALYZER, -sW SEARCHWAPPALYZER
                        Search for technology by name (from Wappalyzer Tech
                        DB) in WebStor DB.
  --NO-TSIG-KEY, -tN    Do not use DNSSec TSIG key stored in database or a
                        file, even if present.
  --TSIG-KEY-IMPORT IMPORTTSIGFILE, -tI IMPORTTSIGFILE
                        Import a specified TSIG key file into the database
  --TSIG-KEY-REPLACE REPLACEMENTTSIGFILE, -tR REPLACEMENTTSIGFILE
                        Replace a TSIG key in the database with a specified
                        file
  --DELETE-TSIG TSIGTODELETE, -dT TSIGTODELETE
                        Delete a TSIG key from the database by name.
  --USE-TSIG-FILE-ONLY USETSIGFILEONLY, -tF USETSIGFILEONLY
                        Only use tsig file specified (full path), do not use
                        TSIGs stored in the DB. Applies to all domains,
                        limiting WebStor to one TSIG for zone transfers in the
                        current execution.
  --DOWNLOAD-NEW-WAPPALYZER, -w
                        Download a new Wappalyzer fingerprints file directly
                        from GitHub. Overwrites existing Wappalyzer
                        fingerprint data.
  --LIST-WAPPALYZER-TECH-NAMES, -wL
                        List the names of all Wappalyzer technologies in the
                        database.
  --ZONE-XFER, -z       Forces a new zone transfer using all domains, servers,
                        and associated TSIG keys in DB
  --ADD-DOMAIN DOMAINDETAILS, -zA DOMAINDETAILS
                        Add a domain in the form <Domain name>,<Server>,<TSIG
                        Key Name>.
  --DELETE-DOMAIN DOMAINTODELETE, -zD DOMAINTODELETE
                        Delete a DNS domain from the database by name.
  --IMPORT-ZONE-FILE IMPORTZONEFILE, -zI IMPORTZONEFILE
                        Add domains for zone transfers from a file.
  --CLEAR-DOMAINS, -zC  Clears all DNS domains stored in DB.
  --LIST-DOMAINS, -zL   Lists all DNS domains stored in DB.
  --LIST-OUTSIDE, -e    Prints a list of all names and IPs from our zone
                        transfers that are outside defined net ranges.
  --SQL-CREDS SQLCREDSFILE, -q SQLCREDSFILE
                        Use SQL credentials in file at specified path.

Steps to initially configure WebStor and populate database:

N