xurlfind3r

xurlfind3r is a command-line utility designed to discover URLs for a given domain in a simple, efficient way. It works by gathering information from a variety of passive sources, meaning it doesn't interact directly with the target but instead gathers data that is already publicly available. This makes xurlfind3r a powerful tool for security researchers, IT professionals, and anyone looking to gain insights into the URLs associated with a domain.

Resource

• Features
• Installation
  • Install release binaries (Without Go Installed)
  • Install source (With Go Installed)
    • go install ...
    • go build ... the development version
  • Install on Docker (With Docker Installed)
• Post Installation
• Usage
• Contributing
• Licensing

Features

• Fetches URLs from multiple online passive sources to provide extensive results
• Supports stdin and stdout for easy integration in automated workflows
• Supports multiple output formats (JSONL, file, stdout)
• Cross-Platform (Windows, Linux, and macOS)

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

  wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz
  

• ...or, with curl:

  curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz
  

...then, extract the binary:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

[!TIP] The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv

[!NOTE] On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.

...move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

[!NOTE] Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

go build ... the development version

• Clone the repository

  git clone https://github.com/hueristiq/xurlfind3r.git 
  

• Build the utility

  cd xurlfind3r/cmd/xurlfind3r && \
  go build .
  

• Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

  sudo mv xurlfind3r /usr/local/bin/
  

  Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

[!CAUTION] While the development version is a good way to take a peek at xurlfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Install on Docker (With Docker Installed)

To install xurlfind3r on docker:

• Pull the docker image using:

  docker pull hueristiq/xurlfind3r:latest
  

• Run xurlfind3r using the image:

  docker run --rm hueristiq/xurlfind3r:latest -h
  

Post Installation

xurlfind3r will work right after installation. However, some sources require API keys to work. These keys can be added to a configuration file at $HOME/.config/xurlfind3r/config.yaml, created upon first run, or set as environment variables.

Example of environment variables for API keys:

XURLFIND3R_KEYS_BEVIGIL=your_bevigil_key
XURLFIND3R_KEYS_ONTELX=your_intelx_key

Usage

To start using xurlfind3r, open your terminal and run the following command for a list of options:

xurlfind3r -h

Here's what the help message looks like:

                 _  __ _           _ _____
__  ___   _ _ __| |/ _(_)_ __   __| |___ / _ __
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
 >  <| |_| | |  | |  _| | | | | (_| |___) | |
/_/\_\\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_|
                                          v1.3.0

USAGE:
 xurlfind3r [OPTIONS]

CONFIGURATION:
 -c, --configuration string           (default: $HOME/.config/xurlfind3r/config.yaml)

INPUT:
 -d, --domain string[]                target domain
 -l, --list string                    target domains file path

 For multiple domains, use comma(,) separated value with `--domain`,
 specify multiple `--domains`, load from file with `--list` or load from stdin.

SCOPE:
     --include-subdomains bool        match subdomain's URLs

SOURCES:
     --sources bool                   list supported sources
 -u, --sources-to-use string[]        comma(,) separated sources to use
 -e, --sources-to-exclude string[]    comma(,) separated sources to exclude

OUTPUT:
     --jsonl bool                     output in JSONL(ines)
 -o, --output string                  output write file path
 -O, --output-directory string        output write directory path
 -m, --monochrome bool                stdout in monochrome
 -s, --silent bool                    stdout in silent mode
 -v, --verbose bool                   stdout in verbose mode

Contributing

Contributions are welcome and encouraged! Feel free to submit Pull Requests or report Issues. For more details, check out the contribution guidelines.

A big thank you to all the contributors for your ongoing support!

Licensing

This package is licensed under the MIT license. You are free to use, modify, and distribute it, as long as you follow the terms of the license. You can find the full license text in the repository - Full MIT license text.