Censys
Python code to query the Censys public scan database.
Install / Use
/learn @gelim/CensysREADME
Python code to query the Censys public scan database. This script is made around library censys-python (https://github.com/Censys/censys-python) and is inteded to make censys queries quick & easy from command-line.
- Requirements
- Usage
- Examples
- Generic query IP or host (look for anything matching the string in Censys indexed data)
- Count how much web servers have 'SAP' in their
Serverheader - Get geo reparition of server with 'ABAP' in their
Serverheader - Filter hosts by their HTML title
- Retrieve the hosts that have SSL certificate with organization 'Whatsapp'
- Highlight which keys are matching string for a specific search query
- Printing raw JSON record from database for a specific request
- Listing example of fields we only want to dump in the records
- Export to disk sites frontpage containing "Hacked by" in their title
- Use tags provided by censys scanner to look at servers that need to be secured
- But what the heck, you want to hack the planet?!
Requirements
You need to create an account on https://censys.io and get your API key and secret at https://censys.io/account
Important note: your queries will be throttled. What is allowed is 0.2 tokens/second (60.0 per 5 minute bucket).
$ sudo pip install -r requirements.txt
Usage
$ censys_io.py --help
usage: censys_io.py [-h] [-m MATCH] [-f FILTER] [--count] [-r REPORT]
[-B REPORT_BUCKET] [-a ASN] [-c COUNTRY] [-o CERT_ORG]
[-i CERT_ISSUER] [-s CERT_HOST] [-S HTTP_SERVER]
[-t HTML_TITLE] [-b HTML_BODY] [-T TAGS] [--api_id API_ID]
[--api_secret API_SECRET] [-d] [-v] [-l LIMIT] [-H]
[--tsv]
[arguments [arguments ...]]
Censys query via command line
-- gelim
positional arguments:
arguments Censys query
optional arguments:
-h, --help show this help message and exit
-m MATCH, --match MATCH
Highlight a string within an existing query result
-f FILTER, --filter FILTER
Filter the JSON keys to display for each result (use value 'help' for interesting fields)
--count Print the count result and exit
-r REPORT, --report REPORT
Stats on given field (use value 'help' for listing interesting fields)
-B REPORT_BUCKET, --report_bucket REPORT_BUCKET
Bucket len in report mode (default: 50)
-a ASN, --asn ASN Filter with ASN (ex: 36040 for Google Inc.)
-c COUNTRY, --country COUNTRY
Filter with country
-o CERT_ORG, --cert-org CERT_ORG
Cert issued to org
-i CERT_ISSUER, --cert-issuer CERT_ISSUER
Cert issued by org
-s CERT_HOST, --cert-host CERT_HOST
hostname cert is issued to
-S HTTP_SERVER, --http-server HTTP_SERVER
Server header
-t HTML_TITLE, --html-title HTML_TITLE
Filter on html page title
-b HTML_BODY, --html-body HTML_BODY
Filter on html body content
-T TAGS, --tags TAGS Filter on specific tags. E.g: -T tag1,tag2,... (use keyword 'list' to list usual tags
--api_id API_ID Censys API ID (optional if no env defined
--api_secret API_SECRET
Censys API SECRET (optional if no env defined)
-d, --debug Debug informations
-v, --verbose Print raw JSON records
-l LIMIT, --limit LIMIT
Limit to N results
-H, --html Renders html elements in a browser
--tsv Export result of search in TSV format
For full details about the formatting rules for arguments see search syntax in page
https://censys.io/ipv4/help?q=x%3Ax
For a quick and dirty test, you can build queries like:
foo AND bar(will do a smart search by checking all keys with value foo and bar)path.to.key:fookey:foo(shortcut of previous, but will give strange results if there are collision with other keys)key:/regex/(regexp support via operator '/')key:"long string with spaces"(need to quote those strings)key:[200 TO 300](int range queries)key:192.168.0.0/24(IP range query)
Note about looking for substrings
Censys is backed by Elasticsearch, plus they filter the requests for performance reasons.
If you want to look for all values beginning with the string
Whatsapp, you can look for key:Whatsapp*. If you look only for
key:Whatsapp you will get only fields that where analyzed (strings
cut into pieces depending on language rules, specific tokens, etc.)
and contains facets with the exact string "Whatsapp".
An example to highlight that is the values stored in the key
443.https.tls.certificate.parsed.subject.organization. By looking for 443.https.tls.certificate.parsed.subject.organization:Whatsapp you will find ~90 results. Those will be entries with values:
WhatsApp Inc.WhatsApp Company LtdWhatsApp
But you will miss the values WhatsApp, Inc. that has ~350
entries. If we check censys.io "Data definitions" this field should be
analyzed as a "String" and the comma should be removed by the
tokenizer but something is not working as expecting.
So you need to be very careful when looking for substrings and try
different methods by either doing wildcards search (beware that Censys
disable beginning search term with a wildcard) or by using pure regexp
like /.*Whatsapp.*/.
Examples
Generic query IP or host (look for anything matching the string in Censys indexed data)
Let's search for IP entries that contain the string "nmap" in one of their keys.
$ censy_io.py --limit 20 nmap
Number of results: 1002
5.196.225.134 Title: N/A SSL: dawidstachowiak.pl AS: OVH, (16276) Loc: FR / OS: N/A Tags: http, ssh, https
74.115.246.29 Title: BrainDump SSL: philmcclure.duckdns.org AS: ENERGIZE (19215) Loc: US / Pulaski OS: N/A Tags: http, ssh, https
104.237.156.37 Title: Starlight Networking Security Lab SSL: AS: LINODE-AP (63949) Loc: US / Absecon OS: N/A Tags: http, ssh
69.160.84.231 Title: N/A SSL: AS: FIBER (5048) Loc: US / Orem OS: CentOS Tags: http, ssh
45.79.82.183 Title: nweb.io SSL: nweb.io AS: LINODE-AP (63949) Loc: US / Absecon OS: N/A Tags: http, ssh, https
60.32.137.218 Title: Kyodo2.0 Digital-Lab News Map Project SSL: localhost.localdomain AS: OCN (4713) Loc: JP / Tokyo OS: Fedora Tags: dhe-export, rsa-export, http, https
104.237.129.231 Title: Ninja.Style SSL: AS: LINODE-AP (63949) Loc: US / Absecon OS: Ubuntu Tags: http, ssh
192.109.14.42 Title: PASA Pallas SSL: pasa.pallas.com AS: PALLAS-AS, (24861) Loc: DE / OS: N/A Tags: http, https
45.33.32.156 Title: Go ahead and ScanMe! SSL: AS: LINODE-AP (63949) Loc: US / Absecon OS: Ubuntu Tags: http, ssh
104.224.137.222 Title: SSL: AS: IT7NET (25820) Loc: US / Phoenix OS: CentOS Tags: http
119.81.35.59 Title: SL Labs SSL: AS: SOFTLAYER (36351) Loc: SG / Singapore OS: CentOS Tags: http
81.27.98.98 Title: Check for Web Servers and more SSL: AS: UK-NETCETERA (24851) Loc: GB / OS: Debian Tags: http
212.237.16.237 Title: Infosec Notes SSL: 2d8.ru AS: ARUBA-ASN, (31034) Loc: DK / OS: Ubuntu Tags: http, smtp, https
198.23.94.99 Title: SL Labs SSL: AS: SOFTLAYER (36351) Loc: US / San Jose OS: CentOS Tags: http
77.109.162.35 Title: Citrin Toolbox SSL: AS: INIT7, (13030) Loc: CH / OS: N/A Tags: http
121.42.165.133 Title:
