Domain & URLs

Get information on an IP
whois <IP>

# Get IP address associated to a domain
nslookup domain.fr
ping domain.fr

# Knockknock is a small automated script allowing you to find domain names
# For a registrant (person or company)
python3 k2.py -n company -d

# Many tools
https://intelx.io/

# Estimate the creation date of a website
http://carbondate.cs.odu.edu/
https://github.com/Lazza/Carbon14

# Informations about the target
https://securitytrails.com/

# Robtex is a great and complete tool
https://www.robtex.com

# Getting technology informations
https://builtwith.com

# Mapping the target website can be good to get an overview
www.visualsitemapper.com

# Then you can find domains and subdomains associated to an IP by using Passive Reverse DNS
https://www.virustotal.com/gui/home/search

# RiskIQ Passive DNS

# You can identify new domains in the "RelationShip" section
# of Builtwith
https://builtwith.com/relationships/xxx.fr

# When you have a name or a e-mail adress you can perform a reverse whois lookup to find domain
# names oned by a person of a company
https://viewdns.info/reversewhois/

# You can parse and scrape javascript content in a target website to look for hidden subdomains or interesting paths
# Often, endpoints are not public but users can still interact with them
# Tools like dirscraper automates this (https://github.com/Cillian-Collins/dirscraper)

# Classic
python discraper.py -u <url>

# Output mode
python discraper.py -u <url> -o <output>

# Silent mode (you won't see result in term)
python discraper.py -u <url> -s -o <output>

# Relative URL Extractor is another good tool to scrape from JS files (https://github.com/jobertabma/relative-url-extractor)
ruby extract.rb https://hackerone.com/some-file.js

# Extract all API endpoints from AngularJS & Angular javascript files
curl -s URL | grep -Po "(\/)((?:[a-zA-Z\-_\:\.0-9\{\}]+))(\/)*((?:[a-zA-Z\-_\:\.0-9\{\}]+))(\/)((?:[a-zA-Z\-_\/\:\.0-9\{\}]+))" | sort -u

# https://github.com/incogbyte/jsearch
# simple script that grep infos from javascript files
python3 jsearch.py -u https://google.com -n google

# One line
# Getting page title without following redirections
for i in $(cat urls_or_subdomains.txt); do echo "$i | $(curl --connect-timeout 3 $i -s -v 2>&1 | grep -Poz '((?<=title>)(.*)(?=</title>)|(?<=Location:)(.*)/|(Could not resolve host:.*))' | tr -d '\0' | sed -r 's/(https?:\/\/.*\/?)(.*)(301 Moved Permanently)/\3 \2\1/g' )"; done

# Thread by Aware Online about some website investigation methodology
https://twitter.com/aware_online/status/1308312883248467975

# 1 - Tactical informations
# 2 - WHOIS
# 3 - Archives
# 4 - Text
# 5 - Reverse Image Search
# 6 - Images and EXIF data
# 7 - Source code
# 8 - Others TLD
# 9 - Mentions of target
# 10 - Check infos via RSS
# 11 - SSL certificates
# 12 - Robots/Sitemap
# 13 - Port scans
# 14 - Reverse IP lookup
# 15 - Reverse DNS lookup
# 16 - Monitoring changes
# 17 - Malware check

# OSINT Tool used to perform some OSINT tests and generates a report
# HTTP Headers, Whois, Traceroute, DNS, nmap, website on the same server, Reverse IP, Page Links
# Take care, not really passiv
./r3con1z3r.py google.com

# Tools like spoofcheck (https://github.com/BishopFox/spoofcheck) 
# It checks SPF and DMARC records for weak configuration that allow domain spoofing
# Domain is spoofable if lack of an SPF or DMARC record, SPF record never specifies ~all or -all, DMARC policy is set to p=none or is nonexistent
./spoofcheck.py [DOMAIN]

# urlcrazy allow to generate typo for a given domain and will check different elements
# such as IP, country, Nameserver and MX
https://github.com/urbanadventurer/urlcrazy/

# Default search
$ urlcrazy domain.com

# You can also search with popularity estimate
$ urlcrazy -p domain.com

# EyeWitness (https://github.com/FortyNorthSecurity/EyeWitness)
# It can take screenshots of websites, RDP services and open VNC servers, provide some server header info and identify defualt credentials
./EyeWitness.py -f filename --timeout optionaltimeout --open (Optional)
./EyeWitness -f urls.txt --web
./EyeWitness -x urls.xml --timeout 8 --headless
./EyeWitness -f rdp.txt --rdp

# XRay tool (https://github.com/evilsocket/xray)
# Bruteforce subdomains using wordlist and DNS requests, then Shodan, then ViewDNS is key is provided.
# Then it will launch banner grabbing and info collectors (not passiv)
xray -shodan-key yadayadayadapicaboo... -viewdns-key foobarsomethingsomething... -domain fbi.gov