Recon Pipeline: OSINT Automation for Bug Bounties and Pentests

Recon Pipeline: How to Automate OSINT, Asset Discovery and Vulnerability Scanning #

An OSINT pipeline is an automated chain of tools that gathers information about a target—domains, subdomains, open ports, technologies, exposed credentials, endpoints—before even touching the application with an active test.

In bug bounty and professional pentesting, reconnaissance is the phase that separates those who find critical vulnerabilities from those who find nothing. Targets with mature programs have been tested by hundreds of researchers: the obvious vulnerabilities have already been found. The remaining vulnerabilities lie in the attack surface that others haven't looked for—forgotten subdomains, legacy endpoints, assets the company isn't aware of.

An automated pipeline allows you to systematically cover that surface without missing a thing.

The Difference Between Passive and Active Recon #

Before building your pipeline, understand the fundamental distinction:

Passive recon: Gather information without ever directly touching the target's infrastructure. You use third-party sources—search engines, TLS certificates, public datasets, and internet archives. The target never sees any requests coming from you.

text

Esempio:
→ Consulti crt.sh per trovare i certificati SSL emessi per *.target.com
→ Il target non sa che hai cercato quei certificati
→ Nessun rischio di rilevamento, nessun log sul loro server

Active Recon: Interact directly with the target's infrastructure—send DNS queries, port scans, and HTTP requests. You may be detected in logs.

text

Esempio:
→ Esegui nmap su target.com
→ Il firewall del target vede le tuoi probe
→ Rischio rilevamento — in un bug bounty spesso è permesso,
  in un pentest varia in base alle regole d'ingaggio

A good pipeline does all the passive recon first, then switches to the active one with the surface already mapped.

Pipeline Structure #

Here is the logical sequence that the best bug hunters follow:

text

1. SUBDOMAIN ENUMERATION (passiva)
   → Trova tutti i sottodomini del target

2. DNS RESOLUTION + LIVE CHECK
   → Quali sottodomini risolvono? Quali server sono attivi?

3. PORT SCANNING
   → Quali porte sono aperte su quegli IP?

4. HTTP FINGERPRINTING
   → Screenshot, status code, tecnologie, header

5. CONTENT DISCOVERY
   → Directory, file, endpoint nascosti

6. VULNERABILITY SCANNING
   → Template automatici su tecnologie identificate

7. SECRET / CREDENTIAL SCANNING
   → Chiavi API, token, credenziali esposte

Each step feeds into the next. Ultimately, you have a complete map of the attack surface.

Step 1 — Subdomain Enumeration (Passive) #

The goal is to find all the target's subdomains without sending a single request to their server.

TLS Certificates (crt.sh) #

Every SSL certificate issued for a domain is recorded in public logs (Certificate Transparency). crt.sh indexes them all.

bash

# Query manuale su crt.sh
curl -s "https://crt.sh/?q=%.target.com&output=json" | \
  python3 -c "import sys,json; [print(x['name_value']) for x in json.load(sys.stdin)]" | \
  sed 's/\*\.//g' | sort -u

# Spesso rivela sottodomini come:
# staging.target.com
# api-internal.target.com
# dev.target.com
# legacy-admin.target.com
# ← questi sono i target più interessanti

Subfinder — Aggregate Passive Enumeration #

Subfinder queries dozens of sources in parallel: VirusTotal, Shodan, Censys, SecurityTrails, crt.sh, DNSDumpster, and many others.

bash

# Installazione
go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest

# Uso base
subfinder -d target.com -o subdomains.txt

# Con verbosità per vedere da quale fonte arriva ogni subdomain
subfinder -d target.com -v -o subdomains.txt

# Con più fonti (richiede API key per alcune)
subfinder -d target.com -all -o subdomains.txt

# Multiple target da file
subfinder -dL domains.txt -o subdomains.txt

Amass — Deep Passive + Active Enumeration #

Amass is more powerful but slower. It performs both passive and active recon (DNS bruteforce, permutations).

bash

# Installazione
go install -v github.com/owasp-amass/amass/v4/...@master

# Recon passiva (solo fonti esterne, nessun contatto col target)
amass enum -passive -d target.com -o amass_passive.txt

# Recon attiva (bruteforce + permutazioni — contatta i DNS del target)
amass enum -active -d target.com -o amass_active.txt

# Bruteforce con wordlist
amass enum -brute -d target.com -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt

# Combina tutte le fonti
amass enum -d target.com -active -brute -o amass_all.txt

Other Passive Sources Not to Forget #

bash

# Wayback Machine — URL storici indicizzati da internet archive
# Spesso contiene endpoint deprecati ancora attivi
curl -s "http://web.archive.org/cdx/search/cdx?url=*.target.com/*&output=text&fl=original&collapse=urlkey" | \
  sort -u | head -100

# GitHub / GitLab — cerca menzioni del dominio nel codice pubblico
# Tool: github-dorking o manuale
# Cerca: "target.com" in:code site:github.com

# Shodan — trova IP con servizi esposti per un'organizzazione
# Con API key:
shodan search "target.com" --fields ip_str,port,hostnames

# SecurityTrails (API gratuita limitata)
curl -s "https://api.securitytrails.com/v1/domain/target.com/subdomains" \
  -H "APIKEY: TUA_API_KEY" | python3 -m json.tool

Step 2 — DNS Resolution and Live Check #

You have a list of hundreds of subdomains. Many don't resolve. DNS and HTTPS only filter the active ones.

bash

# Installazione
go install -v github.com/projectdiscovery/dnsx/cmd/dnsx@latest
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

# DNS resolution: quali sottodomini hanno un record A/CNAME?
cat subdomains.txt | dnsx -a -resp -o resolved.txt
# Output: subdomain.target.com [IP]

# HTTP check: quali rispondono su porta 80/443?
cat resolved.txt | httpx -o live_hosts.txt

# Con più informazioni (status code, title, tecnologia)
cat resolved.txt | httpx \
  -status-code \
  -title \
  -tech-detect \
  -content-length \
  -o live_full.txt

# Esempio output:
# https://staging.target.com [200] [Staging Environment] [Laravel,PHP,Nginx]
# https://api-internal.target.com [403] [Forbidden] [Express,Node.js]
# https://dev.target.com [200] [Dev Portal] [WordPress,Apache]

Step 3 — Port Scanning #

Now that you have active IPs, look for open ports beyond 80 and 443. Services exposed on non-standard ports are often unprotected.

bash

# nmap — accurato ma lento
nmap -sV -sC -p- --open -T4 IP_TARGET -oN nmap_full.txt

# nmap su lista di IP
nmap -sV --open -iL ip_list.txt -oN nmap_multi.txt

# masscan — velocissimo per range grandi (richiede root)
masscan -p1-65535 IP_TARGET --rate=1000 -oG masscan_output.txt

# Combinazione: masscan per trovare porte aperte, nmap per i dettagli
masscan -p1-65535 IP_TARGET --rate=1000 | \
  awk '/open/ {print $4}' | \
  xargs -I{} nmap -sV -p {} IP_TARGET

# Porte ad alto valore da controllare sempre:
# 8080, 8443, 8888 → web app alternative (spesso meno protette)
# 9200, 9300 → Elasticsearch (spesso senza auth)
# 27017 → MongoDB (spesso senza auth)
# 6379 → Redis (spesso senza auth)
# 5432 → PostgreSQL
# 3306 → MySQL
# 445 → SMB
# 22 → SSH (versione, banner)

Step 4 — Screenshot and Fingerprinting #

With hundreds of active hosts, you can't visit them all manually. Automatic screenshots allow you to perform visual triage in minutes.

bash

# gowitness — screenshot di massa
go install github.com/sensepost/gowitness@latest

# Screenshot da lista di URL
gowitness file -f live_hosts.txt --screenshot-path ./screenshots/

# Apri il report HTML interattivo
gowitness report serve

# EyeWitness (Python, già presente in Kali)
eyewitness -f live_hosts.txt --web --timeout 10 -d ./eyewitness_report/

# httpx con screenshot integrato
cat live_hosts.txt | httpx -screenshot -o httpx_screenshots/

What you're looking for in the screenshots:

Login panels (Grafana, Jenkins, Kibana, phpMyAdmin, Admin panels)
Error pages that reveal technologies and paths
Default applications not configured
“Index of /” → directory listing
Legacy applications (older versions visible in the footer)

Step 5 — Content Discovery #

Now look for hidden paths on every active host: admin endpoints, backup files, exposed configurations.

bash

# ffuf — il più veloce
ffuf -u "https://target.com/FUZZ" \
  -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt \
  -mc 200,301,302,403 \
  -o ffuf_results.json

# Con estensioni (cerca file di backup, config, ecc.)
ffuf -u "https://target.com/FUZZ" \
  -w /usr/share/seclists/Discovery/Web-Content/raft-medium-files.txt \
  -e .php,.bak,.old,.conf,.env,.log,.sql,.zip,.tar.gz \
  -mc 200,301,302 \
  -o ffuf_files.json

# feroxbuster — ricorsivo
feroxbuster -u "https://target.com" \
  -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt \
  --auto-tune \
  -o feroxbuster_results.txt

# Wordlist specifiche per tecnologia (molto più efficaci di quelle generiche)
# Laravel:
ffuf -u "https://target.com/FUZZ" -w /usr/share/seclists/Discovery/Web-Content/Laravel.txt
# WordPress:
ffuf -u "https://target.com/FUZZ" -w /usr/share/seclists/Discovery/Web-Content/CMS/WordPress.fuzz.txt
# Spring Boot Actuator (endpoint esposti di default):
ffuf -u "https://target.com/FUZZ" -w /usr/share/seclists/Discovery/Web-Content/spring-boot.txt

Step 6 — Vulnerability Scanning with Cores #

Nuclei runs thousands of templates on each host — known CVEs, misconfigurations, exposed endpoints, default panels.

bash

# Installazione
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest

# Aggiorna i template (fallo sempre prima di usarli)
nuclei -update-templates

# Scan base su lista di host
nuclei -l live_hosts.txt -o nuclei_results.txt

# Solo severity alta e critica
nuclei -l live_hosts.txt -severity high,critical -o nuclei_critical.txt

# Solo CVE
nuclei -l live_hosts.txt -tags cve -o nuclei_cves.txt

# Solo misconfigurazioni e esposizioni
nuclei -l live_hosts.txt -tags misconfig,exposure -o nuclei_misconfig.txt

# Template specifici per tecnologia identificata (es. WordPress)
nuclei -l live_hosts.txt -tags wordpress -o nuclei_wp.txt

# Con rate limiting per non fare troppo rumore
nuclei -l live_hosts.txt -rate-limit 50 -bulk-size 10 -o nuclei_results.txt

Step 7 — Secret Scanning #

Search for credentials, API keys, hardcoded tokens in exposed source code, JavaScript files, and public repositories.

JavaScript Source Analysis #

bash

# getJS — estrae tutti i file JS da un sito
go install github.com/003random/getJS@latest
getJS -url "https://target.com" -output js_files.txt

# Scarica tutti i JS trovati
cat js_files.txt | xargs -I{} curl -s {} -o js_files/{$(echo {} | md5sum | cut -d' ' -f1)}.js

# secretfinder — cerca secret nei file JS
python3 SecretFinder.py -i "https://target.com" -e -o results.html
# Cerca: AWS keys, Stripe keys, Firebase URL, Google API keys, JWT, token...

# trufflehog — cerca secret in URL e file
trufflehog filesystem ./js_files/ --only-verified

Google Dorking for Secrets Exposed #

bash

# File di configurazione esposti sul dominio
site:target.com ext:env "DB_PASSWORD"
site:target.com ext:json "api_key"
site:target.com ext:xml "password"
site:target.com "Index of" ".env"
site:target.com filetype:log

# GitHub dorking per secret nel codice pubblico
site:github.com "target.com" "api_key"
site:github.com "target.com" "password"
site:github.com "target.com" "secret"
site:github.com "target.com" "DB_HOST"

# Pastebin / paste sites
site:pastebin.com "target.com"
site:gist.github.com "target.com" "password"

GitHub / GitLab Automated Search #

bash

# trufflehog su repository GitHub dell'organizzazione
trufflehog github --org=target-org --only-verified

# gitleaks su un repository specifico
gitleaks detect --source ./cloned-repo/ --report-format json --report-path gitleaks_report.json

# gitrob — enumera repository pubblici di un'organizzazione
gitrob analyze --organization target-org

Step 8 — Subdomain Takeover Check #

A subdomain can point to an external service (GitHub Pages, Heroku, AWS S3, Fastly) that is no longer active. If the service has been deleted but the DNS record still exists, anyone can register that service and take control of the subdomain.

bash

# subjack — check automatico subdomain takeover
go install github.com/haccer/subjack@latest
subjack -w subdomains.txt -t 100 -timeout 30 -o takeover_results.txt -ssl

# nuclei con template takeover
nuclei -l subdomains.txt -tags takeover -o nuclei_takeover.txt

# Segnali manuali da cercare:
# "There isn't a GitHub Pages site here." → GitHub Pages takeover possibile
# "NoSuchBucket" → S3 bucket takeover possibile
# "No such app" → Heroku takeover possibile
# "404 Not Found" su Fastly/CloudFront → CDN takeover possibile

# Vedi: subdomain-takeover per la guida completa

→ subdomain takeover

Automated Pipeline: Complete Script #

This script puts all the steps together in sequence. Save it as recon.sh and use that as a starting point.

bash

#!/bin/bash
# recon.sh — OSINT Pipeline Completa
# Uso: ./recon.sh target.com

TARGET=$1
OUTPUT_DIR="recon_$TARGET"
mkdir -p $OUTPUT_DIR/{subdomains,screenshots,nuclei,secrets,ports}

echo "[*] Avvio pipeline per: $TARGET"

# STEP 1 — Subdomain enumeration passiva
echo "[+] Subdomain enumeration..."
subfinder -d $TARGET -silent -o $OUTPUT_DIR/subdomains/subfinder.txt
amass enum -passive -d $TARGET -o $OUTPUT_DIR/subdomains/amass.txt 2>/dev/null

# crt.sh
curl -s "https://crt.sh/?q=%25.$TARGET&output=json" 2>/dev/null | \
  python3 -c "import sys,json; [print(x.get('name_value','')) for x in json.load(sys.stdin)]" 2>/dev/null | \
  sed 's/\*\.//g' >> $OUTPUT_DIR/subdomains/crtsh.txt

# Unisci e deduplicа
cat $OUTPUT_DIR/subdomains/*.txt | sort -u > $OUTPUT_DIR/subdomains/all.txt
TOTAL=$(wc -l < $OUTPUT_DIR/subdomains/all.txt)
echo "[+] Trovati $TOTAL sottodomini unici"

# STEP 2 — DNS resolution
echo "[+] DNS resolution..."
cat $OUTPUT_DIR/subdomains/all.txt | \
  dnsx -silent -a -resp -o $OUTPUT_DIR/subdomains/resolved.txt

# STEP 3 — HTTP check
echo "[+] HTTP fingerprinting..."
cat $OUTPUT_DIR/subdomains/resolved.txt | \
  httpx -silent -status-code -title -tech-detect \
  -o $OUTPUT_DIR/live_hosts.txt

# STEP 4 — Screenshot
echo "[+] Screenshot..."
gowitness file -f $OUTPUT_DIR/live_hosts.txt \
  --screenshot-path $OUTPUT_DIR/screenshots/ 2>/dev/null

# STEP 5 — Port scan (solo host attivi, porte comuni)
echo "[+] Port scanning..."
cat $OUTPUT_DIR/subdomains/resolved.txt | \
  awk '{print $2}' | sort -u > $OUTPUT_DIR/ports/ip_list.txt
nmap -iL $OUTPUT_DIR/ports/ip_list.txt \
  -p 80,443,8080,8443,8888,9200,27017,6379,5432,3306,22,445 \
  --open -oN $OUTPUT_DIR/ports/nmap.txt 2>/dev/null

# STEP 6 — Nuclei
echo "[+] Nuclei scanning..."
nuclei -l $OUTPUT_DIR/live_hosts.txt \
  -severity medium,high,critical \
  -silent \
  -o $OUTPUT_DIR/nuclei/results.txt 2>/dev/null

# STEP 7 — Subdomain takeover
echo "[+] Takeover check..."
subjack -w $OUTPUT_DIR/subdomains/all.txt \
  -t 50 -timeout 20 \
  -o $OUTPUT_DIR/takeover.txt -ssl 2>/dev/null

echo ""
echo "[*] DONE. Risultati in: $OUTPUT_DIR/"
echo "[*] Live hosts: $(wc -l < $OUTPUT_DIR/live_hosts.txt)"
echo "[*] Nuclei findings: $(wc -l < $OUTPUT_DIR/nuclei/results.txt)"

Advanced Passive Sources #

In addition to the tools, these web sources give you information that no tool finds automatically:

Shodan — a search engine for internet-connected devices. Search for organizations to find IP addresses, open ports, service banners, and certificates.

text

Dork Shodan:
org:"Target Company"
ssl:"target.com"
hostname:"target.com"

Censys — similar to Shodan but more oriented towards certificates and protocols.

text

Dork Censys:
parsed.names: target.com
autonomous_system.name: "Target Inc"

FOFA — Chinese alternative to Shodan, useful for targets that don't appear on Shodan.

GreyNoise — Distinguishes scanner traffic from real traffic. Useful for seeing who's already scanning the target.

Wayback Machine — Historical URLs. Endpoints removed from the site but still active on the server. Forgotten parameters.

bash

# Estrai tutti gli URL storici di un dominio
curl -s "https://web.archive.org/cdx/search/cdx?url=*.target.com/*&output=text&fl=original&collapse=urlkey" | \
  grep -v "^$" | sort -u > wayback_urls.txt

# Filtra per endpoint interessanti
cat wayback_urls.txt | grep -iE "\.(php|asp|aspx|jsp|json|xml|env|bak|sql|zip)$"
cat wayback_urls.txt | grep -iE "(admin|login|api|dashboard|config|backup|upload)"

Checklist #

text

SETUP
☐ Tool installati: subfinder, amass, dnsx, httpx, nuclei, gowitness, ffuf, trufflehog
☐ API key configurate: SecurityTrails, Shodan, VirusTotal, GitHub
☐ Wordlist aggiornate: SecLists clonato in /usr/share/seclists/

PASSIVE RECON
☐ crt.sh interrogato per certificati
☐ subfinder eseguito con tutte le fonti
☐ amass enum --passive eseguito
☐ Wayback Machine interrogato per URL storici
☐ Shodan/Censys cercato per org e hostname
☐ GitHub/GitLab cercato per menzioni del dominio

DNS + LIVE CHECK
☐ dnsx per filtrare sottodomini che risolvono
☐ httpx per identificare host HTTP/HTTPS attivi
☐ Status code e tecnologie annotati

ACTIVE RECON
☐ Port scan (nmap o masscan) su IP attivi
☐ Screenshot (gowitness o eyewitness) per triage visivo
☐ Content discovery (ffuf o feroxbuster) su host interessanti
☐ Wordlist specifica per tecnologia identificata

VULNERABILITY + SECRET
☐ Nuclei con template high/critical
☐ Subdomain takeover check (subjack)
☐ JS file scaricati e analizzati per secret
☐ Google dork per file esposti (.env, .log, .bak)
☐ GitHub dork per credenziali hardcodate
☐ trufflehog su repository pubblici dell'organizzazione

DOCUMENTAZIONE
☐ Mappa della superficie d'attacco
☐ Sottodomini interessanti evidenziati
☐ Tecnologie identificate per ogni host
☐ Finding prioritizzati per impatto

FAQ #

Where do I start if the target has thousands of subdomains? Triage by priority: first look at the screenshots and filter for login panels, non-standard applications, and verbose error pages. Then focus on subdomains with known vulnerable technologies (WordPress, Jenkins, Grafana, phpMyAdmin) or with unusual open ports. Ignore generic subdomains with CDNs in front of them for now.

Can I use this pipeline in a bug bounty without violating the rules? Passive recon (crt.sh, Shodan, Wayback Machine) never touches the target's servers—it's always allowed. Active recon (port scan, content discovery, nuclei) must be within the program's scope. Always read the rules before running active scans.

How many API keys do I need for an effective pipeline? For starters: GitHub (free, required for rooting), Shodan (limited free plan). For a complete pipeline: SecurityTrails, VirusTotal, Censys. Many tools like subfinder have API-key-free modes that use only public sources—which is already very useful.

Amass vs. Subfinder: which should I use? Use both and combine the results. Subfinder is faster and finds more subdomains in passive mode thanks to its numerous sources. Amass is slower but does bruteforce and permutations that Subfinder doesn't. The combination covers much more than either one alone.

How do I manage an output of 10.000 subdomains? Filter immediately with DNSX (only those that resolve), then HTTPX (only those that respond to HTTP). Typically, 10% to 30% are actually active. Then use GoWitness for screenshots and perform visual triage—in 30 minutes, you can identify the interesting targets from hundreds.