parser for fqdn by Mzack9999 · Pull Request #2459 · projectdiscovery/httpx

Mzack9999 · 2026-03-21T17:48:37Z

Proposed changes

Close #1773

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

Summary by CodeRabbit

Bug Fixes
- Improved domain/FQDN extraction from response bodies by supporting HTML attributes (including srcset and meta refresh targets) and embedded scripts, with better normalization/validation and reduced false positives.
- Added safer handling for very large script contents by limiting deep script parsing.
Tests
- Expanded coverage with a new HTML fixture and extensive unit tests covering many URL/JS patterns, edge cases, and negative cases (deduping, protocol-relative URLs, auth segments, query/fragment handling).
- Updated benchmarks for the enhanced extraction behavior.

neo-by-projectdiscovery-dev · 2026-03-21T17:48:50Z

Neo - PR Security Review

No security issues found

Highlights

Adds HTML parsing with goquery to extract domains from URL-bearing attributes (href, src, action, formaction, poster, etc.)
Adds JavaScript AST parsing with goja to extract domains from string literals and template literals in inline scripts
Implements comprehensive domain validation including TLD verification, package name rejection, and file extension filtering
Includes 512KB size limit for JS parsing and panic recovery to prevent DoS

Hardening Notes

Consider adding a timeout for HTML/JS parsing operations in domains.go to prevent resource exhaustion on extremely large or deeply nested documents
The regex pattern [a-z0-9.-]* in potentialDomainRegex at line 19 is safe due to Go's RE2 engine, but consider documenting this explicitly in a comment
In extractDomainsFromHTML (line 70), consider adding a maximum depth limit for nested HTML structures as an additional safeguard

_{Comment @pdneo help for available commands. · Open in Neo}

coderabbitai · 2026-03-21T17:48:57Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: dc88c6f5-6a1f-4d0d-b3ad-7abe1bb8a962

📥 Commits

Reviewing files that changed from the base of the PR and between 7c8c2e0 and 569a751.

📒 Files selected for processing (2)

common/httpx/domains.go
common/httpx/domains_test.go

🚧 Files skipped from review as they are similar to previous changes (2)

common/httpx/domains.go
common/httpx/domains_test.go

Walkthrough

BodyDomainGrab now classifies response bodies and extracts candidate domains from HTML attributes, metadata, inline JavaScript, JavaScript-like content, or regex fallback, with centralized hostname normalization and validation. Tests and benchmarks cover parser behavior, false positives, and edge cases.

Changes

Response domain extraction

Layer / File(s)	Summary
Extraction routing and dependencies `go.mod`, `common/httpx/domains.go`	Adds goja and parser utilities, then routes response bodies through HTML, JavaScript, or regex extraction based on content heuristics and size limits.
HTML and JavaScript parsing `common/httpx/domains.go`	Extracts hosts from HTML URL attributes, metadata, srcset values, JSON scripts, and inline JavaScript string or template literals.
Candidate normalization and coverage `common/httpx/domains.go`, `common/httpx/domains_test.go`, `common/httpx/test-data/sample_with_js.html`	Centralizes candidate validation and domain/FQDN classification, with extensive parser tests, fixtures, false-positive checks, edge cases, and benchmarks.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
    participant Response as HTTP Response
    participant BodyDomainGrab
    participant goquery
    participant goja
    participant Validator as Candidate Validator

    Response->>BodyDomainGrab: provide response body
    BodyDomainGrab->>goquery: parse HTML content
    goquery->>BodyDomainGrab: return attributes, metadata, and inline scripts
    BodyDomainGrab->>goja: parse eligible JavaScript
    goja->>BodyDomainGrab: return string and template literal candidates
    BodyDomainGrab->>Validator: submit URL and regex candidates
    Validator->>BodyDomainGrab: return normalized domains and FQDNs

Suggested reviewers: dogancanbakir

Poem

🐇 I hopped through tags and scripts today,
Found hidden hosts along the way.
Goquery watched, goja shone bright,
Regex caught what escaped the sight.
Domains danced in tidy arrays.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 15.71% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is brief but still directly reflects the parser-based FQDN extraction change.
Linked Issues check	✅ Passed	The PR implements parser-based HTML and JavaScript extraction for FQDNs, matching issue `#1773`'s core objective.
Out of Scope Changes check	✅ Passed	The added tests, fixture, and goja dependency all support the parser-based extraction work and do not appear unrelated.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch improve-fqdn-extraction

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@common/httpx/domains.go`:
- Line 151: The deferred anonymous recover in common/httpx/domains.go currently
calls recover() without using its return value which fails lint; update the
defer to explicitly handle the result (e.g., capture into a variable and ignore
or check for nil) so the return value is used — locate the defer func() {
recover() }() and change it to call recover into a variable (for example r :=
recover()) and either assign it to the blank identifier (_ = r) or wrap it with
if r != nil { /* no-op or minimal handling */ } to satisfy errcheck.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 42d71ac6-683a-49d5-a24a-a7beed6f8603

📥 Commits

Reviewing files that changed from the base of the PR and between 9836829 and 090f0c2.

⛔ Files ignored due to path filters (1)

go.sum is excluded by !**/*.sum

📒 Files selected for processing (4)

common/httpx/domains.go
common/httpx/domains_test.go
common/httpx/test-data/sample_with_js.html
go.mod

coderabbitai

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@common/httpx/domains.go`:
- Around line 41-53: The current flow always runs extractDomainsFromRegex after
the HTML branch, causing JS-only bodies to miss extractDomainsFromJS and
reintroducing parser false positives; change the control flow around the
looksLikeHTML check to a single conditional switch on r.Data: if
looksLikeHTML(r.Data) use extractDomainsFromHTML and then extractDomainsFromJS
for inline scripts (respecting maxInlineScriptSize), else if
looksLikeJavaScript(r.Data) (or Content-Type if available) and len(r.Data) <=
maxInlineScriptSize call extractDomainsFromJS(string(r.Data), domains, fqdns,
r.Input), otherwise call extractDomainsFromRegex(r.Raw, domains, fqdns,
r.Input); update or add a looksLikeJavaScript helper if needed and remove the
unconditional extractDomainsFromRegex call after the HTML branch.
- Around line 58-64: The looksLikeHTML function fails for UTF-8 BOM-prefixed
documents because bytes.TrimSpace doesn't remove the BOM; update looksLikeHTML
to strip a leading UTF-8 BOM (bytes.TrimPrefix(..., []byte{0xEF,0xBB,0xBF}) or
equivalent) from the trimmed prefix before testing the first byte, so inputs
like "\ufeff<!DOCTYPE html>" are correctly detected as HTML.
- Around line 194-198: Normalize resp.Input's host portion before doing equality
checks: extract hostname from the existing input value (e.g., using
net.SplitHostPort or strings.Split on ':' into an inputHost) and then use
inputHost in the comparisons instead of raw input; replace the checks around
domains[val] and fqdns[d] so they read "if inputHost != val { domains[val] =
struct{}{} }" and "if d != val && d != inputHost { fqdns[d] = struct{}{} }" and
add the appropriate import (net or strings) so the normalization occurs before
the comparisons that involve input, d, and val.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6d51af93-2eb2-4a8a-8815-9829a7bfd1ce

📥 Commits

Reviewing files that changed from the base of the PR and between 090f0c2 and c3b06b5.

📒 Files selected for processing (1)

common/httpx/domains.go

parser for fqdn

090f0c2

coderabbitai Bot reviewed Mar 21, 2026

View reviewed changes

Comment thread common/httpx/domains.go Outdated

fixing lint

c3b06b5

coderabbitai Bot reviewed Mar 21, 2026

View reviewed changes

Comment thread common/httpx/domains.go Outdated

Comment thread common/httpx/domains.go Outdated

Comment thread common/httpx/domains.go Outdated

Merge branch 'dev' into improve-fqdn-extraction

7c8c2e0

Mzack9999 self-assigned this May 16, 2026

Mzack9999 requested a review from dogancanbakir May 16, 2026 21:49

fix domains

569a751

Mzack9999 merged commit a11a7be into dev Jul 16, 2026
13 checks passed

Mzack9999 deleted the improve-fqdn-extraction branch July 16, 2026 09:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

parser for fqdn#2459

parser for fqdn#2459
Mzack9999 merged 4 commits into
devfrom
improve-fqdn-extraction

Mzack9999 commented Mar 21, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

neo-by-projectdiscovery-dev Bot commented Mar 21, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Mar 21, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Mzack9999 commented Mar 21, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Checklist

Summary by CodeRabbit

Uh oh!

neo-by-projectdiscovery-dev Bot commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Neo - PR Security Review

Highlights

Uh oh!

coderabbitai Bot commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Mzack9999 commented Mar 21, 2026 •

edited by coderabbitai Bot

Loading

neo-by-projectdiscovery-dev Bot commented Mar 21, 2026 •

edited

Loading

coderabbitai Bot commented Mar 21, 2026 •

edited

Loading