Skip to content

[Feature] Add web search context support to GPT plugin#5732

Merged
vstakhov merged 10 commits into
masterfrom
vstakhov-llm-search-context
Nov 7, 2025
Merged

[Feature] Add web search context support to GPT plugin#5732
vstakhov merged 10 commits into
masterfrom
vstakhov-llm-search-context

Conversation

@vstakhov

@vstakhov vstakhov commented Nov 5, 2025

Copy link
Copy Markdown
Member

Summary

Adds optional web search context to the GPT plugin, allowing it to query a search API for domains found in email URLs and provide contextual information to the LLM for better spam classification.

Changes

  • New llm_search_context.lua module that:

    • Extracts domains from email URLs using lua_util.extract_specific_urls()
    • Queries Leta Mullvad search API for domain information
    • Caches results using lua_cache module with messagepack serialization
    • Formats search results as context for LLM prompts
  • Updated gpt.lua plugin to integrate search context:

    • Added search_context configuration section
    • Calls llm_search_context.fetch_and_format() when enabled
  • Configuration in conf/modules.d/gpt.conf:

    • search_context.enabled - enable/disable feature (default: false)
    • search_context.search_url - search API endpoint
    • search_context.search_engine - search engine to use (brave, google, etc.)
    • search_context.max_domains - maximum domains to search
    • search_context.max_results_per_query - maximum results per domain
    • search_context.timeout - HTTP request timeout
    • search_context.cache_ttl - Redis cache TTL

Implementation Details

  • Uses lua_cache module for Redis caching with automatic key hashing
  • Properly handles Leta Mullvad's nested JSON response format with pointer indices
  • Debug module propagation for better diagnostics
  • Clean code following Rspamd idioms (override_defaults pattern, etc.)

Testing

Tested with Leta Mullvad search API and verified:

  • Domain extraction from email URLs
  • Search API queries and response parsing
  • Redis caching behavior
  • Debug logging

- New module llm_search_context.lua: extracts domains from email URLs and queries search API
- Integrated into gpt.lua with parallel context fetching (user + search)
- Redis caching with configurable TTL (default 1 hour)
- Retry logic with exponential backoff for search API failures
- Disabled by default for backward compatibility
- Configuration options in gpt.conf for customization
- Use local N = 'llm_search_context' idiom instead of constant string reuse
- Replace rspamd_logger.debugm with lua_util.debugm (rspamd_logger has no debugm method)
- Use extract_specific_urls instead of task:get_urls()
- Add task parameter to query_search_api for proper logging and HTTP requests
- Remove retry logic using non-existent rspamd_config:add_delayed_callback
- Simplify to single HTTP attempt with graceful failure
- Remove retry_count and retry_delay options from config
- Replace manual Redis operations with lua_cache API for better consistency
- Use messagepack serialization and automatic key hashing
- Fix Leta Mullvad API URL to /search/__data.json endpoint
- Add search_engine parameter support
- Remove redundant 'or DEFAULTS.xxx' patterns (opts already has defaults merged)
- Add proper debug_module propagation throughout call chain
- Improve JSON parsing to handle Leta Mullvad's nested pointer structure
- C code (message.c): collect top CTA URLs per HTML part by button weight,
  store in task mempool variable "html_cta_urls"
- Lua API (lua_task.c): add task:get_cta_urls([max_urls]) method
- llm_search_context: use new API instead of reimplementing CTA logic in Lua
- Benefits: single source of truth for CTA logic, uses C knowledge of HTML
  structure and button weights, cleaner Lua code

This provides proper architecture where C code handles HTML structure analysis
and Lua adds domain filtering (blacklists, infrastructure domains, etc.)
Also refactor all absurdic logic it has previously
Expose rspamd_cryptobox_fast_hash via Lua API to allow hash-based
URL deduplication without string conversion overhead. This is critical
for handling messages with large numbers of URLs (100k+) where tostring()
would fill the Lua string interning table and cause memory issues.

Returns 64-bit hash as Lua number, using the same hash function as
internal URL storage for consistency.
Change cache_key_prefix from 'gpt_search' to 'llm_search' to align
with the module's current name (llm_search_context).
HTML_DISPLAYED URLs are phishing bait text (display-only) and should
not be considered for CTA (call-to-action) detection. Only real link
targets should be analyzed for CTA purposes.
…ased deduplication

Replace tostring() with url:get_hash() throughout URL extraction to avoid
filling the Lua string interning table. Critical for handling malicious
messages with 100k+ URLs where each tostring() would create an interned
string causing memory exhaustion.

Key changes:
- Use dual data structure: array for results + hash set for O(1) dedup
- Add max_urls_to_process=50000 limit with warning for DoS protection
- Track url_index for stable sorting when priorities are equal
- Fix CTA priority preservation: prevent generic phished handling from
  overwriting CTA priorities which include phished/subject bonuses
- Add verbose flag to test suite for debugging

This ensures memory usage is strictly bounded regardless of malicious
input while maintaining correct URL prioritization for spam detection.
@vstakhov
vstakhov merged commit 5d58065 into master Nov 7, 2025
27 checks passed
@vstakhov
vstakhov deleted the vstakhov-llm-search-context branch November 7, 2025 16:48
@alex081512

alex081512 commented Nov 8, 2025

Copy link
Copy Markdown
Contributor

@vstakhov

Copy link
Copy Markdown
Member Author

Yeah, but we can probably use paid api from google/bing for that purposes. This PR also has quite a lot of important fixes not very relevant to llm search context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants