[Feature] Add web search context support to GPT plugin#5732
Merged
Conversation
- New module llm_search_context.lua: extracts domains from email URLs and queries search API - Integrated into gpt.lua with parallel context fetching (user + search) - Redis caching with configurable TTL (default 1 hour) - Retry logic with exponential backoff for search API failures - Disabled by default for backward compatibility - Configuration options in gpt.conf for customization
- Use local N = 'llm_search_context' idiom instead of constant string reuse - Replace rspamd_logger.debugm with lua_util.debugm (rspamd_logger has no debugm method) - Use extract_specific_urls instead of task:get_urls() - Add task parameter to query_search_api for proper logging and HTTP requests - Remove retry logic using non-existent rspamd_config:add_delayed_callback - Simplify to single HTTP attempt with graceful failure - Remove retry_count and retry_delay options from config
- Replace manual Redis operations with lua_cache API for better consistency - Use messagepack serialization and automatic key hashing - Fix Leta Mullvad API URL to /search/__data.json endpoint - Add search_engine parameter support - Remove redundant 'or DEFAULTS.xxx' patterns (opts already has defaults merged) - Add proper debug_module propagation throughout call chain - Improve JSON parsing to handle Leta Mullvad's nested pointer structure
- C code (message.c): collect top CTA URLs per HTML part by button weight, store in task mempool variable "html_cta_urls" - Lua API (lua_task.c): add task:get_cta_urls([max_urls]) method - llm_search_context: use new API instead of reimplementing CTA logic in Lua - Benefits: single source of truth for CTA logic, uses C knowledge of HTML structure and button weights, cleaner Lua code This provides proper architecture where C code handles HTML structure analysis and Lua adds domain filtering (blacklists, infrastructure domains, etc.)
Also refactor all absurdic logic it has previously
Expose rspamd_cryptobox_fast_hash via Lua API to allow hash-based URL deduplication without string conversion overhead. This is critical for handling messages with large numbers of URLs (100k+) where tostring() would fill the Lua string interning table and cause memory issues. Returns 64-bit hash as Lua number, using the same hash function as internal URL storage for consistency.
Change cache_key_prefix from 'gpt_search' to 'llm_search' to align with the module's current name (llm_search_context).
HTML_DISPLAYED URLs are phishing bait text (display-only) and should not be considered for CTA (call-to-action) detection. Only real link targets should be analyzed for CTA purposes.
…ased deduplication Replace tostring() with url:get_hash() throughout URL extraction to avoid filling the Lua string interning table. Critical for handling malicious messages with 100k+ URLs where each tostring() would create an interned string causing memory exhaustion. Key changes: - Use dual data structure: array for results + hash set for O(1) dedup - Add max_urls_to_process=50000 limit with warning for DoS protection - Track url_index for stable sorting when priorities are equal - Fix CTA priority preservation: prevent generic phished handling from overwriting CTA priorities which include phished/subject bonuses - Add verbose flag to test suite for debugging This ensures memory usage is strictly bounded regardless of malicious input while maintaining correct URL prioritization for spam detection.
Contributor
|
@vstakhov Bad Timing .. :-( https://mullvad.net/en/blog/shutting-down-our-search-proxy-leta |
Member
Author
|
Yeah, but we can probably use paid api from google/bing for that purposes. This PR also has quite a lot of important fixes not very relevant to llm search context. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional web search context to the GPT plugin, allowing it to query a search API for domains found in email URLs and provide contextual information to the LLM for better spam classification.
Changes
New
llm_search_context.luamodule that:lua_util.extract_specific_urls()lua_cachemodule with messagepack serializationUpdated
gpt.luaplugin to integrate search context:search_contextconfiguration sectionllm_search_context.fetch_and_format()when enabledConfiguration in
conf/modules.d/gpt.conf:search_context.enabled- enable/disable feature (default: false)search_context.search_url- search API endpointsearch_context.search_engine- search engine to use (brave, google, etc.)search_context.max_domains- maximum domains to searchsearch_context.max_results_per_query- maximum results per domainsearch_context.timeout- HTTP request timeoutsearch_context.cache_ttl- Redis cache TTLImplementation Details
lua_cachemodule for Redis caching with automatic key hashingTesting
Tested with Leta Mullvad search API and verified: