Multi Search Engine

Integration of 16 search engines for web crawling without API keys.

Workflow

1. Preparation: AI Agent initializes an empty in-memory cookie store. Cookies are only acquired dynamically during search operations when access is denied

2. Language Evaluation: Detect the language attribute of the search query. If the query is in Chinese, use Domestic search engines (Baidu, Bing CN, Bing INT, 360, Sogou, WeChat, Shenma). If the query is non-Chinese, use International search engines (Google, Google HK, DuckDuckGo, Yahoo, Startpage, Brave, Ecosia, Qwant, WolframAlpha). Select engines based on query relevance and availability.

3. Controlled Search: Use web_fetch to execute search requests with rate limiting:

   • Add 1-2 second delay between requests to respect server load
   • Batch requests in groups of 3-4 engines with sequential execution between batches
   • Include standard browser headers to identify as legitimate user agent
   • If access is denied (403/429), fetch engine homepage to obtain fresh session cookies

4. Cookie Management:

   • Cookies are stored ONLY in memory during runtime
   • Cookies are acquired on-demand when search requests fail