algo-seo-crawl
Web Crawler
Overview
A web crawler systematically traverses web pages by discovering URLs, fetching content, parsing HTML, and storing results. Uses BFS or priority-based frontier management. Performance is I/O-bound, typically limited by politeness constraints rather than compute.
When to Use
Trigger conditions:
- Building a site audit tool to discover all pages and their link structure
- Collecting structured data from websites at scale
- Mapping site architecture for SEO analysis
When NOT to use:
- When you need data from a single API endpoint (use HTTP client directly)
- When a sitemap.xml provides all needed URLs (parse sitemap instead)
Algorithm
More from asgard-ai-platform/skills
xborder-logistics
Design cross-border logistics strategies including direct mail, overseas warehousing, and bonded warehouse models for international e-commerce. Use this skill when the user needs to ship products internationally, choose a logistics model for cross-border sales, optimize shipping costs, or set up fulfillment in a foreign market — even if they say 'ship to Southeast Asia', 'overseas warehouse vs direct shipping', 'customs clearance', or 'reduce international shipping time'.
65tw-stock-analysis
Analyze Taiwan-listed stocks using fundamental analysis including EPS, P/E ratio, dividend yield, and financial statement review. Use this skill when the user needs to evaluate a Taiwan stock, compare TWSE/TPEx-listed companies, assess valuation, or review earnings — even if they say 'should I buy this stock', 'analyze 台積電', 'is this stock overvalued', or 'compare these two Taiwan companies'.
47econ-macro-indicators
Interpret macroeconomic indicators including GDP, inflation, unemployment, interest rates, and exchange rates to assess economic health and predict trends. Use this skill when the user needs to evaluate a country's economic outlook, understand monetary/fiscal policy impacts, or contextualize business decisions within the macroeconomic environment — even if they say 'is the economy doing well', 'what do rising interest rates mean for us', or 'explain today's economic data'.
43ecom-multilingual-listing
Optimize multilingual product listings for international e-commerce including SEO localization, machine translation workflows, and cultural adaptation. Use this skill when the user needs to create product listings in multiple languages, optimize for local search, or adapt marketing content for different markets — even if they say 'translate our listings', 'optimize for local SEO', 'adapt for the Japanese market', or 'our translated listings don't convert'.
42law-ip
Analyze intellectual property rights across patents, trademarks, copyrights, and trade secrets. Use this skill when the user needs to understand IP protection options, evaluate whether their work is protectable, assess infringement risk, or design an IP strategy — even if they say 'can I patent this', 'someone copied our design', 'how do we protect our brand name', or 'what IP do we have'.
34pr-social-copywriting
Write platform-optimized social media copy for Instagram, Facebook, LinkedIn, and X/Twitter with hooks, CTAs, and hashtag strategies. Use this skill when the user needs to create social media posts, adapt content across platforms, improve engagement rates, or develop a social content strategy — even if they say 'write a post for IG', 'our social engagement is low', 'adapt this for LinkedIn', or 'how do we write better captions'.
33