llm-monitoring-dashboard
Auto-generates a data-driven LLM usage monitoring dashboard with cost tracking, user ranking, and PM insights.
- Tracks token counts, API costs, and latency across OpenAI, Anthropic, Gemini, and OpenRouter using Tokuin CLI; stores metrics in JSONL format with user context and prompt categorization
- Provides two deployment options: Next.js admin dashboard with per-user drilldown pages, or lightweight single-file HTML dashboard with charts, ranking tables, and search
- Includes automatic PM insights (adoption rate, inactive user alerts, success rate monitoring, cost optimization recommendations) and weekly Markdown report generation
- Built-in cost threshold alerts via Slack, cron-based data collection, and privacy-first design (prompt hashing, no raw text storage, environment-based API key management)
LLM Usage Monitoring Dashboard
Tracks LLM API costs, tokens, and latency using Tokuin CLI, and auto-generates a data-driven admin dashboard with PM insights.
When to use this skill
- LLM cost visibility: When you want to monitor API usage costs per team or individual in real time
- PM reporting dashboard: When you need weekly reports on who uses AI, how much, and how
- User adoption management: When you want to track inactive users and increase AI adoption rates
- Model optimization evidence: When you need data-driven decisions for model switching or cost reduction
- Add monitoring tab to admin dashboard: When adding an LLM monitoring section to an existing Admin page
Prerequisites
More from supercent-io/skills-template
security-best-practices
Implement security best practices for web applications and infrastructure. Use when securing APIs, preventing common vulnerabilities, or implementing security policies. Handles HTTPS, CORS, XSS, SQL Injection, CSRF, rate limiting, and OWASP Top 10.
14.1Kdata-analysis
Analyze datasets to extract insights, identify patterns, and generate reports. Use when exploring data, creating visualizations, or performing statistical analysis. Handles CSV, JSON, SQL queries, and Python pandas operations.
13.8Kweb-accessibility
Implement web accessibility (a11y) standards following WCAG 2.1 guidelines. Use when building accessible UIs, fixing accessibility issues, or ensuring compliance with disability standards. Handles ARIA attributes, keyboard navigation, screen readers, semantic HTML, and accessibility testing.
12.7Kworkflow-automation
Automate repetitive development tasks and workflows. Use when creating build scripts, automating deployments, or setting up development workflows. Handles npm scripts, Makefile, GitHub Actions workflows, and task automation.
12.6Kcode-review
Conduct thorough, constructive code reviews for quality and security. Use when reviewing pull requests, checking code quality, identifying bugs, or auditing security. Handles best practices, SOLID principles, security vulnerabilities, performance analysis, and testing coverage.
12.5Kdatabase-schema-design
Design and optimize database schemas for SQL and NoSQL databases. Use when creating new databases, designing tables, defining relationships, indexing strategies, or database migrations. Handles PostgreSQL, MySQL, MongoDB, normalization, and performance optimization.
12.2K