spark-version-upgrade
Upgrade Apache Spark applications between major versions with a structured, phase-by-phase workflow.
When to Use
- Migrating from Spark 2.x → 3.x or Spark 3.x → 4.x
- Updating PySpark, Spark SQL, or Structured Streaming applications
- Resolving deprecation warnings before a Spark version bump
Workflow Overview
- Inventory & Impact Analysis — Scan the codebase and assess scope
- Build File Updates — Bump Spark/Scala/Java dependencies
- API Migration — Replace deprecated and removed APIs
- Configuration Migration — Update Spark config properties
- SQL & DataFrame Migration — Fix query-level breaking changes
- Test Validation — Compile, run tests, verify results
More from openhands/skills
ssh
Establish and manage SSH connections to remote machines, including key generation, configuration, and file transfers. Use when connecting to remote servers, executing remote commands, or transferring files via SCP.
477codereview-roasted
Brutally honest code review in the style of Linus Torvalds, focusing on data structures, simplicity, and pragmatism. Use when you want critical, no-nonsense feedback that prioritizes engineering fundamentals over style preferences.
119jupyter
Read, modify, execute, and convert Jupyter notebooks programmatically. Use when working with .ipynb files for data science workflows, including editing cells, clearing outputs, or converting to other formats.
88code-review
Rigorous code review focusing on data structures, simplicity, security, pragmatism, and risk/safety evaluation. Provides brutally honest, actionable feedback on pull requests or merge requests, including a risk assessment for every review. Use when reviewing code changes.
81readiness-report
Evaluate how well a codebase supports autonomous AI development. Analyzes repositories across eight technical pillars (Style & Validation, Build System, Testing, Documentation, Dev Environment, Debugging & Observability, Security, Task Discovery) and five maturity levels. Use when users request `/readiness-report` or want to assess agent readiness, codebase maturity, or identify gaps preventing effective AI-assisted development.
74security
Security best practices for secure coding, authentication, authorization, and data protection. Use when developing features that handle sensitive data, user authentication, or require security review.
69