--- name: gcp-managed-spark-upgrades description: | Upgrades Spark workloads on Google Cloud Managed Service for Apache Spark (formerly known as Dataproc) clusters and Serverless to a newer image or runtime version. Supports static compatibility analysis, code upgrade, build checks against the target version, and optional verification runs. Use when: - Upgrading Managed Spark versions (major, minor, or sub-minor) or assessing upgrade readiness. Example prompt: "Upgrade Managed Spark job [job_id] in project [project_id], region [region] to target version [target_version]. Code location: [local path or GCS path]. Target GCS path: [gcs_target_path]" Don't use when: - Writing new Spark code. - Migrating non-Spark workloads (e.g., Hive/Flink to Spark). - Troubleshooting a failed Spark job that is not part of an upgrade. Limitation: This skill does not support data/result validation currently. license: Apache-2.0 metadata: version: v3 publisher: google --- ## Scope gate - **PySpark**: Cluster job field `pysparkJob`, Serverless batch field `pysparkBatch`. Status: **Supported**. Analysis reference: [pyspark_analysis.md](references/pyspark_analysis.md). - **Scala/Java**: Cluster job field `sparkJob`, Serverless batch field `sparkBatch`. Status: **Supported**. Analysis references: [scala_analysis.md](references/scala_analysis.md) for Scala sources and [java_analysis.md](references/java_analysis.md) for Java sources (both for a project with both), plus [jvm_build_reference.md](references/jvm_build_reference.md), which every Scala/Java job uses for its jars, Java runtime, dependencies, packaging, and build checks. - **Spark SQL**: Cluster job field `sparkSqlJob`, Serverless batch field `sparkSqlBatch`. Status: Not yet supported. - **SparkR**: Cluster job field `sparkRJob`, Serverless batch field `sparkRBatch`. Status: Not yet supported. - **PySpark notebook**: Cluster job field —, Serverless batch field `pysparkNotebookBatch`. Status: Not yet supported. If the workload's job type isn't **Supported**, stop: tell the user which job types are currently supported, and don't change code, create clusters, or submit jobs. Non-Spark jobs (e.g. `hiveJob`, `pigJob`) are out of scope. Everything else in this skill applies to every supported job type. Checks that depend on the job type live in its analysis references and in the parts of the other references labeled with the job type (e.g. "For PySpark", "For Scala/Java"). ## Terminology - **Managed Spark**: Google Cloud Managed Service for Apache Spark, formerly Dataproc. Commands (`gcloud dataproc …`) and many docs URLs still say `dataproc`. - **Cluster**: A job on a Managed Spark cluster (Compute Engine). Its version is an **image version**, `..-` (e.g. `2.2.50-debian12`). - **Serverless**: A Managed Spark Serverless batch. Its version is a **runtime version**, `.` only (e.g. `2.2`): every batch runs the newest sub-minor of its runtime, which can't be pinned. - **Job type**: The kind of Spark workload (e.g. PySpark, Scala/Java), read from the job or batch's type-specific field. See the scope gate. - **Track**: The `.` part of a version. The same track can run a different Spark version on clusters and on Serverless. - **``**: The directory that contains this SKILL.md. Script paths below are relative to it. - **``**: The sanitized job name used in temporary paths: the job's main code file name without its extension (the main `.py` file for PySpark; for Scala/Java, the main JAR name, or the main class's simple name), lowercased, with every character other than `a-z`, `0-9`, `_`, and `-` replaced by `_` (e.g. `daily_orders` for `daily_orders.py`, `order-metrics-app` for `order-metrics-app.jar`). - **``**: The Target GCS path, `gs:///`. It receives the final code and report; temporary verification artifacts are staged under it. - **Build checks**: Local checks that the code and its dependencies build against the target versions (language runtime, Spark, Scala). The job type's references define them (see the scope gate). - **Verification run**: The optional Step 2 job or batch on GCP. It shows that the job runs; it does not validate output data. ## Rules 1. **Zero defaults.** Take every input from the user or from the source job/batch (its `describe` output or audit logs). Never take values from local `gcloud` configuration (e.g. `gcloud config get-value project`), environment variables, or project defaults such as the `default` network: Managed Spark silently fills in omitted values (default runtime, default subnet), so a guessed value produces a run that looks valid but tests the wrong environment. A value the user supplied or the source job/batch actually used is not a default, even if it is named `default`. Record where each value came from in the report. 2. **Compatibility changes only.** Change only what the target version requires: removed or deprecated APIs, dependency versions, Spark properties, and edits that keep the source behavior on the target (for example, stating explicitly a default that the target changed, or casting an argument to the type that the target now enforces). Don't change business logic, filters, joins, schemas, or production input/output paths. Prefer code changes over the flags that restore the previous behavior (e.g. `spark.sql.legacy.*`), because later Spark releases can remove those flags; use a flag only when the code can't keep the old behavior or can't be edited, and list every flag in the report. 3. **Edit a local working copy.** Never overwrite the source objects in GCS, and don't push to a Git remote unless the user asks. Step 1.3 describes how to get the working copy. If a write to the Target GCS path would replace one of the job's source objects or go into a production input or output location, ask for another Target GCS path first. 4. **Isolate verification output.** A verification run must never write to production destinations. Send every output to `/test-output__temp/`: pass that path through job arguments or Spark properties when the job accepts them; otherwise submit a temporary copy of the job's code whose output paths are redirected, named `__temp` (for PySpark, `