Spark CLI

MicrosoftOfficialPublished by Microsoft

Skill
Tool integration
Software development
MIT
Low risk

From the author

Manages Fabric Spark work, including notebook cell code with %%configure, %%sql, PySpark and notebookutils, named notebook runs, Livy sessions, triage of failed or OOM notebook and pipeline Spark runs, and the Materialized Lake View (MLV) lifecycle. Load it before writing MLV SQL, since CREATE MATERIALIZED LAKE VIEW and its CONSTRAINT clause are Fabric-only. KQL materialized views belong to eventhouse-cli.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

spark-cli is a mode-based dispatcher designed to handle the full lifecycle of Fabric Spark operations. It provides structured guidance for authoring notebook code, managing interactive Livy sessions, diagnosing Spark job failures, and operating Materialized Lake Views (MLV). The skill enforces strict operational boundaries, requiring users to select a specific mode (authoring, consumption, operations, or mlv) and consult the corresponding reference documentation before executing any commands to ensure correct API usage and payload construction.

What it does

Authors and manages Fabric notebook cell code including PySpark, Scala, and %%sql.
Facilitates interactive data analysis through Livy session management.
Diagnoses failed, slow, or OOM Spark jobs and pipeline runs.
Manages the complete lifecycle of Materialized Lake Views, including definition, scheduling, and refresh monitoring.
Provides standardized patterns for data engineering, schema management, and error handling in Fabric.
Ensures proper telemetry and authentication for all Fabric API interactions.

When to use it

When writing or updating code within Fabric notebooks.
When performing ad-hoc data analysis using interactive Livy sessions.
When investigating performance bottlenecks or failures in Spark jobs.
When creating, scheduling, or troubleshooting Materialized Lake Views.
When you need to ensure your Fabric API calls follow mandatory telemetry and authentication standards.

Advantages

Provides a structured, mode-based approach to prevent configuration errors.
Centralizes complex Spark and MLV management tasks into a single interface.
Enforces best practices for data engineering and schema management.
Reduces ambiguity by clearly defining the scope of Spark-related operations versus other CLI tools.

Usage examples

“Write a notebook cell that reads a bronze table and writes a silver Delta table.”: The agent enters 'authoring' mode, reads the relevant reference, and generates the appropriate PySpark code using Delta Lake patterns.
“My notebook run failed with an OOM error, can you help me diagnose it?”: The agent enters 'operations' mode, reads the diagnostic workflow reference, and guides the user through analyzing the job logs and metrics.
“Schedule a nightly refresh for the lakehouse MLV and show me its run history.”: The agent enters 'mlv' mode, reads the MLV reference, and executes the necessary API calls to configure the schedule and retrieve the history.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/spark-cli/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip spark-cli.zip -d .agents/skills
```

Documentation by Microsoft

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: spark-cli (az rest: --headers "x-ms-fabric-skill=spark-cli"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES
To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
Skill disambiguation: use spark-cli for notebook cell code (including %%sql cells), named notebook runs, Livy sessions, Spark failure triage, and everything about a Materialized Lake View -- writing the definition, reviewing a query for incremental-refresh readiness, and scheduling, refreshing, monitoring or diagnosing an existing one. A KQL materialized view in an Eventhouse is eventhouse-cli; plain read-only T-SQL against a Warehouse or Lakehouse SQL endpoint is sqldw-cli.
Hard routing boundary: never execute an Eventhouse/KQL materialized-view request from this skill. Route it to eventhouse-cli; if that skill is unavailable, state that the request cannot be completed in the current skill context and stop without calling Fabric APIs or creating artifacts.

Fabric Spark and Materialized Lake Views -- CLI Skill

This one skill owns Fabric Spark: notebook cell authoring, notebook runs, Livy-session analysis, Spark failure diagnostics, and the whole Materialized Lake View lifecycle.

It is a mode dispatcher and contains NO procedures. Pick the mode that matches the request from the table below, then read the matching references/<mode>.md file end to end with your file-reading tool BEFORE issuing a single command. That file holds the endpoints, payload shapes, templates and gotchas; acting without it produces wrong payloads and wrong results.

Mode selection

Mode	Use when the request ...	Example triggers	Read this first
`authoring`	writes notebook cell code (PySpark, Scala, SparkR, `%%sql`, `%%configure`), runs a notebook by name and reports a NORMAL run status, or authors a Materialized Lake View definition or reviews an MLV query for incremental-refresh readiness	write notebook code, notebook cell code, %%sql cell, run notebook, execute notebook, notebookutils, create materialized lake view, is this MLV query incremental-refresh ready	references/authoring.md
`consumption`	runs interactive ad-hoc PySpark in a Lakehouse Livy session -- never a notebook	create a Livy session, run calculation in Livy, PySpark, DataFrame analysis, join tables across lakehouses, Delta time-travel	references/consumption.md
`operations`	diagnoses a FAILED, unhealthy, throttled or slow Spark notebook / pipeline / Livy run	failed notebook, Spark Livy health, Spark OOM, why is my notebook slow, job diagnostics, 430 throttling	references/operations.md
`mlv`	discovers or operates EXISTING Materialized Lake Views: Spark SQL discovery, refresh schedules, on-demand refresh, run history, cancellation, and refresh-failure classification	discover MLVs, list materialized lake views, schedule MLV, MLV run history, cancel refresh, trigger MLV refresh, diagnose MLV refresh failure	references/mlv.md

Mode boundary rule

Mode is decided by the artefact and the outcome, not by the language. A notebook cell is always authoring even when the cell is %%sql. A Livy session is always consumption. A Spark run that FAILED or is unhealthy is operations; a run that succeeded is reported by authoring. For a Materialized Lake View the VERB decides: writing or reviewing the definition is authoring, while discovering, scheduling, refreshing, monitoring or diagnosing existing MLVs is mlv. If discovery must be executed, switch to consumption for Livy or authoring for a notebook only after reading the discovery command from references/mlv.md.

If a request genuinely spans modes, handle them one at a time and read each reference before you start that part. If the mode is ambiguous after reading this table, ask one short clarifying question instead of guessing.

Terminal write -- the step you must not skip

Reading the reference and planning the change is NOT completing the task. Each mutating mode ends with one state-changing call. If you did not issue it, nothing was persisted -- say so explicitly rather than reporting success.

Mode	Terminal write
`authoring`	for an EXISTING notebook, `POST .../notebooks/{id}/updateDefinition` to save the cell; a NEW notebook needs `POST /v1/workspaces/{ws}/items` first; for a run request, trigger the job via the Jobs API. Printing cell code into the chat is not saving or running it.
`consumption`	none -- this mode is read-only
`operations`	none -- this mode is read-only
`mlv`	`POST /v1/workspaces/{ws}/lakehouses/{lakehouse}/jobs/refreshMaterializedLakeViews/instances` for an on-demand refresh, or the schedule create/update/delete call for a scheduling request. Reporting what the schedule would be is not creating it.

Before you report the task done, confirm the terminal call returned success and, where the reference documents a readback, read the artefact back to prove the change landed.

Shared essentials (all modes)

Resolve the workspace and item first; every mode depends on it.

Task	Reference	Notes
Finding Workspaces and Items in Fabric	COMMON-CLI.md	Mandatory -- read before resolving any workspace or item id
Fabric Topology & Key Concepts	COMMON-CORE.md	Item types, workspaces, capacities
Environment URLs	COMMON-CORE.md	Sovereign / non-public cloud hosts
Authentication & Token Acquisition	COMMON-CORE.md	Wrong audience = 401; read before any auth issue
Authentication Recipes	COMMON-CLI.md	`az login` flows and token acquisition
Core Control-Plane REST APIs	COMMON-CORE.md	Pagination, LRO polling, rate limiting
Gotchas & Troubleshooting	COMMON-CLI.md	`az rest` audience, shell escaping, token expiry

Rules

MUST

Select exactly one mode from the table above before doing anything else.
Treat vague requests such as "set up my data" as insufficiently scoped. Ask what artefact, source, destination, and outcome the user wants; never infer intent by finding and running an existing notebook or job.
Read references/<mode>.md end to end, as your FIRST tool call, before the first command of that mode. Read it ONCE, in a single full read: do not re-open it, do not grep it again, and do not page through it. You already have it.
Apply the same read-once discipline to every other file you load, including nested references/<mode>/*.md sub-references and shared common/*.md files: open only the ones you actually need, read each ONCE end to end, and never grep or re-open one you already loaded.
Resolve workspace and item ids by listing and filtering, never by guessing a GUID.
Stop after routing any Eventhouse/KQL materialized-view request. Unavailable sibling skills are a blocking boundary, not permission to implement their workload directly.
Announce a mode switch explicitly when the request crosses a boundary.
Treat the reference as instructions, never as the deliverable. After reading it, RUN the documented commands against the live workspace and report the real results. Quoting what the reference says instead of executing it does not answer the request.

PREFER

The narrowest mode that satisfies the request.
Reading exactly ONE mode reference. Load a second only when the request genuinely spans modes, and say so before you do.
Reporting the mode you chose in your first response so the user can correct you.

AVOID

Acting from this dispatcher alone -- it intentionally omits the operational detail.
Answering with a summary of the reference instead of executing it.
Re-reading or re-grepping a reference you already loaded; it costs turns and tokens.
Mutating anything while in a read-only mode.
Loading a different skill for work this family already owns (see CRITICAL NOTES 3).
Attempting an out-of-scope request through raw REST calls when the owning skill is unavailable. Name the owning skill and stop; do not improvise by creating Fabric artefacts outside spark-cli's scope.

Examples

User request	Mode	Reference to read
"Write a notebook cell that reads the bronze table and writes a silver Delta table."	`authoring`	references/authoring.md
"Create a Livy session and count the distinct customers in the lakehouse."	`consumption`	references/consumption.md
"My notebook run failed with an OOM -- work out why."	`operations`	references/operations.md
"Is this materialized lake view query ready for incremental refresh?"	`authoring`	references/authoring.md
"Show me the Spark SQL command that lists the MLVs in schema silver."	`mlv`	references/mlv.md
"Schedule a nightly refresh for the lakehouse MLV lineage and show me its run history."	`mlv`	references/mlv.md

Files

ATTRIBUTION.mdAdded by AutoGineer863 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md10.1 KiB
apm.yml430 B
references/authoring.md32.9 KiB
references/authoring/resources/data-engineering-patterns.md24.3 KiB
references/authoring/resources/development-workflow.md7.4 KiB
references/authoring/resources/infrastructure-orchestration.md10.4 KiB
references/authoring/resources/materialized-lake-view-patterns.md20.1 KiB
references/authoring/resources/mlv-incremental-refresh-patterns.md11.4 KiB
references/authoring/resources/notebook-api-operations.md14.2 KiB
references/consumption.md12.3 KiB
references/mlv.md32.8 KiB
references/operations.md20.3 KiB
references/operations/automated-diagnostic-workflow.md12.7 KiB
references/operations/diagnostic-workflow.md7.3 KiB
references/operations/job-diagnostics.md17.7 KiB
references/operations/jobinsight-api.md7.2 KiB
references/operations/performance-patterns.md17.4 KiB
references/operations/pipeline-diagnosis.md20.8 KiB
references/operations/session-health.md8.7 KiB
references/operations/spark-history-server.md6.5 KiB

references/authoring/resources/mlv-incremental-refresh-patterns.md

MLV Incremental Refresh Patterns — Skill Resource

Skill resource for reviewing and improving incremental refresh readiness for Microsoft Fabric Materialized Lake Views (MLVs). Contains user-facing guidance and an agent routing section at the end of the file.

Use this resource when the task is about:

why an MLV may be doing a full refresh
how to rewrite an MLV without changing business logic
source-table readiness for incremental refresh
which SQL patterns are safer for refresh-friendly MLV design

Note: PySpark-authored MLVs always default to full refresh — incremental refresh applies only to Spark SQL MLV definitions. If the user's MLV is PySpark-based, the incremental readiness review does not apply.

IR-friendly syntax guide

Goal: Help users write MLV queries that qualify for incremental refresh from the start. The authoritative Must/Prefer/Avoid for MLV authoring is in materialized-lake-view-patterns.md. The rules below are the incremental-refresh-specific subset — follow both.

IR-friendly SQL patterns (use these)

These SQL patterns are compatible with incremental refresh per the official supported constructs list:

Pattern	Example	Notes
Simple SELECT with filters	`SELECT col1, col2 FROM src WHERE col1 IS NOT NULL`	Best case — flat projection + deterministic filter
`COUNT(*)`, `SUM(col)`	`SELECT region, COUNT(*) AS cnt, SUM(amount) AS total FROM src GROUP BY region`	Preferred aggregates for IR
`GROUP BY` with simple columns	`SELECT region, status, COUNT(*) FROM src GROUP BY region, status`	Keep grouping keys simple
`INNER JOIN`	`SELECT a.id, b.name FROM src_a a INNER JOIN src_b b ON a.id = b.id`	Safest join type for IR
`LEFT OUTER JOIN`	`SELECT a.*, b.name FROM src_a a LEFT JOIN src_b b ON a.id = b.id`	Supported — IR works only if the right-side table remains unchanged during the refresh cycle (MS Learn)
`LEFT SEMI JOIN`	`SELECT a.* FROM src_a a LEFT SEMI JOIN src_b b ON a.id = b.id`	Same right-side-unchanged constraint as LEFT OUTER JOIN; returns only left-side columns (right-side columns are not projected)
`UNION ALL`	`SELECT * FROM src_a UNION ALL SELECT * FROM src_b`	Supported for combining multiple sources
`CAST` / type conversions	`SELECT CAST(amount AS DOUBLE) FROM src`	Schema reshaping is fine
`CASE WHEN` in SELECT	`SELECT CASE WHEN amount > 100 THEN 'High' ELSE 'Low' END AS tier FROM src`	Deterministic expressions are safe
`CONSTRAINT ... ON MISMATCH DROP`	See data quality patterns in materialized-lake-view-patterns.md	Row-level data quality constraints with deterministic functions
Subquery alias (inline view)	`SELECT sub.col FROM (SELECT col FROM src WHERE ...) sub`	Subqueries and CTEs work if they use only supported clauses
Non-recursive `WITH ... AS` (CTE)	`WITH clean AS (SELECT ... FROM src WHERE ...) SELECT * FROM clean`	Keep CTEs simple; avoid nesting blockers inside

Caution patterns (test before relying on IR)

These are not explicitly listed as supported in the official docs. They may work in some cases but deserve extra validation:

Pattern	Guidance
`HAVING`	Aggregate filter — not listed as supported; test whether it affects refresh eligibility
`GROUP BY` with `CASE WHEN` expressions	Adds complexity; test carefully
`Multi-level INNER JOIN chains (3+)`	May work but adds risk; consider staged Silver MLVs
`Subqueries in SELECT or WHERE (scalar subqueries, EXISTS)`	Per docs, triggers full refresh if any referenced table has changes
`AVG()`, `MIN()`, `MAX()`, `STDDEV()` (and similar aggregates other than `SUM` / `COUNT`)	IR-eligible only when every source table is partitioned AND the partition column is included in the MLV `GROUP BY`. Without that, falls back to full refresh. `SUM()` and `COUNT()` (without `DISTINCT`) are the special case that don't need partitioning.

Patterns that force full refresh (rewrite these)

If you need incremental refresh, rewrite queries that use these patterns:

Pattern	Why it blocks IR	IR-friendly alternative
`SELECT DISTINCT`	Non-incremental by nature	Use `GROUP BY` instead, or move DISTINCT to a downstream view
`ROW_NUMBER()`, `RANK()`, `LAG()`, `LEAD()`	Window functions require full recomputation	Keep the MLV flat; apply windowing in a downstream query
`RIGHT JOIN`, `FULL OUTER JOIN`, `CROSS JOIN`	Not eligible for incremental refresh	Rewrite as `INNER JOIN` or `LEFT JOIN` where semantics allow
`ORDER BY`, `LIMIT`	Not on the IR-supported constructs list — forces full refresh (and adds unnecessary sort cost since materialized output ordering is not guaranteed to consumers)	Remove from the MLV; apply ordering in the consuming query
`current_timestamp()`, `current_date()`, `rand()`	Non-deterministic — result changes each refresh	Remove or move to a downstream view
`COUNT(DISTINCT col)`	DISTINCT forces full refresh	Use `COUNT(*)` on a pre-deduplicated Silver MLV
`date_sub(current_date(), 90)`	Rolling window changes each refresh	Use a fixed filter; manage the window in a pipeline parameter
`EXCEPT`, `INTERSECT`	Set operations (other than `UNION ALL`)	Rewrite as `LEFT JOIN ... WHERE ... IS NULL` or `INNER JOIN`
`QUALIFY`, `LATERAL VIEW`, `TABLESAMPLE`	Advanced clauses not IR-compatible	Simplify to basic SELECT/JOIN/GROUP BY
`WITH RECURSIVE`	Recursive CTEs	Break recursion into staged MLVs
User-defined functions (UDFs)	Non-deterministic or unsupported	Use built-in Spark SQL functions

Source table prerequisites

Before any syntax review, confirm the source tables are IR-ready:

Prerequisite	Required	How to check
Delta format	Yes	Non-Delta sources (CSV, Parquet, JSON) force full refresh
Change Data Feed (CDF) enabled	Required for IR	`ALTER TABLE src SET TBLPROPERTIES (delta.enableChangeDataFeed = true)`
Append-only pattern	Required (per cycle)	Updates/deletes on sources fall back to full refresh for that cycle, even with CDF enabled

Readiness workflow

When reviewing an MLV for incremental refresh readiness:

Step 1: Check source tables

Confirm each source meets the prerequisites above (Delta format, CDF, append-only).

Step 2: Compare the query against the IR-friendly patterns

Walk through the MLV SQL and check each clause against the tables above. Identify which patterns are IR-friendly, which need caution, and which force full refresh.

Step 3: Suggest rewrites using the IR-friendly alternatives

Only suggest changes that preserve the business meaning of the query. Use the "IR-friendly alternative" column from the "force full refresh" table.

Step 4: Produce the report

Use this structure:

## IR Readiness Report

**Overall Assessment:** [IR-Ready | Partially Ready | Not IR-Eligible]

### Blockers
### Warnings
### Good Practices Detected
### Source Table Checklist
### Top Recommendations

Safe rewrite patterns

Pattern 1: Move ranking downstream

❌ Avoid in the MLV:

CREATE MATERIALIZED LAKE VIEW gold.latest_orders AS
SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date DESC) AS rn
FROM silver.orders;

✅ Keep the MLV deterministic:

CREATE MATERIALIZED LAKE VIEW gold.orders_base AS
SELECT customer_id, order_date, amount
FROM silver.orders;

Then apply ranking in a notebook or consuming query.

Pattern 2: Remove moving time windows from the MLV

❌ Avoid:

CREATE MATERIALIZED LAKE VIEW gold.recent_sales AS
SELECT product_id, sale_date, amount
FROM silver.sales
WHERE sale_date >= date_sub(current_date(), 90);

✅ Prefer:

CREATE MATERIALIZED LAKE VIEW gold.sales_base AS
SELECT product_id, sale_date, amount
FROM silver.sales;

Then filter for “last 90 days” in the BI or notebook layer.

Pattern 3: Prefer simpler aggregates

✅ Good refresh-friendly shape:

CREATE MATERIALIZED LAKE VIEW gold.daily_sales AS
SELECT
    order_date,
    region,
    COUNT(*) AS order_count,
    SUM(amount) AS total_revenue
FROM silver.orders
GROUP BY order_date, region;

If users request averages, explain the tradeoff and prefer storing totals and counts when that still meets the business need.

Pattern 4: Keep presentation logic downstream

Avoid turning the MLV into a reporting layer. Prefer raw business measures inside the MLV and format later.

Source-table readiness guidance

Enable CDF on all source tables before relying on incremental refresh:

ALTER TABLE bronze.orders SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
ALTER TABLE bronze.customers SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

Prefer append-only ingestion for fact tables
Deletes or updates on source data cause fallback to full refresh for that cycle
Verify the Optimal refresh toggle is enabled (default: On) in schedule settings
For MLV chains, enable CDF on intermediate MLVs too

Sweeping refresh cadence: incremental daily + full weekly

The most cost-effective production pattern for MLVs with CDF-enabled sources:

Cadence	Method	Purpose
Daily (scheduled)	Automatic (optimal refresh)	Incremental refresh via CDF — fast, low-cost, processes only new/changed rows
Weekly (manual or scheduled)	`REFRESH MATERIALIZED LAKE VIEW schema.view_name FULL`	Full rebuild — catches edge cases IR misses (schema drift, orphaned deletes, compaction)

When to use this pattern:

Source tables are append-heavy with occasional updates/deletes
MLVs serve dashboards that need daily freshness
You want cost control (IR = ~5% of full refresh cost) with weekly safety net

Setup:

Enable CDF on all source tables: ALTER TABLE src SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
Create a Daily schedule via REST API or Lakehouse UI (Optimal Refresh = On)
Run REFRESH MATERIALIZED LAKE VIEW ... FULL weekly via notebook pipeline or manual trigger

Why the weekly FULL matters:

Incremental refresh cannot detect: schema changes in sources, vacuum/compaction side effects, or changes that bypass CDF (e.g., direct file manipulation)
FULL rebuild guarantees consistency — catches anything IR missed during the week
Cost: one FULL per week ≈ same as 7 incremental refreshes (~14% overhead for 100% correctness guarantee)

Tip: Schedule the weekly FULL on a low-traffic window (e.g., Sunday 3 AM) to avoid capacity contention with dashboards.

Example assessment language

IR-Ready ✅ — no hard blockers, source prerequisites satisfied, deterministic query shape
Partially Ready ⚠️ — no obvious blockers but source readiness unknown or caution areas remain
Not IR-Eligible ❌ — one or more hard blockers present in the current definition

Routing guidance for the agent

Use this resource when the user asks about incremental refresh readiness, full-refresh debugging, or refresh-friendly SQL rewrites. Pair with materialized-lake-view-patterns.md for broader MLV design guidance. See that file's reference table for full documentation links.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills-for-fabric · skills/spark-cli · commit f8fac11

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

microsoft/skills-for-fabric

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download spark-cli.zip