Dataflows CLI
MicrosoftOfficialPublished by Microsoft
- Skill
- Tool integration
- Automation and integrations
- MIT
- Low risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The dataflows-cli skill provides a structured interface for interacting with Fabric Dataflow Gen2 items. It operates as a mode-based dispatcher, requiring users to select between authoring, consumption, or upgrade modes. The skill handles complex tasks such as M language definition updates, connection management, output destination configuration, and the migration of legacy Gen1 dataflows to Gen2.1. It emphasizes safe, intentional operations by requiring users to read specific reference documentation before executing state-changing API calls.
What it does
- Creates and updates Fabric Dataflow Gen2 items and their definitions.
- Manages connections, credentials, and output destinations for dataflows.
- Previews Power Query M definitions using executeQuery to validate logic before persistence.
- Facilitates the upgrade of Gen1 dataflows to Gen2.1 via save-as operations.
- Retrieves refresh history and inspects existing dataflow definitions.
- Provides a framework for executing and testing ad-hoc M queries.
When to use it
- When you need to programmatically create or modify Fabric Dataflow Gen2 items.
- When performing maintenance on Power Query M definitions or data connections.
- When migrating legacy Gen1 dataflows to the Gen2.1 architecture.
- When you need to audit refresh history or validate dataflow configurations.
- When testing M query logic against live data sources before saving changes.
Advantages
- Enforces a structured, mode-based workflow to prevent accidental mutations.
- Provides granular control over Dataflow Gen2 definitions and metadata.
- Supports complex lifecycle tasks like Gen1-to-Gen2 upgrades.
- Integrates directly with Fabric REST APIs for automated data pipeline management.
Usage examples
- “Show me the refresh history and parameters of the SalesIngest dataflow.”
- The agent enters consumption mode, reads the consumption reference, and retrieves the requested refresh history and parameters for the specified dataflow.
- “Preview this M query, then save it into the SalesIngest dataflow definition.”
- The agent enters authoring mode, previews the M query to validate it, and then executes the updateDefinition call to persist the changes.
- “Is my Gen1 dataflow ready to upgrade to Gen2.1?”
- The agent enters upgrade mode, performs a readiness assessment, and reports on the risk and requirements for upgrading the specified Gen1 dataflow.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/dataflows-cli/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip dataflows-cli.zip -d .agents/skills
Documentation by Microsoft
Telemetry — MANDATORY. Every
api.fabric.microsoft.comcall must carryx-ms-fabric-skill: dataflows-cli(az rest:--headers "x-ms-fabric-skill=dataflows-cli"), including every LRO poll,fabric_lroand retry. Snippets omit it — add it anyway.
CRITICAL NOTES
To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
Skill disambiguation: use
dataflows-clifor every Fabric Dataflow Gen2 item and Power Query M concern. Synapse / ADF pipeline and dataset JSON translation ispipeline-migration; Spark notebook transformations arespark-cli; T-SQL transformations aresqldw-cli.
Fabric Dataflow Gen2 -- CLI Skill
This one skill owns Fabric Dataflow Gen2 items: connections, Power Query M, output destinations, saved definitions, refresh history, and Gen1 upgrades.
It is a mode dispatcher and contains NO procedures. Pick the mode that matches the request from the table below, then read the matching references/<mode>.md file end to end with your file-reading tool BEFORE issuing a single command. That file holds the endpoints, payload shapes, templates and gotchas; acting without it produces wrong payloads and wrong results.
Read it once per session. A file you have already read stays in context, so do not re-read it on a later turn; open the deeper files under references/<mode>/ only when a row in that file's reference table matches what you still need, and open only that row's file.
Mode selection
| Mode | Use when the request ... | Example triggers | Read this first |
|---|---|---|---|
authoring | creates or changes a dataflow: connections, credentials, M edits previewed before save, output destinations, updateDefinition | executeQuery preview, updateDefinition, connection setup, supportedConnectionTypes, credentialType, add output destination | references/authoring.md |
consumption | reads a saved dataflow: definition, parameters, refresh history, or runs a saved/ad-hoc read-only M query and parses Arrow | getDefinition, executeQuery saved query, ad-hoc query, Arrow IPC, refresh history | references/consumption.md |
upgrade | upgrades a Gen1 dataflow to Gen2.1 via save-as and rebind, or assesses upgrade readiness/risk | Gen1 Gen2 readiness, saveAsNativeArtifact, clone Gen1 dataflow, upgrade risk assessment | references/upgrade.md |
Mode boundary rule
consumption is read-only. A preview run that exists to validate M before updateDefinition, and any capability lookup (supportedConnectionTypes, credentialType), belong to authoring even though they issue the same executeQuery call -- classify by intent, not by endpoint.
If a request genuinely spans modes, handle them one at a time and read each reference before you start that part. If the mode is ambiguous after reading this table, ask one short clarifying question instead of guessing.
Terminal write -- the step you must not skip
Reading the reference and planning the change is NOT completing the task. Each mutating mode ends with one state-changing call. If you did not issue it, nothing was persisted -- say so explicitly rather than reporting success.
| Mode | Terminal write |
|---|---|
authoring | POST /v1/workspaces/{ws}/dataflows/{id}/updateDefinition to persist the M, connections and output destinations; a NEW dataflow needs POST /v1/workspaces/{ws}/items (or /dataflows) first. executeQuery is a preview and writes nothing. If the user asked for a refresh, also POST .../jobs/instances?jobType=Refresh. |
consumption | none -- this mode is read-only |
upgrade | saveAsNativeArtifact to create the Gen2.1 artefact, then rebind. A readiness report alone does not upgrade anything. |
Before you report the task done, confirm the terminal call returned success and, where the reference documents a readback, read the artefact back to prove the change landed.
upgrade generation boundary -- HARD STOP (mandatory)
Save-as runs only from Gen1 to Gen2.1. For an execution request whose source is Gen2, or whose source generation is not established, state that no public save-as or in-place upgrade endpoint is available and stop before any API call. This does not block a read-only readiness scan whose purpose is to discover and classify Gen1 candidates.
Do not read "choose the closest endpoint and proceed" as approval to export a definition and create a copy, do not switch to authoring to do it instead, and do not mutate anything. Ask the user to clarify the intended outcome and to approve explicitly.
consumption reporting
consumption has no terminal write, so its deliverable is the answer itself. When that mode runs executeQuery, name the operation you called and state the QueryName you sent, verbatim, next to the results, plus whether it was a saved shared member or a non-persisted ad-hoc customMashupDocument. Rows alone do not tell the user which call produced them.
This applies to consumption only. In authoring, executeQuery is an intermediate preview step -- do not add reporting turns around it; go on to the terminal write.
Shared essentials (all modes)
Resolve the workspace and item first; every mode depends on it.
| Task | Reference | Notes |
|---|---|---|
| Finding Workspaces and Items in Fabric | COMMON-CLI.md | Mandatory -- read before resolving any workspace or item id |
| Fabric Topology & Key Concepts | COMMON-CORE.md | Item types, workspaces, capacities |
| Environment URLs | COMMON-CORE.md | Sovereign / non-public cloud hosts |
| Authentication & Token Acquisition | COMMON-CORE.md | Wrong audience = 401; read before any auth issue |
| Authentication Recipes | COMMON-CLI.md | az login flows and token acquisition |
| Core Control-Plane REST APIs | COMMON-CORE.md | Pagination, LRO polling, rate limiting |
| Gotchas & Troubleshooting | COMMON-CLI.md | az rest audience, shell escaping, token expiry |
Rules
MUST
Select exactly one mode from the table above before doing anything else.
Read
references/<mode>.mdend to end, as your FIRST tool call, before the first command of that mode. Read it ONCE, in a single full read: do not re-open it, do not grep it again, and do not page through it. You already have it.Resolve workspace and item ids by listing and filtering, never by guessing a GUID.
Announce a mode switch explicitly when the request crosses a boundary.
Treat the reference as instructions, never as the deliverable. After reading it, RUN the documented commands against the live workspace and report the real results. Quoting what the reference says instead of executing it does not answer the request.
When a question about what Fabric supports IS the deliverable -- supported connection types, a connector's parameters, available credential types -- answer it from the endpoint the reference names rather than from the reference's own list, and name that endpoint in your answer. Whichever mode you are in: the reader cannot tell a live value from a quoted one unless you say where it came from. This is wording in the final answer, not an extra step.
Produce every artefact the user asked for, under the name they used. Running the right API calls is not a substitute for the requested report: if the request names a deliverable, emit it, and keep its heading even when the finding is "none" or "not applicable".
PREFER
The narrowest mode that satisfies the request.
Reading exactly ONE mode reference. Load a second only when the request genuinely spans modes, and say so before you do.
Reporting the mode you chose in your first response so the user can correct you.
AVOID
Acting from this dispatcher alone -- it intentionally omits the operational detail.
Answering with a summary of the reference instead of executing it.
Re-reading or re-grepping a reference you already loaded; it costs turns and tokens.
Mutating anything while in a read-only mode.
Loading a different skill for work this skill already owns (see CRITICAL NOTES 3).
Examples
| User request | Mode | Reference to read |
|---|---|---|
| "Show me the refresh history and parameters of the SalesIngest dataflow." | consumption | references/consumption.md |
| "Preview this M query, then save it into the SalesIngest dataflow definition." | authoring | references/authoring.md |
| "Is my Gen1 dataflow ready to upgrade to Gen2.1?" | upgrade | references/upgrade.md |
Files
- ATTRIBUTION.mdAdded by AutoGineer871 B
- LICENSELicence file added by AutoGineer1.1 KiB
- SKILL.md9.5 KiB
- apm.yml445 B
- references/authoring.md55.3 KiB
- references/authoring/authoring-cli-quickref.md23.4 KiB
- references/authoring/authoring-script-templates.md79.7 KiB
- references/authoring/charts/bar_chart.pyScript3.9 KiB
- references/authoring/charts/line_chart.pyScript3.5 KiB
- references/authoring/connection-management.md39.8 KiB
- references/authoring/connectors.md14.7 KiB
- references/authoring/m-language.md9.7 KiB
- references/authoring/mashup-preview.md28.3 KiB
- references/authoring/output-destinations.md39.6 KiB
- references/consumption.md28.7 KiB
- references/consumption/chart-visualization.md2.6 KiB
- references/consumption/charts/bar_chart.pyScript3.9 KiB
- references/consumption/charts/line_chart.pyScript3.5 KiB
- references/consumption/consumption-cli-quickref.md6.7 KiB
- references/consumption/discovery-queries.md7.0 KiB
- references/consumption/script-templates.md17.1 KiB
- references/upgrade.md21.0 KiB
- references/upgrade/risk-assessment-guide.md11.2 KiB
- references/upgrade/upgrade-cli-quickref.md8.8 KiB
references/authoring/connection-management.md
Connection Management via CLI
Operational guide for the full Fabric data source connection lifecycle — discover, create, list, find, inspect, test, and bind to dataflows — using az rest against the Fabric REST API. Pairs with the existing bind connection to dataflow workflows under authoring-cli-quickref.md § Connection Binding Quick Patterns and connection discovery/validation under authoring-cli-quickref.md § Connection Discovery and Validation.
Scope. This reference covers
GET /v1/connections,GET /v1/connections/{id},GET /v1/connections/supportedConnectionTypes,POST /v1/connections,POST /v1/connections/{id}/testConnection, and the discover → inspect → create → bind → test → refresh flow needed when a Dataflow Gen2 references a connection that does not yet exist. Sharing, role assignments,PATCH, andDELETEof connections are related lifecycle tasks but are out of scope here.API-only. Every step uses
az restagainst the Fabric REST API. Portal/UI-driven connection creation (Fabric portal "New connection" dialog, OAuth2 browser consent flows) is out of scope — this reference targets headless, scriptable, CI-friendly automation.
Authoritative API spec. All schemas, field names, defaults, and credential shapes in this document are taken from Microsoft Learn:
Table of Contents
| Section | Why |
|---|---|
| Decision Tree | When to call which API in what order |
| Required Permissions | Scopes for read / create / gateway operations |
| Concept Model | The type system underlying every connection: connectivityType, credentialType, encryption, privacy |
| Step 1 — List Supported Connection Types | Always call this before create, never guess parameters |
| Step 2 — Create Connection (Cloud) | POST /v1/connections for ShareableCloud |
| Credential Type Schemas | Schema-accurate body shapes per credential type |
| Connection Type Examples | SQL, AzureBlobs, Web, Lakehouse, Warehouse — examples only |
| Step 3 — Verify and Get the Connection ID | GET /v1/connections/{id} to inspect; list+filter for discovery |
| Step 3b — Test the Connection (optional) | POST /v1/connections/{id}/testConnection LRO pre-bind sanity check |
| Step 4 — Bind to Dataflow and Refresh | Hand-off to existing bind workflow + post-save verification |
| Connection ID Format Cheat Sheet | REST id vs queryMetadata composite vs plain DatasourceId |
| Operational Pitfalls | Re-bind-after-save; multi-source; AllowCombine; Lakehouse consolidation |
| Gateways (Appendix) | List gateways; VNet create; on-prem caveat |
| Troubleshooting | DuplicateConnectionName, Invalid*, IncorrectCredentials |
| Out of Scope | Sharing, PATCH, DELETE, OAuth2 create, on-prem encrypted credentials |
Decision Tree
Need a connection for a dataflow?
│
├─► (1) GET /v1/connections — list and filter by displayName / path / connectionDetails.type
│ │
│ ├─ FOUND → (1a) GET /v1/connections/{id}
│ │ Inspect connectivityType, connectionDetails.type, credentialDetails, gatewayId.
│ │ │
│ │ ├─ Matches required source + network path → reuse. Take `id`, skip to (4).
│ │ └─ Wrong type / wrong network path / wrong credentials → create a new one.
│ │
│ └─ NOT FOUND → continue.
│
├─► (2) GET /v1/connections/supportedConnectionTypes
│ (filter to the type you need; capture parameter names + supported credential types)
│
├─► (3) POST /v1/connections
│ (use a `connectivityType` matching the network path: ShareableCloud / OnPremisesGateway / VirtualNetworkGateway)
│ Capture `id` from the 201 response.
│
├─► (3b) POST /v1/connections/{id}/testConnection (optional)
│ LRO. Sanity-check credentials/network before binding. Skip for connection types where
│ `skipTestConnection` was used at create time or `supportsSkipTestConnection: false`.
│
├─► (4) Bind it to the dataflow definition (`queryMetadata.json connections[]`) + `updateDefinition`,
│
└─► (5) Verify `connections[]` survived the save and trigger refresh.Why list-types-before-create? Each connection type (
SQL,AzureBlobs,Lakehouse,Warehouse,Web,Dataverse, …) has a different set of required parameters and supports a different set of credential types. The same connection type may also expose multiplecreationMethodvariants (e.g., several forWeb). Step 2 below covers the exact endpoint.
Required Permissions
Delegated scopes (or service-principal equivalents):
| Scope | Needed for |
|---|---|
Connection.Read.All | GET /v1/connections, GET /v1/connections/supportedConnectionTypes |
Connection.ReadWrite.All | POST /v1/connections (create) |
Gateway.Read.All | GET /v1/gateways |
Gateway.ReadWrite.All | POST /v1/gateways (VNet gateway only — see appendix) |
Additional caller requirements:
Service principals creating connections require Fabric tenant admin enablement: see Service principals can create workspaces, connections, and deployment pipelines.
Gateway connections require the caller to have permission on the target gateway.
For az login recipes and token audience rules, see COMMON-CLI.md § Authentication Recipes and COMMON-CORE.md § Authentication & Token Acquisition.
Concept Model
A Fabric connection is a credential-bearing object that data sources reference for authenticated access. Every connection is defined along the following dimensions — getting any of them wrong is the most common cause of InvalidInput on POST:
| Dimension | Values | Notes |
|---|---|---|
connectivityType | ShareableCloud · PersonalCloud · OnPremisesGateway · OnPremisesGatewayPersonal · VirtualNetworkGateway | Drives routing and which credential-details schema applies. ShareableCloud / PersonalCloud are cloud-only (no gatewayId). The three gateway variants require gatewayId. |
connectionDetails.type | SQL, AzureBlobs, Lakehouse, Web, Dataverse, … | The connector kind. Discover the full set via Step 1. The M kind recorded in queryMetadata.json may differ in casing — see Connection Type Examples. |
connectionDetails.creationMethod | per-type; often equal to type | Some types expose multiple creation methods (Web has several). Read each method's parameters[] from Step 1 — never guess. |
credentialType | Anonymous, Basic, Key, KeyPair, ServicePrincipal, SharedAccessSignature, WorkspaceIdentity, Windows, WindowsWithoutImpersonation | OAuth2 may appear in a connector's supportedCredentialTypes but cannot be created via this API — see Credential Type Schemas. |
privacyLevel | None, Public, Organizational, Private | Default: Organizational. Affects multi-source folding (whether queries from different sources can be combined). |
connectionEncryption | NotEncrypted, Encrypted, Any | Per-connector support varies — confirm via Step 1's supportedConnectionEncryptionTypes. |
skipTestConnection | bool | false (default) runs a test at create time; failure surfaces as IncorrectCredentials. Set true to defer validation to first use (rare). |
A cloud connection identity has two IDs: the plain Fabric GUID (.id from the POST/GET response) and a Power BI gateway ClusterId. They're embedded differently depending on context — see Connection ID Format Cheat Sheet.
Step 1 — List Supported Connection Types
GET /v1/connections/supportedConnectionTypes returns, per type:
creationMethods[].parameters[]— the names + dataTypes +requiredflag forconnectionDetails.parameterssupportedCredentialTypes[]— whichcredentialDetails.credentials.credentialTypevalues are validsupportedConnectionEncryptionTypes[]— whichconnectionEncryptionvalues are validsupportsSkipTestConnectionsupportedCredentialTypesForUsageInUserControlledCode— subset usable from notebooks (allowUsageInUserControlledCode)
Cloud (no gateway):
RESOURCE="https://api.fabric.microsoft.com"
API="https://api.fabric.microsoft.com/v1"
az rest --method get \
--resource "$RESOURCE" \
--url "$API/connections/supportedConnectionTypes" \
--query "value[?type=='SQL']"For a specific gateway (cloud + on-prem types may differ):
GW_ID="<gatewayId>"
az rest --method get \
--resource "$RESOURCE" \
--url "$API/connections/supportedConnectionTypes?gatewayId=$GW_ID&showAllCreationMethods=true"Pagination: response includes continuationToken + continuationUri; reuse the continuationToken query parameter to fetch the next page. Page until continuationToken is null — stopping early hides connector types that may only appear on later pages.
showAllCreationMethods=false (default) returns only the recommended creation methods for each type. Set to true if you need a less-common method.
⚠ Send
dataTypecorrectly per parameter. Each parameter'sdataType(Text,Boolean,Number,Date,DateTime,DateTimeZone,Duration, …) is part of the connector's contract. The FabricPOST /v1/connectionsendpoint is lenient about type coercion at create time — most connectors acceptdataType: "Text"for a numeric / boolean param and the create returns201. The failures surface later: at refresh /executeQuerytime the engine binds parameters by declared type, and a string passed where a Boolean / Number was expected produces silent NULL substitution orEntityUserFailure. ReadcreationMethods[].parameters[].dataTypefrom this Step 1 response and pass it back exactly — that is the only forward-compatible behavior.
Casing matters. REST
connectionDetails.typeisSQL. The corresponding M connector kind inqueryMetadata.json connections[].kindisSql(camel case). Do not assume 1-to-1 casing between the REST type and the dataflow definition.
Step 2 — Create Connection (Cloud)
POST /v1/connections — body shape depends on the value of connectivityType:
connectivityType | Routing | Credential schema |
|---|---|---|
ShareableCloud | Cloud, can be shared | CreateCredentialDetails (this section) |
VirtualNetworkGateway | Cloud via VNet data gateway | CreateCredentialDetails (same shape; requires gatewayId) |
OnPremisesGateway | On-prem data gateway | Different schema — see Gateways (Appendix); requires RSA-encrypted credentials |
Cloud body — minimum fields
{
"connectivityType": "ShareableCloud",
"displayName": "ContosoSqlConnection",
"connectionDetails": {
"type": "SQL",
"creationMethod": "SQL",
"parameters": [
{ "dataType": "Text", "name": "server", "value": "contoso.database.windows.net" },
{ "dataType": "Text", "name": "database", "value": "sales" }
]
},
"privacyLevel": "Organizational",
"credentialDetails": {
"singleSignOnType": "None",
"connectionEncryption": "Encrypted",
"skipTestConnection": false,
"credentials": {
"credentialType": "Basic",
"username": "admin",
"passwordReference": {
"connectionId": "<keyVaultConnectionId>",
"secretName": "sql-admin-password"
}
}
}
}Important defaults and optional fields:
| Field | Default | Notes |
|---|---|---|
privacyLevel | Organizational | Other values: None, Public, Organizational, Private. Set explicitly to make intent obvious. |
connectionEncryption | not encrypted | Allowed: NotEncrypted, Encrypted, Any. |
skipTestConnection | false | Test runs at create time; failure → IncorrectCredentials. |
singleSignOnType | None | Kerberos, MicrosoftEntraID, SecurityAssertionMarkupLanguage, KerberosDirectQueryAndRefresh. |
allowUsageInUserControlledCode | false | Set true only if the connection should be usable from Notebooks (and only when the credential type is in the supported subset for that). |
allowConnectionUsageInGateway | unset | Allow this cloud connection to also be used through a gateway. |
Bash — create cloud SQL connection (Basic auth via Key Vault reference)
#!/usr/bin/env bash
set -euo pipefail
RESOURCE="https://api.fabric.microsoft.com"
API="https://api.fabric.microsoft.com/v1"
DISPLAY_NAME="${DISPLAY_NAME:?Set DISPLAY_NAME}"
SQL_SERVER="${SQL_SERVER:?Set SQL_SERVER}"
SQL_DATABASE="${SQL_DATABASE:?Set SQL_DATABASE}"
SQL_USER="${SQL_USER:?Set SQL_USER}"
KV_CONN_ID="${KV_CONN_ID:?Set KV_CONN_ID — Fabric Key Vault connection ID}"
KV_SECRET_NAME="${KV_SECRET_NAME:?Set KV_SECRET_NAME}"
BODY=$(jq -n \
--arg name "$DISPLAY_NAME" \
--arg server "$SQL_SERVER" --arg db "$SQL_DATABASE" \
--arg user "$SQL_USER" \
--arg kvConn "$KV_CONN_ID" --arg secret "$KV_SECRET_NAME" \
'{
connectivityType: "ShareableCloud",
displayName: $name,
connectionDetails: {
type: "SQL",
creationMethod: "SQL",
parameters: [
{ dataType: "Text", name: "server", value: $server },
{ dataType: "Text", name: "database", value: $db }
]
},
privacyLevel: "Organizational",
credentialDetails: {
singleSignOnType: "None",
connectionEncryption: "Encrypted",
skipTestConnection: false,
credentials: {
credentialType: "Basic",
username: $user,
passwordReference: { connectionId: $kvConn, secretName: $secret }
}
}
}')
az rest --method post \
--resource "$RESOURCE" \
--url "$API/connections" \
--body "$BODY" \
--query "{id:id, name:displayName, type:connectionDetails.type, path:connectionDetails.path}"Secret-safe generation. Use
passwordReferencein generated examples; do not render a plaintext credential field even as an unsafe or local-testing example. Never commit plaintext credentials, and never echo or log runtime secret values.
⚠ Unknown credential keys are silently dropped. Fabric does not return
400for unknown keys insidecredentialDetails.credentials— it discards them and proceeds with the recognized fields that remain. A typo likepassWordinstead ofpasswordproduces a201 Createdwith no usable credential, and the first refresh then fails with a genericIncorrectCredentials. Validate your request body's credential keys against the discovered schema from Step 1 before POSTing.
Caching guidance.
supportedConnectionTypesis stable per tenant — cache aggressively (24h+).gatewayClusterDatasourcesupdates whenever a connection is created — cache ~5 min per process. The/v1/connectionslist changes whenever anyone with overlapping visibility creates a connection — cache short (<1m) or not at all if precision matters.
Credential Type Schemas
The full union accepted by CreateCredentialDetails.credentials for ShareableCloud and VirtualNetworkGateway connections. Always check the type's supportedCredentialTypes from Step 1 first — not every type accepts every credential.
| Credential type | Required fields | KV reference variant |
|---|---|---|
Anonymous | credentialType | — |
Basic | username, password OR passwordReference | passwordReference |
Key | key OR keyReference | keyReference |
KeyPair | identifier, privateKey (PKCS #8), passphrase | — |
ServicePrincipal | tenantId, servicePrincipalClientId, servicePrincipalSecret OR servicePrincipalSecretReference | servicePrincipalSecretReference |
SharedAccessSignature | token OR tokenReference | tokenReference |
WorkspaceIdentity | credentialType only (no other fields) | — |
Windows / WindowsWithoutImpersonation | On-prem gateway only | — |
OAuth2 is not creatable via
POST /v1/connections. It can appear insupportedCredentialTypesfor some connector types, but the Create Connection schema does not define an OAuth2 credentials body. Live-verified — the API rejects anyOAuth2payload with:400 InvalidInput { "errorCode": "InvalidInput", "message": "The request has an invalid input", "moreDetails": [{ "errorCode": "InvalidParameter", "message": "The CredentialType input is not supported for this API" }], "isRetriable": false }Use an existing OAuth2-backed connection that was authored interactively in the portal/UI, or pick a different credential type from the connector's
supportedCredentialTypes.
Key Vault reference shape (KeyVaultSecretReference)
{
"connectionId": "<keyVaultConnectionIdAsUuid>", // not a Key Vault URL — a Fabric KV connection ID
"secretName": "<secret-name-in-vault>",
"version": "<optional secret version>"
}Chicken-and-egg. A Key Vault reference points at another Fabric connection (a Key Vault connection) that already exists. If you do not have one, create the Key Vault connection first via this same API or pause and request that prerequisite; generated templates should not silently downgrade to plaintext credentials.
Service principal example
"credentials": {
"credentialType": "ServicePrincipal",
"tenantId": "<tenantId>",
"servicePrincipalClientId": "<appId>",
"servicePrincipalSecretReference": {
"connectionId": "<kvConnectionId>",
"secretName": "sp-client-secret"
}
}Workspace identity example (Lakehouse / Warehouse / Eventhouse common case)
"credentials": { "credentialType": "WorkspaceIdentity" }KeyPair example (Snowflake-style)
"credentials": {
"credentialType": "KeyPair",
"identifier": "admin",
"privateKey": "-----BEGIN ENCRYPTED PRIVATE KEY-----\n...\n-----END ENCRYPTED PRIVATE KEY-----",
"passphrase": "<passphrase>"
}Connection Type Examples
Examples only — always confirm via
supportedConnectionTypes. Connector parameters and supported credential types vary by tenant, gateway, and over time. Casing ofconnectionDetails.typemay also differ from the Mkindrecorded inqueryMetadata.json.
connectionDetails.type | Common required parameters | Typical credentials | M kind (in queryMetadata) |
|---|---|---|---|
SQL | server, database (optional) | Basic, ServicePrincipal, WorkspaceIdentity | Sql |
AzureBlobs | account, domain | Key, SharedAccessSignature, WorkspaceIdentity | AzureStorage |
Web | url | Anonymous, Basic, Key | Web |
Lakehouse | workspaceId, lakehouseId | WorkspaceIdentity, OAuth2 (not creatable here) | Lakehouse |
Warehouse | workspaceId, warehouseId | WorkspaceIdentity, ServicePrincipal | Sql (DW SQL endpoint) |
For Fabric-source connections (Lakehouse, Warehouse, Eventhouse) the workspace ID is mandatory because artifact names are scoped to the parent workspace and would otherwise be ambiguous.
Step 3 — Verify and Get the Connection ID
The POST /v1/connections 201 Created response includes the new connection's id. Capture it directly:
NEW_CONN_ID=$(az rest --method post \
--resource "$RESOURCE" \
--url "$API/connections" \
--body @body.json \
--query "id" --output tsv)
echo "Created connection: $NEW_CONN_ID"Inspect an existing connection — GET /v1/connections/{id}
When the caller already knows the connection's GUID (e.g., captured at create time, copied from another script, or extracted from queryMetadata.json connections[].connectionId's DatasourceId), prefer GET /v1/connections/{id} over GET /v1/connections + filter. It is a direct lookup, returns the full connectivityType / connectionDetails / credentialDetails shape, and surfaces EntityNotFound clearly:
az rest --method get \
--resource "$RESOURCE" \
--url "$API/connections/$NEW_CONN_ID" \
--query "{id:id, name:displayName, connectivity:connectivityType, type:connectionDetails.type, path:connectionDetails.path, credType:credentialDetails.credentialType, gw:gatewayId}"Use this to:
Confirm the connection still exists and is reachable for this caller (
403/404⇒ no access or deleted).Check
connectivityTypebefore binding — preferShareableCloud/PersonalCloudfor cloud-reachable sources overOnPremisesGateway/VirtualNetworkGatewayto avoid an unnecessary gateway-online failure surface (see Picking between PersonalCloud and OnPremisesGateway when both exist).Verify
credentialDetails.credentialTypematches the Mkindrecorded inqueryMetadata.json.
Discover by name — GET /v1/connections + filter
When the GUID is unknown (e.g., from another shell, or after POST without capturing the response), list-and-filter via the standard recipe in authoring-cli-quickref.md § Connection Discovery and Validation:
az rest --method get \
--resource "$RESOURCE" \
--url "$API/connections" \
--query "value[?displayName=='ContosoSqlConnection'].id" --output tsvPer-caller visibility.
GET /v1/connectionsonly returns connections the caller has at least read permission on; an empty result is not proof the connection is absent from the tenant. Request access from the connection owner if expected results are missing.
Step 3b — Test the Connection (optional)
After create (or before binding a connection whose credentials may have rotated), call POST /v1/connections/{id}/testConnection as a pre-bind sanity check. This catches IncorrectCredentials, offline gateways, and network-path issues before they surface as a generic EntityUserFailure on the next refresh.
This endpoint is LRO — a successful test may return either 200 OK (synchronous, with ConnectionStatusResponse) or 202 Accepted with the following headers (live-verified):
| Header | Purpose |
|---|---|
Location | Absolute URL of the LRO operation resource; GET it to poll status. |
Retry-After | Server-recommended polling interval (seconds; typically 5). |
x-ms-operation-id | Operation GUID — also the suffix of the Location URL. Use this to correlate logs across the request and any retries. |
Body of the 202 is null; all state lives on the LRO resource at Location. Final ConnectionStatusResponse payload is at ${Location}/result once status is terminal.
# az rest cannot capture response headers — use curl for the initial call so we can read Location.
TOKEN=$(az account get-access-token --resource "$RESOURCE" --query accessToken -o tsv)
HDR=$(mktemp); BODY=$(mktemp)
CODE=$(curl -sS -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Length: 0" \
"$API/connections/$NEW_CONN_ID/testConnection" \
-D "$HDR" -o "$BODY" -w "%{http_code}")
if [ "$CODE" = "200" ]; then
jq '.' "$BODY"
elif [ "$CODE" = "202" ]; then
LOC=$(tr -d '\r' < "$HDR" | grep -i "^location:" | awk '{print $2}')
RETRY=$(tr -d '\r' < "$HDR" | grep -i "^retry-after:" | awk '{print $2}'); RETRY=${RETRY:-5}
# Poll the LRO until terminal status, then GET /result for the ConnectionStatusResponse payload.
# Full polling helper: authoring-cli-quickref.md § LRO Polling Helper.
while :; do
sleep "$RETRY"
OP=$(az rest --method get --resource "$RESOURCE" --url "$LOC")
case "$(echo "$OP" | jq -r '.status // empty')" in
Succeeded) az rest --method get --resource "$RESOURCE" --url "${LOC%/}/result"; break ;;
Failed|Cancelled) echo "ERROR: testConnection $(echo "$OP" | jq -r '.status')" >&2; exit 1 ;;
esac
done
else
echo "ERROR: testConnection HTTP $CODE" >&2; cat "$BODY" >&2; exit 1
fi
rm -f "$HDR" "$BODY"Skip Step 3b when:
The connection was created with
skipTestConnection: trueand the connector advertisessupportsSkipTestConnection: falsefor the chosen credential type (call will return400).The source is rate-limited and a probe call is unsafe.
The agent records an explicit skip reason (e.g., test traffic billed per call).
Common failures: IncorrectCredentials (rotate or re-create), EntityNotFound (connection deleted or no access), gateway-offline (the response will surface the gateway error).
Step 4 — Bind to Dataflow and Refresh
Once you have the connection's id GUID, bind it into queryMetadata.json and update the dataflow definition. The bind mechanics already live in this skill — do not reinvent:
authoring-cli-quickref.md § Connection Binding Quick Patterns —
az restsnippets to fetch ClusterId, editqueryMetadata.json, andupdateDefinition.authoring-script-templates.md § Connection Binding Templates — full end-to-end Bash/PowerShell flow.
After updateDefinition, always verify that queryMetadata.json connections[] still contains your binding (see Operational Pitfalls below). Then trigger refresh and poll using the existing helpers.
Connection ID Format Cheat Sheet
Three different ID forms appear in this flow. Confusing them is the most common cause of "connection not found" at refresh.
| Where | Field | Format |
|---|---|---|
POST /v1/connections response | id | Plain GUID, e.g. eeec9a3a-6ef5-4e2b-bb6a-0060bd2f0172 |
GET /v1/connections response | value[].id | Plain GUID |
GET /v1/connections/{id} response | id | Plain GUID |
queryMetadata.json connections[].connectionId (dataflow definition) | connectionId | Stringified composite JSON, e.g. "{\"ClusterId\":\"<guid>\",\"DatasourceId\":\"<guid>\"}" |
Power BI v2 gatewayClusterDatasources response | clusterId (camelCase) | Plain GUID — the ClusterId to embed in the composite above. See Resolving ClusterId below. |
Rule of thumb:
REST
POST/GET/PATCH/DELETEoperations on/v1/connections/...want the plain GUID (the.idfrom any of the above responses).Dataflow definition
queryMetadata.json connections[].connectionIdwants the stringified composite{"ClusterId":"…","DatasourceId":"…"}.
Resolving ClusterId (Power BI v2)
The ClusterId value embedded in the composite comes from the Power BI v2 control-plane endpoint myorg/me/gatewayClusterDatasources. Use the list-and-filter pattern — list the user's gateway-cluster datasources, then filter the response by the connection's plain GUID (value[?id=='{connId}'].clusterId):
| URL | Returns | Notes |
|---|---|---|
GET https://api.powerbi.com/v2.0/myorg/me/gatewayClusterDatasources | { value: [...] } — flat, paginated array of cluster + datasource records, each with clusterId, id, datasourceReference, … | Canonical pattern. Filter by id == <connectionId> to obtain clusterId. Newly-created connections may take a few seconds to surface here — retry the list call if the filter returns empty. |
Don't use the per-id route.
GET .../gatewayClusterDatasources/{datasourceId}returnsPowerBIEntityNotFoundfor cloud connections (verified live againstconnectivityType: ShareableCloud+ Web). Use list+filter only.
Token audience. The list endpoint accepts the Power BI audience (
https://analysis.windows.net/powerbi/api— no trailing slash; the slashed form fails AADSTS500011). The Fabric token (https://api.fabric.microsoft.com/) is also accepted in current tenants — for scripts that already hold a Fabric token, no separate token acquisition is needed.
PBI_RESOURCE="https://analysis.windows.net/powerbi/api"
CONN_ID="<datasourceId>"
CLUSTER_ID=$(az rest --method get \
--resource "$PBI_RESOURCE" \
--url "https://api.powerbi.com/v2.0/myorg/me/gatewayClusterDatasources" \
--query "value[?id=='$CONN_ID'] | [0].clusterId" --output tsv)Empty result after retries. If the filter still returns empty after ~30s of retries, the connection is not visible to PBI v2 — that's a connection-lifecycle problem (e.g., orphaned record after a failed create), not a ClusterId-lookup problem. Verify the connection exists via
GET /v1/connections/{id}; if Fabric shows it but PBI v2 doesn't, recreate the connection.
Picking between PersonalCloud and OnPremisesGateway when both exist
When multiple connections target the same source (e.g., a public OData feed registered both as connectivityType: PersonalCloud and connectivityType: OnPremisesGateway), prefer the PersonalCloud connection for cloud-only / publicly reachable sources:
connectivityType | When to use | Failure surface |
|---|---|---|
PersonalCloud | Source is reachable from the Fabric cloud (public REST/OData/Web endpoints, Azure-hosted sources) | None beyond the cloud → source path. |
OnPremisesGateway (or VirtualNetwork) | Source sits behind a firewall / on-prem network and a gateway hop is mandatory | Adds gateway-online and gateway-permission as additional failure modes. executeQuery and refresh both fail (often with the same generic "Credentials required" / EntityUserFailure error) when the gateway is offline or misconfigured. |
If a cloud-reachable source happens to be registered against a gateway connection, picking that connection forces an unnecessary gateway hop and an unnecessary failure surface — the preview and refresh will fail in non-obvious ways if the gateway is cold or the cluster is unhealthy. Inspect each candidate connection with GET /v1/connections/{id} (see Step 3 — Inspect an existing connection) and check connectivityType before binding.
Operational Pitfalls
These are observed pitfalls, not formal API guarantees — verify each one in your environment before treating it as load-bearing.
Verify connections survived updateDefinition
updateDefinition is a full replacement: any 3-part payload that omits or scrubs queryMetadata.json connections[] will leave the dataflow with no bindings. If a workflow modifies mashup.pq and re-builds queryMetadata.json from a stale snapshot, bindings get silently dropped.
Mandatory step after every updateDefinition:
⚠ LRO caveat: the
getDefinitioncall below assumes a synchronous 200 response. For production code, handle 202 + Location with the LRO-aware curl pattern (see authoring-cli-quickref.md § Validate All Connections in a Dataflow or authoring-script-templates.md § Bash — Read-Modify-Write Dataflow Definition).
# Re-fetch and confirm bindings survived (happy-path 200 only — see caveat)
RESULT=$(az rest --method post \
--resource "$RESOURCE" \
--url "$API/workspaces/$WS_ID/dataflows/$DF_ID/getDefinition" \
--headers "Content-Length=0")
echo "$RESULT" \
| jq -r '.definition.parts[] | select(.path=="queryMetadata.json") | .payload' \
| base64 -d \
| jq '.connections // []'If connections[] is empty or missing the entry you expected, re-bind and updateDefinition again before triggering refresh.
Multi-source dataflows
If a dataflow reads from multiple distinct sources, each source needs its own connection bound. The M section also needs to opt in to combining queries:
[AllowCombine = true]
section Section1;Without [AllowCombine = true], the engine refuses to fold queries that touch multiple sources and refresh fails with privacy-level errors.
For multiple Lakehouse reads in the same workspace, consolidate into one Lakehouse.Contents([...]) call rather than calling it once per table — the engine treats each call as a separate source for combine-rule purposes.
Don't convert published single-source dataflows to multi-source in place
Editing a published single-source dataflow to add a second source frequently leaves the dataflow in an inconsistent binding state. Create a fresh dataflow that is multi-source from the start, then retire the old one.
Duplicate connection names
POST /v1/connections with a displayName that already exists returns 409 DuplicateConnectionName. Recovery:
# 1. List existing connections with the same name
az rest --method get --resource "$RESOURCE" --url "$API/connections" \
--query "value[?displayName=='ContosoSqlConnection']"
# 2. Either reuse the existing id, or pick a unique name (e.g. add an environment suffix)
DISPLAY_NAME="ContosoSqlConnection-$(date +%Y%m%d-%H%M)"Gateways (Appendix)
List gateways
az rest --method get \
--resource "$RESOURCE" \
--url "$API/gateways" \
--query "value[].{id:id, name:displayName, type:type}"VNet gateway connection
VNet gateway connections use the same CreateCredentialDetails schema as ShareableCloud, plus a top-level gatewayId:
{
"connectivityType": "VirtualNetworkGateway",
"gatewayId": "<vnetGatewayId>",
"displayName": "ContosoVnetSqlConnection",
"connectionDetails": { /* same as cloud */ },
"privacyLevel": "Organizational",
"credentialDetails": {
"singleSignOnType": "None",
"connectionEncryption": "Encrypted",
"skipTestConnection": false,
"credentials": { /* Basic / ServicePrincipal / WorkspaceIdentity / etc. */ }
}
}Creating the VNet gateway itself (
POST /v1/gateways) requires a Fabric capacity, an Azure subscription, resource group, VNet, subnet, and subnet delegation — that's gateway provisioning, outside the dataflows authoring scope. Treat the VNet gateway as a prerequisite that exists before you author the connection.
On-premises gateway connection — different credential flow
OnPremisesGateway connections use CreateOnPremisesCredentialDetails, where credentials are not plaintext but RSA-encrypted with each gateway member's public key:
"credentialDetails": {
"singleSignOnType": "None",
"connectionEncryption": "NotEncrypted",
"skipTestConnection": false,
"credentials": {
"credentialType": "Windows",
"values": [
{ "gatewayId": "<gatewayMember1>", "encryptedCredentials": "<RSA-encrypted-blob>" },
{ "gatewayId": "<gatewayMember2>", "encryptedCredentials": "<RSA-encrypted-blob>" }
]
}
}Out of scope for ready-to-run templates. Generating
encryptedCredentialsrequires fetching the gateway member's RSA public key and serializing the credential payload according to Microsoft's encryption format. See Configure credentials programmatically for the algorithm. Plaintext templates do not work for on-prem.
For Dataflow Gen2 use, prefer either ShareableCloud (when the source is reachable from the cloud) or VirtualNetworkGateway over on-prem when feasible.
Troubleshooting
| Error / Symptom | Root cause | Fix |
|---|---|---|
409 DuplicateConnectionName | A connection with that displayName already exists in the tenant. | List existing → reuse id or rename. |
400 InvalidConnectionDetails | Wrong type, missing/extra parameters, or wrong creationMethod. | Re-run GET /v1/connections/supportedConnectionTypes; copy parameter names exactly. |
400 InvalidCredentialDetails | credentialType not in supportedCredentialTypes for that connection type, or required fields missing (e.g., SP without tenantId). | Verify schema in Credential Type Schemas; re-check supportedConnectionTypes. |
400 InvalidInput — "The CredentialType input is not supported for this API" | Tried to create with credentialType: OAuth2 (or another non-API-creatable type) via POST /v1/connections. | OAuth2 connections must be authored interactively in the portal — see Out of Scope. Pick a different credential type from the source's supportedCredentialTypes. |
400 IncorrectCredentials | Test connection failed at create time. | Verify credentials by hand against the source; or set skipTestConnection: true if the type supports it. |
400 CreateGatewayConnectionFailed | connectivityType is gateway-bound but gatewayId is wrong, the caller lacks gateway permission, or (on-prem) encryptedCredentials is malformed. | Confirm gatewayId exists; check caller's gateway role; for on-prem, regenerate RSA-encrypted credentials. |
403 Forbidden on POST | Caller lacks Connection.ReadWrite.All, or service principal not enabled by tenant admin. | Check delegated scope or grant admin enablement. |
429 Too Many Requests | Tenant connection-create rate limit. | Honor Retry-After; back off. |
| Refresh after create reports "connection not found" | The dataflow was bound using the wrong ID format (e.g., plain GUID where composite is required, or vice versa). | See Connection ID Format Cheat Sheet. |
connections[] missing after updateDefinition | Read-modify-write rebuilt queryMetadata.json from a snapshot that did not include bindings. | Re-bind, updateDefinition again, verify before refresh. |
Out of Scope
The following endpoints and flows are live-confirmed to exist at the API level, but their lifecycle, governance, and integration surfaces differ from the discover → create → bind path covered here:
Update a connection (
PATCH /v1/connections/{id}) — rotate credentials, changedisplayName, adjustprivacyLevel. Endpoint exists (returns typedInvalidInputon bad payload, not405).Delete a connection (
DELETE /v1/connections/{id}) — destructive; not idempotent across active bindings. Endpoint exists (returnsEntityNotFoundon missing id, not405).Connection sharing / role assignments — granting other principals access to use the connection.
OAuth2 connection authoring — interactive only (portal / UI); the API rejects
OAuth2inPOST /v1/connectionswithInvalidInput(see Troubleshooting).On-premises gateway credential encryption — see Microsoft's Configure credentials programmatically for the RSA encryption format.
Creating gateways — VNet/on-prem gateway provisioning is an admin/network task, not a dataflows authoring task.
For dataflow → connection binding (where you already have a connection ID), see:
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python
Security
What our scanner noticed· Low risk
- Instructions that use the network
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Microsoft
- Publisher
- Microsoft (Official)
- Source
- microsoft/skills-for-fabric · skills/dataflows-cli · commit f8fac11
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- microsoft/skills-for-fabric
AutoGineer is not affiliated with or endorsed by Microsoft.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download dataflows-cli.zip