Managing Data Catalog

Describing assets: descriptions, README, ownership, columns

The operations here are well known and you probably know them already. What goes wrong is the merge, the surface, and the content. This chapter is about those three.

What a good description says

A description that lists the columns has documented nothing: the column names were already in the catalog. State what the data means and what it shows, including where it is incomplete or surprising.

• You MUST look at the data or its quality results before writing, not only at the schema. A table's meaning is often not guessable from its name.

• You MUST state the finding, not the framing. Naming what the table holds describes it; saying what the data turned out to show is the documentation the user asked for. The contrast pair below is the same table written both ways.

• If a data quality score, a row count or a coverage gap changes how the table should be read, say so. A table that covers only part of the population, or starts partway through the history, or whose key column is nullable in practice, is a table a consumer will misuse unless the description says so.

• Keep it to what a consumer needs to decide whether to use the table. No credentials, no PII values, no cross-account ARNs.

The failure has a shape, so here it is twice

The wrong version is not lazy or short. It is usually fluent, accurate about the structure, and written without looking at a single value, which is why it passes a self-review. Both of these describe the same imaginary table.

Not this. "Monthly active users by region. Each row is one region and one month, and reports the count of active users along with the month-over-month change."

That is a schema restated in sentences. Every fact in it was already in the catalog: a consumer learns nothing they could not have read off the column list, and nothing in it could have come out differently whatever the data said.

This. "Monthly active users by region. Usage is concentrated in two regions that together account for about four fifths of activity, so a region-level average is misleading. The EMEA series begins partway through the history because collection started then, and those earlier months are absent rather than zero, so the month-over-month change is not meaningful for its first month."

Two tests that separate them, and both are cheap to apply to your own draft:

• Could you have written it without reading the data? If yes, you have documented the schema. Every clause in the second version is a claim that could have turned out false.

• Could a consumer decide NOT to use the table from what you wrote? A description that can only encourage use is a sales page. The limits are the most valuable part of it, and they are the part a schema cannot supply.

If the table exists to answer a question, the description states the answer, including when the answer is that the effect the table was built to measure is not present in the data, or that the data cannot settle it. A null result is a finding, and it is the one most often left unwritten, because it feels like having found nothing. It is not: a consumer who does not know it will read the table as evidence for the thing it disproves.

SageMaker Catalog

The merge is the whole hazard

CreateAssetRevision replaces the entire record. Anything you do not send is dropped, the call still succeeds, and the console still shows the field you asked for. AssetCommonDetailsForm is the sharp edge, because it carries summary, readMe and sourceIdentifier in one form, so writing a README by sending only that field unbinds the asset from the table it describes.

The API protects the technical form (GlueTableForm) and refuses a revision that omits it. It does not protect AssetCommonDetailsForm, which is the reverse of what you would expect. An omitted glossaryTerms clears every association.

Where each field lives:

The canonical read-modify-write

Run this as given, substituting the four variables. It preserves every form, patches one field, and prints a before/after comparison. Do not compose your own from memory.

It patches a form that is already on the asset. It cannot add one: if $FORM is not in formsOutput, as with ColumnBusinessMetadataForm on a freshly crawled asset, the rebuild changes nothing and the write still succeeds with the field never written. The guard after the snapshot turns that into a stop. To add a form, follow "Applying it" in references/metadata-forms.md, which sources the form's type revision rather than guessing it.

DOMAIN=<domain-id>; ASSET=<asset-id>
FORM=AssetCommonDetailsForm            # the form to patch
FIELD=readMe                           # the field inside it
VALUE='<the new value>'                # a string; for an object or array, see below

# 1. SNAPSHOT
aws datazone get-asset --domain-identifier "$DOMAIN" --identifier "$ASSET" \
  > /tmp/asset-before.json
jq -e --arg form "$FORM" 'any(.formsOutput[]; .formName == $form)' /tmp/asset-before.json >/dev/null \
  || { echo "Form $FORM is not on this asset -- add it (metadata-forms.md), do not write"; exit 1; }

# 2. Rebuild every form, patching only the target field. jq keeps the rest byte
#    for byte, and `readOnlyFormsOutput` is deliberately not read -- no form
#    input can write one. Omit --glossary-terms entirely when there are none:
#    the parameter has a minimum length of one and [] is refused client-side.
#    --arg inserts VALUE as a STRING. For an object or array, use --argjson.
jq --arg form "$FORM" --arg field "$FIELD" --arg value "$VALUE" '
  [ .formsOutput[]
    | . as $f
    | {formName, typeIdentifier: .typeName, typeRevision}
      + ( if $f.content then {content: $f.content} else {} end )
    | if .formName == $form
      then .content = ((.content // "{}") | fromjson | . + {($field): $value} | tojson)
      else . end ]
' /tmp/asset-before.json > /tmp/forms-input.json

TERMS=$(jq -r '[.glossaryTerms // [] | .[]] | join(" ")' /tmp/asset-before.json)

# 3. WRITE
if [ -n "$TERMS" ]; then
  aws datazone create-asset-revision --domain-identifier "$DOMAIN" \
    --identifier "$ASSET" --name "$(jq -r .name /tmp/asset-before.json)" \
    --forms-input file:///tmp/forms-input.json --glossary-terms $TERMS
else
  aws datazone create-asset-revision --domain-identifier "$DOMAIN" \
    --identifier "$ASSET" --name "$(jq -r .name /tmp/asset-before.json)" \
    --forms-input file:///tmp/forms-input.json
fi

# 4. VERIFY -- these three MUST all be non-empty and unchanged
aws datazone get-asset --domain-identifier "$DOMAIN" --identifier "$ASSET" \
  > /tmp/asset-after.json
echo "forms before/after:"; jq -r '[.formsOutput[].formName]|sort|join(",")' \
  /tmp/asset-before.json /tmp/asset-after.json
echo "sourceIdentifier:"; jq -r '.formsOutput[]|select(.formName=="AssetCommonDetailsForm")
  |(.content // "{}")|fromjson|.sourceIdentifier' /tmp/asset-after.json
echo "glossaryTerms:"; jq -c '.glossaryTerms' /tmp/asset-before.json /tmp/asset-after.json

If the form list shrank, or sourceIdentifier is now null, or the term list is shorter, restore from /tmp/asset-before.json by running the same create-asset-revision with the untouched forms, and tell the user what was lost. A dropped term is recoverable. A dropped AssetCommonDetailsForm is not recoverable from anything but your snapshot or a fresh data source run.

Column-level business metadata

Two lookup traps, both of which read as a missing capability:

• The form type is amazon.datazone.ColumnBusinessMetadataFormType; the form name on an asset is ColumnBusinessMetadataForm. Querying the type by the form name answers ResourceNotFoundException.

• ListIterableForms and BatchGetIterableForms are Glue operations. They do not exist on the datazone client, and their absence there is not evidence that column metadata is unavailable.

Written through the same merge above, with FORM=ColumnBusinessMetadataForm, FIELD=columnsBusinessMetadata and VALUE set to the array below. Bind it with --argjson value "$VALUE" in place of --arg value "$VALUE". --arg always inserts a string, so the array would be stored as a quoted JSON string that no consumer can parse. Read back the form after the write and check that columnsBusinessMetadata is an array: jq '.formsOutput[]|select(.formName=="ColumnBusinessMetadataForm") |(.content // "{}")|fromjson|.columnsBusinessMetadata|type' /tmp/asset-after.json prints "array".

Content shape:

{"columnsBusinessMetadata": [
  {"columnIdentifier": "<column name>", "name": "<business name>",
   "description": "<what the column means>"}]}

Ignore entries that carry a bare columnIdentifier with no name or description: a rejected AI prediction leaves stubs of exactly that shape, and counting them reads as a documented column.

Glue Data Catalog

Check availability first. Before using the Glue business-context operations in this section (glossaries, terms, custom form types, catalog assets and attachments), check their availability and any stated limitations in the AWS Glue documentation, and pass any caveat it gives on to the user. The table-layer operations (Description, column Comment and Parameters via update-table) are not part of that surface.

There are two descriptions and they diverge permanently

Initially the asset layer mirrors the table. After the first update-asset, that mirroring stops for good: a later update-table no longer propagates, and no API can clear the asset-layer value: update-asset refuses an empty description. So the two disagree permanently.

The trade to state to the user rather than decide silently: update-asset takes only an identifier, a name and a description, so it cannot drop anything, but query engines never see it. update-table is what a consumer sees, and it is the destructive path. When the request says "the table", prefer update-table with the recipe below and mention the other surface. When it says "the catalog asset", use update-asset.

The allow-list read-modify-write

glue get-table output cannot be fed back into update-table: six read-only fields are rejected client-side, and stripping them reveals a seventh. Build the input from an allow-list of the fourteen keys TableInput accepts, not a deny-list you discover one error at a time.

The API rewards the wrong choice: the minimal write succeeds silently, the correct one fails loudly. update-table with only a name and a description returns exit 0 and leaves the table with no columns, no Parameters, no Location and no TableType, which on an Iceberg table means no schema and no pointer to its metadata. Athena then reports Table StorageDescriptor is null.

DB=<database>; TBL=<table>; REGION=<region>
DESC='<the new table description>'

aws glue get-table --region "$REGION" --database-name "$DB" --name "$TBL" \
  > /tmp/table-before.json

jq --arg desc "$DESC" '
  .Table
  | {Name, Description, Owner, LastAccessTime, LastAnalyzedTime, Retention,
     StorageDescriptor, PartitionKeys, ViewOriginalText, ViewExpandedText,
     TableType, Parameters, TargetTable, ViewDefinition}
  | with_entries(select(.value != null))
  | .Description = $desc
' /tmp/table-before.json > /tmp/table-input.json

aws glue update-table --region "$REGION" --database-name "$DB" \
  --table-input file:///tmp/table-input.json

# VERIFY on get-table, NOT on the asset surface -- see below
aws glue get-table --region "$REGION" --database-name "$DB" --name "$TBL" \
  > /tmp/table-after.json
jq -r '.Table | "cols=\(.StorageDescriptor.Columns|length) params=\(.Parameters|keys|join(","))
  loc=\(.StorageDescriptor.Location) type=\(.TableType)"' \
  /tmp/table-before.json /tmp/table-after.json

To add column comments, patch StorageDescriptor.Columns[].Comment inside the same jq filter: there is no per-column write on the table layer, so they travel in the same full TableInput.

Verify on get-table and nowhere else. The column metadata layer is a separate store: with a table's StorageDescriptor destroyed and get-table returning null, glue list-iterable-forms still returns every column with its comment intact, ninety seconds later. That is the surface an agent naturally reaches for, and it cannot see the damage.

Recovery. If a definition is already destroyed, Glue's version history is the repair path: glue get-table-versions --database-name <db> --table-name <table>, pick the last good VersionId, and send its TableInput back through the allow-list above. This restores columns, Parameters, Location and TableType, and an Iceberg table becomes readable again. It depends on the version still being in history, so it is a repair, not a substitute for the snapshot.

Iceberg: the two write paths conflict, and order matters

Measured on a live Iceberg table. This is the trap that no amount of general reasoning gets right, because both halves look safe on their own.

The mechanism: update-table writes only the Glue catalog's mirror of the schema, while an Iceberg DDL commit re-derives that mirror from Iceberg's own schema. So the two do not compose, and a comment written one way is erased by the next write of the other kind.

So for an Iceberg table where the column comments must be visible to Athena and Spark:

1. Set every column comment through Athena DDL, one CHANGE COLUMN statement per column, in one batch. Repeat the column's exact current name and type and change only the COMMENT, because the statement redefines the column. A comment you do not restate is lost.

2. Then write Table.Description with the allow-list update-table above. Doing it in the other order loses the description to step 1.

3. Verify with both aws glue get-table and an Athena DESCRIBE, because they read different stores.

If the column comments only need to be visible in the Glue catalog and console, the allow-list update-table alone is enough and no DDL is needed. Say which audience you optimised for.

Columns on the Glue asset layer

amazon::BasicAttribute is reserved and put-attachment with it is refused. The asset layer's column Description mirrors the Glue column Comment read-only. So there is no safe asset-layer route to a column description: the paths above are the only ones. Custom forms can attach to a column, which is references/metadata-forms.md.