Query Unity Catalog tables from Python DataFrame libraries

Find, read, and append to external Delta tables in Unity Catalog from Polars, Daft, and pandas, with storage credentials vended by the server.

Polars, Daft, and pandas can find Unity Catalog tables by name and read and append to them. Each library looks up the table, asks the server for temporary storage credentials, and then reads or writes the Delta files directly in storage.

All three work with external Delta tables, through delta-rs. For catalog-managed tables, use Spark or DuckDB, which commit through the server.

The Polars and pandas examples use a table on S3, and the Daft examples a table on local storage, which is what each library is tested with here. The local environment above simulates S3. On real AWS, leave its AWS_ENDPOINT_URL and AWS_ALLOW_HTTP variables unset.

Connect to the server

uv run --with polars==1.44.2 --with deltalake==1.6.6 python

import polars as pl

catalog = pl.Catalog("http://localhost:8080", require_https=False)

uv run --with "daft[unity]==0.7.25" --with tenacity python

import daft
from daft.catalog import Catalog
from daft.catalog.__unity import UnityCatalogClient

client = UnityCatalogClient("http://localhost:8080", token="not-used")
catalog = Catalog.from_unity(client)

uv run --with pandas==2.3.3 --with deltalake==1.6.6 python

import pandas as pd
from deltalake import DeltaTable, write_deltalake

UNITY = {
    "unity_workspace_url": "http://localhost:8080",
    "unity_access_token": "not-used",
    "unity_allow_http_url": "true",
}

List tables

for table in catalog.list_tables("retail", "sales"):
    print(table.name, table.table_type, table.storage_location)

orders EXTERNAL s3://uc-docs/retail/orders

print(catalog.list_tables("retail.sales"))

[Identifier('retail.sales.orders')]

deltalake can't list tables. With pandas, use the Python client to find them.

Read a table

orders = catalog.scan_table("retail", "sales", "orders")
print(orders.filter(pl.col("amount") > 5).collect())

orders = catalog.get_table("retail.sales.orders").read()
orders.where(daft.col("amount") > 5).show()

orders = DeltaTable("uc://retail.sales.orders", storage_options=UNITY).to_pandas()
print(orders[orders["amount"] > 5])

Each library requests read credentials for the table's location from the server, and then reads the Delta files with them. For how vending works, see Credential vending.

Append rows

new_orders = pl.DataFrame(
    {"order_id": [4], "customer": ["margaret"], "amount": [12.0]}
)
catalog.write_table(new_orders, "retail", "sales", "orders", delta_mode="append")

new_orders = pd.DataFrame(
    {"order_id": [4], "customer": ["margaret"], "amount": [12.0]}
)
write_deltalake(
    "uc://retail.sales.orders", new_orders, mode="append", storage_options=UNITY
)

The library requests read-write credentials, writes new Parquet files, and commits a new version to the table's Delta log. The server isn't involved in the commit, so it doesn't coordinate writers on an external table.

Other table operations

Create and drop tables with the Python client or Spark.

Required privileges

With authorization enabled, reading needs USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT on the table. Appending also needs MODIFY on the table. Owners have all of these. See Core concepts.

Next steps

• External tables and catalog-managed Delta tables explains why path-based libraries can't use managed tables.
• The Polars, Daft, and delta-rs documentation cover other storage options and authentication methods.