unity catalog

GuideUse Unity CatalogTables

Seed TPC-H data into Unity Catalog

Generate the eight-table TPC-H benchmark dataset with DuckDB, write it as Delta tables, and register them in Unity Catalog as samples.tpch.

TPC-H is the standard benchmark schema. Its eight related tables model a wholesale supplier: region, nation, supplier, customer, part, partsupp, orders, and lineitem. Seeded into Unity Catalog, it gives you realistic business data to try joins, access grants, and cross-engine queries on.

This guide generates TPC-H with DuckDB, writes each table as Delta, and registers it as an external table, three steps per table:

generate (DuckDB) → write Delta (deltalake) → register (Unity Catalog)

Unity Catalog only records where an external table lives. It never reads the files itself, so the script writes the Delta tables to a local directory it owns. Set TPCH_STORAGE_ROOT to use another directory.

  1. Generate TPC-H

    DuckDB ships a tpch extension that generates the benchmark data in-process. dbgen(sf = ...) builds every table at the given scale factor, and each table comes straight back as an Arrow table. A scale factor of 0.01 keeps it small and fast; raise it for more data.

    seed_tpch.pyFull script
    con = duckdb.connect()  # in-memory
con.execute("INSTALL tpch; LOAD tpch;")
con.execute(f"CALL dbgen(sf = {sf})")  # deterministic for a given scale factor
tables = {name: con.table(name).to_arrow_table() for name in TPCH_TABLES}

  2. Write each table as Delta

    Point write_deltalake at a location under the storage root and hand it the Arrow table. That writes a real Delta table — transaction log and all — and returns its file:// URI.

    seed_tpch.pyFull script
    location = root / name
write_deltalake(location, arrow, mode="overwrite")

  3. Register the tables in Unity Catalog

    Connect with the async SDK, then create the catalog samples with the schema tpch. For each Delta table written, register an external table, with the column list derived from the table's Arrow schema. The tables are then samples.tpch.lineitem, samples.tpch.orders, and so on.

    seed_tpch.pyFull script
    await catalogs.create_catalog(
    CreateCatalog(name="samples", comment="TPC-H dataset")
)
await schemas.create_schema(CreateSchema(name="tpch", catalog_name="samples"))

# Register each Delta table we wrote as an EXTERNAL table Unity Catalog
# governs — UC stores the schema and location, the data stays in Delta.
for name, arrow in arrow_tables.items():
    await tables.create_table(
        CreateTable(
            name=name,
            catalog_name="samples",
            schema_name="tpch",
            table_type=TableType.EXTERNAL,
            data_source_format=DataSourceFormat.DELTA,
            storage_location=locations[name],
            columns=arrow_to_columns(arrow.schema),
        )
    )
    print(f"registered samples.tpch.{name}")

  4. Verify the catalog

    List the schema's tables to confirm all eight registered, then read one straight back from Delta to prove the data is really there.

    seed_tpch.pyFull script
    listed = await tables.list_tables(catalog_name="samples", schema_name="tpch")
registered = sorted(t.name for t in listed.tables or [])
print(f"{len(registered)} tables in samples.tpch: {registered}")

# Read a table straight back from Delta to prove the data is really there.
lineitem_rows = DeltaTable(locations["lineitem"]).to_pyarrow_table().num_rows
print(f"lineitem has {lineitem_rows} rows")

Next steps

You now have an eight-table TPC-H dataset in Unity Catalog. To regenerate it at a larger scale, raise the sf argument to main and run the script again. For example, 0.1 gives about 600,000 lineitem rows. The new run replaces the earlier tables.