unity catalog

GuideUse Unity CatalogTables

Seed TPC-H data into Unity Catalog

Generate the eight-table TPC-H benchmark dataset with DuckDB, write it as Delta tables, and register them in Unity Catalog as samples.tpch.

TPC-H is the standard benchmark schema: eight related tables (region, nation, supplier, customer, part, partsupp, orders, and lineitem) modeling a wholesale supplier. Seeded into Unity Catalog, it gives joins, access grants, and cross-engine queries data that looks like a real business.

This guide generates TPC-H with DuckDB, writes each table as Delta, and registers it as an external table, three steps per table:

generate (DuckDB) → write Delta (deltalake) → register (Unity Catalog)

Unity Catalog only records where an external table lives and never reads the files itself, so the script writes the Delta tables to a local directory it owns (override with TPCH_STORAGE_ROOT).

  1. Generate TPC-H

    DuckDB ships a tpch extension that generates the benchmark data in-process. dbgen(sf = ...) builds every table at the given scale factor (0.01 here keeps it small and fast; bump it for more data), and each table comes straight back as an Arrow table.

    seed_tpch.pyFull script
    con = duckdb.connect()  # in-memory
con.execute("INSTALL tpch; LOAD tpch;")
con.execute(f"CALL dbgen(sf = {sf})")  # deterministic for a given scale factor
tables = {name: con.table(name).to_arrow_table() for name in TPCH_TABLES}

  2. Write each table as Delta

    Point write_deltalake at a location under the storage root and hand it the Arrow table. That writes a real Delta table — transaction log and all — and returns its file:// URI.

    seed_tpch.pyFull script
    location = root / name
write_deltalake(location, arrow, mode="overwrite")

  3. Register the tables in Unity Catalog

    Connect with the async SDK, then create the catalog samples with the schema tpch. For each Delta table written, register an external table, with the column list derived from the table's Arrow schema. The tables are then samples.tpch.lineitem, samples.tpch.orders, and so on.

    seed_tpch.pyFull script
    await catalogs.create_catalog(
    CreateCatalog(name="samples", comment="TPC-H dataset")
)
await schemas.create_schema(CreateSchema(name="tpch", catalog_name="samples"))

# Register each Delta table we wrote as an EXTERNAL table Unity Catalog
# governs — UC stores the schema and location, the data stays in Delta.
for name, arrow in arrow_tables.items():
    await tables.create_table(
        CreateTable(
            name=name,
            catalog_name="samples",
            schema_name="tpch",
            table_type=TableType.EXTERNAL,
            data_source_format=DataSourceFormat.DELTA,
            storage_location=locations[name],
            columns=arrow_to_columns(arrow.schema),
        )
    )
    print(f"registered samples.tpch.{name}")

  4. Verify the catalog

    List the schema's tables to confirm all eight registered, then read one straight back from Delta to prove the data is really there.

    seed_tpch.pyFull script
    listed = await tables.list_tables(catalog_name="samples", schema_name="tpch")
registered = sorted(t.name for t in listed.tables or [])
print(f"{len(registered)} tables in samples.tpch: {registered}")

# Read a table straight back from Delta to prove the data is really there.
lineitem_rows = DeltaTable(locations["lineitem"]).to_pyarrow_table().num_rows
print(f"lineitem has {lineitem_rows} rows")

Next steps

You now have an eight-table TPC-H dataset in Unity Catalog. To regenerate it at a larger scale, raise the sf argument to main (for example 0.1 for about 600,000 lineitem rows) and run the script again; it replaces the earlier tables.