Inspect robotics dataset snapshots from HFlow¶
HFlow processes robotics recordings into canonical MCAP episodes and a Parquet catalog of metadata, quality measurements, tags, intervals, and derived artifacts. Its dataset snapshot export packages a selected catalog view as standard Parquet tables and, optionally, copies the referenced media into the same directory.
Use Spotlight to inspect the snapshot through a pandas.DataFrame. This
workflow doesn't require an HFlow-specific Spotlight adapter.
Before you begin¶
Install HFlow, Spotlight, and the pandas Parquet reader on your local machine:
pip install hflow pandas pyarrow renumics-spotlight
Export a self-contained snapshot¶
To select episodes and export their media, complete the following steps:
-
Select the episodes that you want to inspect:
hflow curate \ "SELECT episode_id FROM episodes WHERE status = 'ok'" \ --output data/snapshot-manifest.parquet -
Create a self-contained snapshot that includes the selected media:
hflow export snapshot \ --manifest data/snapshot-manifest.parquet \ --output data/dataset-snapshot \ --media copy
The export creates the following files:
samples.parquet: One wide row per episode.measurements.parquet: Long-form quality evidence.media.parquet: An index of the exported artifacts.
Because the command uses copy mode, each media URI is a path relative to the snapshot directory.
Load image samples¶
To prepare the image samples for Spotlight, read the sample table, select rows whose representative artifact is an image, and resolve the copied media paths:
from pathlib import Path
import pandas as pd
snapshot_directory = Path("data/dataset-snapshot").resolve()
samples = pd.read_parquet(snapshot_directory / "samples.parquet")
image_samples = samples.loc[samples["media_kind"] == "image"].copy()
image_samples["media_uri"] = image_samples["media_uri"].map(
lambda media_uri: str(snapshot_directory / media_uri)
)
HFlow uses a contact sheet as the representative artifact when one is available. Otherwise, it selects an image before video, audio, and other artifact types. The remaining sample columns include episode metadata, status, version stamps, and numeric or boolean quality measurements. You can filter and compare these columns alongside the preview.
Open the snapshot in Spotlight¶
Declare the media column as an image and pass the pandas.DataFrame to
Spotlight:
from renumics import spotlight
spotlight.show(image_samples, dtype={"media_uri": spotlight.Image})
Optional: Load all media artifacts¶
If you need every camera or derived artifact, load
media.parquet, filter it by artifact_name, producer, role, or
media_kind, and join the selected rows to samples.parquet on episode_id.
You can also read the same snapshot with DuckDB, Polars, pandas, and other
tools.
For the complete table contract and reference-versus-copy behavior, see the HFlow dataset snapshot guide.