Fred Hutch Cancer Center
Restructured hundreds of millions of genomic data points from .loom into a parquet-style columnar format so the data-visualization lead could work with them interactively.
- Data Engineering
- Genomics
- Parquet
- Python
Genomic datasets are large enough that the storage format decides what analysis is possible. I
restructured hundreds of millions of data points out of .loom and into a parquet-style
columnar format for Fred Hutch’s data-visualization lead.
Why the format change
.loom is a sensible container for single-cell genomics — it holds a big matrix plus its row and
column annotations in one HDF5 file. It’s less good when the access pattern is “give me these three
features across every cell, right now, while someone is dragging a slider.”
Columnar storage inverts that. Reading three columns means touching three columns’ worth of bytes rather than streaming the matrix. At this scale that’s the difference between a visualization that responds and one where you start a query and go get coffee.
The actual work
Converting hundreds of millions of points isn’t a one-liner, because you can’t hold the dataset in memory to do it. The job is chunking the read, preserving the annotations that make the values meaningful, choosing a schema and partitioning that matches how the data will be queried, and verifying that nothing was silently dropped or mistyped along the way — on data where a wrong answer looks exactly like a right one.
The takeaway
This was the project that taught me data engineering is a design discipline. Nobody sees the conversion. They see whether the visualization is fast, and that was decided before a single chart was drawn.