Fred Hutch Cancer Center

Genomic datasets are large enough that the storage format decides what analysis is possible. I worked toward restructuring hundreds of millions of data points out of .loom and into a parquet-style columnar format for Fred Hutch’s data-visualization lead.

Why the format change

.loom is a sensible container for single-cell genomics: it holds a gigantic matrix with row and column annotations in one HDF5 file. However, it doesn’t work as well when the access pattern is something along the lines of “give me these three features across every cell, right now, while someone is dragging a slider.”

Columnar storage flips that: reading three columns means touching three columns’ worth of bytes instead of streaming the whole matrix for every query. At this scale, that gap decides whether a visualization responds instantly or leaves you waiting for minutes or even hours, depending on the size of the matrix.

What I actually got from it

This was a short, unofficial internship: three or four weeks, as a student rather than a developer. I was there to learn from the lab’s data-visualization lead how programs like this get built, and how to write software where efficiency truly matters against large-form datasets.

My biggest learning was that you can’t just load hundreds of millions of points into memory and convert them, because no machine can remember that much data at once. You must read the data in chunks, keep the annotations that give each value its meaning, and pick a layout that matches how the data will actually be queried. Get those three right, and a dataset too big to open starts to feel small.