The brief sounded almost contradictory: let 40 independent research labs jointly analyze a shared genomic dataset, without any single lab losing control over its own data or being forced to hand raw data to a central authority. That constraint shaped nearly every technical decision that followed.
The constraint that shaped everything
Most data platforms are built around a simple assumption: there’s one owner, and everyone else is a consumer. A 40-lab consortium breaks that assumption immediately. Each lab had its own governance requirements, its own institutional review board constraints, and understandably, no appetite to simply export its raw variant and phenotype data into someone else’s system.
That ruled out the obvious approach — centralize everything into one warehouse — and pointed toward a federated architecture instead, where each lab’s data stays within its own access boundary, and queries move to the data rather than data moving to a central store.
What federated actually means in practice
- A harmonization layer that normalizes formats and identifiers across labs without requiring any lab to change its own internal systems.
- Row- and column-level access control scoped per institution, so a cross-cohort query only surfaces what each contributing lab has explicitly agreed to share.
- Full data lineage tracking, so any result can be traced back to which labs contributed which records, without exposing the underlying raw data itself.
For the first time, our 40 partner labs can query one dataset instead of comparing forty different spreadsheets.
What we’d do differently next time
The single biggest lesson: onboard the governance conversation before the technical one. Every delay in the project came from institutional data-sharing agreements taking longer than expected, never from the underlying architecture. On our next federated platform, we started drafting standard data-sharing agreement templates in parallel with the first lines of code, and it cut onboarding time roughly in half.
The technical pattern itself held up well: 40 labs onboarded within two academic terms, with cross-cohort queries that used to take weeks of manual spreadsheet reconciliation now running in seconds.