Governing Patient Data Across a Private Databricks Lakehouse and Palantir Foundry
A healthcare provider network needed governed, privacy-preserving analytics over longitudinal patient records, with data moving between Azure Databricks and Palantir Foundry without public egress.
The challenge
Data scientists and clinical researchers needed access to aggregated electronic health records (EHR) to train clinical decision models. Fragmented access permissions across legacy Hive metastores left protected health information (PHI) open to over-privileged querying.
The client's compliance team did not allow public IP addresses on Databricks clusters.
Regulatory exposure
PHI exposure carries regulatory and financial consequences under HIPAA. The platform work could not disrupt clinical operations or compromise patient privacy.
What we built
We deployed our private Azure Databricks with Foundry and Unity Catalog implementation package:
- VNet-injected workspace with no public IPs. Dedicated host and container subnets with secure cluster connectivity, so worker compute traffic stays inside the client's virtual network.
- Fine-grained governance in Unity Catalog. Dynamic row filters and column masks let researchers analyze trends without seeing raw Social Security numbers, names, or unmasked medical identifiers.
- Private ingestion into Palantir Foundry. A private link connection between the Azure Data Lake Storage Gen2 container and Palantir Foundry, so data movement does not cross the public internet.
- Databricks cluster governance. Cluster policies that enforce single-user access modes and terminate idle clusters automatically.
Where it landed
The client has a single governed data platform: compute runs without public IPs, access to PHI is controlled centrally in Unity Catalog, and data reaches Palantir Foundry over a private connection. The client's team owns the architecture code and runbooks in its own repository.