Context
An enterprise data platform on Databricks, used by multiple data engineering and analytics teams with different needs for access, compute and deployment.
Challenge
Improve governance and create consistent deployment and access patterns for Databricks users and workloads — strengthening control without slowing data teams down.
Approach
We started from the people using the platform and worked down to the controls that serve them.
- 01PersonasWho uses the platform, and for what
- 02AccessACLs and Unity Catalog controls
- 03ComputeCluster policies and instance pools
- 04ServingSQL warehouses for analytics
- 05DeploymentDatabricks Asset Bundles
- Defined personas and the access each one genuinely needs.
- Applied ACLs and Unity Catalog controls to govern data access consistently.
- Standardised compute through cluster policies, SQL warehouses and instance pools.
- Established deployment practices using Databricks Asset Bundles.
Delivery
Standards were delivered as reusable definitions and documented patterns rather than one-off configuration, so new teams and workloads could adopt them from day one.
Outcome
Improved platform consistency, governance and repeatability for data teams.
Consistent access
One model, applied the same way across teams.
Predictable compute
Policies and pools instead of ad-hoc clusters.
Repeatable deployment
A standard path from development to production.
Technologies
- Databricks
- Unity Catalog
- Databricks Asset Bundles
- SQL Warehouses
- Cluster policies
- Instance pools