US Energy Retailer Transforms Its Power Valuation Engine from Dremio to a Verified Enterprise Data Product with Databricks on AWS
At a glance
A US energy retailer and trader values its wholesale and retail power positions across four US energy markets in a valuation engine its Risk team relies on. The engine ran on Dremio, outside the Databricks platform the business had standardized on, with no automated tests guarding its logic. As the business moved under its parent’s enterprise IT, the valuations Risk used rested on code no one could verify automatically.
New Math Data migrated the engine to Databricks on AWS in eight weeks, with every in-scope model passing the client’s acceptance testing, then built the test suite, permissions framework, and data sharing the platform lacked.
The company now runs its core valuation engine as a governed enterprise data product: tested on every run, shared directly with Risk, and architected to serve usage data to customers through an API.
Industry
Use Case
Data Platform Migration, Data Engineering, Modern Data Architecture, Compliance & Governance
Solution implemented
- Migrated the dbt project from Dremio SQL to Databricks SQL on AWS with Unity Catalog references.
- Integrated dbt on Databricks into existing Apache Airflow orchestration across development, pre-production, and production.
- Ported Apache Iceberg table schemas to Databricks and validated outputs against Dremio through client UAT.
- Built a dbt test suite covering schema, referential integrity, business logic, freshness, and volume anomalies.
- Configured Delta Sharing from Databricks on AWS to the enterprise data platform with freshness monitoring.
- Implemented permissions-as-code with CI/CD drift detection and tuned high-cost models for runtime and cost.
The value equation
- Risk team valuations run on the enterprise-standard platform, with every in-scope model validated.
- Automated tests check valuation logic on every scheduled run, replacing zero prior coverage.
- Risk team reads reporting models directly from the enterprise data platform through governed sharing.
- Access to valuation data follows a defined, auditable permissions model managed as code.
- Usage data product has a target architecture for API delivery to external customers.
Company Snapshot
A US retail and wholesale electricity provider serving commercial customers across four ISO markets, valuing its energy books for risk and reporting.
Location
United States
Customer Situation
The company’s valuation engine is its primary valuation and reporting platform, calculating exposures across its wholesale and retail books in four US grid markets. The Risk team works from those outputs.
The engine ran on Dremio, outside the Databricks platform the business had chosen as its unified environment, and its dbt models carried no automated tests. Access to its outputs followed no defined permissions model. At the same time, the business was moving under its parent’s central IT, with stricter delivery commitments and a changing team.
Energy retailers absorbed into larger parents face the same problem: valuation logic built on standalone tools must move onto enterprise platforms without breaking the numbers risk depends on.
NMD Solution
NMD reviewed the engine and found that its valuation logic was tied to Dremio SQL, guarded by no automated tests, and unreachable from the parent’s enterprise data platform. The solution required porting the dbt project to Databricks SQL on AWS with Unity Catalog, running it inside the existing Apache Airflow orchestration, and validating every in-scope model against Dremio outputs through client UAT. NMD then added a dbt test suite, Delta Sharing to the enterprise platform, permissions managed as infrastructure-as-code, and performance tuning of the costliest models.
What We Delivered
The valuation engine runs on Databricks on AWS, migrated in eight weeks with every in-scope model accepted through client UAT. A dbt test suite checks valuation logic on each scheduled run and alerts on failure. The Risk team reads reporting models from the parent’s enterprise data platform through Delta Sharing, under a permissions framework managed as code. The costliest models are tuned for runtime and cost, and the company holds target architectures for consolidating operational systems and serving its usage data product to customers over API.
Ready to Transform Your Energy & Utilities Business?
See how New Math Data can transform your energy & utilities business with AWS-powered innovation.