Climate Tech Startup Migrates to AWS Batch to Run 2 to 5x More Simulations at Flat Spend
At a glance
A New York climate startup permanently removes carbon by injecting CO2 into volcanic rock. Its ML model learns injection schedules from simulation back-data, but one engineer could run only 1,000 simulations per cycle on ad hoc cloud infrastructure costing $8,000 to $10,000 monthly, propped up by an expiring $100,000 credit. That ceiling limited how much data the model could learn from.
New Math Data migrated the company’s pipeline to automated AWS Batch jobs on Spot Instances. The new pipeline runs 2 to 5x more simulations at the same spend, and every run becomes back-data the engineer uses to sharpen the model.
The company now generates data fast enough to match its Series A-funded growth and builds model confidence ahead of field decisions. A follow-on AI initiative with NMD is planned to extend that gain.
Industry
Use Case
Data Platform Migration, Machine Learning, Cost Optimization
Solution implemented
- Migrated virtual machine simulation workloads to AWS Batch using Amazon EC2 Spot and On-Demand Instances.
- Containerized the physics simulator with automatic checkpoint recovery for Spot Instance interruptions.
- Replaced 170 million-row data warehouse tables with Parquet on Amazon S3, the AWS Glue Data Catalog, and Amazon Athena.
- Built event-driven AWS Lambda pipelines for ETL and curve fitting, replacing manual notebook execution.
- Deployed the Streamlit ML application on Amazon ECS with AWS Fargate, with Amazon Cognito authentication and an Amazon Route 53 domain.
- Automated Amazon S3 Lifecycle policies to S3 Glacier Deep Archive, with all infrastructure managed in Terraform.
The value equation
- Multiplies simulation throughput 2 to 5x, generating far more back-data for the injection-schedule model.
- Ends a $100,000 cloud credit cliff with a cost-optimized, production-grade AWS platform.
- Frees the company's sole technologist from manual VM babysitting to focus on model validation and science.
- Replaces single-person infrastructure knowledge with documented, automated systems ready for Series A team growth.
- Positions the company to extend this productivity gain through a follow-on AI initiative with NMD.
Company Snapshot
A New York-based climate tech startup that permanently removes atmospheric CO2 by injecting it into volcanic rock, running AWS-based reservoir simulations to train the ML model that recommends injection strategies.
Location
United States
Customer Situation
A climate tech company’s ML surrogate model recommends CO2 injection schedules by learning from reservoir simulation back-data, and its confidence scales with how many simulations feed it. One engineer ran roughly 1,000 simulations per cycle on its previous cloud provider, manually duplicating VM images because the simulator can’t parallelize, at $8,000 to $10,000 a month. The $100,000 cloud credit absorbing that cost was expiring, capping volume right as the company’s Series A was set to fund a larger team.
The binding constraint was data throughput. The company could not generate more back-data without raising cloud spend or hand-managing more infrastructure. Climate tech startups running physics-heavy ML models on thin cloud engineering hit this ceiling, where the model’s need for data outpaces the infrastructure feeding it.
NMD Solution
NMD reviewed the client’s existing cloud environment and found that manual VM duplication was capping the simulation volume feeding the client’s ML surrogate model. The fix required containerizing the physics simulator for AWS Batch on Amazon EC2 Spot Instances with On-Demand fallback, replacing the previous data warehouse with Parquet on Amazon S3 queried through AWS Glue and Amazon Athena, and building event-driven AWS Lambda pipelines to automate the ETL and curve-fitting steps that generate the back-data the model learns from. The Streamlit optimization application now runs on Amazon ECS with AWS Fargate behind Amazon Cognito authentication, with Terraform managing the full infrastructure as code.
What We Delivered
The client’s reservoir simulations now run as fully automated AWS Batch jobs on Spot Instances with On-Demand fallback and automatic checkpoint recovery, generating 2 to 5x more back-data per cycle than the previous manual process. Outputs flow through an event-driven pipeline into Parquet on Amazon S3, queryable through Amazon Athena, feeding the model that recommends CO2 injection schedules. The optimization app runs on Amazon ECS with AWS Fargate behind its own secured domain. The client’s team completed UAT against a documented runbook, and now generates simulation data fast enough to keep pace with its Series A-funded growth.
Ready to Transform Your CleanTech Business?
See how New Math Data can transform your CleanTech business with AWS-powered innovation.