How to think about cost savings for machine learning models
Introduction
The rise of organizations attempting to deploy and train custom Machine Learning models for specific purposes (such as demand forecasting, fraud detection, and more) has led to an increase in dependency on third-party cloud providers to host these models for training and inference purposes due to the scalability, availability and storage they offer.
The complexity and commitment required to purchase and maintain hardware to train a model locally is not feasible for small to mid-scale organizations. In such cases, cloud-based solutions are preferred due to their offerings like Amazon SageMaker for managed ML workflows, and also due to the ability to build managed custom stacks using EC2, Lambda, and more based on specific business requirements.
However, this flexibility comes with the requirement to monitor the costs associated with these services. All AWS services are pay-as-you-go and can become expensive if not monitored. Cost management becomes particularly crucial when deploying persistent inference endpoints or high-compute training jobs. Training ML models with large datasets can also be expensive due to prolonged compute usage and storage needs. This balancing act of cost vs flexibility requires planning and effective automation.
Costs associated with ML Models
Generally, the costs associated with hosting a Machine Learning model can be traced back to two main categories:
Training Costs
- Training Costs are costs associated with “teaching” your ML model.
- The main cost associated with training is that of the compute-heavy resources like GPUs, RAM, and storage.
- If the training job for a user’s ML model takes 4 hours to complete on a powerful instance, the user will be responsible for provisioning all 4 hours, especially costlier in a pay-as-you-go approach.
- This training cost can multiply if re-training is required.
Inference Costs
- A trained model can be deployed to be treated as an endpoint to respond to predictions which is known as inference.
- The inference cost is associated with the provisioning of the server or endpoint irrespective of how frequently it is utilized.
Smart Data Management
Apart from the algorithm, a Machine Learning model is heavily dependent on the quality and quantity of the data it’s trained on. While simpler models like linear regression can be trained with relatively small datasets that don’t strain storage budgets, more complex models like neural networks require much more data to generalize effectively. Amazon S3 is a popular choice of storage for training data and artifacts for cloud-based ML workflows, but the cost of storage for large volumes of data can escalate quickly without proper data management.
Use S3 Intelligent Tiering
AWS offers an intelligent-tiering feature which moves data between different storage tiers according to the access patterns. The storage tiers in AWS S3 balance cost vs latency. This allows for infrequently accessed data for which low retrieval time is not a priority to be stored in storage tiers which cost less like S3-InfrequentAccess or Glacier. Since training data may not be frequently accessed after initial model development, it can often be moved to lower-cost tiers, thus allowing a user to retain the training data while maintaining low costs.
Pre-Processing & Cleanup
Additionally, AWS services like Glue and Lambda can be used for pre-processing raw data such as cleaning, transforming, and formatting, before its use for model training. This can be done on-premise in cases where intermediate data is not used by the model for training or validation purposes. Furthermore, only the processed data can be stored to reduce storage costs. This step also allows for easier management of data as raw, unversioned, duplicate data can accumulate as data bloat thereby leading to an increase in cost.
Comparing ML Inference Deployment Options
The use of the “right” AWS Service heavily depends on the usage of the model. AWS provides multiple options for inference including EC2, Lambda and SageMaker, each with its own trade-offs in terms of scalability, control, cost and complexity.
Amazon EC2
Amazon EC2 offers full control over the deployment environment making it suitable for machine learning models that require custom OS or network configurations. However, if you are not utilizing the compute resources continuously, you will end up paying for the infrastructure whether it’s idle or active. If underutilized, it makes EC2 one of the costliest options in ML deployment. On the other hand, if the model is handling constant, high-volume inference, EC2 can be the most cost-effective option justifying a continuously running instance.
AWS Lambda
AWS Lambda, in contrast, is extremely cost-efficient for low traffic and occasional inference workloads. As pricing is based on the execution time and memory usage, you pay only when the function is invoked.
If your model:
- Is under 500MB (uncompressed)
- Doesn’t need GPU
- Handles occasional requests
… Lambda could be the ideal choice.
It is the recommended option in case of event driven applications with unpredictable traffic. While it has low devops overhead, it is not the best when it comes to scalability. It can get expensive at high scale and is not suitable for large models or long inference times due to memory and timeout limits.
AWS SageMaker
Amazon SageMaker offers a balance between scalability and cost. Real-time endpoints in SageMaker are priced similar to EC2 such that you pay for uptime, but they include built-in scaling and monitoring tailored for ML.
SageMaker also supports several cost-saving modes:
- Serverless Inference — pay only during model invocation
- Batch Transform / Asynchronous Inference — ideal for non-real-time, large-scale jobs
- Multi-model endpoints — deploy multiple models to a single instance
In addition to this, SageMaker offers AWS SageMaker Neo for faster inference. Neo optimizes the model based on the hardware including OS and processors, memory access and data patterns. Edge deployment with SageMaker and Neo allows you to run inference on mobile, IoT and embedded systems, reducing data transfer to the cloud and saving bandwidth costs. It also results in lower latency without expensive cloud infra.
If application moves to the edge, high inference charges are avoided from services like SageMaker endpoints or Lambda. If running on the cloud, Neo optimizes the model allowing smaller, cheaper EC2 instances thus making it feasible to switch from expensive GPU instances to the CPU ones. This results in lower compute costs and better resource utilization for high-throughput inference tasks.
Monitoring Costs
Monitoring and managing the costs is equally important as picking the right model. AWS tools like CloudWatch and Cost Explorer help track these costs and analyze them.
Use AWS Native Tools
For instance, AWS Cost Explorer can be used to visualize storage costs. It also helps compare and break down SageMaker, Lambda and EC2 costs for better understanding of the costs associated with the model. Combining this with CloudWatch enables you to have a comparative view of resource utilization metrics like CPU, memory and execution rates, such that unutilized or idle endpoints or instances can be downscaled or shut down. To avoid billing surprises, you can set up AWS Budgets alerts that notify you when the billing amount exceeds the custom threshold.
Automate Shutdowns & Cleanups
Another alternative to manage costs proactively is connecting AWS Lambda to EventBridge, to trigger workflows that automatically shutdown idle endpoints or instances or clean up outdated resources or models. This ensures you pay what you use. Additionally, regularly archiving old logs, datasets and training artifacts to data tiers like S3 Glacier helps reduce storage costs and improve traceability.
Thus, in this article we discuss how deploying ML models in the cloud offers great flexibility, but costs can add up fast. With the right service choices, smart data management, and proactive monitoring, you can keep expenses in check without sacrificing performance.
Find this article useful or interesting? Please comment below or reach out to us at NewMathData.com .