Databricks Asset Bundles

Databricks offers several ways to deploy resources like jobs, notebooks, clusters — each suited to different levels of automation, complexity and levels of control. Whether it’s managing infrastructure, orchestrating jobs, or promoting code across environments, choosing the right deployment tool is key to building scalable and maintainable workflows. From interactive development in the Databricks UI to fully automated CI/CD pipelines powered by Terraform, REST APIs, or other tools, the “best” deployment tool is one that aligns with your team’s workflow, scale, and operational maturity.

In this blog, we’ll explore the landscape of deployment tools available within the Databricks ecosystem and take a closer look at one in particular — Databricks Asset Bundles (DABs) — to examine its key benefits as well as its current limitations.

Deployment Tools in the Databricks Ecosystem

Below is an overview of the most widely used tools for deploying resources in the Databricks Ecosystem:

• Databricks UI — The web-based interface is ideal for manual or exploratory development. It allows users to create clusters, run notebooks, configure jobs, and manage workspace objects with minimal setup. While convenient for prototyping, the UI lacks version control and reproducibility, making it less suitable for production environments.
• Terraform —Terraform, in conjunction with the Databricks provider, is a widely adopted tool for managing infrastructure as code. It supports declarative provisioning of resources such as clusters, jobs, users, instance pools, and access controls. Terraform is particularly effective for standardizing configurations across workspaces and enforcing governance in enterprise environments. While powerful, Terraform is primarily used by infrastructure or DevOps teams due to its complexity and the depth of knowledge required to use it effectively. It has a steep learning curve, requiring familiarity with infrastructure design patterns, configuration workflows, and lifecycle management of cloud resources — areas that are often outside the day-to-day responsibilities of Data/ML engineers and analysts. As a result, these users may find Terraform challenging to adopt and use productively without close collaboration with DevOps or platform engineering teams.
• Databricks CLI (v1 & v2) — The Databricks Command Line Interface (CLI) provides a scripting-friendly way to interact with the platform. It is a powerful tool that allows for direct interaction with the platform to trigger jobs, copy over files (like notebooks or DBFS files), handle secrets etc, tasks that are not supported by Terraform. Terraform is designed to be declarative and stateful, focusing on provisioning and managing infrastructure, while Databricks CLI performs runtime operations and manages the content within those resources. For eg, Terraform can create the underlying cluster from cloud resources, but the CLI can also start, restart, edit and terminate a cluster and manage its operations.
• Databricks Asset Bundles (DABs)— DABs are the newest tooling specifically designed for the Databricks ecosystem and is officially developed by Databricks. It comes seamlessly integrated with the Databricks CLI v2 and uses Terraform under the hood. It truly is the best of both worlds abstracting away Terraform’s complexity while providing the powerful native resource management features of the CLI in an easy to use declarative YAML configuration to deploy and manage Databricks resources across different environments. It is also a version-controlled approach to managing everything from notebooks and jobs to clusters and libraries in the Databricks workspace. Unlike traditional infrastructure-as-code tools like Terraform, which manage underlying infrastructure, DABs are purpose-built for orchestrating code assets and job execution logic — making them ideal for teams focused on CI/CD pipelines and application-level management within the Databricks platform.
• DBX (Databricks Extensions) — is a deployment framework developed by Databricks Labs to simplify job launches and deployment process across multiple environments. It also helps to package the project and deliver it to the Databricks environment in a versioned fashion. Designed in a CLI-first manner, it is built to be actively used both inside CI/CD pipelines and as a part of local tooling for rapid prototyping. Unlike Terraform, dbx doesn’t provide infrastructure management capabilities, but brings more flexible deployment and launch options. It is now replaced by Databricks Asset bundles and is therefore no longer actively maintained.
• Databricks REST APIs and SDKs — The REST APIs offer fine-grained, programmatic access to nearly all aspects of the Databricks platform. Paired with SDKs in languages such as Python and Go, they are ideal for custom automation, integrations with other systems, or embedding Databricks operations into internal tools.

Databricks Asset Bundles(DABs)

Although many tools exist for deployment, the most recent one that’s both officially supported and growing in popularity and adoption is Databricks Asset Bundles. According to the official docs:

Databricks Asset Bundles are a tool to facilitate the adoption of software engineering best practices, including source control, code review, testing, and continuous integration and delivery (CI/CD), for your data and AI projects. Bundles provide a way to include metadata alongside your project’s source files and make it possible to describe Databricks resources such as jobs and pipelines as source files. Ultimately a bundle is an end-to-end definition of a project, including how the project should be structured, tested, and deployed. This makes it easier to collaborate on projects during active development.

Key Benefits of Databricks Asset Bundles

• Terraform under the hood: DABs use Terraform but abstracts away complexities like managing terraform versions, state files, providers and complex file structures.
• Declarative Configuration: templates are written in YAML which is straightforward to learn and use
• Version Control : Bundles are file-based and designed to work seamlessly with Git, allowing teams to track changes, perform rollbacks when needed and collaborate more effectively on complex projects with multiple collaborators.
• Easy migration and CLI support: For teams previously using tools like dbx, transitioning to DABs is straightforward. Even in environments where infrastructure-as-code tools were not used and jobs or workloads were created manually via the UI, the Databricks CLI provides commands to automatically generate DAB configuration files from existing resources. This makes it easy to adopt DABs incrementally and standardize deployment workflows across existing projects and workspaces.
• CI/CD Integration and best practices: DABs follow CI/CD best practices for managing the Databricks ecosystem and can be easily integrated with various CI/CD tools.
• Parameterization and environment management: Support for parameterized deployments allows the same bundle to be used in different environments (dev, staging, prod) with minimal changes.
• Templates: There are many pre-existing templates packaged into the CLI that can be used to customize and initialize DABs.

What’s still missing in DABs?

DAB is a tool that is both new and still evolving (and quite quickly too!). As of writing this article, several gaps remain in its features.

• No support for all resource types : DABs offer strong support for automating the deployment (and related artifacts) of jobs, notebooks, dashboards, and MLOps etc. However, many critical resource types — such as workspaces, user groups, service principals, and access controls — must still be pre-created, as they cannot currently be provisioned through DABs. Managing these foundational resources requires separate tooling, typically via Terraform, Databricks REST APIs, or the Databricks UI.
• Not suited to implement Multi tenant architectures: There’s no native support for custom or dynamic environments. Say for example you’re a PaaS or consulting company deploying a Databricks environment for each of your customers as part of your solution, it is not possible to dynamically create workspaces or client-specific environments. DABs assumes that these workspaces are pre-existing. This means teams have to manually define each environment in the configuration for each client and duplicate the bundles with hardcoded values, leading to maintenance headaches.
• Limited testing and validation support: The current validation capabilities in Databricks Asset Bundles are relatively basic. While the CLI offers a validate command and provides some error messaging, feedback is often limited to high-level or cryptic errors, making it difficult to diagnose issues in complex deployments. There is no native support for detailed dry runs, environment diffs, or structured error logs. In cases where deployments fail, users are often left interpreting terse CLI output or state file discrepancies (tfstate-style errors) without access to granular logs or actionable debugging information. This lack of visibility can significantly slow down development and troubleshooting in more advanced CI/CD workflows.
• No Native Support for Dependencies Beyond Wheels: Databricks Asset Bundles support packaging Python wheels, but do not currently offer a way to manage more complex dependency trees (e.g., conda environments, system packages, or private repos) within the bundle itself. This adds friction for teams with non-trivial runtime environments.
• No Support for Pre/Post Deployment Hooks: Unlike tools such as dbx, which allow custom pre- and post-deployment scripts, DABs do not currently support lifecycle hooks. This means there’s no built-in mechanism to:
  – Run setup logic (e.g., provisioning test data, setting feature flags) before a job is deployed.
  – Execute cleanup steps (e.g., archiving logs, notifying monitoring systems) after deployment completes.
  – Integrate with external systems like Slack, Jira, or monitoring tools without external scripting. With DABs, this logic must be implemented manually outside the bundle, usually in a CI/CD pipeline.
• Limited Parameterization and Logic: DABs use YAML for configuration, which limits dynamic logic (e.g., loops, conditions, or parameter resolution). This makes it difficult to:
  – Share reusable templates across teams or projects.
  – Generate jobs dynamically based on metadata or other input files.
  – Conditionally deploy jobs or resources based on environment-specific flags.
  Example: A team wants to deploy different job clusters depending on whether the environment is staging or prod. In dbx, this can be easily handled with Jinja templates or environment-aware Python logic. In DABs, this requires duplication or fragile YAML manipulation.

Are Asset Bundles an alternative to Terraform?

This is a common question — and the answer is yes and no.

✅ Yes, in the sense that both DABs and Terraform enable you to define Databricks resources as code. They allow for repeatable, version-controlled deployments and support automation in managing Databricks environments. There’s significant overlap in the types of project-level resources both tools can deploy — such as jobs, notebooks, and workflows. DABs also use terraform under the hood while abstracting away Terraform’s complexity.

🚫 No, in that Terraform goes much further. It is better suited for deploying workspace-level resources, managing access policies, and integrating with cloud infrastructure like VPCs, IAM roles, and storage in AWS, Azure, or GCP. Terraform offers deeper control over the full infrastructure stack, and supports advanced features like remote state management, modularization, and multi-cloud orchestration.

DABs do not currently replace Terraform for all use cases, but they are rapidly closing the gap for Databricks-native automation.

DABs are still evolving, still valuable

Databricks Asset Bundles are still in their early stages, but they’re rapidly maturing. As an actively developed feature, DABs are continuously gaining new capabilities — expanding resource coverage, improving extensibility, and strengthening integration with the broader Databricks platform. Frequent updates and enhancements reflect Databricks’ commitment to making DABs a reliable, full-featured deployment tool. While some gaps remain, the pace of development suggests that DABs are on track to become a robust and versatile solution for managing production workflows. Early adopters not only benefit from the current functionality but also have a chance to shape its evolution by providing feedback and contributing to best practices.

Why DABs Are a Smart Next Step for Most Teams

Despite some limitations around extensibility, resource coverage, and multi-tenant support, Databricks Asset Bundles (DABs) remain a powerful addition to the modern data and ML deployment toolkit. For many organizations, DABs strike the right balance between ease of use and operational control — especially for teams looking to move beyond manual UI-based deployments without adopting the full complexity of Terraform or custom tooling. It brings much-needed consistency and automation to Databricks workflows — especially for organizations looking to grow their DevOps maturity on Databricks without starting from scratch. With built-in support for environment separation, Git integration, and CLI-driven deployment, DABs lower the barrier to CI/CD adoption for teams of all sizes. As the ecosystem matures, DABs are well-positioned to become a central piece of scalable, version-controlled data platform operations.

Next Steps

• Check out the official Databricks Asset Bundle project templates with examples for mlops-stacks, python, sql, and dbt project templates.
• Explore how to migrate existing resources to a bundle

Enjoy this article? Want to discuss more with us? Please comment below, or reach out to us at NewMathData.com.