Introduction After exploring how vector databases work and trying the AWS OpenSearch vector database in this blog post, the goal of this article is to experiment with a similar product from Databricks – Mosaic AI Vector Search. We will create a new Vector Search endpoint and index, load the toy dataset, try to perform similarity […]
Clean Room Monitoring, VS Code Extension, and System Tables GA Introduction As organizations continue to scale their data and AI operations, effective monitoring, streamlined development, and transparent billing have become crucial for maintaining efficiency. This September, Databricks has introduced several powerful updates designed to address these needs. From the ability to track clean room usage […]
Streamlined UX, Powerful Tools, and AI Integration Introduction Having powerful and intuitive tools is crucial for success. Databricks has recently unveiled the next generation of its Notebooks, bringing a host of new features designed to enhance productivity and ease of use. This update includes a modernized user interface, advanced Python capabilities, and AI-powered authoring tools, […]
Exploring the Power of Seamless Data Integration and Enhanced Security with Databricks Introduction In the fast-evolving landscape of data analytics, staying updated with the latest platform enhancements is crucial. The August 2024 release from Databricks brings a suite of impactful updates designed to boost security, compliance, and performance. Among these, Lakehouse Federation stands out, offering […]
Your Quick Start Guide Introduction Databricks provides an AI Playground that is a chat-like environment for testing and comparing models. It supports foundation models like Databricks DBRX Instruct or Meta Llama 3, on a pay-per-token-basis. Also, it supports models you’ve created (via the Model Registry with a Serving Endpoint). The AI Playground reduces time spent […]
Explore the integration of PySpark with DataBricks Delta Live Tables Introduction Welcome to our exploration of how PySpark integrates with Databricks Delta Live Tables, a powerful combination for managing big data workflows with enhanced reliability and integrity. This post is designed for data engineers and scientists who are looking to leverage the capabilities of Jupyter […]
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the third part, we will talk about one specific example of how to load noSQL data into Databricks(originally coming from MongoDB). Task: We have […]
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the second part, we will talk about working with python models. Starting from version 1.3 python support is added to DBT. As for now […]
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the first part, we will talk about how to set up a profile when using dbt-databricks python package. Install python package dbt-databricks using pip […]
This article highlights a specific use case where one might need to run dbt on Databricks while utilizing tables in Snowflake. Typically, dbt runs on top of the database where it is instantiated. However, if a table needed to run dbt in Databricks does not exist in the hive-metastore and instead exists in an external […]