Experimenting with the Databricks Mosaic AI Vector Search using python: a beginner example
Introduction After exploring how vector databases work and trying the AWS OpenSearch vector database in this blog post, the goal of this article is to experiment with a similar product from Databricks – Mosaic AI Vector Search. We will create a new Vector Search endpoint and index, load the toy dataset, try to perform similarity […]
September Databricks Updates
Clean Room Monitoring, VS Code Extension, and System Tables GA Introduction As organizations continue to scale their data and AI operations, effective monitoring, streamlined development, and transparent billing have become crucial for maintaining efficiency. This September, Databricks has introduced several powerful updates designed to address these needs. From the ability to track clean room usage […]
Databricks Notebooks Reimagined: Simplicity Meets Power
Streamlined UX, Powerful Tools, and AI Integration Introduction Having powerful and intuitive tools is crucial for success. Databricks has recently unveiled the next generation of its Notebooks, bringing a host of new features designed to enhance productivity and ease of use. This update includes a modernized user interface, advanced Python capabilities, and AI-powered authoring tools, […]
Databricks First Look: August 2024 Release, Deep Dive into Lakehouse Federation
Exploring the Power of Seamless Data Integration and Enhanced Security with Databricks Introduction In the fast-evolving landscape of data analytics, staying updated with the latest platform enhancements is crucial. The August 2024 release from Databricks brings a suite of impactful updates designed to boost security, compliance, and performance. Among these, Lakehouse Federation stands out, offering […]
Databricks AI Playground
Your Quick Start Guide Introduction Databricks provides an AI Playground that is a chat-like environment for testing and comparing models. It supports foundation models like Databricks DBRX Instruct or Meta Llama 3, on a pay-per-token-basis. Also, it supports models you’ve created (via the Model Registry with a Serving Endpoint). The AI Playground reduces time spent […]
Harnessing the Power of PySpark in DataBricks Delta Live Tables
Explore the integration of PySpark with DataBricks Delta Live Tables Introduction Welcome to our exploration of how PySpark integrates with Databricks Delta Live Tables, a powerful combination for managing big data workflows with enhanced reliability and integrity. This post is designed for data engineers and scientists who are looking to leverage the capabilities of Jupyter […]
DBT and Databricks part 3: Loading noSQL data (from MongoDB) into Databricks
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the third part, we will talk about one specific example of how to load noSQL data into Databricks(originally coming from MongoDB). Task: We have […]
DBT and Databricks Part 2: Working with python models
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the second part, we will talk about working with python models. Starting from version 1.3 python support is added to DBT. As for now […]
DBT and Databricks Part 1: Setting up DBT profile for connecting to Azure Databricks using…
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the first part, we will talk about how to set up a profile when using dbt-databricks python package. Install python package dbt-databricks using pip […]
Running DBT on Databricks while using dbt_external_tables package to utilize Snowflake Tables
This article highlights a specific use case where one might need to run dbt on Databricks while utilizing tables in Snowflake. Typically, dbt runs on top of the database where it is instantiated. However, if a table needed to run dbt in Databricks does not exist in the hive-metastore and instead exists in an external […]
