Data Engineering Methodology: From requirements to hand-off
Introduction Joining or starting data projects in large enterprise environments with many stakeholders can be stressful, not to mention a technical implementation nightmare. When the primary stakeholders can’t (or won’t) give the project team clear requirements, the onus falls to the technical implementation team to create order from the chaos and organize the delivery team […]
Remote Development in Sagemaker Studio with VS Code
Disclaimer about Changes to Sagemaker Studio As of Nov. 30 2023, there have been major changes to Sagemaker Studio. Existing customers of Sagemaker Studio will get the default experience now called Sagemaker Studio Classic — this is the Studio experience this article was written for. New Sagemaker Studio customers (and existing customers that choose to […]
The Data Journey
Many organizations share similar challenges with growing their operational capabilities with data. I have given several talks on data lake design and avoiding the “swampiness” of your data lake, invariably there are various pockets of mess or a “junk drawer” where people hide little bits of critical information. A complex data environment with myriad source […]
Boost AI Fairness and Explainability with Amazon SageMaker Clarify
From hiring decisions to loan approvals and even healthcare recommendations, machine learning (ML) impacts our lives daily. Fairness and explainability are crucial in this context. Fairness means data is balanced, and model predictions are fair across groups. Checking for fairness ensures that negative outcomes are fair across all groups, such as age or gender. Explainability […]
DBT and Databricks part 3: Loading noSQL data (from MongoDB) into Databricks
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the third part, we will talk about one specific example of how to load noSQL data into Databricks(originally coming from MongoDB). Task: We have […]
DBT and Databricks Part 2: Working with python models
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the second part, we will talk about working with python models. Starting from version 1.3 python support is added to DBT. As for now […]
DBT and Databricks Part 1: Setting up DBT profile for connecting to Azure Databricks using…
This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the first part, we will talk about how to set up a profile when using dbt-databricks python package. Install python package dbt-databricks using pip […]
Running DBT on Databricks while using dbt_external_tables package to utilize Snowflake Tables
This article highlights a specific use case where one might need to run dbt on Databricks while utilizing tables in Snowflake. Typically, dbt runs on top of the database where it is instantiated. However, if a table needed to run dbt in Databricks does not exist in the hive-metastore and instead exists in an external […]
External Knowledge Base for LLMs: Leveraging Retrieval Augmented Generation Framework with AWS…
Introduction In the realm of artificial intelligence and language models, the pursuit of enhancing their capabilities is a constant endeavor. Today, we embark on a journey to explore an innovative approach that has the potential to revolutionize the way we harness the power of AI. Picture a scenario where your language model not only excels […]
Use AWS Bedrock language models with a Slack-powered chatbot
co-authors: Meghana Venkataswamy, Sean Cahill, Salman Ahmed Mian Architecture What is a Foundational Model? How Do we customize the FM to our Data and needs? Our /hr-bot uses RAG Technique FAISS (Facebook AI Similarity Search) What is AWS Kendra LangChain AWS Bedrock Slack integration Terraform Challenges and opportunities Navigating policy and rules in a large […]