Boost AI Fairness and Explainability with Amazon SageMaker Clarify

From hiring decisions to loan approvals and even healthcare recommendations, machine learning (ML) impacts our lives daily. Fairness and explainability are crucial in this context. Fairness means data is balanced, and model predictions are fair across groups. Checking for fairness ensures that negative outcomes are fair across all groups, such as age or gender. Explainability […]

DBT and Databricks part 3: Loading noSQL data (from MongoDB) into Databricks

This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the third part, we will talk about one specific example of how to load noSQL data into Databricks(originally coming from MongoDB). Task: We have […]

DBT and Databricks Part 2: Working with python models

This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the second part, we will talk about working with python models. Starting from version 1.3 python support is added to DBT. As for now […]

DBT and Databricks Part 1: Setting up DBT profile for connecting to Azure Databricks using…

This series of blog posts will illustrate how to use DBT with Azure Databricks: set up a connection profile, work with python models, and copy noSQL data into Databricks(from MongoDB). In the first part, we will talk about how to set up a profile when using dbt-databricks python package. Install python package dbt-databricks using pip […]

Running DBT on Databricks while using dbt_external_tables package to utilize Snowflake Tables

This article highlights a specific use case where one might need to run dbt on Databricks while utilizing tables in Snowflake. Typically, dbt runs on top of the database where it is instantiated. However, if a table needed to run dbt in Databricks does not exist in the hive-metastore and instead exists in an external […]

External Knowledge Base for LLMs: Leveraging Retrieval Augmented Generation Framework with AWS…

Introduction In the realm of artificial intelligence and language models, the pursuit of enhancing their capabilities is a constant endeavor. Today, we embark on a journey to explore an innovative approach that has the potential to revolutionize the way we harness the power of AI. Picture a scenario where your language model not only excels […]

Use AWS Bedrock language models with a Slack-powered chatbot

co-authors: Meghana Venkataswamy, Sean Cahill, Salman Ahmed Mian Architecture What is a Foundational Model? How Do we customize the FM to our Data and needs? Our /hr-bot uses RAG Technique FAISS (Facebook AI Similarity Search) What is AWS Kendra LangChain AWS Bedrock Slack integration Terraform Challenges and opportunities Navigating policy and rules in a large […]

Harnessing the Power of Large Language Models for Knowledge Graph Creation

· The role of large language models in creating knowledge graphs from unstructured data.· Comparison of Top Models· Storage Platforms for Knowledge Graphs· Practical Example: Knowledge Graph from Wikipedia Text· Best Practices and Tips for Using Large Language Models in Knowledge Graph Creation· Conclusion In the vast realm of data, the ability to meaningfully connect […]

Migrating Apache Cassandra to AWS Keyspaces

Apache Cassandra is an open-source, NoSQL database with a distributed architecture to maximize availability and reliability. Due to the relative ease with which clusters may be scaled, Cassandra is a popular choice for use cases involving heavy writes with simple, pre-defined querying needs. Despite this ease of use, the effort involved in managing a Cassandra […]

Creating Custom Hadoop Events in EventBridge on AWS

Introduction: Event driven architecture as defined by Amazon is a system whose architecture uses events to trigger and communicate between decoupled services and is common in modern applications built with microservices. An event is a change in state or an update. Event-driven architectures have three key components: event producers, event routers, and event consumers. A […]