Computer Vision: Image Generator Networks
ChatGPT Generated Understanding of this Article A Look at the Core Concepts and How T2I has Evolved I am a Machine Learning Engineer by trade and a Computer Vision scientist by education, so any use case where computers are trying to learn the wonderful world outside (machine learning), primarily using visual information (computer vision) gives […]
Optimizing AWS Workloads: Insights from the Well-Architected Tool
TL;DR Building in the cloud? Make sure your workloads aren’t held together with duct tape and wishful thinking. The AWS Well-Architected Framework helps you design reliable, secure, cost-effective systems by following six key principles: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability. Use the AWS Well-Architected Tool to review workloads, tailor assessments with […]
Taking the Leap: How a Bias for Action Drives Innovation
Unleashing Creativity Through Decisive Action Introduction In today’s fast-paced and competitive business landscape, the ability to innovate swiftly and effectively is crucial for success. One key principle that separates thriving organizations and individuals from the rest is a “bias for action.” This mindset prioritizes taking decisive, proactive steps over endless deliberation and analysis. A bias […]
A Technical Comparison of Apache Parquet, ORC, and Arrow: Storage Formats for Big Data Workloads
In the world of Big Data, choosing the right storage format is critical for the performance, scalability, and the efficiency of analytics and processing tasks. Apache Parquet, Apache ORC, and Apache Arrow are three popular formats commonly used for data storage and processing within the ecosystem. While each of these formats serves a distinct purposes […]
Chatting with Your PDFs and VTTs at Lightning Speed
A Deep Dive into an AWS OpenSearch + Bedrock + Chainlit RAG Pipeline Introduction This article explains how to build a question-answering system that processes PDF and Video Text Track (VTT) files, index them in AWS OpenSearch, and serve fast answers to users. It uses AWS Bedrock (Anthropic Claude) for language processing and Chainlit for […]
S3 Custom Lifecycles: An AWS Glue solution to incorporate read operations into S3 Lifecycles
Introduction S3 Lifecycles provide a fantastic way to manage the lifetimes of S3 Objects. Working in AWS and in bigger organizations, you will inevitably have objects lingering (often in standard storage), costing you money. Lifecycle rules offer an easy way to trim some fat, which can both help you save money and exist in a […]
An In-Depth Overview of Electrical Generation, Transmission, Distribution, and Future Innovations
High level conceptual picture of Generation, Transmission and Distribution Systems Electricity powers modern civilization, enabling technological advancements and industrial growth. Its journey from generation to end-users involves intricate systems of infrastructure and management. This whitepaper explores the critical stages of this journey — generation, transmission, and distribution — while examining the transformative potential of Distributed […]
The Art of Data Engineering: Applying Sun Tzu’s Principles
How Ancient Wisdom Can Transform Modern Data Practices Sun Tzu’s “The Art of War” offers timeless wisdom that transcends the battlefield, providing insights applicable to various domains, including data engineering. In this field, success is not merely about coming up with innovative ideas or implementing initial solutions. True success is measured by how effectively these […]
Needle in a Haystack: A Rag Architecture and Use Case for Biotech
AI Acceleration with Pluton Bioscience If you’ve ever had to navigate a complex research process, you likely understand the challenges of sifting through vast amounts of information to find specific, valuable insights. Our partners at Pluton Biosciences have faced similar hurdles in their work. These research processes, while essential for gaining critical knowledge, can be […]
Core Concepts for Small Language Models
Foundational Computational Linguistics Concepts and how they relate to Language Models LLMs are all the rage, and to use them effectively as engineers and users of any kind we could benefit from thinking about them in the context of some core Natural Language Processing (NLP) concepts. At New Math Data, we’ve been building a lot […]