Introduction In the realm of distributed computing and batch processing, operational challenges frequently arise that necessitate innovative solutions. A particular challenge we encountered involved a scenario where multiple jobs within our AWS environment were generating tens of thousands of files and storing them in an Amazon S3 bucket. Subsequently, a specific job was tasked with […]
Introduction In the digital era, where data is king, the choice of table format for data storage and processing is crucial. Common file formats like CSV, Avro, and Parquet have long been the go-to solutions in various data handling scenarios. However, with the evolving needs of big data and cloud computing, newer and more efficient […]
From hiring decisions to loan approvals and even healthcare recommendations, machine learning (ML) impacts our lives daily. Fairness and explainability are crucial in this context. Fairness means data is balanced, and model predictions are fair across groups. Checking for fairness ensures that negative outcomes are fair across all groups, such as age or gender. Explainability […]
Introduction: Event driven architecture as defined by Amazon is a system whose architecture uses events to trigger and communicate between decoupled services and is common in modern applications built with microservices. An event is a change in state or an update. Event-driven architectures have three key components: event producers, event routers, and event consumers. A […]
- 1
- 2