Risk Management in Cloud Data Projects: Strategies for Success
Introduction to Risk Management in Cloud Data Projects Risk management is a critical component of any cloud data project. As organizations increasingly rely on cloud technologies to store, process, and analyze data, understanding the unique risks associated with these projects becomes essential. Cloud data projects involve various stakeholders and technologies, which introduce complexities in data […]
Effective Project Planning for Cloud Data Engineering
Introduction to Project Planning in Cloud Data Engineering Project planning is a critical component of successful data engineering projects, especially when these projects are executed in the cloud. The unique characteristics of cloud computing, such as scalability, on-demand resources, and geographic distribution, offer both opportunities and challenges that must be carefully managed through effective planning. […]
Developing Future Leaders in Cloud Data Engineering
Exploring the Path to Leadership in the Evolving Field of Cloud Data Engineering Introduction The field of cloud data engineering is rapidly evolving, driven by the relentless pace of technological advancements and the increasing reliance on data-driven decision-making. As organizations continue to migrate their operations and data storage to cloud platforms, the demand for skilled […]
Navigating the Ethical Landscape: Generative AI in Data Engineering
Exploring the integration of generative AI in data engineering and its ethical implications. Introduction The integration of generative AI into data engineering marks a significant evolution in the way data ecosystems are managed and utilized. As these advanced technologies take on more complex tasks traditionally performed by humans, they bring forth not only enhanced efficiency […]
Unleashing Potential: High-Performing Cloud Data Engineering Teams
Introduction The ability to efficiently process, store, and analyze vast amounts of data in real-time is not just a competitive advantage but a necessity for survival and growth. This transformative potential, however, hinges on the capabilities of the teams at the helm, tasked with navigating the intricacies of cloud data systems to unlock valuable insights […]
Data Engineering Methodology: From requirements to hand-off
Introduction Joining or starting data projects in large enterprise environments with many stakeholders can be stressful, not to mention a technical implementation nightmare. When the primary stakeholders can’t (or won’t) give the project team clear requirements, the onus falls to the technical implementation team to create order from the chaos and organize the delivery team […]
The Data Journey
Many organizations share similar challenges with growing their operational capabilities with data. I have given several talks on data lake design and avoiding the “swampiness” of your data lake, invariably there are various pockets of mess or a “junk drawer” where people hide little bits of critical information. A complex data environment with myriad source […]
Running DBT on Databricks while using dbt_external_tables package to utilize Snowflake Tables
This article highlights a specific use case where one might need to run dbt on Databricks while utilizing tables in Snowflake. Typically, dbt runs on top of the database where it is instantiated. However, if a table needed to run dbt in Databricks does not exist in the hive-metastore and instead exists in an external […]