SummaryDatabases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

10 days ago 56m 23s

SummaryMaintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable this u

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

18 days ago 50m 44s

SummaryWorking with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

24 days ago 55m 39s

SummaryA core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the

Reconciling The Data In Your Databases With Datafold

Mar 17th, 2024 58m 14s

SummaryA significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and

Version Your Data Lakehouse Like Your Software With Nessie

Mar 10th, 2024 40m 55s

SummaryData lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the cat

When And How To Conduct An AI Program

Mar 3rd, 2024 46m 25s

SummaryArtificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment required. Colleen Tartow has wor

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Feb 25th, 2024 56m

SummaryBuilding a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are share

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Feb 18th, 2024 58m 46s

SummaryA data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to

Data Sharing Across Business And Platform Boundaries

Feb 11th, 2024 59m 55s

SummarySharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be made, particularly if the pro

Tackling Real Time Streaming Data With SQL Using RisingWave

Feb 4th, 2024 56m 55s

SummaryStream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage

Build A Data Lake For Your Security Logs With Scanner

Jan 29th, 2024 1h 2m

SummaryMonitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, o

Modern Customer Data Platform Principles

Jan 22nd, 2024 1h 1m

SummaryDatabases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. In this ep

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Jan 7th, 2024 50m 26s

SummaryData processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilit

Designing Data Platforms For Fintech Companies

Jan 1st, 2024 47m 56s

SummaryWorking with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of desig

Troubleshooting Kafka In Production

Dec 24th, 2023 1h 14m

SummaryKafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has experienced these challenges first-

Adding An Easy Mode For The Modern Data Stack With 5X

Dec 18th, 2023 56m 12s

SummaryThe "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering e

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Dec 11th, 2023 49m 51s

SummaryIf your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortu

Designing Data Transfer Systems That Scale

Dec 4th, 2023 1h 3m

SummaryThe first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes

Addressing The Challenges Of Component Integration In Data Platform Architectures

Nov 27th, 2023 29m 42s

SummaryBuilding a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the

Unlocking Your dbt Projects With Practical Advice For Practitioners

Nov 20th, 2023 1h 16m

SummaryThe dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your db

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Nov 13th, 2023 1h 7m

SummarySoftware development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is on

Shining Some Light In The Black Box Of PostgreSQL Performance

Nov 6th, 2023 54m 51s

SummaryDatabases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this episode Lukas Fittl shares some

Surveying The Market Of Database Products

Oct 30th, 2023 46m 24s

SummaryDatabases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episo

Defining A Strategy For Your Data Products

Oct 23rd, 2023 1h 3m

SummaryThe primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for inf