Databricks
Interview Prep

Traditional data lakes store raw files cheaply but lack ACID transactions, schema enforcement, and query performance. Traditional data warehouses have great query performance and structure but are expensive and lack support for ML workloads. The Lakehouse (coined by Databricks) combines both: ACID transactions and schema enforcement on open file formats (Delta Lake on Parquet) stored in cheap object storage, enabling both BI and ML workloads on the same data with performance comparable to data warehouses.

Delta Lake adds: ACID transactions (serializable isolation via optimistic concurrency with a transaction log), schema enforcement (rejects writes that don't match the schema), schema evolution (adding columns without rewriting history), time travel (query data as of any historical version using the transaction log), and performance optimizations (Z-ordering for multi-dimensional clustering, data skipping via file-level statistics, compaction for small files). Each of these addresses a specific pain point of plain Parquet on object storage.

Walk through: the job's purpose (ETL, aggregation, ML feature engineering), data volume and cluster size, the performance bottleneck you identified (shuffle operations, data skew, small files problem, unnecessary data scans), the optimization you applied (repartitioning, broadcast joins for small tables, predicate pushdown, caching hot datasets, AQE — Adaptive Query Execution in Spark 3.x), and the before/after improvement in runtime and cost.

Walk through: feature registry (catalog of feature definitions with metadata, lineage, owners), feature computation (offline pipeline for historical features using Spark, online pipeline for real-time features using streaming), storage (offline store in Delta Lake, online store in Redis or DynamoDB for low-latency serving), point-in-time correct joins for training data (avoiding future leakage), feature sharing and discovery across teams, and monitoring for feature drift.

Walk through: experimentation tracking (MLflow experiment tracking — logging parameters, metrics, artifacts), model validation (offline metrics, business metric estimation, A/B test setup), packaging (MLflow Models for reproducibility), deployment (MLflow Model Registry for versioning and lifecycle management, Databricks Model Serving for REST endpoint), monitoring (feature drift, prediction drift, business metric monitoring), and retraining trigger design.

Walk through: 1) Logical plan construction (DataFrame transformations build a DAG), 2) Catalyst optimizer (logical plan → optimized logical plan via rule-based and cost-based optimization, predicate pushdown, constant folding), 3) Physical plan generation (join strategy selection — sort-merge vs. broadcast, partitioning decisions), 4) Code generation (Tungsten's whole-stage codegen for CPU efficiency), 5) DAG scheduling (stages defined by shuffle boundaries, tasks executed in parallel on partitions), 6) Shuffle (sort-based shuffle with in-memory sort buffers).

Walk through: the streaming source (Kafka, Kinesis, or Auto Loader for cloud files), the processing framework (Spark Structured Streaming), the windowing strategy (tumbling vs. sliding windows, session windows for user behavior), late data handling (watermarks — defining how late data can arrive and still be included, stateful operations with state cleanup), output sink (Delta Lake for ACID writes with streaming), and monitoring (streaming query metrics, lag monitoring).

Walk through your methodology: 1) Identify the anomaly (data quality metric alert, downstream model performance drop, stakeholder report), 2) Isolate the scope (which tables, which time range, which dimension is affected), 3) Trace the lineage upstream (Delta Lake's history, data lineage in Unity Catalog), 4) Inspect the transformation logic at the failing stage (Spark job logs, intermediate output inspection), 5) Identify root cause (upstream data change, schema change, business logic change, infrastructure issue), 6) Fix and prevent (add data quality checks at each stage, use Great Expectations or Databricks data quality constraints).