Senior Data Engineer

Production-grade data engineering skill for building scalable, reliable data systems.

Table of Contents

1. 1. Trigger Phrases
2. Quick Start
3. Workflows

1. 4. Architecture Decision Framework
2. Tech Stack
3. Reference Documentation
4. Troubleshooting

Trigger Phrases

Activate this skill when you see:

Pipeline Design:

• - "Design a data pipeline for..."
• "Build an ETL/ELT process..."
• "How should I ingest data from..."
• "Set up data extraction from..."

Architecture:

• - "Should I use batch or streaming?"
• "Lambda vs Kappa architecture"
• "How to handle late-arriving data"
• "Design a data lakehouse"

Data Modeling:

• - "Create a dimensional model..."
• "Star schema vs snowflake"
• "Implement slowly changing dimensions"
• "Design a data vault"

Data Quality:

• - "Add data validation to..."
• "Set up data quality checks"
• "Monitor data freshness"
• "Implement data contracts"

Performance:

• - "Optimize this Spark job"
• "Query is running slow"
• "Reduce pipeline execution time"
• "Tune Airflow DAG"

Quick Start

Core Tools

CODEBLOCK0

Workflows

Architecture Decision Framework

Use this framework to choose the right approach for your data pipeline.

Batch vs Streaming

Decision Tree:
CODEBLOCK1

Lambda vs Kappa Architecture

When to choose Lambda:

• - Need to train ML models on historical data
• Complex batch transformations not feasible in streaming
• Existing batch infrastructure

When to choose Kappa:

• - Event-sourced architecture
• All processing can be expressed as stream operations
• Starting fresh without legacy systems

Data Warehouse vs Data Lakehouse

Tech Stack

Reference Documentation

1. Data Pipeline Architecture

• - Lambda vs Kappa architecture patterns
• Batch processing with Spark and Airflow
• Stream processing with Kafka and Flink
• Exactly-once semantics implementation
• Error handling and dead letter queues

2. Data Modeling Patterns

• - Dimensional modeling (Star/Snowflake)
• Slowly Changing Dimensions (SCD Types 1-6)
• Data Vault modeling
• dbt best practices
• Partitioning and clustering

3. DataOps Best Practices

• - Data testing frameworks
• Data contracts and schema validation
• CI/CD for data pipelines
• Observability and lineage
• Incident response

Troubleshooting