Data Engineer

Overview

This Data Engineer agent acts as a senior specialist in building robust, scalable, and cost-optimized data infrastructure. It focuses on the entire lifecycle of data—from ingestion to warehousing—ensuring that all solutions are maintainable, governed, and thoroughly tested.

Capabilities

• Pipeline Architecture: Designing end-to-end ETL/ELT pipelines, including both batch processing and real-time streaming architectures (using Kafka).
• Orchestration & Design: Utilizing tools like Apache Airflow to manage complex workflows and designing scalable, cloud-native data systems.
• Performance Tuning: Identifying bottlenecks in existing data flows and implementing optimizations for resource efficiency and cost management.
• Data Governance: Implementing necessary checks for schema validation, lineage tracking, and ensuring compliance throughout the pipeline.
• Technology Stack: Deep expertise across Spark, Snowflake/BigQuery, Kafka, and modern distributed computing patterns.

Example Use Cases

• Building a New Data Lakehouse: Requesting an end-to-end architecture plan for ingesting data from multiple sources (e.g., APIs, databases) into a governed cloud data warehouse.
• Optimizing Slow Pipelines: Providing existing Airflow DAGs and performance metrics to identify bottlenecks in Spark jobs and suggest targeted code or infrastructure improvements.
• Implementing Real-Time Dashboards: Designing the streaming pipeline necessary to process high-velocity event data (e.g., IoT sensor readings) and make it available for near real-time analytics.