Data Platform Architect

Overview

This agent functions as a senior Data Platform Architect, specializing in the entire lifecycle of data infrastructure. It moves beyond simple scripting to design comprehensive, production-grade systems that guarantee scalability, reliability (targeting 99.9% SLA), and cost efficiency.

Whether you are building a real-time streaming system or a complex batch warehouse, this agent applies best practices across modern big data stacks, including cloud-native services and open-source frameworks.

Capabilities

• End-to-End Pipeline Design: Architecting the full flow from source ingestion to consumption layer, covering orchestration (Airflow/Prefect) and monitoring.
• ETL/ELT Development: Implementing robust data transformation logic with advanced error handling, validation, and incremental processing strategies.
• Data Lakehouse Implementation: Designing optimal storage solutions using formats like Delta Lake or Iceberg, focusing on partitioning, compaction, and metadata management.
• Stream Processing Mastery: Building real-time pipelines using Kafka/Flink, ensuring exactly-once semantics and handling schema evolution.
• Cloud Optimization: Providing specific architectural guidance for Snowflake, BigQuery, Redshift, and Databricks environments to ensure cost-per-TB efficiency.

Example Use Cases

1. Migrating a Legacy Data Warehouse: You can provide existing schemas and data volume estimates, and the agent will design a modern ELT pipeline using cloud services (e.g., AWS Glue/Snowflake) while recommending necessary governance layers.
2. Building Real-Time Analytics: If you need to process IoT sensor data immediately, this agent will architect a Kafka stream processing job with proper windowing and state management for near real-time dashboards.
3. Establishing Data Governance: For any new platform, the agent can create a detailed checklist covering monitoring setup, disaster recovery planning, and data quality checks required to meet enterprise SLAs.