Principal Engineer - Data Engineering

About the role

WD builds the storage systems behind the AI-driven data economy, serving hyperscale data centers, cloud platforms and enterprise infrastructure. This Principal Engineer role focuses on ML-ready scientific data pipelines for experimental measurement data from precision product development instruments, a challenge distinct from standard web-scale or financial data work.

Key responsibilities

• Build and maintain versioned feature engineering pipelines that turn raw engineering, sensor and operational data into structured ML-ready features.
• Design and run data quality checks (completeness, schema consistency, distribution stability, label accuracy) and alert the ML team when quality degrades.
• Maintain training data versioning and lineage tracking, with early alerts when deployment data diverges from training distributions.
• Implement data contracts between upstream producers and downstream ML consumers, following governance standards set with the ML Engineer, and establish access control and retention for AI data assets.
• Contribute to real-time sensor data ingestion pipelines, developing operational ownership over 6-12 months.
• Build pipeline infrastructure for synthetic data generation, including ingestion, storage and versioning.
• Maintain the training dataset registry, feature store and model input validation, integrated with the AI platform (AWS Kubernetes, PortKey, Agent Gateway, LangFuse, AWS Guardrails, Elastic Search).

Required profile

• Bachelor's or Master's degree in AI, computer science, data engineering, electrical engineering, applied mathematics or a related field; AI major preferred.
• At least 2-3 years of experience; demonstrated end-to-end data pipeline projects (academic, personal or internship) are the main evaluation criterion.
• Solid Python and SQL, and demonstrable pipeline design fundamentals.
• Basic understanding of how data pipelines connect to ML model training, including risks such as train/test leakage and label quality.

Required skills

• Python and SQL, including complex queries and window functions.
• Batch pipeline architecture, reliability, schema management and fault tolerance.
• Pipeline orchestration (Airflow, Prefect, AWS Glue Jobs).
• Data quality principles, data versioning and lineage.
• Good to have: Feast or Tecton, DBT, DLT, Label Studio or CVAT, Iceberg, Dremio, AWS Lake Formation.