# Data Pipeline taste of a data engineer
- Prefer ELT: land raw data unchanged first, then transform inside the warehouse. Confidence: 0.85
- Keep raw, staging and mart layers separate, and never let dashboards read from raw tables. Confidence: 0.90
- Model transformations in dbt (or SQLMesh) as version-controlled SQL with code review like any other code. Confidence: 0.90
- Make every job idempotent: rerunning a day must produce the same result without duplicates. Confidence: 0.95
- Partition large tables by event date and process data in date-bounded increments. Confidence: 0.85
- Backfills are a first-class operation with a documented command, not a manual one-off script. Confidence: 0.85
- Add tests for uniqueness, not-null, accepted values and referential integrity on every important model. Confidence: 0.90
- Define data contracts with upstream producers so schema changes fail in their CI, not in your pipeline. Confidence: 0.80
- Track freshness per source and alert when data is late, not only when a job fails. Confidence: 0.90
- Orchestrate with Airflow, Dagster or Prefect, and keep heavy computation out of the orchestrator workers. Confidence: 0.80
- Prefer asset-based orchestration that knows which tables depend on which over plain task graphs. Confidence: 0.70
- Store timestamps in UTC and convert to local time only in the presentation layer. Confidence: 0.95
- Use Parquet or another columnar format with explicit schemas for files in the lake. Confidence: 0.85
- Separate event time from processing time and handle late-arriving data explicitly. Confidence: 0.85
- Keep PII in as few tables as possible, mask or hash it early and document access in one place. Confidence: 0.90
- Use incremental models only after a full refresh becomes too slow; correctness first, cost second. Confidence: 0.75
- Name models consistently (stg_, int_, fct_, dim_) and document every column people actually query. Confidence: 0.80
- Use slowly changing dimension type 2 snapshots when history matters, not ad hoc audit columns. Confidence: 0.70
- Run pipelines in CI against a sample or zero-copy clone before deploying changes to production. Confidence: 0.80
- Monitor warehouse cost per model and kill queries that scan whole tables without partition filters. Confidence: 0.75
- Every metric has one owner and one definition in a semantic layer, not five versions in five dashboards. Confidence: 0.80