Ask ten companies what is data engineering and most will describe a tool: a warehouse, a pipeline product, a Spark cluster. The discipline is wider than any of them. Data engineering is the work of moving data from the systems that create it to the people and applications that use it, in a shape they can trust, on a schedule they can rely on, at a cost the business accepts.
The stakes are easy to underestimate because the failures are quiet. Gartner estimates that poor data quality costs organizations an average of 12.9 million dollars a year, and most of that cost shows up far from the pipeline that caused it: a forecast that missed, a dashboard two departments read differently, a model trained on duplicated rows. Data engineering is where those problems are prevented or created.
This article is the center of our data engineering material. It covers the lifecycle every data team runs, whether it names it or not, the concerns that cut across that lifecycle, how the work splits between roles, how to choose a processing model, and where to start when the current setup is a collection of scripts nobody wants to touch.
What is data engineering?
Data engineering is the design, construction and operation of the systems that collect, store, transform and serve data for analysis, applications and machine learning. The output is a dependable flow of data, with known freshness, known quality and a known owner. Reports, models and AI agents sit downstream of that flow.
A useful way to see the scope is to follow one record. An order is created in an e-commerce database. It has to be copied out without slowing the store, landed somewhere durable, joined with customer and product data, checked for duplicates and impossible values, reshaped into tables analysts understand, and delivered to a dashboard, a demand forecast and a fraud model. Every one of those steps is data engineering, and each can fail independently.
That is also why the field resists a single tool. Ingestion, storage, transformation and serving each have mature products, and a data platform is the way they are combined, governed and operated. The engineering lives in the combination.
The data engineering lifecycle, from source to serving
Joe Reis and Matt Housley, in the O'Reilly book Fundamentals of Data Engineering, describe the work as a lifecycle with five stages: generation, storage, ingestion, transformation and serving. The framing has become a common reference because it holds regardless of vendor. The table below follows the same sequence and adds what tends to break at each step.
| Stage | What happens | Typical technologies | What usually breaks |
|---|---|---|---|
| Generation | Source systems create data: application databases, SaaS tools, events, files, sensors | PostgreSQL, SAP, Salesforce, application logs | Schema changes nobody announced; fields reused for new meanings |
| Ingestion | Data is extracted and moved into the platform, in batches or as a stream | Airbyte, Fivetran, Azure Data Factory, Kafka, change data capture | Silent partial loads; API rate limits; duplicated records after retries |
| Storage | Data lands in a durable, queryable layer | Object storage, data warehouses, lakehouse tables | Unbounded costs; files nobody can find; no retention policy |
| Transformation | Raw data is cleaned, joined and modeled for use | SQL, dbt, Spark, warehouse-native transformations | Business logic duplicated in many places; untested models |
| Serving | Data reaches dashboards, applications, models and agents | BI tools, semantic layers, reverse ETL, feature stores, APIs | Two numbers for the same metric; stale data presented as current |
Two practical points follow from the table. First, most incidents start at the boundary between stages, where one team's output becomes another team's input. Second, the stage that gets the least design attention is usually generation, because the source systems belong to application teams. A pipeline is only as stable as the contract it has with its sources.
The five stages of the lifecycle and the concerns that run under every one of them. Source: BIX Tech, based on Reis and Housley, Fundamentals of Data Engineering.
Layering data inside the platform
Most modern platforms organize stored data in layers of increasing refinement. Databricks documents this as the medallion architecture: a bronze layer with raw data as it arrived, a silver layer with validated and deduplicated records, and a gold layer modeled for specific business uses. The medallion pattern in the Databricks documentation is vendor language, yet the idea applies to any warehouse or lakehouse: keep the raw copy so you can reprocess, and make each layer's quality promise explicit.
Open table formats changed the storage stage in the last few years. Apache Iceberg, described by the project as an open table format for huge analytic datasets, adds schema evolution, time travel and atomic commits to files in object storage, and several engines can read the same tables. For a data team, the consequence is less lock-in at the storage layer and more freedom to choose compute per workload. The choice between a lake, a warehouse and a lakehouse is covered in our guide to data lake, data warehouse or lakehouse.
The undercurrents: quality, security, orchestration and cost
The same book names a set of undercurrents that run beneath every stage: security, data management, DataOps, data architecture, orchestration and software engineering. In practice, these are the concerns that decide whether a platform still works after the second year, when the original engineers have moved on.
Quality and observability. Each layer needs tests that run automatically: row counts, uniqueness, freshness, accepted values, referential integrity. Tests catch what you predicted; observability catches the rest, by tracking volume, schema and distribution changes over time. The dimensions and the tooling are covered in what data quality means in practice.
Orchestration. Pipelines depend on each other, and something has to run them in order, retry what failed and alert someone when a dependency is late. Apache Airflow, which its documentation describes as a platform to programmatically author, schedule and monitor workflows, is the most common answer, and Dagster and Prefect are frequent alternatives. Orchestration in depth is the subject of our guide to workflow orchestration with Apache Airflow.
Security and governance. Access control by role and by row, masking of personal data, lineage that shows where a field came from, and retention rules that satisfy regulators. Governance works when it is built into the platform rather than reviewed after the fact, a point developed in our guide to decentralized data governance.
Data contracts. A data contract is an agreement between the team that produces data and the teams that consume it, covering schema, quality rules and freshness. Written as code and checked in the pipeline, it turns a silent upstream change into a failed check with an owner. Contracts are one of the most effective ways to stabilize the generation stage that the table above flags as fragile.
Cost. Cloud platforms bill for storage, compute and data movement, and a careless transformation can scan the same terabytes every hour. Cost belongs in the design: partitioning, incremental models, scheduled versus on-demand compute, and a named owner for each expensive job.
Who does what: data engineer, analytics engineer, data scientist
The work splits differently in every company, but three roles appear in most mature teams. The boundaries matter less than knowing who owns each handoff.
| Role | Owns | Main questions | Typical tools |
|---|---|---|---|
| Data engineer | Ingestion, storage, platform reliability, performance and cost | Is the data arriving, complete and on time? Can the platform scale? | Python, SQL, Spark, Kafka, Airflow, cloud services |
| Analytics engineer | Transformation and modeling for analysis, metric definitions, tests | Do the tables match how the business works? Is each metric defined once? | SQL, dbt, semantic layers, version control |
| Data scientist | Models, experiments and statistical analysis built on modeled data | What predicts the outcome? Did the change work? | Python, notebooks, ML frameworks, feature stores |
The analytics engineer is the newest of the three. The term was popularized by the team behind dbt to describe people who apply software practices, such as version control, tests and code review, to the transformation layer. In small teams one person covers all three columns; what matters is that the transformation layer has an owner, because that is where business definitions live. The trade-offs between the roles are detailed in our comparison of data engineer, analytics engineer and data scientist, and the transformation layer itself in our guide to dbt and semantic models.
Batch, streaming and ELT: choosing the processing model
Batch or streaming. Batch processing moves data on a schedule, every hour or every night, and is simpler to build, test and pay for. Streaming processes events as they occur, with latency measured in seconds, and it earns its cost when a decision loses value within minutes: fraud checks, stock levels, operational alerts. Most companies run both, and the mistake is choosing streaming because it sounds modern. The criteria are laid out in when to use Apache Kafka, and when batch is enough.
ETL or ELT. In ETL, data is transformed before it is loaded into the warehouse. In ELT, raw data is loaded first and transformed inside the warehouse or lakehouse, which became the default once cloud compute made in-place transformation cheap and fast. ELT keeps the raw copy available for reprocessing; ETL still makes sense when sensitive fields must be removed before data leaves its source, or when the target cannot handle heavy transformation.
| Situation | Usual choice | Why |
|---|---|---|
| Daily reporting on sales, finance or operations | Batch, ELT | Freshness of hours is enough, and the raw copy simplifies corrections |
| Fraud detection or real-time inventory | Streaming | The decision loses value within seconds or minutes |
| Personal data that must not leave the source unmasked | ETL for the sensitive fields | Masking happens before data reaches the shared platform |
| Many SaaS sources feeding one warehouse | Managed ingestion, ELT | Connectors are maintained by the vendor; the team focuses on modeling |
| Machine learning features needed online | Streaming plus batch backfill | Fresh features for inference, full history for training |
How to start or fix a data engineering practice
Most teams are not building from zero. They inherit scheduled scripts, a warehouse with tables nobody can explain, and reports that disagree. The order of work below assumes that starting point.
- Inventory what exists. List sources, pipelines, tables and the dashboards that depend on them, with an owner for each. Undocumented dependencies are the main reason migrations stall.
- Pick one business question with a sponsor. Rebuild the path for that question end to end, from source to dashboard, instead of migrating everything at once.
- Standardize ingestion. Replace hand-written extract scripts with a maintained connector or a documented pattern, and land raw data unchanged in a bronze layer.
- Put transformations under version control with tests. One definition per metric, reviewed like code, with tests that block a deploy when they fail.
- Add orchestration and alerting. Every pipeline runs on a scheduler, retries on failure and notifies a named person when it is late.
- Measure and expand. Track freshness, failed runs and cost per pipeline, then repeat the path for the next business question.
The most common failure is skipping step two and attempting a full platform rebuild before anyone sees value. The second is treating the platform as a one-time project; teams that keep improving it run it like a product, with users, a backlog and service levels.
A practical sequence for teams that inherit an existing setup: one business question at a time, end to end. Source: BIX Tech.
Where to go deeper
Getting data in. Ingestion is where most data platforms first break. How data ingestion can empower your organization covers the basics, Airbyte vs Fivetran compares open source and managed ELT, and Airbyte in practice shows how to build reliable integrations. For enterprise sources, see metadata-driven ingestion in Azure Data Factory and SAP Integration Suite for enterprise data.
Keeping pipelines stable as sources change. Schema evolution in data pipelines explains how to absorb upstream changes without downtime, data versioning covers reproducibility, and why latency matters in modern data pipelines addresses freshness.
Designing the platform as a whole. What makes a data platform enterprise-ready lists the capabilities that matter at scale, and data platforms as products, not projects covers how to run one. For a Microsoft stack, see Microsoft Fabric data architecture, from ingestion to insights.
The people and the direction of the field. The skills that set senior data engineers apart in 2026 describes what seniority looks like now, and how AI is reshaping data engineering workflows covers what changes for the team.
Data engineering decides whether the rest of the data strategy works. Dashboards, forecasts and AI agents all inherit the freshness, quality and cost of the pipelines underneath them, and no model or visualization recovers what the platform lost. Treating the lifecycle as one system, with owners for each handoff and tests at each layer, is what separates a platform that scales from a collection of scripts that only its author can run.
If your company is untangling inherited pipelines or planning a data platform that has to grow, our specialists can help you structure the right architecture for your context. Talk to our team and move your data maturity forward.
Frequently asked questions about data engineering
What is data engineering in simple terms? Data engineering is the work of moving data from the systems that create it to the people and tools that use it, cleaned, organized and on time. It covers ingestion, storage, transformation and delivery, plus the testing, orchestration and security that keep that flow reliable as sources and volumes change.
What does a data engineer do day to day? A data engineer builds and maintains pipelines that bring data into the platform, models storage for performance and cost, fixes failed or late jobs, and works with source teams on schema changes. Much of the job is operational: monitoring freshness and quality, tuning expensive queries and documenting what each pipeline delivers.
What is the difference between data engineering and data science? Data engineering builds and operates the systems that deliver reliable data; data science uses that data to build models and answer statistical questions. Data scientists depend on engineered data: without dependable pipelines, most of their time goes to cleaning inputs instead of analysis, and models fail in production when inputs drift.
What is the data engineering lifecycle? The data engineering lifecycle, as described by Joe Reis and Matt Housley, has five stages: generation, ingestion, storage, transformation and serving. Under all of them run concerns such as security, data management, DataOps, architecture, orchestration and software engineering, which decide whether the platform stays reliable over time.
Should we use ETL or ELT? Most cloud data platforms default to ELT: load raw data first, then transform it inside the warehouse or lakehouse, keeping the raw copy for reprocessing. ETL still fits when sensitive fields must be masked before data leaves its source, or when the target system cannot run heavy transformations efficiently.
