Data Pipeline

Origin & evolution

The classical shape is ETL: extract, transform, load. It dates to the data-warehousing practice of the 1980s and 1990s, when warehouse compute was scarce and expensive, so data was cleaned and reshaped on a separate staging server before being loaded into the warehouse in a nightly batch. Transformation came first because the warehouse could not afford to do it.

Cloud warehouses inverted that constraint🔒ConstraintStrategyA constraint entityView reference →. When Redshift, BigQuery, and Snowflake made warehouse compute cheap and elastic, it became practical to load raw data first and transform it inside the warehouse using its own SQL engine. ETL became ELT. As Fivetran describes the shift, the reordering is small on paper and large in consequence: raw data lands intact, and transformation becomes versioned, testable SQL in place of brittle pre-load scripts. This reordering anchored what the industry came to call the modern data stack, a set of specialised tools where managed connectors such as Fivetran or Airbyte handle extract-and-load and dbt handles transformation as code. (The category keeps consolidating; Fivetran and dbt Labs announced a merger in 2025.)

A second axis runs alongside the ETL/ELT one: batch versus streaming. Batch pipelines move data on a schedule, which suits most analytics. Streaming pipelines built on systems such as Apache Kafka process events continuously as they arrive, transforming in flight, which suits fraud detection and live operational metrics where minutes-old data is already stale. Most organisations run both, choosing per use case and treating neither as a default.

How it works in practice

A subscription🔂SubscriptionSales & RevenueA recurring subscriptionView reference → business needs a daily revenue dashboard. The pipeline is ELT and batch. At 02:00 a managed connector extracts rows from the billing database, the payments processor's API, and the product-event log, and loads them raw into a cloud warehouse. A dbt job🔨JobUserJob To Be Done: what the user is trying to accomplishView reference → then runs: it deduplicates events, reconciles currencies, joins payments to accounts, and builds a clean daily_revenue table, with tests asserting that no row has a negative amount and that yesterday's total reconciles to the processor's settlement report. The dashboard reads only that final table.

The discipline pays off the morning revenue appears to drop 12%. Because the raw load is preserved and the transformations are tested code, an engineer traces the dip in twenty minutes to a currency-conversion step that mishandled a new market, with no need to re-derive the number by hand. The pipeline is debuggable because each stage is explicit.

Data pipeline vs. its neighbours

• Data source🛢️Data SourceData & AnalyticsA data source or integrationView reference → is an origin or a destination of data: a database, an API, an event stream. A data pipeline is the moving machinery between sources; the source holds data at rest, the pipeline carries it.
• Data product🎁Data ProductData & AnalyticsA curated, reusable data assetView reference → is a curated, reusable dataset packaged for consumers with an owner and a contract, such as a trusted "customer 360" table. A pipeline is what builds and maintains a data product; the product is the output, the pipeline is the means.
• Dashboard is a presentation surface that reads processed data. A pipeline is upstream of it. When a dashboard is wrong, the cause usually lives in the pipeline that fed it.

In the graph

In the Unified Product Graph, a data pipeline sits in the data and analytics domain and is defined by its flow. It connects to its inputs through data_pipeline_reads_from_data_sourceData Pipelinereads fromData Sourcecross-domain and to its outputs through data_pipeline_writes_to_data_sourceData Pipelinewrites toData Sourcecross-domain, while the inverse data_source_processed_via_data_pipelineData Sourceprocessed viaData Pipelinehierarchy lets any source name the pipelines acting on it. Downstream, data_pipeline_feeds_data_productData PipelinefeedsData Productcross-domain links the machinery to the curated dataset it produces. Modelling direction explicitly (reads-from versus writes-to) is what makes the graph traceable: when a metric looks wrong, the edges let you walk back from the dashboard to the data product to the exact pipeline and source, which is the question every data team is actually asking.