Lineage
The lineage graph shows where data comes from and where it goes. Give it OpenLineage events - the
open standard Airflow, Spark, dbt and others emit - a dbt manifest.json or a simple JSON list, and the
assets are laid out from left to right: sources, the models built from them, and the dashboards, ML features and
applications at the end, each with its columns, owner, system and health.
Choose an asset to light up everything upstream and downstream; choose a column to trace it, row by row, both ways. Impact analysis lists everything a change would break and the owners to tell. Classifications such as PII are carried downstream through every column built from a classified one and highlighted wherever they land. Search, focus with a number of hops, an assets table and checks for gaps in the lineage itself.
A shop's data platform
Sixteen assets of a fictional shop: Postgres, Stripe and Salesforce sources, dbt models, two dashboards, a churn feature table and a marketing audience, with PII highlighted. Choose customers, then its email column, to see where personal data ends up; then open Impact for the owners to tell.
Get the code
<script src="/assets/bmx-components.min.js"></script>
<bmx-lineage-graph id="lg-platform" highlight="PII" style="inline-size: 100%; --bmx-lineage-graph-height: 32rem"></bmx-lineage-graph>
<script>
const lineageGraph = document.getElementById('lg-platform');
lineageGraph.data = [ /* your data */ ];
</script>
From a dbt manifest
A small dbt project's manifest.json: sources, staging and mart models, and an exposure (the
dashboard that reads them), with owners and a PII tag. The Checks view reports the model nothing reads.
// Node 18 or later: the lineage engine, from the Gold package - run in CI on every pull request.
import { readLineage, indexLineage, impactOf } from './dist/node/bmx-lineage-graph.mjs';
const ix = indexLineage(readLineage(await readFile('target/manifest.json', 'utf8')).graph);
for (const model of changedModels()) {
const dashboards = impactOf(ix, model).items.filter(i => i.node.kind === 'dashboard');
if (dashboards.length) console.log(`${model} feeds ${dashboards.map(d => d.node.name).join(', ')}: ask ${[...new Set(dashboards.map(d => d.node.owner))].join(', ')} to review.`);
}