What is Data Lineage
Track the path of data from its source through transformations to a report, model, or solution.
Definition
Data Lineage is tracing the path of data from its source through transformations to a report, model, or solution. Simply put, this concept helps build reliable services around models: data, compute, access, deployment and monitoring. In practice, it helps to understand what capabilities the tool actually has, what data it will need, and what limitations are worth checking before implementation.
Example
The auditor sees from which tables and transformations the data set for training the model was obtained.
Why it matters
Data provenance helps verify errors, compliance, and reproducibility of AI projects. This helps you choose AI tools not by big promises, but by how they work in a real problem.
How it works
Typically, the process starts with data sources and the environment, then sets up calculations, access, automation, monitoring, and security rules. In the case of the term “Data provenance”, it is important to look separately at the data, quality criteria and application conditions.
Where it is used
- It is found in projects where data storage, computing, integration, deployment, security and stable operation of AI services are important.
Limitations
Limitations are related to computational cost, security, data quality, latency, service availability, and maintenance complexity.