An Automated Data Lineage-Based Impact Analysis Framework for Assessing Upstream and Downstream Dependencies of Changes across Enterprise Financial Data Architectures
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V4I4P122Keywords:
Change Impact Analysis, Data Governance, Data Lineage, Data Provenance, Metadata Management, Column-Level Lineage, Dependency Graph, Impact Analysis Framework, Provenance Semirings, Regulatory Compliance, Enterprise Data ArchitectureAbstract
Modern enterprise financial data architectures are distributed assemblies of core banking systems, risk engines, data warehouses, data lakes, and streaming pipelines, in which a single schema change or threshold adjustment can propagate silently across dozens of downstream datasets and regulatory reports. While this dependency structure needs to be made explicit and auditable under supervisory frameworks like the Basel Committee on Banking Supervision (BCBS 239) principles for effective risk data aggregation and risk reporting, it is mostly manual, person-dependent, and not documented in most banks. This article introduces a conceptual framework for impact analysis of enterprise financial data architectures (DLBIAF). It makes three contributions. It defines a formal graph model with dependency types, propagation factors, and optional impact conditions on the lineage edges, as well as a normalized criticality-weighted scoring function, conservative rules for cycles, duplicate paths and unknown conditions. Second, it integrates condition-aware propagation, as used in variability-aware change impact analysis, with provenance-semiring path semantics. This integration makes pruning decisions and impact rankings explicit design choices represented through path annotations. A structured comparison with existing lineage systems, metadata platforms, and software change-impact analysis identifies the specific gap addressed within the reviewed corpus. Third, it sets out a governance mapping that views impact assessments as supporting evidence for BCBS 239 compliance activities, but not as proof of compliance. The framework is developed as a focused literature synthesis of 25 sources, which is evaluated through requirement-coverage analysis and the illustrative analytical scenario. No implementation or experiments were conducted; all quantitative findings are attributed to their original studies and contexts. The tractability of simple-path aggregation at enterprise scale is stated as an open design question rather than as an established property. An empirical evaluation plan with baselines and metrics is outlined as future work.
References
1. M. Ahmed, A. R. Dar, M. Helfert, A. Khan, and J. Kim, "Data provenance in healthcare: Approaches, challenges, and future directions," Sensors, vol. 23, no. 14, p. 6495, 2023.
2. R. Hai, C. Koutras, C. Quix, and M. Jarke, "Data lakes: A survey of functions and systems," IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 12, pp. 12571-12590, Dec. 2023.
3. A. A. Harby and F. Zulkernine, "From data warehouse to lakehouse: A comparative review," in Proc. IEEE Int. Conf. on Big Data (Big Data), 2022, pp. 389-395, doi: 10.1109/BigData55660.2022.10020719.
4. Financial Stability Board, "Feasibility study on approaches to aggregate OTC derivatives data," Financial Stability Board, Sep. 19, 2014.
5. B. Otto, "Data governance," Business & Information Systems Engineering, vol. 3, no. 4, pp. 241-244, 2011, doi: 10.1007/s12599-011-0162-8.
6. R. Abraham, J. Schneider, and J. vom Brocke, "Data governance: A conceptual framework, structured review, and research agenda," International Journal of Information Management, vol. 49, pp. 424-438, 2019.
7. M. Interlandi, K. Shah, S. D. Tetali, M. A. Gulzar, S. Yoo, M. Kim, T. Millstein, and T. Condie, "Titian: Data provenance support in Spark," Proc. VLDB Endowment, vol. 9, no. 3, pp. 216-227, 2015.
8. M. Herschel, R. Diestelkamper, and H. Ben Lahmar, "A survey on provenance: What for? What form? What from?" The VLDB Journal, vol. 26, no. 6, pp. 881-906, 2017, doi: 10.1007/s00778-017-0486-1.
9. M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia, "Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics," in Proc. 11th Conf. on Innovative Data Systems Research (CIDR), 2021.
10. P. Vassiliadis, A. Simitsis, and S. Skiadopoulos, "Conceptual modeling for ETL processes," in Proc. 5th ACM Int. Workshop on Data Warehousing and OLAP (DOLAP), 2002, pp. 14-21.
11. Basel Committee on Banking Supervision, "Principles for effective risk data aggregation and risk reporting," Bank for International Settlements, Basel, Switzerland, Jan. 2013.
12. V. Khatri and C. V. Brown, "Designing data governance," Communications of the ACM, vol. 53, no. 1, pp. 148-152, Jan. 2010, doi: 10.1145/1629175.1629210.
13. C. Batini, C. Cappiello, C. Francalanci, and A. Maurino, "Methodologies for data quality assessment and improvement," ACM Computing Surveys, vol. 41, no. 3, article 16, 2009.
14. N. Noy, Y. Gao, A. Jain, A. Narayanan, A. Patterson, and J. Taylor, "Industry-scale knowledge graphs: Lessons and challenges," Communications of the ACM, vol. 62, no. 8, pp. 36-43, 2019, doi: 10.1145/3331166.
15. D. Petrik, D. Dhungana, S. Kloeckner, and C. Dannecker, "Functional requirements for enterprise data catalogs: A systematic literature review," in Software Business (ICSOB 2023), Lecture Notes in Business Information Processing, vol. 500, Springer, 2024, pp. 3-18.
16. A. Halevy, F. Korn, N. Noy, C. Olston, N. Polyzotis, S. Roy, and S. E. Whang, "Goods: Organizing Google's datasets," in Proc. ACM SIGMOD Int. Conf. on Management of Data, 2016, pp. 795-806.
17. J. M. Hellerstein, V. Sreekanti, J. E. Gonzalez, et al., "Ground: A data context service," in Proc. 8th Biennial Conf. on Innovative Data Systems Research (CIDR), 2017.
18. F. Angerer, A. Grimmer, P. Grunbacher, and H. Prahofer, "Change impact analysis for maintenance and evolution of variable software systems," Automated Software Engineering, vol. 26, no. 2, pp. 417-461, 2019, doi: 10.1007/s10515-019-00253-7.
19. K. A. Alam, R. Ahmad, A. Akhunzada, M. H. N. Md Nasir, and S. U. Khan, "Impact analysis and change propagation in service-oriented enterprises: A systematic review," Information Systems, vol. 54, pp. 43-73, Dec. 2015, doi: 10.1016/j.is.2015.06.003.
20. R. Goeritzer, "Using impact analysis in industry," Dept. of Informatics Systems, Univ. of Klagenfurt, Klagenfurt, Austria, Tech. Rep., 2018.
21. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proc. 9th USENIX Symp. on Networked Systems Design and Implementation (NSDI), 2012, pp. 15-28.
22. J. Cheney, L. Chiticariu, and W.-C. Tan, "Provenance in databases: Why, how, and where," Foundations and Trends in Databases, vol. 1, no. 4, pp. 379-474, 2009, doi: 10.1561/1900000006.
23. T. J. Green, G. Karvounarakis, and V. Tannen, "Provenance semirings," in Proc. 26th ACM SIGMOD-SIGACT-SIGART Symp. on Principles of Database Systems (PODS), 2007, pp. 31-40.
24. S. Karkoskova, "Data governance model to enhance data quality in financial institutions," Information Systems Management, vol. 40, no. 1, pp. 90-110, 2023, doi: 10.1080/10580530.2022.2042628.
25. J. Martins, H. S. Mamede, and J. Correia, "Risk compliance and master data management in banking - A novel BCBS 239 compliance action-plan proposal," Heliyon, vol. 8, no. 6, e09627, Jun. 2022, doi: 10.1016/j.heliyon.2022.e09627.
26. L. G. Valiant, “The complexity of enumeration and reliability problems,” SIAM Journal on Computing, vol. 8, no. 3, pp. 410-421, 1979, doi: 10.1137/0208032.
