From Legacy Data to Intelligent Enterprises: Transforming Data Architecture for the AI-Driven Future
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V2I1P114Keywords:
Artificial Intelligence, Big Data Analytics, Cloud Computing, Data Governance, Data Lake, Data Warehouse, Enterprise Data Architecture, Intelligent Enterprise, Machine LearningAbstract
Enterprise competitiveness increasingly depends on the ability of organizations to transform heterogeneous legacy data estates into scalable, intelligent analytical platforms. Traditional enterprise data architectures, where operational systems are tied to a centralized data warehouse via scheduled extract-transform-load (ETL) pipelines, have well-established drawbacks such as data silos, inflexible schemas, batch data integration, limited horizontal scalability, and limited machine learning capabilities. This paper investigates the transformation of enterprise data architecture from legacy warehouse-centric designs toward AI-driven intelligent enterprise platforms, using 2020 as the historical evidence cutoff. A reproducible, literature-based methodology is employed, comprising the systematic collection of peer-reviewed literature, benchmark publications, and official technical documents published before 2021, followed by the extraction of quantitative and qualitative evidence and the preparation of structured evidence tables. Distributed in-memory processing and cloud data object storage significantly enhance processing throughput and elasticity compared with legacy relational, enterprise data warehouse, Hadoop-based, data lake, cloud-based, and AI-enabled architectures, and lakehouse-style managed metadata layers bring lake flexibility together with warehouse reliability. The paper presents (i) a unified overview of the architectural evolution from legacy databases to AI-based platforms, (ii) a comparative evidence-based coverage of the various aspects of processing, storage, integration, governance, and AI readiness, and (iii) a seven-layer reference architecture of the intelligent enterprise based on the use of AI. The findings provide enterprise architects with evidence-based guidance for planning legacy modernization and highlight unresolved challenges related to governance, migration, and data quality.
References
1. R. Agrawal, A. Ailamaki, P. A. Bernstein, E. A. Brewer, M. J. Carey, and S. Chaudhuri, "The Claremont report on database research," Communications of the ACM, vol. 52, no. 6, pp. 56-65, Jun. 2009, doi: 10.1145/1516046.1516064.
2. B. Inmon, Building the Data Warehouse, 4th ed. Indianapolis, IN, USA: Wiley, 2005.
3. R. Kimball and M. Ross, The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, 3rd ed. Indianapolis, IN, USA: Wiley, 2013.
4. S. Chaudhuri and U. Dayal, "An overview of data warehousing and OLAP technology," ACM SIGMOD Record, vol. 26, no. 1, pp. 65-74, Mar. 1997, doi: 10.1145/248575.248591.
5. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, Jan. 2008, doi: 10.1145/1327452.1327492.
6. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System," in Proc. IEEE 26th Symp. Mass Storage Systems and Technologies (MSST), Incline Village, NV, USA, May 2010, pp. 1-10, doi: 10.1109/MSST.2010.5496972.
7. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient Distributed Datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proc. 9th USENIX Symp. Networked Systems Design and Implementation (NSDI), San Jose, CA, USA, Apr. 2012, pp. 15-28.
8. M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, and M. Zaharia, "Spark SQL: Relational data processing in Spark," in Proc. ACM SIGMOD Int. Conf. Management of Data, Melbourne, Australia, May-Jun. 2015, pp. 1383-1394, doi: 10.1145/2723372.2742797.
9. W. Inoubli, S. Aridhi, H. Mezni, M. Maddouri, and E. M. Nguifo, "An experimental survey on big data frameworks," Future Generation Computer Systems, vol. 86, pp. 546-564, Sep. 2018, doi: 10.1016/j.future.2018.04.032.
10. J. Shi, Y. Qiu, U. F. Minhas, L. Jiao, C. Wang, B. Reinwald, and W. Wang, "Clash of the titans: MapReduce vs. Spark for large scale data analytics," Proc. VLDB Endowment, vol. 8, no. 13, pp. 2110-2121, 2015, doi: 10.14778/2824032.2824053.
11. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster computing with working sets," in Proc. 2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud), Boston, MA, USA, Jun. 2010.
12. S. Kulkarni, N. Bhagat, M. Fu, V. Kedigehalli, C. Kellogg, S. Mittal, J. M. Patel, K. Ramasamy, and S. Taneja, "Twitter Heron: Stream processing at scale," in Proc. ACM SIGMOD Int. Conf. Management of Data, Melbourne, Australia, May-Jun. 2015, pp. 239-250, doi: 10.1145/2723372.2742788.
13. M. Armbrust, T. Das, J. Torres, L. Yavuz, S. Zhu, R. Xin, A. Ghodsi, I. Stoica, and M. Zaharia, "Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics," in Proc. 8th Annual Conf. Innovative Data Systems Research (CIDR), Chaminade, CA, USA, Jan. 2021. (Position paper outlining the lakehouse vision; retained as retrospective context.)
14. F. Nargesian, E. Zhu, R. J. Miller, K. Q. Pu, and P. C. Arocena, "Data lake management: Challenges and approaches," Proc. VLDB Endowment, vol. 12, no. 12, pp. 1986-1989, 2019, doi: 10.14778/3352063.3352116.
15. C. Quix and S. Hai, "Data lakes," in Encyclopedia of Big Data Technologies, vol. 1, S. Sakr and A. Y. Zomaya, Eds. Cham, Switzerland: Springer, 2019, pp. 596-602, doi: 10.1007/978-3-319-63962-8_134-1.
16. M. Armbrust, T. Das, L. Sun, B. Yavuz, S. Zhu, R. Xin, M. Murthy, J. Torres, H. Yavuz, H. Zhou, R. Bose, and A. Ghodsi, "Delta Lake: High-performance ACID table storage over cloud object stores," Proc. VLDB Endowment, vol. 13, no. 12, pp. 3411-3424, 2020, doi: 10.14778/3417633.3417660.
17. S. Landset, T. M. Khoshgoftaar, A. N. Richter, and T. Hasanin, "A survey of open source tools for machine learning with big data in the Hadoop ecosystem," Journal of Big Data, vol. 2, no. 1, art. 24, Dec. 2015, doi: 10.1186/s40537-015-0032-1.
18. X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman, D. Tsai, M. Amde, S. Owen, D. Xin, R. Xin, M. J. Franklin, R. Zadeh, M. Zaharia, and A. Talwalkar, "MLlib: Machine learning in Apache Spark," Journal of Machine Learning Research, vol. 17, no. 34, pp. 1-7, 2016.
19. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., "TensorFlow: A system for large-scale machine learning," in Proc. 12th USENIX Symp. Operating Systems Design and Implementation (OSDI), Savannah, GA, USA, Nov. 2016, pp. 265-283.
20. V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, et al., "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs," JAMA, vol. 316, no. 22, pp. 2402-2410, Dec. 2016, doi: 10.1001/jama.2016.17216.
21. R. Buyya and K. Ramamohanarao, Eds., Big Data Computing and Clouds: Concepts, Technologies, Tools, and Trends, ser. Big Data series. Boca Raton, FL, USA: CRC Press, 2018.
22. S. Kaisler, F. Armour, J. A. Espinosa, and W. Money, "Big data: Issues and challenges moving forward," in Proc. 46th Hawaii Int. Conf. System Sciences (HICSS), Wailea, Maui, HI, USA, Jan. 2013, pp. 995-1004, doi: 10.1109/HICSS.2013.645.
23. M. D. Assuncao, R. N. Calheiros, S. Bianchi, M. A. S. Netto, and R. Buyya, "Big Data computing and clouds: Trends and future directions," Journal of Parallel and Distributed Computing, vol. 79-80, pp. 3-16, May 2015, doi: 10.1016/j.jpdc.2014.08.003.
24. F. Alam and S. M. Mehmood, "Technologies and analytics for extracting value from big data: A systematic mapping study," in Semantic Big Data Integration, vol. 78, Springer Briefs, 2017.
25. P. P. Chen and A. C. F. Cheung, "The requirements for big data analytics in enterprise information systems: A systematic literature review," in Proc. 19th Int. Conf. Enterprise Information Systems (ICEIS), vol. 3, Porto, Portugal, Apr. 2017, pp. 210-221, doi: 10.5220/0006351002100221.
26. V. Khatri and C. V. Brown, "Designing data governance," Communications of the ACM, vol. 60, no. 3, pp. 43-50, Mar. 2017, doi: 10.1145/3024918.
27. N. Golchha, P. Jain, and M. F. Mokbel, "Automating metadata management in data lakes: Opportunities and challenges," Proc. VLDB Endowment, vol. 12, no. 12, pp. 1900-1903, 2019, doi: 10.14778/3352063.3352064.
28. Databricks, "Databricks documentation: Delta Lake guide," Databricks, 2020. [Online]. Available: https://databricks.com/learn/delta/deep-dive, 2020.
