KATTA HAJMLI TA’LIM MA’LUMOTLARINI (BIG DATA) INTEGRATSIYALASH VA TAQSIMLANGAN MA’LUMOTLAR BAZALARINI OPTIMALLASHTIRISH
Xusenov M. Z. & Istamov B. B.
Annotatsiya
Ushbu maqolada ta’lim tizimida shakllanayotgan katta hajmli ma’lumotlarni yig‘ish, integratsiyalash, saqlash, qayta ishlash va tahlil qilish masalalari yoritiladi. Elektron ta’lim platformalari, o‘quv jarayonini boshqarish tizimlari, onlayn testlar, raqamli kutubxonalar, talabalarning faollik jurnallari, mobil ilovalar va ma’muriy axborot tizimlari ta’lim muassasalarida turli formatdagi ma’lumotlar oqimini hosil qilmoqda. Bunday ma’lumotlardan samarali foydalanish uchun integratsiya arxitekturasi, metama’lumotlar boshqaruvi, taqsimlangan ma’lumotlar bazalari, sharding, replikatsiya, indekslash, kesh, so‘rovlarni optimallashtirish, xavfsizlik va boshqaruv tamoyillari kompleks yondashuvda ko‘rib chiqilishi zarur. Maqolada Big Data tushunchasi, ta’lim ma’lumotlarining xususiyatlari, ETL/ELT jarayonlari, data lake va lakehouse yondashuvlari, Hadoop, Spark, Cassandra, MongoDB, PostgreSQL kabi texnologik asoslar, shuningdek taqsimlangan tizimlarda CAP, konsistensiya, mavjudlik va kechikish o‘rtasidagi muvozanat masalalari tahlil qilinadi. Ish yakunida ta’lim muassasalari uchun amaliy arxitektura modeli va optimallashtirish bo‘yicha tavsiyalar beriladi.
Adabiyotlar
- NIST. Big Data Interoperability Framework: Volume 6, Reference Architecture.https://www.nist.gov/publications/nist-big-data-interoperability-framework-volume-6-reference-architecture
- NIST. NBDIF Version 2.0 Final. https://www.nist.gov/itl/big-data-nist/big-data-nist-documents/nbdif-version-20-final
- Apache Hadoop. HDFS Architecture. https://hadoop.apache.org/docs/r2.10.2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
- Apache Hadoop. HDFS Users Guide. https://hadoop.apache.org/docs/r3.3.5/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
- Apache Spark. Documentation. https://spark.apache.org/documentation
- Apache Spark. Cluster Mode Overview. https://spark.apache.org/docs/latest/cluster-overview.html
- Apache Cassandra. Architecture Overview. https://cassandra.apache.org/doc/latest/cassandra/architecture/overview.html
- MongoDB. Sharding. https://www.mongodb.com/docs/current/sharding/
- PostgreSQL. Table Partitioning. https://www.postgresql.org/docs/current/ddl-partitioning.html
- Chang, F., Dean, J., Ghemawat, S. et al. Bigtable: A Distributed Storage System for Structured Data. Google Research. https://research.google/pubs/pub27898
- Bigtable PDF. https://research.google.com/archive/bigtable-osdi06.pdf
- DeCandia, G. et al. Dynamo: Amazon’s Highly Available Key-value Store. https://web.stanford.edu/class/cs244/papers/amazon-dynamo-sosp2007.pdf
- Srivastava, D. Big Data Integration. Proceedings of the VLDB Endowment. https://www.vldb.org/pvldb/vol6/p1188-srivastava.pdf
- Nargesian, F. et al. Data Lake Management: Challenges and Opportunities. PVLDB. https://www.vldb.org/pvldb/vol12/p1986-nargesian.pdf
- Giebler, C. et al. Data Lakes: A Survey of Concepts and Architectures. Computers, MDPI. https://www.mdpi.com/2073-431X/13/7/183
- Hai, R. et al. Data Lakes: A Survey of Functions and Systems. arXiv. https://arxiv.org/abs/2106.09592
- Brewer, E. Towards Robust Distributed Systems. CAP keynote reference. https://people.eecs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf
- Kleppmann, M. A Critique of the CAP Theorem. University of Cambridge. https://www.cl.cam.ac.uk/research/dtg/archived/files/publications/public/mk428/cap-critique.pdf
- Abadi, D. PACELC design principle reference. https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf
- Zaharia, M. et al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf