KATTA HAJMLI TA’LIM MA’LUMOTLARINI (BIG DATA) INTEGRATSIYALASH VA TAQSIMLANGAN MA’LUMOTLAR BAZALARINI OPTIMALLASHTIRISH

Xusenov M. Z. & Istamov B. B.


Annotatsiya

Ushbu maqolada ta’lim tizimida shakllanayotgan katta hajmli ma’lumotlarni yig‘ish, integratsiyalash, saqlash, qayta ishlash va tahlil qilish masalalari yoritiladi. Elektron ta’lim platformalari, o‘quv jarayonini boshqarish tizimlari, onlayn testlar, raqamli kutubxonalar, talabalarning faollik jurnallari, mobil ilovalar va ma’muriy axborot tizimlari ta’lim muassasalarida turli formatdagi ma’lumotlar oqimini hosil qilmoqda. Bunday ma’lumotlardan samarali foydalanish uchun integratsiya arxitekturasi, metama’lumotlar boshqaruvi, taqsimlangan ma’lumotlar bazalari, sharding, replikatsiya, indekslash, kesh, so‘rovlarni optimallashtirish, xavfsizlik va boshqaruv tamoyillari kompleks yondashuvda ko‘rib chiqilishi zarur. Maqolada Big Data tushunchasi, ta’lim ma’lumotlarining xususiyatlari, ETL/ELT jarayonlari, data lake va lakehouse yondashuvlari, Hadoop, Spark, Cassandra, MongoDB, PostgreSQL kabi texnologik asoslar, shuningdek taqsimlangan tizimlarda CAP, konsistensiya, mavjudlik va kechikish o‘rtasidagi muvozanat masalalari tahlil qilinadi. Ish yakunida ta’lim muassasalari uchun amaliy arxitektura modeli va optimallashtirish bo‘yicha tavsiyalar beriladi.


Adabiyotlar

  1. NIST. Big Data Interoperability Framework: Volume 6, Reference Architecture.https://www.nist.gov/publications/nist-big-data-interoperability-framework-volume-6-reference-architecture
  2. NIST. NBDIF Version 2.0 Final. https://www.nist.gov/itl/big-data-nist/big-data-nist-documents/nbdif-version-20-final
  3. Apache Hadoop. HDFS Architecture. https://hadoop.apache.org/docs/r2.10.2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
  4. Apache Hadoop. HDFS Users Guide. https://hadoop.apache.org/docs/r3.3.5/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
  5. Apache Spark. Documentation. https://spark.apache.org/documentation
  6. Apache Spark. Cluster Mode Overview. https://spark.apache.org/docs/latest/cluster-overview.html
  7. Apache Cassandra. Architecture Overview. https://cassandra.apache.org/doc/latest/cassandra/architecture/overview.html
  8. MongoDB. Sharding. https://www.mongodb.com/docs/current/sharding/
  9. PostgreSQL. Table Partitioning. https://www.postgresql.org/docs/current/ddl-partitioning.html
  10. Chang, F., Dean, J., Ghemawat, S. et al. Bigtable: A Distributed Storage System for Structured Data. Google Research. https://research.google/pubs/pub27898
  11. Bigtable PDF. https://research.google.com/archive/bigtable-osdi06.pdf
  12. DeCandia, G. et al. Dynamo: Amazon’s Highly Available Key-value Store. https://web.stanford.edu/class/cs244/papers/amazon-dynamo-sosp2007.pdf
  13. Srivastava, D. Big Data Integration. Proceedings of the VLDB Endowment. https://www.vldb.org/pvldb/vol6/p1188-srivastava.pdf
  14. Nargesian, F. et al. Data Lake Management: Challenges and Opportunities. PVLDB. https://www.vldb.org/pvldb/vol12/p1986-nargesian.pdf
  15. Giebler, C. et al. Data Lakes: A Survey of Concepts and Architectures. Computers, MDPI. https://www.mdpi.com/2073-431X/13/7/183
  16. Hai, R. et al. Data Lakes: A Survey of Functions and Systems. arXiv. https://arxiv.org/abs/2106.09592
  17. Brewer, E. Towards Robust Distributed Systems. CAP keynote reference. https://people.eecs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf
  18. Kleppmann, M. A Critique of the CAP Theorem. University of Cambridge. https://www.cl.cam.ac.uk/research/dtg/archived/files/publications/public/mk428/cap-critique.pdf
  19. Abadi, D. PACELC design principle reference. https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf
  20. Zaharia, M. et al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf