Structure of the data

[NurElHuda]

@ZibanPirate @Fcmam5

I was wondering why the structure of the data is set as a tree of directories with info.json leaf instead of one main JSON file that contains all data.

1. The overhead of traversing a JSON object compared to opening nested directories files to extract data is negligible.
2. The name of the directories of the department does not make sense, the acronyms for the faculties are set by the universities and somewhat distinguishable, but once you dive into the departments you get lost.
3. What is more important, is search. I can't think of an efficient way to search for, let's say, all the departments of geology amidst all the data.
4. The moment the data is stored into one main JSON file will make automating, scraping, and extracting the data from the internet much faster.

So.. what do you guys say? I am happy to convert the available data into a JSON file, than continue with the rest.

Or is there other reasons for why the data is stored like this that I am missing?

[ZibanPirate]

hey @NurElHuda thanks for raising this concern, I will answer your questions, however, that doesn't mean we are not open to changing the structure if we found a better solution, so feel free to suggest improvement over the current solution, or maybe provide an entirely different approach that may work better for this project. cc @dzcode-io/maintainers @dzcode-io/members

---

I was wondering why the structure of the data is set as a tree of directories with info.json leaf instead of one main JSON file that contains all data.

Having one big JSON file is not scalable for contribution, if we have two or more open PRs for adding new data to the file, we will end up with conflict whenever we merge each one. The solution for that is having directories instead. this way, we can also better Git blame (or credit) whoever modified that portion of the data.

---

1. The overhead of traversing a JSON object compared to opening nested directories files to extract data is negligible.

Maybe you didn't understand the way we handle data correctly here 😅, we don't actually consume the directories at all in the produced package, we instead consume a data.json file that is shipped with the package during build time, that's why we have two main steps to generating a Kulia package no matter the language, for instance, the JS library:

• First, prepare the data using this prepare.js script, which will compile the _data directory into a javascript/src/_data/main.json file (this is not committed)
• Second, the actual API implementation for this library, such as getNodeByPath which consumes the generated JSON file in step 1.

---

2. The name of the directories of the department does not make sense, the acronyms for the faculties are set by the universities and somewhat distinguishable, but once you dive into the departments you get lost.

Right, these names are not necessarily official codes, and they are not meant to be, they instead act as an identifier for each node, this is much needed if you want to store this in a database (or we export it into an SQL database file), or consider a case where, for example, we build a website that uses kuliya, and has to create a Page URL to Electrical engineering department of UMKB, we can write something like: https://www.example-academy.dz/UMKB/FST/GE/ETECH/...you-can-dive-deeper.
I would like more feedback here if you can give 🙏🏽

---

3. What is more important, is search. I can't think of an efficient way to search for, let's say, all the departments of geology amidst all the data.
4. The moment the data is stored into one main JSON file will make automating, scraping, and extracting the data from the internet much faster.

I guess I answered that as part of the first question's answer. Still, I don't have a specific way to do the search here, I thought we could just go for a library that searches JSON file content? or do we have to build our custom solution?

[NurElHuda]

@ZibanPirate Thank you for the through answer.

For the department names, I found that the used acronyms pattern is different for different scenarios:

1. "Department of Spanish, German & Italian" is "dsgi", the patter is first letter of each word except prepositions.
2. "Department of Arts" is "dart", which is the whole word.
3. "Department of English" is "den", the patter is the first two letters of the word.

I am trying to create a script that scrap and create the nested folders, so I have to set one pattern for all possibilities.
I am planning to use "letter of each word except prepositions", but it would make departments like "art" and "English" be "da" and "de", respectively. Ang suggestions for improvements?

For the search, I don't have an idea, I will look for possible solution.
On the top of my head, I find using regexp to search for words in JSON very effective, for example:

function getDepartementByName(strJSON, nameOfDept){
  DEP_REGEXP = '"name": { .*  "fr": ".*${nameOfDept}.*"  }, "type": "DEPARTMENT"'
  return [...strJSON.matchAll(regexp)];
}

Or something like this.