Merge pull request #89 from MonashDeepNeuron/dev

Dev

src/SUMMARY.md

@@ -3,6 +3,7 @@
 [Welcome](home.md)
 
 - [Installation & Set-up](./chapter1/getting-started.md)
+
   - [GitHub](./chapter1/github.md)
   - [Windows](./chapter1/windows.md)
   - [Mac](./chapter1/mac.md)
@@ -13,6 +14,7 @@
   - [Challenges](./chapter1/challenges.md)
 
 - [Intro to C](./chapter2/intro-to-c.md)
+
   - [Hello World](./chapter2/helloworld.md)
   - [Compilation](./chapter2/compilation.md)
   - [Types & Variables](./chapter2/vars.md)
@@ -25,52 +27,47 @@
   - [Challenges](./chapter2/challenges.md)
 
 - [Operating Systems](./chapter3/chapter3.md)
+
   - [Computer Architecture](./chapter3/computer-architecture.md)
-  - [Pointers & Memory](./chapter3/memory-pointers.md)
+  - [Pointers](./chapter3/memory-pointers.md)
+  - [Dynamic Memory](./chapter3/dynamic-memory.md)
+  - [Structures & Macros](./chapter3/structs-macros.md)
   - [Intro to Linux](./chapter3/linux-intro.md)
-  - [Threading & Concurrency](./chapter3/threads-concurrency.md)
-  - [Processes](./chapter3/processes.md)
-  - [Scheduling Algorithms](./chapter3/scheduling.md)
+  - [VMs & Containers](./chapter3/vms-containers.md)
   - [Challenges](./chapter3/challenges.md)
 
-- [More C](./chapter4/chapter4.md)
-  - [Dynamic Memory](./chapter4/memory.md)
-  - [Structures](./chapter4/structs.md)
-  - [Macros & The Preprocessor](./chapter4/macros.md)
-  - [System Calls](./chapter4/syscalls.md)
-  - [Spawning Processes & Threads](./chapter4/spawn-procs.md)
+- [M3 & SLURM](./chapter4/chapter4.md)
+
+  - [Batch Processing vs. Cloud Computing](./chapter4/batch-cloud.md)
+  - [Parallel & Distributed Computing](./chapter4/parallel-distributed.md)
+  - [M3 Login - SSH & Strudel](./chapter4/login.md)
+  - [Intro to SLURM](./chapter4/slurm_intro.md)
+  - [M3 Interface & Usage](./chapter4/m3-interface.md)
+  - [Software & Tooling](./chapter4/software-tooling.md)
   - [Challenges](./chapter4/challenges.md)
 
-- [M3 & SLURM](./chapter5/chapter5.md)
+- [Introduction to Parallel Computing](./chapter5/chapter5.md)
 
-  - [Batch Processing vs. Cloud Computing](./chapter5/batch-cloud.md)
-  - [Parallel & Distributed Computing](./chapter5/parallel-distributed.md)
-  - [M3 Login - SSH & Strudel](./chapter5/login.md)
-  - [Intro to SLURM](./chapter5/slurm_intro.md)
-  - [M3 Interface & Usage](./chapter5/m3-interface.md)
-  - [Software & Tooling](./chapter5/software-tooling.md)
+  - [Multithreading](./chapter5/multithreading.md)
+  - [Synchronisation](./chapter5/synchronisation.md)
+  - [Locks](./chapter5/locks.md)
+  - [Message Passing](./chapter5/message-passing.md)
   - [Challenges](./chapter5/challenges.md)
 
-- [Introduction to Parallel Computing](./chapter6/chapter6.md)
-  - [Multithreading](./chapter6/multithreading.md)
-  - [Synchronisation](./chapter6/synchronisation.md)
-  - [Locks](./chapter6/locks.md)
-  - [Message Passing](./chapter6/message-passing.md)
+- [Parallellisation of Algorithms](./chapter6/chapter6.md)
+
+  - [Parallel Search](./chapter6/parallel-search.md)
+  - [Parallel Sort](./chapter6/parallel-sort.md)
+  - [Other Parallel Algorithms](./chapter6/other-parallel-algos.md)
+  - [Machine Learning & HPC](./chapter6/machine-learning-and-hpc.md)
+  - [Optimisation Algorithms](./chapter6/optim-algos.md)
   - [Challenges](./chapter6/challenges.md)
 
-- [Parallellisation of Algorithms](./chapter7/chapter7.md)
-  - [Parallel Search](./chapter7/parallel-search.md)
-  - [Parallel Sort](./chapter7/parallel-sort.md)
-  - [Other Parallel Algorithms](./chapter7/other-parallel-algos.md)
-  - [Machine Learning & HPC](./chapter7/machine-learning-and-hpc.md)
-  - [Optimisation Algorithms](./chapter7/optim-algos.md)
+- [Apache Spark](./chapter7/chapter7.md)
+  - [Installation & Cluster Set-up](./chapter7/set-up.md)
+  - [Internal Architecture](./chapter7/internals.md)
+  - [Data Processing](./chapter7/data-processing.md)
+  - [Job Batching](./chapter7/job-batching.md)
   - [Challenges](./chapter7/challenges.md)
 
-- [Apache Spark](./chapter8/chapter8.md)
-  - [Installation & Cluster Set-up](./chapter8/set-up.md)
-  - [Internal Architecture](./chapter8/internals.md)
-  - [Data Processing](./chapter8/data-processing.md)
-  - [Job Batching](./chapter8/job-batching.md)
-  - [Challenges](./chapter8/challenges.md)
-
 [Acknowledgements](./acknowledgements.md)

src/chapter3/challenges.md

@@ -1,3 +1,19 @@
 # Challenges
 
-![under-const](../imgs/under-const.gif)
+## Challenge 1 - Sum and Product Algorithms
+
+This challenge involves implementing the sum and product reductions on an array or memory block of integers. As a bonus challenge, try and make the algorithms more generic and work with any binary operator.
+
+## Challenge 2 - Array Concatenation
+
+In this challenge you have to implement an array concatenation function. This should join two arrays of the same type into a single array, similar to `strcat()`. You will need to allocate a new block of memory and in order to store the concatenated arrays which will requires the sizes of the two input arrays to be known by the function. This function should return a pointer to the resulting array.
+
+> Note: The type of the array this function concatenates can be any type except `char`.
+
+## Challenge 3 - Doing it in Docker
+
+Pull an Ubuntu image from the Docker registry, install any required dependencies and execute the same C code that you wrote for challenge 2 within that running container instance. You will have to consult the Docker documentation and be resourceful in learning and completing this task.
+
+## Challenge 4 - Launch a VM instance on Nectar Cloud
+
+This is an extension challenge in which you will have to go through the [ARDC's cloud starter tutorial series](https://tutorials.rc.nectar.org.au/cloud-starter/01-overview) in order to launch a VM instance and connect to it.

src/chapter4/memory.md → src/chapter3/dynamic-memory.md

@@ -48,6 +48,8 @@ int main()
 }
 ```
 
+> If you recall the previous subchapter, `calloc()` allocates memory in the BSS segment instead of the heap/stack.
+
 ## Reallocated Memory
 
 We can also reallocate data to fit a larger or smaller amount. The elements from the old block will be copied to the new location until the new array is full or there are no more elements to copy. `realloc()` my not actual allocate memory in a new locating if there is free space next to the existing array. `realloc()` also works like `malloc()` where the new memory is left uninitialised. `realloc()` takes two parameters, the old pointer address and the new size.

src/chapter3/linux-intro.md

@@ -1,10 +1,100 @@
 # Introduction to Linux
 
-Linux is one of the most popular versions of the UNIX operating System. It is open source as its source code is freely available. It is free to use. Linux was designed considering UNIX compatibility. Its functionality list is quite similar to that of UNIX.
+Linux is a freely available open-source operating system built by Linux Torvalds back in 1991. It's based off of Bell Lab's development of Unix in the 1970s which also forms the basis of Android and MacOS. It's an extremely popular operating system, especially for servers (nearly all nodes in M3 use Linux). We will be learning about it and using it to gain both low-level programming skills and an understanding of Operating Systems theory.
 
-Linux Operating System has primarily three components:
-- **Kernel:** The kernel is the core part of Linux. It is responsible for all major activities of this operating system. It consists of various modules and it interacts directly with the underlying hardware. Kernel provides the required abstraction to hide low level hardware details to system or application programs.
-- **System Library:** System libraries are special functions or programs using which application programs or system utilities access Kernel’s features. These libraries implement most of the functionalities of the operating system and do not require kernel module’s code access rights.
-- **System Utility:** System Utility programs are responsible to do specialised, individual level tasks.
+There are various implementations (distributions) of Linux. We won't go into detail on them but here's a comparison of some of the popular ones.
 
-![linux-struct](./imgs/Structure-Of-Linux-Operating-System.png)
+![linux-distros](./imgs/linux-distros.png)
+
+You can think of Linux as having 3 layers or components. Here they are from the highest to the lowest level (how removed they are from the hardware):
+- **System Libraries:** System libraries are special functions or programs using which application programs or system utilities access the Kernel’s features. This is the topmost layer of the operating system. It allows access to the deeper parts of the machine without exposing them to direct user access. One such example for Linux and other Unix-like operating systems is `unistd.h` which provides C functions to access the POSIX (Portable Operating System Interface) API.
+- **Kernel:** The kernel is the core part of Linux. It is responsible for all major activities of this operating system. It consists of various modules which are all hidden and protected from the user. The only way that a user can access the kernel is through the system library. I encourage you all to check out the [Linux kernel code on GitHub](https://github.com/torvalds/linux) (you can see Linus still actively approving all the PRs into master).
+- **Device Drivers (Kernel Modules)**: If you wondered how an operating system actually controls the hardware this is how they do it. They use device drivers/kernel modules which are software programs written to act as an interface between the OS kernel and the device's firmware.
+
+![linux-struct](./imgs/Structure-Of-Linux-Operating-System.png)
+
+
+## What does an operating system actually do?
+Pretty much all interactions you have with your machine is facilitated by an operating system. I find it useful to break it down into 2 functional areas - compute (algorithms/processes/dynamic stuff) and storage (data structures/memory/static stuff).
+
+### Compute
+The operating system is what ultimately controls the CPU. Time on the CPU is a scarce resource for user applications that need to get code/instructions executed as quick as possible. Hence why compute is considered a "resource" and the OS is responsible for the fair and efficient allocation of this and all resources. 
+
+Some more terminology - a program is a file with some non-executing (static) code while a process can be thought of as a live program that's being executed on the CPU.
+
+![program-vs-process](./imgs/program-process.png)
+
+Operating systems have to enable the creation of new processes, schedule them time on a CPU, manage and keep track of processes while also handling their completion. To do this there are a lot of attributes, structures and behaviours implemented by the Linux kernel.
+
+#### Process Characteristics & Attributes
+A process has the following attributes:
+
+- **Process Id:** A unique identifier assigned by the operating system.
+- **Process State:** Can be ready, running, etc.
+- **CPU registers:** Like the Program Counter (CPU registers must be saved and restored when a process is swapped in and out of the CPU)
+- **Accounts information:** Amount of CPU used for process execution, time limits, execution ID, etc
+- **I/O (input/output) status information:** For example, devices allocated to the process, open files, etc
+- **CPU scheduling information:** For example, Priority (Different processes may have different priorities, for example, a shorter process assigned high priority in the shortest job first scheduling)
+
+A process is in one of the following states at any given time:
+
+- **New:** Newly Created Process (or) being-created process.
+- **Ready:** After the creation process moves to the Ready state, i.e. the process is ready for execution.
+- **Run:** Currently running process in CPU (only one process at a time can be under execution in a single processor)
+- **Wait (or Block):** When a process requests I/O access.
+- **Complete (or Terminated):** The process completed its execution.
+- **Suspended Ready:** When the ready queue becomes full, some processes are moved to a suspended ready state
+- **Suspended Block:** When the waiting queue becomes full.
+
+![proc-states](./imgs/process-states.png)
+
+#### Context Switching
+The process of saving the context of one process and loading the context of another process is known as Context Switching. In simple terms, it is unloading a running process into the ready state in order to load another ready process into the running state.
+
+*When Does Context Switching Happen?*
+
+1. When a high-priority process comes to a ready state (i.e. with higher priority than the running process). 
+2. An Interrupt occurs (some I/O device tells the kernel that it needs CPU time).
+3. User and kernel-mode switch.
+4. Preemptive CPU scheduling is used (context switches at regular time intervals).
+
+There is a lot more involved in how compute is managed by the OS (eg. process scheduling, threading, etc...) which will be covered in a later chapter.
+
+### Storage
+If you recall chapter 3.1, this area can be further subdivided into two - temporary storage (main memory i.e. RAM) and permenant storage (hard drives and SSDs). 
+
+#### Linux File Systems
+As you all know, computers manage the permenant storage of information using a system of files and directories. The Linux file system is a multifaceted structure comprised of three essential layers. At its foundation, the **Logical File System** serves as the interface between user applications and the file system, managing operations like opening, reading, and closing files. Below this layer, the **Virtual File System** facilitates the concurrent operation of multiple physical file systems, providing a standardized interface for compatibility. Finally, the **Physical File System** is responsible for the tangible management and storage of physical memory blocks on the disk, ensuring efficient data allocation and retrieval. Together, these layers form a cohesive architecture, orchestrating the organized and efficient handling of data in the Linux operating system.
+
+![linux-file-sys](./imgs/file-system-arch.png)
+
+#### Paging & Memory Allocation
+
+Paging is a memory management technique in operating systems that enables processes to access more memory than is physically available. The system improves performance and resource utilization using virtual memory. A process has access to the pages it needs without waiting for them to be loaded into physical memory. The technique stores and retrieves data from a computer's secondary or virtual storage (hard drive, SSD, etc.) to the primary storage (RAM). 
+
+When a process tries to access a page that is not in RAM, the OS raises a page fault and brings in the page from virtual memory.
+
+![paging](./imgs/paging-basic-scheme.jpg)
+
+Paging improves the efficiency of memory management. By dividing memory into pages, the operating system moves pages in and out of memory as needed. Keeping only the frequently used pages reduces the number of page faults, which improves system performance and responsiveness. This is a key HPC optimisation concept known as **locality of reference**.
+
+#### Cache Optimisation
+A lot of you must be familiar with the concept of caching. It basically means storing data temporarily in an easily accessible place in order to be more efficient when accessing it. Nearly all modern PCs use caches for efficiency. If you recall the memory heirarchy in chapter 3.1, caches sit between CPU registers and main memory (RAM) in terms of speed and cost. There are usually 3 levels of caches (depending on computer architecture) - L1, L2 and L3 with L1 being the smallest, most expensive, fastest and closest to the CPU.
+
+![cpu-caches](./imgs/cpu-cache.jpg)
+
+In the above figure, you can see that the CPU wants to read or fetch the data or instruction. First, it will access the cache memory as it is near to it and provides very fast access. If the required data or instruction is found, it will be fetched. This situation is known as a cache hit. But if the required data or instruction is not found in the cache memory then this situation is known as a cache miss.
+
+![cpu-caches2](./imgs/cache-hit-miss.jpg)
+
+The aim is to store data that any given process is likely to access in the future, in the cache. Cache optimisation involves minimising the no. of cache misses while maximizing cache hits. The benefits are obvious - reduced memory access times resulting in a faster program. Cache optimisation is done by implementing locality of reference and there are two localities:
+
+1. **Temporal locality** is when current data or instruction that is being fetched frequently may be needed soon. It's based on the same assumption that if a program is accessing the same location (using pointers) again and again then it's likely to access it in the immediate future as well.
+
+2. **Spatial locality**, on the other hand, assumes that memory addresses that are closer to currently accessed addresses are more likely to be accessed again.
+
+![localities](./imgs/spatial-vs-temporal.gif)
+
+### Accessing the Kernel's API
+
+As mentioned earlier, user space programs (code that a programmer writes for an application/script) will need to use a system library to access the kernel and it's lower-level functionality. For Linux, the main library is `unistd.h` which only runs on POSIX-compatible (Unix-like) operating systems and unfortunately Windows is not one of them. To get around this, we will be using a Docker container with an Ubuntu image. But first let's finish this chapter by learning about Virtual Machines and Containers.