diff --git a/src/.chapter7/challenges.md b/src/.chapter7/challenges.md deleted file mode 100644 index 9358534..0000000 --- a/src/.chapter7/challenges.md +++ /dev/null @@ -1 +0,0 @@ -# Challenges diff --git a/src/SUMMARY.md b/src/SUMMARY.md index cedb798..248055a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -2,72 +2,75 @@ [Welcome](home.md) -- [Installation & Set-up](./chapter1/getting-started.md) +- [Installation & Set-up](./getting-started/getting-started.md) - - [GitHub](./chapter1/github.md) - - [Windows](./chapter1/windows.md) - - [Mac](./chapter1/mac.md) - - [Linux](./chapter1/linux.md) - - [WSL](./chapter1/wsl.md) - - [M3 MASSIVE](./chapter1/m3.md) - - [Nectar Cloud](./chapter1/nectar.md) - - [Challenges](./chapter1/challenges.md) + - [GitHub](./getting-started/github.md) + - [Windows](./getting-started/windows.md) + - [Mac](./getting-started/mac.md) + - [Linux](./getting-started/linux.md) + - [WSL](./getting-started/wsl.md) + - [M3 MASSIVE](./getting-started/m3.md) + - [Nectar Cloud](./getting-started/nectar.md) + - [Challenges](./getting-started/challenges.md) -- [Intro to C](./chapter2/intro-to-c.md) +- [Intro to C](./intro-to-c/intro-to-c.md) - - [Hello World](./chapter2/helloworld.md) - - [Compilation](./chapter2/compilation.md) - - [Types & Variables](./chapter2/vars.md) - - [Printing](./chapter2/printing.md) - - [Input](./chapter2/input.md) - - [Arrays & Strings](./chapter2/array.md) - - [Control Flow](./chapter2/ctrl-flow.md) - - [Loops](./chapter2/loops.md) - - [Functions](./chapter2/functions.md) - - [Challenges](./chapter2/challenges.md) + - [Hello World](./intro-to-c/helloworld.md) + - [Compilation](./intro-to-c/compilation.md) + - [Types & Variables](./intro-to-c/vars.md) + - [Printing](./intro-to-c/printing.md) + - [Input](./intro-to-c/input.md) + - [Arrays & Strings](./intro-to-c/array.md) + - [Control Flow](./intro-to-c/ctrl-flow.md) + - [Loops](./intro-to-c/loops.md) + - [Functions](./intro-to-c/functions.md) + - [Challenges](./intro-to-c/challenges.md) -- [Operating Systems](./chapter3/chapter3.md) +- [Operating Systems](./operating-systems/operating-systems.md) - - [Computer Architecture](./chapter3/computer-architecture.md) - - [Pointers](./chapter3/memory-pointers.md) - - [Dynamic Memory](./chapter3/dynamic-memory.md) - - [Structures & Macros](./chapter3/structs-macros.md) - - [Intro to Linux](./chapter3/linux-intro.md) - - [VMs & Containers](./chapter3/vms-containers.md) - - [Challenges](./chapter3/challenges.md) + - [Computer Architecture](./operating-systems/computer-architecture.md) + - [Pointers](./operating-systems/memory-pointers.md) + - [Dynamic Memory](./operating-systems/dynamic-memory.md) + - [Structures & Macros](./operating-systems/structs-macros.md) + - [Intro to Linux](./operating-systems/linux-intro.md) + - [VMs & Containers](./operating-systems/vms-containers.md) + - [Bash CMDs & Scripts](./operating-systems/bash-cmds-scripts.md) + - [Processes & Threads](./operating-systems/processes-threads.md) + - [Scheduling Algorithms](./operating-systems/scheduling.md) + - [Challenges](./operating-systems/challenges.md) -- [M3 & SLURM](./chapter4/chapter4.md) +- [M3 & SLURM](./m3-slurm/m3-slurm.md) - - [Batch Processing vs. Cloud Computing](./chapter4/batch-cloud.md) - - [Parallel & Distributed Computing](./chapter4/parallel-distributed.md) - - [M3 Login - SSH & Strudel](./chapter4/login.md) - - [Intro to SLURM](./chapter4/slurm_intro.md) - - [M3 Interface & Usage](./chapter4/m3-interface.md) - - [Software & Tooling](./chapter4/software-tooling.md) - - [Challenges](./chapter4/challenges.md) + - [Batch Processing vs. Cloud Computing](./m3-slurm/batch-cloud.md) + - [Parallel & Distributed Computing](./m3-slurm/parallel-distributed.md) + - [M3 Login - SSH & Strudel](./m3-slurm/login.md) + - [Intro to SLURM](./m3-slurm/slurm_intro.md) + - [M3 Interface & Usage](./m3-slurm/m3-interface.md) + - [Software & Tooling](./m3-slurm/software-tooling.md) + - [Challenges](./m3-slurm/challenges.md) -- [Introduction to Parallel Computing](./chapter5/chapter5.md) +- [Intro to Parallel Computing](./intro-to-parallel-comp/intro-to-parallel-comp.md) - - [Multithreading](./chapter5/multithreading.md) - - [Synchronisation](./chapter5/synchronisation.md) - - [Locks](./chapter5/locks.md) - - [Message Passing](./chapter5/message-passing.md) - - [Challenges](./chapter5/challenges.md) + - [OpenMP: Multithreading](./intro-to-parallel-comp/multithreading.md) + - [Synchronisation Issues](./intro-to-parallel-comp/synchronisation.md) + - [Dead & Live Locks](./intro-to-parallel-comp/locks.md) + - [MPI: Message Passing](./intro-to-parallel-comp/message-passing.md) + - [Challenges](./intro-to-parallel-comp/challenges.md) -- [Parallellisation of Algorithms](./chapter6/chapter6.md) +- [Parallellisation of Algorithms](./parallel-algos/parallel-algos.md) - - [Parallel Search](./chapter6/parallel-search.md) - - [Parallel Sort](./chapter6/parallel-sort.md) - - [Other Parallel Algorithms](./chapter6/other-parallel-algos.md) - - [Machine Learning & HPC](./chapter6/machine-learning-and-hpc.md) - - [Optimisation Algorithms](./chapter6/optim-algos.md) - - [Challenges](./chapter6/challenges.md) + - [Parallel Search](./parallel-algos/parallel-search.md) + - [Parallel Sort](./parallel-algos/parallel-sort.md) + - [Other Parallel Algorithms](./parallel-algos/other-parallel-algos.md) + - [Machine Learning & HPC](./parallel-algos/machine-learning-and-hpc.md) + - [Optimisation Algorithms](./parallel-algos/optim-algos.md) + - [Challenges](./parallel-algos/challenges.md) -- [Apache Spark](./chapter7/chapter7.md) - - [Installation & Cluster Set-up](./chapter7/set-up.md) - - [Internal Architecture](./chapter7/internals.md) - - [Data Processing](./chapter7/data-processing.md) - - [Job Batching](./chapter7/job-batching.md) - - [Challenges](./chapter7/challenges.md) +- [Apache Spark](./apache-spark/apache-spark.md) + - [Installation & Cluster Set-up](./apache-spark/set-up.md) + - [Internal Architecture](./apache-spark/internals.md) + - [Data Processing](./apache-spark/data-processing.md) + - [Job Batching](./apache-spark/job-batching.md) + - [Challenges](./apache-spark/challenges.md) [Acknowledgements](./acknowledgements.md) \ No newline at end of file diff --git a/src/chapter7/chapter7.md b/src/apache-spark/apache-spark.md similarity index 100% rename from src/chapter7/chapter7.md rename to src/apache-spark/apache-spark.md diff --git a/src/chapter7/challenges.md b/src/apache-spark/challenges.md similarity index 100% rename from src/chapter7/challenges.md rename to src/apache-spark/challenges.md diff --git a/src/chapter7/data-processing.md b/src/apache-spark/data-processing.md similarity index 100% rename from src/chapter7/data-processing.md rename to src/apache-spark/data-processing.md diff --git a/src/chapter7/imgs/jupyterlab.png b/src/apache-spark/imgs/jupyterlab.png similarity index 100% rename from src/chapter7/imgs/jupyterlab.png rename to src/apache-spark/imgs/jupyterlab.png diff --git a/src/chapter7/imgs/spark-architecture.png b/src/apache-spark/imgs/spark-architecture.png similarity index 100% rename from src/chapter7/imgs/spark-architecture.png rename to src/apache-spark/imgs/spark-architecture.png diff --git a/src/chapter7/imgs/spark-cluster-overview.png b/src/apache-spark/imgs/spark-cluster-overview.png similarity index 100% rename from src/chapter7/imgs/spark-cluster-overview.png rename to src/apache-spark/imgs/spark-cluster-overview.png diff --git a/src/chapter7/imgs/spark-sql.png b/src/apache-spark/imgs/spark-sql.png similarity index 100% rename from src/chapter7/imgs/spark-sql.png rename to src/apache-spark/imgs/spark-sql.png diff --git a/src/chapter7/internals.md b/src/apache-spark/internals.md similarity index 100% rename from src/chapter7/internals.md rename to src/apache-spark/internals.md diff --git a/src/chapter7/job-batching.md b/src/apache-spark/job-batching.md similarity index 100% rename from src/chapter7/job-batching.md rename to src/apache-spark/job-batching.md diff --git a/src/chapter7/set-up.md b/src/apache-spark/set-up.md similarity index 100% rename from src/chapter7/set-up.md rename to src/apache-spark/set-up.md diff --git a/src/chapter3/processes.md b/src/chapter3/processes.md deleted file mode 100644 index ce5138d..0000000 --- a/src/chapter3/processes.md +++ /dev/null @@ -1,3 +0,0 @@ -# Processes - -![under-const](../imgs/under-const.gif) \ No newline at end of file diff --git a/src/chapter3/scheduling.md b/src/chapter3/scheduling.md deleted file mode 100644 index ddc6525..0000000 --- a/src/chapter3/scheduling.md +++ /dev/null @@ -1,3 +0,0 @@ -# Scheduling Algorithms - -![under-const](../imgs/under-const.gif) \ No newline at end of file diff --git a/src/chapter3/threads-concurrency.md b/src/chapter3/threads-concurrency.md deleted file mode 100644 index b92708b..0000000 --- a/src/chapter3/threads-concurrency.md +++ /dev/null @@ -1,3 +0,0 @@ -# Threading & Concurrency - -![under-const](../imgs/under-const.gif) \ No newline at end of file diff --git a/src/chapter3/vms-containers.md b/src/chapter3/vms-containers.md deleted file mode 100644 index 95d5fb4..0000000 --- a/src/chapter3/vms-containers.md +++ /dev/null @@ -1,42 +0,0 @@ -# VMs & Containers - -## What is a virtual machine (VM)? - -A virtual machine is not a physical machine. It’s a file that replicates the computing environment of a physical machine. It’s similar to how virtual reality (VR) environments replicate the real world. VR isn’t a physical space; it’s a virtual imitation. Still, we can perform real-world functions in VR, such as exploring and interacting with objects. Instead of imitating video game functions, virtual machine software emulates computer system functions i.e. the operating system. To achieve this VMs use a technology called **virtualisation** as shown below. - -![vms](./imgs/vms.png) - -At the base, you have the host hardware and OS. This is the physical machine that is used to create the virtual machines. On top of this, you have the hypervisor. This allows multiple virtual machines, each with their own operating systems (OS), to run on a single physical server. - -VMs have a few downsides, though, which containers address. Two downsides particularly stand out: - -- **VMs consume more resources:** VMs have a higher resource overhead due to the need to run a full OS instance for each VM. This can lead to larger memory and storage consumption. This in turn can have a negative effect on performance and startup times of the virtual machine. -- **Portability:** VMs are typically less portable due to differences in underlying OS environments. Moving VMs between different hypervisors or cloud providers can be more complex. - -## What is a container? - -A container is a lightweight, standalone, and executable software package that includes everything needed to run a piece of software, including the code, runtime, system tools, and libraries. - -Containers are designed to isolate applications and their dependencies, ensuring that they can run consistently across different environments. Whether the application is running from your computer or in the cloud, the application behaviour remains the same. - -Unlike VMs which virtualise the hardware, containers *virtualise the operating system*. This simply means that a container uses a single OS to create a virtual application and its libraries. Containers run on top of a shared OS provided by the host system. - -![containers](./imgs/containers.png) - -The container engine allows you to spin up containers. It provides the tools and services necessary for building, running, and deploying containerised applications. - -## Docker -As you might know, Docker is an open-source containerization platform by which you can pack your application and all its dependencies into a standardized unit called a container. Let's clarify some Docker terminology, - -- **Docker Image**: An image is an inert, immutable, file that's essentially a snapshot of a container. Images are created with the [build](https://docs.docker.com/reference/cli/docker/image/build/) command, and they'll produce a container when started with [run](https://docs.docker.com/reference/cli/docker/container/run/). Images are stored in a Docker registry such as [registry.hub.docker.com](https://registry.hub.docker.com). -- **Docker Container**: To use a programming metaphor, if an image is a class, then a container is an instance of a class—a runtime object. Multiple containers can run from the same image simultaneously. -- **Docker Daemon**: The Docker daemon (`dockerd`) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. -- **Docker Client**: The Docker client (`docker`) is the primary way that many Docker users interact with Docker. When you use commands such as `docker run`, the client sends these commands to `dockerd`, which carries them out. -- **Docker Desktop**: This is an easy-to-install application for your Mac, Windows or Linux environment that enables you to build and share containerized applications and microservices. - -### Docker Architecture -Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. - -![docker-arch](./imgs/docker-architecture.jpg) - -Go through [this guide](https://www.docker.com/get-started/) to download and install Docker. If you'd like to learn more about it's functionality and how to work with it, [check out their documentation website](https://docs.docker.com/get-started/overview/). \ No newline at end of file diff --git a/src/chapter4/job-scripting.md b/src/chapter4/job-scripting.md deleted file mode 100644 index 1777b9d..0000000 --- a/src/chapter4/job-scripting.md +++ /dev/null @@ -1 +0,0 @@ -# Job Scripting diff --git a/src/chapter1/aaf.png b/src/getting-started/aaf.png similarity index 100% rename from src/chapter1/aaf.png rename to src/getting-started/aaf.png diff --git a/src/chapter1/challenges.md b/src/getting-started/challenges.md similarity index 100% rename from src/chapter1/challenges.md rename to src/getting-started/challenges.md diff --git a/src/chapter1/getting-started.md b/src/getting-started/getting-started.md similarity index 100% rename from src/chapter1/getting-started.md rename to src/getting-started/getting-started.md diff --git a/src/chapter1/github.md b/src/getting-started/github.md similarity index 100% rename from src/chapter1/github.md rename to src/getting-started/github.md diff --git a/src/chapter1/hpcid.png b/src/getting-started/hpcid.png similarity index 100% rename from src/chapter1/hpcid.png rename to src/getting-started/hpcid.png diff --git a/src/chapter1/join_project.png b/src/getting-started/join_project.png similarity index 100% rename from src/chapter1/join_project.png rename to src/getting-started/join_project.png diff --git a/src/chapter1/linux.md b/src/getting-started/linux.md similarity index 100% rename from src/chapter1/linux.md rename to src/getting-started/linux.md diff --git a/src/chapter1/m3.md b/src/getting-started/m3.md similarity index 100% rename from src/chapter1/m3.md rename to src/getting-started/m3.md diff --git a/src/chapter1/mac.md b/src/getting-started/mac.md similarity index 100% rename from src/chapter1/mac.md rename to src/getting-started/mac.md diff --git a/src/chapter1/nectar-login.png b/src/getting-started/nectar-login.png similarity index 100% rename from src/chapter1/nectar-login.png rename to src/getting-started/nectar-login.png diff --git a/src/chapter1/nectar.md b/src/getting-started/nectar.md similarity index 100% rename from src/chapter1/nectar.md rename to src/getting-started/nectar.md diff --git a/src/chapter1/taskmanager.png b/src/getting-started/taskmanager.png similarity index 100% rename from src/chapter1/taskmanager.png rename to src/getting-started/taskmanager.png diff --git a/src/chapter1/windows.md b/src/getting-started/windows.md similarity index 100% rename from src/chapter1/windows.md rename to src/getting-started/windows.md diff --git a/src/chapter1/wsl.md b/src/getting-started/wsl.md similarity index 100% rename from src/chapter1/wsl.md rename to src/getting-started/wsl.md diff --git a/src/chapter2/array.md b/src/intro-to-c/array.md similarity index 100% rename from src/chapter2/array.md rename to src/intro-to-c/array.md diff --git a/src/chapter2/challenges.md b/src/intro-to-c/challenges.md similarity index 100% rename from src/chapter2/challenges.md rename to src/intro-to-c/challenges.md diff --git a/src/chapter2/compilation.md b/src/intro-to-c/compilation.md similarity index 100% rename from src/chapter2/compilation.md rename to src/intro-to-c/compilation.md diff --git a/src/chapter2/ctrl-flow.md b/src/intro-to-c/ctrl-flow.md similarity index 100% rename from src/chapter2/ctrl-flow.md rename to src/intro-to-c/ctrl-flow.md diff --git a/src/chapter2/functions.md b/src/intro-to-c/functions.md similarity index 100% rename from src/chapter2/functions.md rename to src/intro-to-c/functions.md diff --git a/src/chapter2/helloworld.md b/src/intro-to-c/helloworld.md similarity index 100% rename from src/chapter2/helloworld.md rename to src/intro-to-c/helloworld.md diff --git a/src/chapter2/imgs/lang-family-tree-light.png b/src/intro-to-c/imgs/lang-family-tree-light.png similarity index 100% rename from src/chapter2/imgs/lang-family-tree-light.png rename to src/intro-to-c/imgs/lang-family-tree-light.png diff --git a/src/chapter2/imgs/lang-family-tree.png b/src/intro-to-c/imgs/lang-family-tree.png similarity index 100% rename from src/chapter2/imgs/lang-family-tree.png rename to src/intro-to-c/imgs/lang-family-tree.png diff --git a/src/chapter2/input.md b/src/intro-to-c/input.md similarity index 100% rename from src/chapter2/input.md rename to src/intro-to-c/input.md diff --git a/src/chapter2/intro-to-c.md b/src/intro-to-c/intro-to-c.md similarity index 100% rename from src/chapter2/intro-to-c.md rename to src/intro-to-c/intro-to-c.md diff --git a/src/chapter2/loops.md b/src/intro-to-c/loops.md similarity index 100% rename from src/chapter2/loops.md rename to src/intro-to-c/loops.md diff --git a/src/chapter2/printing.md b/src/intro-to-c/printing.md similarity index 100% rename from src/chapter2/printing.md rename to src/intro-to-c/printing.md diff --git a/src/chapter2/structs.md b/src/intro-to-c/structs.md similarity index 100% rename from src/chapter2/structs.md rename to src/intro-to-c/structs.md diff --git a/src/chapter2/vars.md b/src/intro-to-c/vars.md similarity index 100% rename from src/chapter2/vars.md rename to src/intro-to-c/vars.md diff --git a/src/chapter5/challenges.md b/src/intro-to-parallel-comp/challenges.md similarity index 100% rename from src/chapter5/challenges.md rename to src/intro-to-parallel-comp/challenges.md diff --git a/src/chapter5/imgs/barrier-end.png b/src/intro-to-parallel-comp/imgs/barrier-end.png similarity index 100% rename from src/chapter5/imgs/barrier-end.png rename to src/intro-to-parallel-comp/imgs/barrier-end.png diff --git a/src/chapter5/imgs/barrier-wait.png b/src/intro-to-parallel-comp/imgs/barrier-wait.png similarity index 100% rename from src/chapter5/imgs/barrier-wait.png rename to src/intro-to-parallel-comp/imgs/barrier-wait.png diff --git a/src/chapter5/imgs/barrier.png b/src/intro-to-parallel-comp/imgs/barrier.png similarity index 100% rename from src/chapter5/imgs/barrier.png rename to src/intro-to-parallel-comp/imgs/barrier.png diff --git a/src/chapter5/imgs/deadlock.png b/src/intro-to-parallel-comp/imgs/deadlock.png similarity index 100% rename from src/chapter5/imgs/deadlock.png rename to src/intro-to-parallel-comp/imgs/deadlock.png diff --git a/src/chapter5/imgs/explicit-barrier.png b/src/intro-to-parallel-comp/imgs/explicit-barrier.png similarity index 100% rename from src/chapter5/imgs/explicit-barrier.png rename to src/intro-to-parallel-comp/imgs/explicit-barrier.png diff --git a/src/chapter5/imgs/fork-join.png b/src/intro-to-parallel-comp/imgs/fork-join.png similarity index 100% rename from src/chapter5/imgs/fork-join.png rename to src/intro-to-parallel-comp/imgs/fork-join.png diff --git a/src/chapter5/imgs/htop.png b/src/intro-to-parallel-comp/imgs/htop.png similarity index 100% rename from src/chapter5/imgs/htop.png rename to src/intro-to-parallel-comp/imgs/htop.png diff --git a/src/chapter5/imgs/mpi-routines.png b/src/intro-to-parallel-comp/imgs/mpi-routines.png similarity index 100% rename from src/chapter5/imgs/mpi-routines.png rename to src/intro-to-parallel-comp/imgs/mpi-routines.png diff --git a/src/chapter5/imgs/one-thread-counter.png b/src/intro-to-parallel-comp/imgs/one-thread-counter.png similarity index 100% rename from src/chapter5/imgs/one-thread-counter.png rename to src/intro-to-parallel-comp/imgs/one-thread-counter.png diff --git a/src/chapter5/imgs/program-structure.png b/src/intro-to-parallel-comp/imgs/program-structure.png similarity index 100% rename from src/chapter5/imgs/program-structure.png rename to src/intro-to-parallel-comp/imgs/program-structure.png diff --git a/src/chapter5/imgs/time.png b/src/intro-to-parallel-comp/imgs/time.png similarity index 100% rename from src/chapter5/imgs/time.png rename to src/intro-to-parallel-comp/imgs/time.png diff --git a/src/chapter5/imgs/two-threads-counter.png b/src/intro-to-parallel-comp/imgs/two-threads-counter.png similarity index 100% rename from src/chapter5/imgs/two-threads-counter.png rename to src/intro-to-parallel-comp/imgs/two-threads-counter.png diff --git a/src/chapter5/chapter5.md b/src/intro-to-parallel-comp/intro-to-parallel-comp.md similarity index 100% rename from src/chapter5/chapter5.md rename to src/intro-to-parallel-comp/intro-to-parallel-comp.md diff --git a/src/chapter5/locks.md b/src/intro-to-parallel-comp/locks.md similarity index 100% rename from src/chapter5/locks.md rename to src/intro-to-parallel-comp/locks.md diff --git a/src/chapter5/message-passing.md b/src/intro-to-parallel-comp/message-passing.md similarity index 100% rename from src/chapter5/message-passing.md rename to src/intro-to-parallel-comp/message-passing.md diff --git a/src/chapter5/multithreading.md b/src/intro-to-parallel-comp/multithreading.md similarity index 100% rename from src/chapter5/multithreading.md rename to src/intro-to-parallel-comp/multithreading.md diff --git a/src/chapter5/synchronisation.md b/src/intro-to-parallel-comp/synchronisation.md similarity index 100% rename from src/chapter5/synchronisation.md rename to src/intro-to-parallel-comp/synchronisation.md diff --git a/src/chapter4/batch-cloud.md b/src/m3-slurm/batch-cloud.md similarity index 100% rename from src/chapter4/batch-cloud.md rename to src/m3-slurm/batch-cloud.md diff --git a/src/chapter4/challenges.md b/src/m3-slurm/challenges.md similarity index 100% rename from src/chapter4/challenges.md rename to src/m3-slurm/challenges.md diff --git a/src/chapter4/imgs/aaf.png b/src/m3-slurm/imgs/aaf.png similarity index 100% rename from src/chapter4/imgs/aaf.png rename to src/m3-slurm/imgs/aaf.png diff --git a/src/chapter4/imgs/aaf_strudel.png b/src/m3-slurm/imgs/aaf_strudel.png similarity index 100% rename from src/chapter4/imgs/aaf_strudel.png rename to src/m3-slurm/imgs/aaf_strudel.png diff --git a/src/chapter4/imgs/auth_strudel.png b/src/m3-slurm/imgs/auth_strudel.png similarity index 100% rename from src/chapter4/imgs/auth_strudel.png rename to src/m3-slurm/imgs/auth_strudel.png diff --git a/src/chapter4/imgs/batch-processing.jpeg b/src/m3-slurm/imgs/batch-processing.jpeg similarity index 100% rename from src/chapter4/imgs/batch-processing.jpeg rename to src/m3-slurm/imgs/batch-processing.jpeg diff --git a/src/chapter4/imgs/data_parallelism.jpg b/src/m3-slurm/imgs/data_parallelism.jpg similarity index 100% rename from src/chapter4/imgs/data_parallelism.jpg rename to src/m3-slurm/imgs/data_parallelism.jpg diff --git a/src/chapter4/imgs/distributed_memory_architecture.png b/src/m3-slurm/imgs/distributed_memory_architecture.png similarity index 100% rename from src/chapter4/imgs/distributed_memory_architecture.png rename to src/m3-slurm/imgs/distributed_memory_architecture.png diff --git a/src/chapter4/imgs/distributed_memory_architecture_2.png b/src/m3-slurm/imgs/distributed_memory_architecture_2.png similarity index 100% rename from src/chapter4/imgs/distributed_memory_architecture_2.png rename to src/m3-slurm/imgs/distributed_memory_architecture_2.png diff --git a/src/chapter4/imgs/distributed_vs_shared.png b/src/m3-slurm/imgs/distributed_vs_shared.png similarity index 100% rename from src/chapter4/imgs/distributed_vs_shared.png rename to src/m3-slurm/imgs/distributed_vs_shared.png diff --git a/src/chapter4/imgs/filezilla_connect_m3.png b/src/m3-slurm/imgs/filezilla_connect_m3.png similarity index 100% rename from src/chapter4/imgs/filezilla_connect_m3.png rename to src/m3-slurm/imgs/filezilla_connect_m3.png diff --git a/src/chapter4/imgs/filezilla_sitemanager.png b/src/m3-slurm/imgs/filezilla_sitemanager.png similarity index 100% rename from src/chapter4/imgs/filezilla_sitemanager.png rename to src/m3-slurm/imgs/filezilla_sitemanager.png diff --git a/src/chapter4/imgs/gurobi.png b/src/m3-slurm/imgs/gurobi.png similarity index 100% rename from src/chapter4/imgs/gurobi.png rename to src/m3-slurm/imgs/gurobi.png diff --git a/src/chapter4/imgs/gurobi2.png b/src/m3-slurm/imgs/gurobi2.png similarity index 100% rename from src/chapter4/imgs/gurobi2.png rename to src/m3-slurm/imgs/gurobi2.png diff --git a/src/chapter4/imgs/htop.png b/src/m3-slurm/imgs/htop.png similarity index 100% rename from src/chapter4/imgs/htop.png rename to src/m3-slurm/imgs/htop.png diff --git a/src/chapter4/imgs/interactive-processing.png b/src/m3-slurm/imgs/interactive-processing.png similarity index 100% rename from src/chapter4/imgs/interactive-processing.png rename to src/m3-slurm/imgs/interactive-processing.png diff --git a/src/chapter4/imgs/login-compute-nodes.jpeg b/src/m3-slurm/imgs/login-compute-nodes.jpeg similarity index 100% rename from src/chapter4/imgs/login-compute-nodes.jpeg rename to src/m3-slurm/imgs/login-compute-nodes.jpeg diff --git a/src/chapter4/imgs/memory_architectures.jpg b/src/m3-slurm/imgs/memory_architectures.jpg similarity index 100% rename from src/chapter4/imgs/memory_architectures.jpg rename to src/m3-slurm/imgs/memory_architectures.jpg diff --git a/src/chapter4/imgs/mpi_datatypes.png b/src/m3-slurm/imgs/mpi_datatypes.png similarity index 100% rename from src/chapter4/imgs/mpi_datatypes.png rename to src/m3-slurm/imgs/mpi_datatypes.png diff --git a/src/chapter4/imgs/mpi_routines.png b/src/m3-slurm/imgs/mpi_routines.png similarity index 100% rename from src/chapter4/imgs/mpi_routines.png rename to src/m3-slurm/imgs/mpi_routines.png diff --git a/src/chapter4/imgs/parallel-distributed.png b/src/m3-slurm/imgs/parallel-distributed.png similarity index 100% rename from src/chapter4/imgs/parallel-distributed.png rename to src/m3-slurm/imgs/parallel-distributed.png diff --git a/src/chapter4/imgs/parallel_computing_arrays_eg.png b/src/m3-slurm/imgs/parallel_computing_arrays_eg.png similarity index 100% rename from src/chapter4/imgs/parallel_computing_arrays_eg.png rename to src/m3-slurm/imgs/parallel_computing_arrays_eg.png diff --git a/src/chapter4/imgs/parallel_scalability.jpg b/src/m3-slurm/imgs/parallel_scalability.jpg similarity index 100% rename from src/chapter4/imgs/parallel_scalability.jpg rename to src/m3-slurm/imgs/parallel_scalability.jpg diff --git a/src/chapter4/imgs/ping_pong.png b/src/m3-slurm/imgs/ping_pong.png similarity index 100% rename from src/chapter4/imgs/ping_pong.png rename to src/m3-slurm/imgs/ping_pong.png diff --git a/src/chapter4/imgs/putty_key_not_cached.png b/src/m3-slurm/imgs/putty_key_not_cached.png similarity index 100% rename from src/chapter4/imgs/putty_key_not_cached.png rename to src/m3-slurm/imgs/putty_key_not_cached.png diff --git a/src/chapter4/imgs/putty_start.png b/src/m3-slurm/imgs/putty_start.png similarity index 100% rename from src/chapter4/imgs/putty_start.png rename to src/m3-slurm/imgs/putty_start.png diff --git a/src/chapter4/imgs/scale-out-up.png b/src/m3-slurm/imgs/scale-out-up.png similarity index 100% rename from src/chapter4/imgs/scale-out-up.png rename to src/m3-slurm/imgs/scale-out-up.png diff --git a/src/chapter4/imgs/slurm-arch.gif b/src/m3-slurm/imgs/slurm-arch.gif similarity index 100% rename from src/chapter4/imgs/slurm-arch.gif rename to src/m3-slurm/imgs/slurm-arch.gif diff --git a/src/m3-slurm/imgs/slurm-on-m3.png b/src/m3-slurm/imgs/slurm-on-m3.png new file mode 100644 index 0000000..e3547ff Binary files /dev/null and b/src/m3-slurm/imgs/slurm-on-m3.png differ diff --git a/src/chapter4/imgs/strudel1.png b/src/m3-slurm/imgs/strudel1.png similarity index 100% rename from src/chapter4/imgs/strudel1.png rename to src/m3-slurm/imgs/strudel1.png diff --git a/src/chapter4/imgs/strudel2.png b/src/m3-slurm/imgs/strudel2.png similarity index 100% rename from src/chapter4/imgs/strudel2.png rename to src/m3-slurm/imgs/strudel2.png diff --git a/src/chapter4/imgs/strudel_home.png b/src/m3-slurm/imgs/strudel_home.png similarity index 100% rename from src/chapter4/imgs/strudel_home.png rename to src/m3-slurm/imgs/strudel_home.png diff --git a/src/chapter4/imgs/task_parallelism.jpg b/src/m3-slurm/imgs/task_parallelism.jpg similarity index 100% rename from src/chapter4/imgs/task_parallelism.jpg rename to src/m3-slurm/imgs/task_parallelism.jpg diff --git a/src/chapter4/imgs/time.png b/src/m3-slurm/imgs/time.png similarity index 100% rename from src/chapter4/imgs/time.png rename to src/m3-slurm/imgs/time.png diff --git a/src/chapter4/login.md b/src/m3-slurm/login.md similarity index 100% rename from src/chapter4/login.md rename to src/m3-slurm/login.md diff --git a/src/chapter4/m3-interface.md b/src/m3-slurm/m3-interface.md similarity index 100% rename from src/chapter4/m3-interface.md rename to src/m3-slurm/m3-interface.md diff --git a/src/chapter4/chapter4.md b/src/m3-slurm/m3-slurm.md similarity index 100% rename from src/chapter4/chapter4.md rename to src/m3-slurm/m3-slurm.md diff --git a/src/chapter4/parallel-distributed.md b/src/m3-slurm/parallel-distributed.md similarity index 62% rename from src/chapter4/parallel-distributed.md rename to src/m3-slurm/parallel-distributed.md index ecca38b..b645e0a 100644 --- a/src/chapter4/parallel-distributed.md +++ b/src/m3-slurm/parallel-distributed.md @@ -6,7 +6,7 @@ Nearly all modern computer systems utilise parallel computing to speed up the ex As you can see, in a scenario where a program (job) takes 3 seconds and 3 independent jobs have to be executed by a system, doing it serially in a single computer takes a total of 9 seconds. But doing it simultaneously across 3 computers will only take 3 seconds thus achieving a 3x speedup through parallel computing. -This is the fundamental principle that High Performance Computing is based on. +This is the fundamental principle that High Performance Computing is based on. The trouble (or fun) is when your tasks have dependencies on each other which is gonna be the case for the vast majority of algorithms. That's when things like synchronisation issues, data sharing and all of that comes into play - which we'll explore in later chapters. ## What is Distributed Computing? @@ -52,4 +52,19 @@ And finally, not everything needs to be done on a parallel or distributed system ### Advantages of serial computing: - **More simple** to design & implement algorithms. Parallel algorithms can get quite complex, especially when dealing with more complicated instructions with dependencies. - **Less overhead** involved in managing a parallel & distributed job. No need to manage data sharing between threads, processes, nodes, etc... -- **No synchronisation issues** & headaches involved in concurrent computing. Don't have to deal with race conditions, deadlocks, livelocks, etc... \ No newline at end of file +- **No synchronisation issues** & headaches involved in concurrent computing. Don't have to deal with race conditions, deadlocks, livelocks, etc... + +## Parallel Scalability +The speed up achieved from parallelism is dictated by your algorithm. Notably the serial parts of your algorithm can not be sped up by increasing the number of processors. The diagram below looks at the benefits we can achieve from writing parallel code as the number of processes increases. + +![amdahl](./imgs/parallel_scalability.jpg) + +Amdahl's Law, formulated by computer architect Gene Amdahl in 1967, is a principle used to analyze the potential speedup of parallel computing. It states that the speedup of a program from parallelization is limited by the proportion of the program that must be executed serially. In other words, it helps to determine the maximum performance improvement that can be achieved by using parallel processing. + +The implications of Amdahl's Law for HPC is very significant: + +- **Limitation of Speedup:** Amdahl's Law highlights that even with an increase in the number of processors (parallelization), the overall speedup is limited by the sequential portion of the code. Thus, if a significant portion of the code is inherently serial, the potential speedup achievable through parallelization is restricted. +- **Importance of Identifying Serial Sections:** In HPC, it's crucial to identify the sections of code that are inherently serial and cannot be parallelized. Optimizing these sections can lead to better overall performance. Conversely, focusing solely on parallelizing code without addressing these serial bottlenecks can result in suboptimal speedup. +- **Efficiency vs. Scalability:** Amdahl's Law emphasizes the importance of balancing efficiency and scalability in parallel computing. While increasing the number of processors can improve performance to a certain extent, beyond a certain point, diminishing returns occur due to the overhead of synchronization, communication, and managing parallel tasks. +- **Architectural Considerations:** HPC system architects must consider Amdahl's Law when designing hardware and software architectures. Designing systems that minimize the impact of serial portions of code and optimize parallel execution can lead to better overall performance. +- **Algorithm Selection:** When choosing algorithms for HPC applications, it's essential to consider their parallelizability. Algorithms that can be efficiently parallelized are more suitable for HPC environments, as they can leverage the potential for speedup provided by parallel computing resources more effectively. \ No newline at end of file diff --git a/src/chapter4/slurm_intro.md b/src/m3-slurm/slurm_intro.md similarity index 58% rename from src/chapter4/slurm_intro.md rename to src/m3-slurm/slurm_intro.md index 3dd7e16..c582739 100644 --- a/src/chapter4/slurm_intro.md +++ b/src/m3-slurm/slurm_intro.md @@ -7,68 +7,9 @@ Slurm has three key functions. 2. It provides a framework to start, execute, and check the work on the set of allocated compute nodes. 3. It manages the queue of pending jobs based on the availability of resources. -## Basic Linux Commands +The below diagram shows how Slurm works in the context of M3 MASSIVE. -Since Slurm and M3 nodes are implemented on Linux, it's necessary to know some basic Linux commands. You will have to be comfortable using these commands, writing Bash scripts and navigating the Linux environment in order to be successful in a lot of our HPC projects. - -| Command | Function | -| --- | --- | -| `pwd` | prints current directory | -| `ls` | prints list of files / directories in current directory (add a `-a` to list everything, including hidden files/directories | -| `mkdir` | makes a directory | -| `rm ` | deletes *filename*. add `-r` to delete directory. add `-f` to force deletion (be really careful with that one) | -| `cd ` | move directory. | -| `vim` or `nano` | bring up a text editor | -| `cat ` | prints contents of file to terminal | -| `echo` | prints whatever you put after it | -| `chmod ` | changes permissions of file | -| `cp` | copy a file or directory| -| `mv ` | move or rename file or directory | - -> Note: `.` and `..` are special directories. `.` is the current directory, and `..` is the parent directory. These can be used when using any command that takes a directory as an argument. Similar to these, `~` is the home directory, and `/` is the root directory. For example, if you wanted to copy something from the parent directory to the home directory, you could do `cp ../ ~/`, without having to navigate anywhere. - -## Bash Scripts - -Bash is both a command line interface and a scripting language. Linux commands are generally using Bash. Bash scripts are a series of commands that are executed in order. Bash scripts are useful for automating tasks that you do often, or for running a series of commands that you don't want to type out every time. In our case, Bash scripts are used for running jobs on M3. - -In terms of use, Bash can encapsulate any command you would normally run in the terminal into a script that can be easily reused. For example you could have a script that automatically navigates to a directory and activates a virtual environment, or a script that compiles and runs a C program. - -The basic syntax of a bash file is as follows: - -```bash -#!/bin/bash - -# This is a comment - -echo "Hello World" -``` - -We can save this file as `hello.sh` and run it using the following command: `source hello.sh`. This will print `Hello World` to the terminal. - -Let's walk through the file. The first line is `#!/bin/bash`. This is called a shebang, and it tells the system that this file is a bash script. The second line is a comment, and is ignored by the system. The third line is the actual command that we want to run. In this case, we are using the `echo` command to print `Hello World` to the terminal. - -Bash can do a lot more, including basic arithmetic, if statements, loops, and functions, however these are not really necesary for what we are doing. If you want to learn more about bash, you can find a good tutorial [here](https://linuxconfig.org/bash-scripting-tutorial). - -For our use, the main things we need to be able to do are to run executables and files. We do this the exact same way as if manually running them in the terminal. For example, if we want to run a python script, we can do the following: - -```bash -#!/bin/bash - -# This will run hello.py using the python3 executable -python3 hello.py -``` - -If we want to compile and then run a C program, we can do the following: - -```bash -#!/bin/bash - -# This will compile hello.c and then run it -gcc hello.c -o hello -./hello -``` - -Using bash scripts not only saves a lot of time and effort, but it also makes it easier to run jobs on M3 using SLURM. We will go over how to do this in the next section. +![slurm-on-m3](./imgs/slurm-on-m3.png) ## Slurm Architecture Slurm has a centralized manager, slurmctld, to monitor resources and work. Each compute server (node) has a slurmd daemon, which can be compared to a remote shell: it waits for work, executes that work, returns status, and waits for more work. There is an optional slurmdbd (Slurm DataBase Daemon) which can be used to record job accounting information in a database. @@ -109,6 +50,22 @@ In creating a Slurm script, there are **4 main parts** that are mandatory in ord #SBATCH --mem-per-cpu=500M ``` +Some of the various resource request parameters available for you on Slurm are below: + +- `ntasks`: The number of tasks or processes to run. +- `mem`: The amount of memory to allocate to the job. +- `time`: The maximum amount of time the job can run for. +- `job-name`: The name of the job. Up to 15 characters. +- `partition`: The partition to run the job on. +- `mail-user`: The email address to send job status emails to. +- `mail-type`: The types of emails to send. + +> Note: In the case of M3, a task is essentially the same as a process. This is **not** the same as a cpu core. You can have a task that uses one or multiple cores. You can also have multiple tasks comprising the same job, each with one or multiple cores being utilised. It can get quite confusing, so if you are unsure about what you need, just ask. There is also more information in the M3 docs. + +There are a lot more options that you can use, and you can find a more complete list [here](https://docs.massive.org.au/M3/slurm/simple-batch-jobs.html). + +In particular, if you want to run multithreading or multiprocessing jobs, or you need a gpu, there are more options you need to configure. + 3. **Dependencies:** Load all the software that the project depends on to execute. For example, if you are working on a python project, you’d definitely require the python software or module to interpret and run your code. Go to Chapter 5.6 for more info on this. ``` @@ -150,4 +107,27 @@ In the script above, 1 Node with 1 CPU, 500MB of memory per CPU, 10 minutes of W The first job step will run the Linux echo command and output Start process. The next job step(2) will echo the Hostname of the compute node that executed the job. Then, the next job step will execute the Linux sleep command for 30 seconds. The final job step will just echo out End process. Note that these job steps executed sequentially and not in parallel. -It’s important to set a limit on the total run time of the job allocation, this helps the Slurm manager to handle prioritization and queuing efficiently. The above example is a very simple script which takes less than a second. Hence, it’s important to specify the run time limit so that Slurm doesn’t see the job as one that requires a lot of time to execute. \ No newline at end of file +It’s important to set a limit on the total run time of the job allocation, this helps the Slurm manager to handle prioritization and queuing efficiently. The above example is a very simple script which takes less than a second. Hence, it’s important to specify the run time limit so that Slurm doesn’t see the job as one that requires a lot of time to execute. + +## Interactive jobs + +Sometimes you might want to actually connect to the node that you are running your job on, in order to see what is happening or to set it up before running the job. You can do this using the `smux` command. Similar to regular batch jobs, you can set options when you start the interactive session. An example of this is: + +`smux new-session --ntasks=1 --time=0-00:01:00 --partition=m3i --mem=4GB` + +This will start an interactive session on a node with 1 cpu, 1 minute of time, and 4GB of memory. There are again other options available, and you can find a more complete explanation [here](https://docs.massive.org.au/M3/slurm/interactive-jobs.html). + +### Connecting to interactive jobs + +Typically when you start an interactive job it will not start immediately. Instead, it will be queued up once it has started you will need to connect to it. You can do this by running `smux a`, which will reconnect you to the session. If you want to disconnect from the session but leave it running, you can press `Ctrl + b` followed by `d`. This will disconnect you from the session, but leave it running. You can reconnect to it later using `smux a`. If you want to kill the session, if you are connected just run `exit`, otherwise if you are in a login node run `scancel `. You can find the job id using `show_job`. + +## Checking the status of jobs, finding out job IDs, and killing jobs + +A couple of useful commands for general housekeeping are: + +- `squeue`: This will show you the status of all jobs currently running on M3. +- `show_job`: This will show you the status of all jobs you have submitted. +- `squeue -u `: This will show you the status of all jobs submitted by a particular user currently running. +- `scancel `: This will kill a job with a particular job id. +- `scancel -u `: This will kill all jobs submitted by a particular user. +- `show_cluster`: This will show you the status of the cluster, including any nodes that are offline or in maintenance. \ No newline at end of file diff --git a/src/chapter4/software-tooling.md b/src/m3-slurm/software-tooling.md similarity index 100% rename from src/chapter4/software-tooling.md rename to src/m3-slurm/software-tooling.md diff --git a/src/operating-systems/bash-cmds-scripts.md b/src/operating-systems/bash-cmds-scripts.md new file mode 100644 index 0000000..c3c7016 --- /dev/null +++ b/src/operating-systems/bash-cmds-scripts.md @@ -0,0 +1,77 @@ +# Bash Commands & Scripts + +Bash (the Bourne-Again SHell) is a terminal that allows you to enter commands and write scripts in order to access the Kernel's functionality. It's similar to system libraries (for programming) and the GUI (for non-IT people) in that it's a way to navigate your operating system and use the services that it provides, i.e. it's your kernel's interface. Bash in particular was created in the late 1980s as part of the GNU project and it was intended to replace the Bourne shell, hence Bourne-again shell. + +![shell-kernel](./imgs/shell-kernel.png) + + +## Basic Linux Commands + +These commands will be your bread and better whenever you can't access the GUI or the GUI doesn't provide enough functionality, which will be the case for a lot of HPC work. You will have to be comfortable using these commands, writing Bash scripts and navigating the Linux environment. + +| Command | Function | +| --- | --- | +| `pwd` | prints current directory | +| `ls` | prints list of files / directories in current directory (add a `-a` to list everything, including hidden files/directories | +| `mkdir` | makes a directory | +| `rm ` | deletes *filename*. add `-r` to delete directory. add `-f` to force deletion (be really careful with that one) | +| `cd ` | move directory. | +| `vim` or `nano` | bring up a text editor | +| `cat ` | prints contents of file to terminal | +| `echo` | prints whatever you put after it | +| `chmod ` | changes permissions of file | +| `cp` | copy a file or directory| +| `mv ` | move or rename file or directory | +| `grep` | Search for a pattern in files | +| `touch ` | Create an empty file in current dir | + +> Note: `.` and `..` are special directories. `.` is the current directory, and `..` is the parent directory. These can be used when using any command that takes a directory as an argument. Similar to these, `~` is the home directory, and `/` is the root directory. For example, if you wanted to copy something from the parent directory to the home directory, you could do `cp ../ ~/`, without having to navigate anywhere. + +### Piping + +If you want to use the output of one command as input to another, you can simply "pipe" it to the other command. For example, if you want to search your directory to find a file named `somefile.someextension` you can do this: + +```bash +ls | grep somefile.someextension +``` + +## Bash Scripts + +Bash is both a command line interface and a scripting language. Bash scripts are useful for automating tasks that you do often, or for running a series of commands that you don't want to type out every time. + +In terms of use, Bash can encapsulate any command you would normally run in the terminal into a script that can be easily reused. For example you could have a script that automatically navigates to a directory and activates a virtual environment, or a script that compiles and runs a C program. + +The basic syntax of a bash file is as follows: + +```bash +#!/bin/bash + +# This is a comment + +echo "Hello World" +``` + +We can save this file as `hello.sh` and run it using the following command: `source hello.sh`. This will print `Hello World` to the terminal. + +Let's walk through the file. The first line is `#!/bin/bash`. This is called a shebang, and it tells the system that this file is a bash script. The second line is a comment, and is ignored by the system. The third line is the actual command that we want to run. In this case, we are using the `echo` command to print `Hello World` to the terminal. + +Bash can do a lot more, including basic arithmetic, if statements, loops, and functions. If you want to learn more about bash, you can find a good tutorial [here](https://linuxconfig.org/bash-scripting-tutorial). + +The main things we need to be able to do are to run executables and files. We do this the exact same way as if manually running them in the terminal. For example, if we want to run a python script, we can do the following: + +```bash +#!/bin/bash + +# This will run hello.py using the python3 executable +python3 hello.py +``` + +If we want to compile and then run a C program, we can do the following: + +```bash +#!/bin/bash + +# This will compile hello.c and then run it +gcc hello.c -o hello +./hello +``` diff --git a/src/chapter3/challenges.md b/src/operating-systems/challenges.md similarity index 60% rename from src/chapter3/challenges.md rename to src/operating-systems/challenges.md index 10575e6..177236b 100644 --- a/src/chapter3/challenges.md +++ b/src/operating-systems/challenges.md @@ -12,8 +12,20 @@ In this challenge you have to implement an array concatenation function. This sh ## Challenge 3 - Doing it in Docker -Pull an Ubuntu image from the Docker registry, install any required dependencies and execute the same C code that you wrote for challenge 2 within that running container instance. You will have to consult the Docker documentation and be resourceful in learning and completing this task. +Pull an Ubuntu image from the Docker registry, install any required dependencies and execute the same C code that you wrote for challenge 2 within that running container instance. -## Challenge 4 - Launch a VM instance on Nectar Cloud +## Challenge 4 - Bash Process Dump -This is an extension challenge in which you will have to go through the [ARDC's cloud starter tutorial series](https://tutorials.rc.nectar.org.au/cloud-starter/01-overview) in order to launch a VM instance and connect to it. \ No newline at end of file +Write and execute a Bash script that captures the process id, process (cmd) name and Nice score and outputs it all into a text file. + +## Challenge 5 - Scheduler evaluation + +Write a C program that computes the average turnaround time based on these process stats: + +| Process | Arrival Time | Burst Time | +|---------|--------------|------------| +| P1 | 0 | 3 | +| P2 | 1 | 5 | +| P3 | 2 | 2 | +| P4 | 3 | 4 | +| P5 | 4 | 6 | \ No newline at end of file diff --git a/src/chapter3/computer-architecture.md b/src/operating-systems/computer-architecture.md similarity index 100% rename from src/chapter3/computer-architecture.md rename to src/operating-systems/computer-architecture.md diff --git a/src/chapter3/dynamic-memory.md b/src/operating-systems/dynamic-memory.md similarity index 100% rename from src/chapter3/dynamic-memory.md rename to src/operating-systems/dynamic-memory.md diff --git a/src/chapter3/imgs/Memory-Hierarchy-Design.png b/src/operating-systems/imgs/Memory-Hierarchy-Design.png similarity index 100% rename from src/chapter3/imgs/Memory-Hierarchy-Design.png rename to src/operating-systems/imgs/Memory-Hierarchy-Design.png diff --git a/src/chapter3/imgs/Structure-Of-Linux-Operating-System.png b/src/operating-systems/imgs/Structure-Of-Linux-Operating-System.png similarity index 100% rename from src/chapter3/imgs/Structure-Of-Linux-Operating-System.png rename to src/operating-systems/imgs/Structure-Of-Linux-Operating-System.png diff --git a/src/operating-systems/imgs/Thread vs Processes.png b/src/operating-systems/imgs/Thread vs Processes.png new file mode 100644 index 0000000..a1179ea Binary files /dev/null and b/src/operating-systems/imgs/Thread vs Processes.png differ diff --git a/src/operating-systems/imgs/another-machine.png b/src/operating-systems/imgs/another-machine.png new file mode 100644 index 0000000..8f2fd84 Binary files /dev/null and b/src/operating-systems/imgs/another-machine.png differ diff --git a/src/chapter3/imgs/basic_structure.png b/src/operating-systems/imgs/basic_structure.png similarity index 100% rename from src/chapter3/imgs/basic_structure.png rename to src/operating-systems/imgs/basic_structure.png diff --git a/src/chapter3/imgs/bit-byte-word.jpg b/src/operating-systems/imgs/bit-byte-word.jpg similarity index 100% rename from src/chapter3/imgs/bit-byte-word.jpg rename to src/operating-systems/imgs/bit-byte-word.jpg diff --git a/src/chapter3/imgs/cache-hit-miss.jpg b/src/operating-systems/imgs/cache-hit-miss.jpg similarity index 100% rename from src/chapter3/imgs/cache-hit-miss.jpg rename to src/operating-systems/imgs/cache-hit-miss.jpg diff --git a/src/chapter3/imgs/containers.png b/src/operating-systems/imgs/containers.png similarity index 100% rename from src/chapter3/imgs/containers.png rename to src/operating-systems/imgs/containers.png diff --git a/src/chapter3/imgs/cpu-cache.jpg b/src/operating-systems/imgs/cpu-cache.jpg similarity index 100% rename from src/chapter3/imgs/cpu-cache.jpg rename to src/operating-systems/imgs/cpu-cache.jpg diff --git a/src/operating-systems/imgs/cpu-scheduling-diagram.png b/src/operating-systems/imgs/cpu-scheduling-diagram.png new file mode 100644 index 0000000..9cc2493 Binary files /dev/null and b/src/operating-systems/imgs/cpu-scheduling-diagram.png differ diff --git a/src/chapter3/imgs/docker-architecture.jpg b/src/operating-systems/imgs/docker-architecture.jpg similarity index 100% rename from src/chapter3/imgs/docker-architecture.jpg rename to src/operating-systems/imgs/docker-architecture.jpg diff --git a/src/chapter3/imgs/file-system-arch.png b/src/operating-systems/imgs/file-system-arch.png similarity index 100% rename from src/chapter3/imgs/file-system-arch.png rename to src/operating-systems/imgs/file-system-arch.png diff --git a/src/chapter3/imgs/linux-distros.png b/src/operating-systems/imgs/linux-distros.png similarity index 100% rename from src/chapter3/imgs/linux-distros.png rename to src/operating-systems/imgs/linux-distros.png diff --git a/src/chapter3/imgs/memory-cells.jpg b/src/operating-systems/imgs/memory-cells.jpg similarity index 100% rename from src/chapter3/imgs/memory-cells.jpg rename to src/operating-systems/imgs/memory-cells.jpg diff --git a/src/chapter3/imgs/memory-segments.png b/src/operating-systems/imgs/memory-segments.png similarity index 100% rename from src/chapter3/imgs/memory-segments.png rename to src/operating-systems/imgs/memory-segments.png diff --git a/src/chapter3/imgs/paging-basic-scheme.jpg b/src/operating-systems/imgs/paging-basic-scheme.jpg similarity index 100% rename from src/chapter3/imgs/paging-basic-scheme.jpg rename to src/operating-systems/imgs/paging-basic-scheme.jpg diff --git a/src/chapter3/imgs/pointers-in-c.jpg b/src/operating-systems/imgs/pointers-in-c.jpg similarity index 100% rename from src/chapter3/imgs/pointers-in-c.jpg rename to src/operating-systems/imgs/pointers-in-c.jpg diff --git a/src/operating-systems/imgs/priority-scheduling.jpeg b/src/operating-systems/imgs/priority-scheduling.jpeg new file mode 100644 index 0000000..6237c8b Binary files /dev/null and b/src/operating-systems/imgs/priority-scheduling.jpeg differ diff --git a/src/chapter3/imgs/process-states.png b/src/operating-systems/imgs/process-states.png similarity index 100% rename from src/chapter3/imgs/process-states.png rename to src/operating-systems/imgs/process-states.png diff --git a/src/chapter3/imgs/program-process.png b/src/operating-systems/imgs/program-process.png similarity index 100% rename from src/chapter3/imgs/program-process.png rename to src/operating-systems/imgs/program-process.png diff --git a/src/chapter3/imgs/programming-levels.jpg b/src/operating-systems/imgs/programming-levels.jpg similarity index 100% rename from src/chapter3/imgs/programming-levels.jpg rename to src/operating-systems/imgs/programming-levels.jpg diff --git a/src/operating-systems/imgs/round-robin.jpg b/src/operating-systems/imgs/round-robin.jpg new file mode 100644 index 0000000..abcd4cc Binary files /dev/null and b/src/operating-systems/imgs/round-robin.jpg differ diff --git a/src/operating-systems/imgs/scheduling-fcfs.png b/src/operating-systems/imgs/scheduling-fcfs.png new file mode 100644 index 0000000..ea08168 Binary files /dev/null and b/src/operating-systems/imgs/scheduling-fcfs.png differ diff --git a/src/operating-systems/imgs/scheduling-types.jpeg b/src/operating-systems/imgs/scheduling-types.jpeg new file mode 100644 index 0000000..4639958 Binary files /dev/null and b/src/operating-systems/imgs/scheduling-types.jpeg differ diff --git a/src/operating-systems/imgs/shell-kernel.png b/src/operating-systems/imgs/shell-kernel.png new file mode 100644 index 0000000..78207d8 Binary files /dev/null and b/src/operating-systems/imgs/shell-kernel.png differ diff --git a/src/operating-systems/imgs/shortest-job-first.png b/src/operating-systems/imgs/shortest-job-first.png new file mode 100644 index 0000000..9e38cca Binary files /dev/null and b/src/operating-systems/imgs/shortest-job-first.png differ diff --git a/src/operating-systems/imgs/signals.png b/src/operating-systems/imgs/signals.png new file mode 100644 index 0000000..a302215 Binary files /dev/null and b/src/operating-systems/imgs/signals.png differ diff --git a/src/chapter3/imgs/spatial-vs-temporal.gif b/src/operating-systems/imgs/spatial-vs-temporal.gif similarity index 100% rename from src/chapter3/imgs/spatial-vs-temporal.gif rename to src/operating-systems/imgs/spatial-vs-temporal.gif diff --git a/src/chapter3/imgs/vms.png b/src/operating-systems/imgs/vms.png similarity index 100% rename from src/chapter3/imgs/vms.png rename to src/operating-systems/imgs/vms.png diff --git a/src/chapter3/imgs/vn_cpu.png b/src/operating-systems/imgs/vn_cpu.png similarity index 100% rename from src/chapter3/imgs/vn_cpu.png rename to src/operating-systems/imgs/vn_cpu.png diff --git a/src/chapter3/linux-intro.md b/src/operating-systems/linux-intro.md similarity index 96% rename from src/chapter3/linux-intro.md rename to src/operating-systems/linux-intro.md index acceea1..e0e45ad 100644 --- a/src/chapter3/linux-intro.md +++ b/src/operating-systems/linux-intro.md @@ -97,4 +97,6 @@ The aim is to store data that any given process is likely to access in the futur ### Accessing the Kernel's API -As mentioned earlier, user space programs (code that a programmer writes for an application/script) will need to use a system library to access the kernel and it's lower-level functionality. For Linux, the main library is `unistd.h` which only runs on POSIX-compatible (Unix-like) operating systems and unfortunately Windows is not one of them. To get around this, we will be using a Docker container with an Ubuntu image. But first let's finish this chapter by learning about Virtual Machines and Containers. \ No newline at end of file +As mentioned earlier, user space programs (code that a programmer writes for an application/script) will need to use a system library to access the kernel and it's lower-level functionality. For Linux, the main library is `unistd.h` which only runs on POSIX-compatible (Unix-like) operating systems and unfortunately Windows is not one of them. To get around this, we will be using a Docker container with an Ubuntu image. + +> **Note:** You can also use WSL or launch an Ubuntu VM to get `unistd.h` working. But it's useful to know Docker in case you ever need to replicate the same environment and avoid dependency-related bugs while working in teams or if you need to do [container-based deployments](https://cloudkul.com/blog/how-to-host-website-on-docker-container/). diff --git a/src/chapter3/memory-pointers.md b/src/operating-systems/memory-pointers.md similarity index 100% rename from src/chapter3/memory-pointers.md rename to src/operating-systems/memory-pointers.md diff --git a/src/chapter3/chapter3.md b/src/operating-systems/operating-systems.md similarity index 100% rename from src/chapter3/chapter3.md rename to src/operating-systems/operating-systems.md diff --git a/src/operating-systems/processes-threads.md b/src/operating-systems/processes-threads.md new file mode 100644 index 0000000..49914a0 --- /dev/null +++ b/src/operating-systems/processes-threads.md @@ -0,0 +1,65 @@ +# Processes & Threads +As introduced earlier, when computer runs a program, your source code is loaded into RAM and process is started. +A **process** is a collection of code, memory, data and other resources. +A process runs in a unique address space. So Two processes can not see each other’s memory. + +A **thread** is a sequence of code that is executed inside the scope of the **process**. You can (usually) have multiple **threads** executing concurrently within the same process. +**Threads** can view the memory (i.e. variables) of other threads within the same process + +![procs-threads](./imgs/Thread%20vs%20Processes.png) + +A **multiprocessing** system has more than two processors, whereas **multithreading** is a program execution technique that allows a single process to have multiple code segments. + +## Bash Process Management +There are several commands in Bash for process management. Following are some commonly used commands: + +| Command | Purpose | +|----------|----------------------------------------------------------| +| `ps` | Lists all processes in the system. | +| `top` | Tracks the running processes on the system and provides real-time information. | +| `kill` | Terminates a running process by sending a signal. | +| `pkill` | Sends a termination signal to a process. | +| `jobs` | Lists all the running jobs (tasks) in the system. | +| `fg` | Moves a background process to the foreground. | +| `bg` | Restarts a stopped process to run in the background. | +| `nice` | Adjusts the priority of a process. | +| `renice` | Changes or adjusts the priority of a running process. | +| `free` | Displays the status of RAM. | +| `df` | Shows the disk management of the system. | + +As you might have noticed if you ran `ps` or `top`, each Bash command spawns (creates) a process. That's because all of these commands are just executable binary files that are software programs (exactly the same as compiled C code) that you can find in one of your system directories. + +> **Pro-Tip:** The `-o` flag with the `ps` command allows you to specify the columns to list for processes. For example, to list the process ID, nice score and command name for each process use `ps -o pid,ni,cmd`. + +### Signals +In Linux, a signal is a notification sent to a process to indicate an event. Typically it is triggered by the operating system, hardware, or other processes in order to signify various events, including process termination, user interrupts, errors, and system events. A unique identifier represents each signal and has predefined behaviors associated with it. + +To get a list of all signals in Bash use: +```bash +kill -l +``` + +Which should give you an output similar to below, + +![signals-bash](./imgs/signals.png) + +From the above you can see that to kill/terminate a process (SIGKILL), you need to use signal 9. To kill a process you can use the command `kill -9 `. If you need to kill a deadlocked/non-responsive process on a HPC node with no GUI, this would come in handy. + +## Process Priorities +A process's priority determines when and how much CPU walltime it gets. This is also the case in Slurm or any other multi-user/multi-process system with a scheduler so it's useful to know how process priorities work. + +Managing a process’s priority involves altering the order of priority in which processes are executed by the system. By setting the priority, you can make sure that the most important tasks get the most CPU time to receive the necessary system resources compared to the less important tasks. Some common aspects of prioritizing processes: + +### Process Priority Range + +The priority value of a process is known as the ‘nice value’ that typically ranges from -20 to 19, where -20 indicates the highest priority and 19 indicates the lowest priority. By default, the processes launched from the terminal have a zero (0) value of priority. The negative values (-20 to 0) are reserved for real-time tasks while the positive values (0-19) are for normal user processes or tasks. + +### Using “nice” Command + +The nice command starts a process with a specified priority level. It assigns a lower priority to a process as the lower the nice values, the higher the priority. The syntax to use the nice command is `nice -n `. + +### Using “renice” Command + +The renice command modifies and adjusts the priority of a process that is already running. This command either increases or decreases the priority of a process based on the specific value. The syntax to use the renice command is `renice -p `. + +> Decreasing priority can be done by any user in the system but the priority can only be increased by the process owner or the super user (root). \ No newline at end of file diff --git a/src/operating-systems/scheduling.md b/src/operating-systems/scheduling.md new file mode 100644 index 0000000..4413285 --- /dev/null +++ b/src/operating-systems/scheduling.md @@ -0,0 +1,78 @@ +# Scheduling Algorithms + +As you might've noticed, your computer **runs a large amount of processes** (around 500+) at any given time. Most of them are system utilities and kernel processes that you don't really have access to but a lot of them involve applications you opened or commands you've typed into your terminal. Even though most modern CPUs have 8+ cores so they can run 8 processes/execute 8 threads at the same time, there's obviously a need to **queue these processes** and schedule them time on the CPU in an orderly manner. In order to do this, there are a variety of scheduling algorithms. + +Knowing these scheduling algorithms also helps you understand how scheduling works in a HPC cluster. This is important if you're involved in a HPC project that builds, manages and/or analyses a cluster from an admin perspective and also as a user that's wondering why their job (process) isn't getting much time on compute nodes. + +![cpu-scheduling](./imgs/cpu-scheduling-diagram.png) + +Hopefully the diagram above is self-explanatory at this point. There's a lot of implementation details involved in the specific type of scheduler you're using (i.e. Linux Kernel) but we will focus on high-level concepts that are also generalisable to HPC cluster schedulers. + +## Terminology +Before we start there's some terminology to clarify and define: +- **Preemptive Scheduling:** This is a scheduling policy where a process running in a CPU can be interrupted (blocked/stopped) and moved to the ready state while another process starts running on the CPU. +- **Wait Time:** Wait time refers to the period during which a process remains in the ready queue, waiting for CPU execution. +- **Arrival Time:** Arrival time is the moment when a process enters the ready state and becomes eligible for execution. +- **Burst Time:** Burst time (execution/wall time) is the total time taken by a process to execute on the CPU. +- **Turnaround Time:** Turnaround time is the total time taken for a process to complete its execution from submission to exit. It includes the waiting time, execution time, and any other overhead (such as context switching). Turnaround time is a crucial metric for evaluating scheduling algorithms and system performance (for both PCs & HPC cluster). + +![scheduling-types](./imgs/scheduling-types.jpeg) + +## First Come First Serve (FCFS) +This is the simplest CPU scheduling algorithm that schedules according to arrival times of processes. The first come first serve scheduling algorithm states that the process that requests the CPU first is allocated the CPU first. It is implemented by using a FIFO (first-in, first-out) queue. When a process enters the ready queue, its PCB (process control block) is linked to the tail of the queue. When the CPU is free, it is allocated to the process at the head of the queue. The running process is then removed from the queue. + +![fcfs](./imgs/scheduling-fcfs.png) + + Characteristics of FCFS: +- FCFS supports non-preemptive and preemptive CPU scheduling algorithms. +- Tasks are always executed on a First-come, First-serve concept. +- FCFS is easy to implement and use. +- This algorithm is not very efficient in performance, and the wait time is quite high. + +## Round Robin Scheduling + +Round-robin scheduling allocates each task an equal share of the CPU time. In its simplest form, tasks are in a circular queue and when a task's allocated CPU time expires, the task is put to the end of the queue and the new task is taken from the front of the queue. + +![round-robin](./imgs/round-robin.jpg) + +**Advantages:** + +- It is not affected by the convoy effect or the starvation problem as occurred in First Come First Serve CPU Scheduling Algorithm. + +**Disadvantages:** + +- Low Operating System slicing times will result in decreased CPU output. +- Round Robin CPU Scheduling approach takes longer to swap contexts. +- Time quantum has a significant impact on its performance. +- The procedures cannot have priorities established. + +## Shortest Job First Algorithm + +In the Shortest Job First (SJF) algorithm, processes are scheduled based on their burst time rather than their arrival time. This algorithm aims to minimize waiting time and maximize throughput. However, a challenge with SJF is that CPU burst times are often not known in advance. To address this, various techniques can be used to estimate burst times, such as analyzing process size or type. + +![sjf](./imgs/shortest-job-first.png) + +## Priority Scheduling + +In priority scheduling, each process is assigned a priority number. Depending on the system, a lower or higher priority number indicates higher priority. There are two types of priority scheduling: preemptive and non-preemptive. + +![priority-sched](./imgs/priority-scheduling.jpeg) + +### Non-Preemptive Priority Scheduling + +In non-preemptive priority scheduling, processes are scheduled based on their priority number. Once a process is scheduled, it runs to completion without interruption. Typically, lower priority numbers indicate higher priority. + +### Preemptive Priority Scheduling + +In preemptive priority scheduling, when a process enters the ready queue, its priority is compared with those of other processes, including the one currently being executed. The process with the highest priority among all available processes is given the CPU. Unlike non-preemptive scheduling, a running process can be interrupted by a higher priority process. + +### Example + +Consider a scenario with 7 processes: P1, P2, P3, P4, P5, P6, and P7. Each process has a priority, arrival time, and burst time. Let's walk through an example to understand how priority scheduling works: + +- At time 0, P1 arrives with a burst time of 1 unit and priority 2. Since no other process is available, P1 is scheduled immediately. +- At time 1, P2 arrives. Since P1 has completed execution, P2 is scheduled regardless of its priority. +- At time 2, P3 arrives with a higher priority than P2. Thus, P2's execution is halted, and P3 is scheduled. +- During the execution of P3, three more processes (P4, P5, and P6) become available. However, since their priorities are lower than that of P3, they cannot preempt P3's execution. +- After P3 completes execution, P5, with the highest priority among the remaining processes, is scheduled. +- Once all processes are available in the ready queue, the algorithm behaves as non-preemptive priority scheduling. The process with the highest priority (P4) is executed till completion without interruption. diff --git a/src/chapter3/structs-macros.md b/src/operating-systems/structs-macros.md similarity index 100% rename from src/chapter3/structs-macros.md rename to src/operating-systems/structs-macros.md diff --git a/src/operating-systems/vms-containers.md b/src/operating-systems/vms-containers.md new file mode 100644 index 0000000..1583cad --- /dev/null +++ b/src/operating-systems/vms-containers.md @@ -0,0 +1,123 @@ +# VMs & Containers + +In order to succeed in HPC, you will need to know how distributed computing works. Virtual machines and containers are a way of doing distributed computing and we'll give a decent introduction to them here. They help in a lot of scenarios both in your MDN work and in the real-world by allowing you to manage dependencies (eg. for a custom benchmarking script) and deploy an application into a server (eg. for a public HPC visualisation website). + +## What is a virtual machine (VM)? + +A virtual machine is not a physical machine. It’s a file that replicates the computing environment of a physical machine. It’s similar to how virtual reality (VR) environments replicate the real world. VR isn’t a physical space; it’s a virtual imitation. Still, we can perform real-world functions in VR, such as exploring and interacting with objects. Instead of imitating video game functions, virtual machine software emulates computer system functions i.e. the operating system. To achieve this VMs use a technology called **virtualisation** as shown below. + +![vms](./imgs/vms.png) + +At the base, you have the host hardware and OS. This is the physical machine that is used to create the virtual machines. On top of this, you have the hypervisor. This allows multiple virtual machines, each with their own operating systems (OS), to run on a single physical server. + +VMs have a few downsides, though, which containers address. Two downsides particularly stand out: + +- **VMs consume more resources:** VMs have a higher resource overhead due to the need to run a full OS instance for each VM. This can lead to larger memory and storage consumption. This in turn can have a negative effect on performance and startup times of the virtual machine. +- **Less portability:** VMs are typically less portable due to differences in underlying OS environments. Moving VMs between different hypervisors or cloud providers can be more complex. + + +There are a few upsides though and they mostly apply if you need **better isolation/security** or **more control** over low level things that containers don't give you access to. If you want to learn how to launch and use VMs, it's covered in [ARDC's Nectar Cloud starter tutorial series](https://tutorials.rc.nectar.org.au/cloud-starter/01-overview). You can also check out other cloud platform providers like Amazon Web Services (AWS) and Microsoft Azure. + +## What is a container? + +A container is a lightweight, standalone, and executable software package that includes everything needed to run a piece of software, including the code, runtime, system tools, and libraries. + +Containers are designed to isolate applications and their dependencies, ensuring that they can run consistently across different environments. Whether the application is running from your computer or in the cloud, the application behaviour remains the same. + +Unlike VMs which virtualise the hardware, containers *virtualise the operating system*. This simply means that a container uses a single OS to create a virtual application and its libraries. Containers run on top of a shared OS provided by the host system. + +![containers](./imgs/containers.png) + +The container engine allows you to spin up containers. It provides the tools and services necessary for building, running, and deploying containerised applications. If you ever had the problem below, containers are a life saver. + +![another-machine-meme](./imgs/another-machine.png) + +## Docker +As you might know, Docker is an open-source containerization platform by which you can pack your application and all its dependencies into a standardized unit called a container. Let's clarify some Docker terminology, + +- **Docker Image**: An image is an inert, immutable, file that's essentially a snapshot of a container. Images are created with the [build](https://docs.docker.com/reference/cli/docker/image/build/) command, and they'll produce a container when started with [run](https://docs.docker.com/reference/cli/docker/container/run/). Images are stored in a Docker registry such as [registry.hub.docker.com](https://registry.hub.docker.com). +- **Docker Container**: To use a programming metaphor, if an image is a class, then a container is an instance of a class—a runtime object. Multiple containers can run from the same image simultaneously. +- **Docker Daemon**: The Docker daemon (`dockerd`) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. +- **Docker Client**: The Docker client (`docker`) is the primary way that many Docker users interact with Docker. When you use commands such as `docker run`, the client sends these commands to `dockerd`, which carries them out. +- **Docker Desktop**: This is an easy-to-install application for your Mac, Windows or Linux environment that enables you to build and share containerized applications and microservices. + +### Docker Architecture +Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. + +![docker-arch](./imgs/docker-architecture.jpg) + +Go through [this guide](https://www.docker.com/get-started/) to download and install Docker. If you'd like to learn more about it's functionality and how to work with it, [check out their documentation website](https://docs.docker.com/get-started/overview/). We will go through the basics of how Docker and it's CLI (command-line interface) works to get you started. + +#### Launch Docker Daemon +Before we get started it's important to make sure the Docker daemon (background process that manages Docker) is running. In order to launch it, go to your docker desktop app and run it. + +#### Check Version +If you want to know what version of Docker you're running, use this command. It's also a useful command for checking if you've correctly installed and launched Docker. +``` +docker version +``` + +#### More Info +For more info about installation and the no. of Docker objects (images, containers, networks, etc...) use this command. +``` +docker info +``` + +#### List Containers +To list all running containers use the command below. To include ones that are not currently running as well, use the command with the `-a` flag. +``` +docker ps +``` + +#### List Images +Likewise to list all images you have downloaded, use the command below. +``` +docker images +``` + +#### Download Image +Similar to GitHub, if you want to download an image you "pull" it from the Docker registry. +``` +docker pull nginx +``` +This command will pull the latest image of [nginx (a HTTP server)](https://nginx.org/en/) from [Docker Hub's registry](https://hub.docker.com/_/nginx). + +#### Stop/Remove Containers & Images +To stop a running container and to remove containers and images you use the following commands. +For stopping a container: +``` +docker stop {container_name_or_id} +``` +For removing a container: +``` +docker rm {container_name_or_id} +``` +For removing an image: +``` +docker rmi {image_name_or_id} +``` + +#### Start and work with containers +To create a new container from an image after you've pulled it, use: +``` +docker run {image_name_or_id} +``` +After the container is created, you can start (launch) it by running: +``` +docker start {container_name_or_id} +``` +If you want to actually run commands inside, for example - to open a Bash terminal: +``` +docker exec -it {container_name_or_id} bash +``` +> The `-it` flag specifies to run it in interactive mode `-i` and to allocate a psueo-TTY `-t`. + +### Advanced Docker Concepts +This is a little out of scope for the training book but if you want to build your Docker image, you will have to use [docker build](https://docs.docker.com/build/). Other docker functionality to be aware of: + +- For managing a lot of containers using declarative config files - [Docker Compose](https://docs.docker.com/compose/) +- To find and share container images (like GitHub) - [Docker Hub](https://docs.docker.com/docker-hub/) +- If you want your containers to talk to each other - [Docker Network](https://docs.docker.com/network/) +- For image scanning, analysis and security-related things - [Docker Scout](https://docs.docker.com/scout/) + +There are a bunch of other extensions and software that's designed to work with Docker that are not mentioned here. Important takeaway here is that if you ever have to manage dependencies and/or work across teams doing that *(which you will definitely have to do in an advanced HPC project)* - containers make your life a lot easier. This is also the case if you want to deploy and scale any web service for which you can use something like [Kubernetes](https://kubernetes.io). Container-based deployments are becoming more and more popular due to how lightweight and flexible they are compared to traditional virtual machines. \ No newline at end of file diff --git a/src/chapter6/challenges.md b/src/parallel-algos/challenges.md similarity index 100% rename from src/chapter6/challenges.md rename to src/parallel-algos/challenges.md diff --git a/src/chapter6/imgs/Beale_contour.svg.png b/src/parallel-algos/imgs/Beale_contour.svg.png similarity index 100% rename from src/chapter6/imgs/Beale_contour.svg.png rename to src/parallel-algos/imgs/Beale_contour.svg.png diff --git a/src/chapter6/imgs/Rastrigin_contour_plot.svg.png b/src/parallel-algos/imgs/Rastrigin_contour_plot.svg.png similarity index 100% rename from src/chapter6/imgs/Rastrigin_contour_plot.svg.png rename to src/parallel-algos/imgs/Rastrigin_contour_plot.svg.png diff --git a/src/chapter6/imgs/Rosenbrock_contour.svg.png b/src/parallel-algos/imgs/Rosenbrock_contour.svg.png similarity index 100% rename from src/chapter6/imgs/Rosenbrock_contour.svg.png rename to src/parallel-algos/imgs/Rosenbrock_contour.svg.png diff --git a/src/chapter6/machine-learning-and-hpc.md b/src/parallel-algos/machine-learning-and-hpc.md similarity index 100% rename from src/chapter6/machine-learning-and-hpc.md rename to src/parallel-algos/machine-learning-and-hpc.md diff --git a/src/chapter6/optim-algos.md b/src/parallel-algos/optim-algos.md similarity index 100% rename from src/chapter6/optim-algos.md rename to src/parallel-algos/optim-algos.md diff --git a/src/chapter6/optimisation-algorithms.md b/src/parallel-algos/optimisation-algorithms.md similarity index 100% rename from src/chapter6/optimisation-algorithms.md rename to src/parallel-algos/optimisation-algorithms.md diff --git a/src/chapter6/other-parallel-algos.md b/src/parallel-algos/other-parallel-algos.md similarity index 100% rename from src/chapter6/other-parallel-algos.md rename to src/parallel-algos/other-parallel-algos.md diff --git a/src/chapter6/chapter6.md b/src/parallel-algos/parallel-algos.md similarity index 100% rename from src/chapter6/chapter6.md rename to src/parallel-algos/parallel-algos.md diff --git a/src/chapter6/parallel-search.md b/src/parallel-algos/parallel-search.md similarity index 100% rename from src/chapter6/parallel-search.md rename to src/parallel-algos/parallel-search.md diff --git a/src/chapter6/parallel-sort.md b/src/parallel-algos/parallel-sort.md similarity index 100% rename from src/chapter6/parallel-sort.md rename to src/parallel-algos/parallel-sort.md