From 6209662d3e5952673cec88389c8af168d48e4990 Mon Sep 17 00:00:00 2001 From: Amadeusz Kryze Date: Thu, 16 Mar 2023 18:14:00 +0100 Subject: [PATCH 1/4] [openzfs/zfs#14611] Fix missing install from source steps. --- ..., build from source till working guide.rst | 243 ++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst diff --git a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst new file mode 100644 index 000000000..33375650d --- /dev/null +++ b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst @@ -0,0 +1,243 @@ +User friendly, zero to hero, build from source till working guide (Ubuntu based) +============ + +GitHub Repositories +~~~~~~~~~~~~~~~~~~~ + +The official source for OpenZFS is maintained at GitHub by the +`openzfs `__ organization. The primary +git repository for the project is the `zfs +`__ repository. + +Installing Dependencies +~~~~~~~~~~~~~~~~~~~~~~~ + +The first thing you'll need to do is prepare your environment by +installing a full development tool chain. In addition, development +headers for both the kernel and the following packages must be +available. It is important to note that if the development kernel +headers for the currently running kernel aren't installed, the modules +won't compile properly. + +The following dependencies should be installed to build the latest ZFS release. + +- **Debian, Ubuntu**: + +.. code:: sh + + sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev linux-headers-generic python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev + +- **Ansible**: + +.. code:: ansible + + - ansible.builtin.package: + name: + - build-essential + - autoconf + - automake + - libtool + - gawk + - alien + - fakeroot + - dkms + - libblkid-dev + - uuid-dev + - libudev-dev + - libssl-dev + - zlib1g-dev + - libaio-dev + - libattr1-dev + - libelf-dev + - linux-headers-generic + - python3 + - python3-dev + - python3-setuptools + - python3-cffi + - libffi-dev + - python3-packaging + - git + - libcurl4-openssl-dev + state: latest + update_cache: true + become: true + +Getting the sources +~~~~~~~~~~~~~~~~~~ + +Clone from GitHub +^^^^^^^^^^^^^^^^^ + +Start by cloning the ZFS repository from GitHub. The repository has a +**master** branch for development and a series of **\*-release** +branches for tagged releases. After checking out the repository your +clone will default to the master branch. Tagged releases may be built +by checking out zfs-x.y.z tags with matching version numbers or +matching release branches. + +- **Debian, Ubuntu**: + +.. code:: sh + + git clone https://github.com/openzfs/zfs + +- **Ansible**: + +.. code:: ansible + + - ansible.builtin.file: + path: /opt + mode: '0777' + become: true + + - ansible.builtin.uri: + url: https://api.github.com/repos/openzfs/zfs/releases/latest + return_content: true + register: json_reponse + + - ansible.builtin.git: + repo: https://github.com/openzfs/zfs.git + dest: /opt/zfs + version: "{{ json_reponse.json.tag_name | default('master') }}" + +Preparing the rest of the system +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now pay attention to how your distribution handles kernel modules. On Ubuntu, +for example, the modules from this repository install in the ``extra`` kernel +module path, which is not in the standard ``depmod`` search path. Therefore, +for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add +``extra`` to the beginning of the search path. + +- **Debian, Ubuntu**: + +.. code:: sh + + sudo vim /etc/depmod.d/ubuntu.conf + +- **Ansible**: + +.. code:: ansible + + - ansible.builtin.lineinfile: + dest: /etc/depmod.d/ubuntu.conf + regexp: '^(search updates ubuntu built-in)$' + line: '\1 extra' + backrefs: yes + become: true + +Building +~~~~~~~~~~~~~~~~~~~~~~~ + +The ZFS build system is based on GNU Autoconf and GNU Automake. So the +first step is to run the ``autogen.sh`` script to generate the +``configure`` script. This script is used to configure the build +environment and generate the ``Makefile`` used to build the ZFS +modules. + +- **Debian, Ubuntu**: + +.. code:: sh + + git clean -fx + ./autogen.sh + ./configure --enable-systemd + make -s -j$(nproc) deb-utils deb-dkms + +- **Ansible**: + +.. code:: ansible + + - ansible.builtin.shell: | + git clean -fx + args: + executable: /bin/bash + chdir: /opt/zfs + + - ansible.builtin.shell: | + ./autogen.sh + args: + executable: /bin/bash + chdir: /opt/zfs + + - ansible.builtin.shell: | + ./configure --enable-systemd + args: + executable: /bin/bash + chdir: /opt/zfs + + - ansible.builtin.shell: | + make -j$(nproc) deb-utils deb-dkms + args: + executable: /bin/bash + chdir: /opt/zfs + +Installing +~~~~~~~~~~~~~~~~~~~~~~~ + +The ZFS packages are built using the `Debian Package`_ format. The +packages are built using the ``make deb-utils deb-dkms`` command. The +``deb-utils`` package contains the ``zfs`` and ``zpool`` user space +utilities. The ``deb-dkms`` package contains the ZFS kernel modules +and a DKMS configuration file. DKMS is used to automatically rebuild +and install the kernel modules when a new kernel is installed. + +- **Debian, Ubuntu**: + +.. code:: sh + + sudo apt install ./*.deb + +- **Ansible**: + + - ansible.builtin.shell: | + shopt -s extglob + apt install ./*.deb + args: + executable: /bin/bash + chdir: /opt/zfs + become: true + +Post Install +~~~~~~~~~~~~~~~~~~~~~~~ + +After installing the ZFS packages, the ZFS services must be enabled +and started. The ``zfs-import-cache`` service is responsible for +importing the ZFS pools during system boot. The ``zfs-mount`` +service is responsible for mounting all filesystems in the system's +root pool. The ``zfs-zed`` service is responsible for monitoring the +system for events and taking appropriate actions. The ``zfs-share`` +service is responsible for automatically sharing any ZFS filesystems +marked as shareable. The ``zfs.target`` is a convenience target that +will start all of the ZFS services. The ``zfs-import.target`` is a +convenience target that will start the ``zfs-import-cache`` and +``zfs-import-scan`` services. + +- **Debian, Ubuntu**: + +.. code:: sh + + sudo service enable zfs-import-cache zfs-import.target zfs-mount zfs-zed zfs-share zfs-volume-wait zfs.target + sudo service start zfs-import-cache zfs-import.target zfs-mount zfs-zed zfs-share zfs-volume-wait zfs.target + +- **Ansible**: + +.. code:: ansible + + - ansible.builtin.service: + name: + - zfs-import-cache + - zfs-import.target + - zfs-mount + - zfs-share + - zfs-zed + - zfs-volume-wait + - zfs.target + state: started + enabled: yes + become: true + +Final step +~~~~~~~~~~~~~~~~~~~~~~~ + +Now reboot, and you should be able to use ZFS. From f312ccf880587bd616c7a4af62652aa367af1ae8 Mon Sep 17 00:00:00 2001 From: Amadeusz Kryze Date: Thu, 16 Mar 2023 18:46:17 +0100 Subject: [PATCH 2/4] Fixed linter errors --- ..., build from source till working guide.rst | 28 ++++++++++--------- docs/Developer Resources/index.rst | 1 + 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst index 33375650d..622e9764b 100644 --- a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst +++ b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst @@ -1,5 +1,5 @@ User friendly, zero to hero, build from source till working guide (Ubuntu based) -============ +================================================================================ GitHub Repositories ~~~~~~~~~~~~~~~~~~~ @@ -29,7 +29,7 @@ The following dependencies should be installed to build the latest ZFS release. - **Ansible**: -.. code:: ansible +.. code:: sh - ansible.builtin.package: name: @@ -63,7 +63,7 @@ The following dependencies should be installed to build the latest ZFS release. become: true Getting the sources -~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~ Clone from GitHub ^^^^^^^^^^^^^^^^^ @@ -83,7 +83,7 @@ matching release branches. - **Ansible**: -.. code:: ansible +.. code:: sh - ansible.builtin.file: path: /opt @@ -101,7 +101,7 @@ matching release branches. version: "{{ json_reponse.json.tag_name | default('master') }}" Preparing the rest of the system -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now pay attention to how your distribution handles kernel modules. On Ubuntu, for example, the modules from this repository install in the ``extra`` kernel @@ -117,7 +117,7 @@ for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add - **Ansible**: -.. code:: ansible +.. code:: sh - ansible.builtin.lineinfile: dest: /etc/depmod.d/ubuntu.conf @@ -127,7 +127,7 @@ for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add become: true Building -~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~ The ZFS build system is based on GNU Autoconf and GNU Automake. So the first step is to run the ``autogen.sh`` script to generate the @@ -146,7 +146,7 @@ modules. - **Ansible**: -.. code:: ansible +.. code:: sh - ansible.builtin.shell: | git clean -fx @@ -173,9 +173,9 @@ modules. chdir: /opt/zfs Installing -~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~ -The ZFS packages are built using the `Debian Package`_ format. The +The ZFS packages are built using the `Debian Package` format. The packages are built using the ``make deb-utils deb-dkms`` command. The ``deb-utils`` package contains the ``zfs`` and ``zpool`` user space utilities. The ``deb-dkms`` package contains the ZFS kernel modules @@ -190,6 +190,8 @@ and install the kernel modules when a new kernel is installed. - **Ansible**: +.. code:: sh + - ansible.builtin.shell: | shopt -s extglob apt install ./*.deb @@ -199,7 +201,7 @@ and install the kernel modules when a new kernel is installed. become: true Post Install -~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~ After installing the ZFS packages, the ZFS services must be enabled and started. The ``zfs-import-cache`` service is responsible for @@ -222,7 +224,7 @@ convenience target that will start the ``zfs-import-cache`` and - **Ansible**: -.. code:: ansible +.. code:: sh - ansible.builtin.service: name: @@ -238,6 +240,6 @@ convenience target that will start the ``zfs-import-cache`` and become: true Final step -~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~ Now reboot, and you should be able to use ZFS. diff --git a/docs/Developer Resources/index.rst b/docs/Developer Resources/index.rst index 3b5d62b74..e0c6d72c5 100644 --- a/docs/Developer Resources/index.rst +++ b/docs/Developer Resources/index.rst @@ -6,6 +6,7 @@ Developer Resources :caption: Contents: :glob: + User friendly, zero to hero, build from source till working guide Custom Packages Building ZFS Buildbot Status From e479e43ecef3da8f5007f996e7cc9faa3dda4529 Mon Sep 17 00:00:00 2001 From: Amadeusz Kryze Date: Thu, 16 Mar 2023 18:51:03 +0100 Subject: [PATCH 3/4] Added more fixes in text. --- ..., build from source till working guide.rst | 95 ++++++++++--------- 1 file changed, 49 insertions(+), 46 deletions(-) diff --git a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst index 622e9764b..b3bf36c3c 100644 --- a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst +++ b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst @@ -4,20 +4,12 @@ User friendly, zero to hero, build from source till working guide (Ubuntu based) GitHub Repositories ~~~~~~~~~~~~~~~~~~~ -The official source for OpenZFS is maintained at GitHub by the -`openzfs `__ organization. The primary -git repository for the project is the `zfs -`__ repository. +The official source for OpenZFS is maintained at GitHub by the `openzfs `__ organization. The primary git repository for the project is the `zfs `__ repository. Installing Dependencies ~~~~~~~~~~~~~~~~~~~~~~~ -The first thing you'll need to do is prepare your environment by -installing a full development tool chain. In addition, development -headers for both the kernel and the following packages must be -available. It is important to note that if the development kernel -headers for the currently running kernel aren't installed, the modules -won't compile properly. +The first thing you'll need to do is prepare your environment by installing a full development tool chain. In addition, development headers for both the kernel and the following packages must be available. It is important to note that if the development kernel headers for the currently running kernel aren't installed, the modules won't compile properly. The following dependencies should be installed to build the latest ZFS release. @@ -25,7 +17,32 @@ The following dependencies should be installed to build the latest ZFS release. .. code:: sh - sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev linux-headers-generic python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev + sudo apt install -y \ + build-essential \ + autoconf \ + automake \ + libtool \ + gawk \ + alien \ + fakeroot \ + dkms \ + libblkid-dev \ + uuid-dev \ + libudev-dev \ + libssl-dev \ + zlib1g-dev \ + libaio-dev \ + libattr1-dev \ + libelf-dev \ + linux-headers-generic \ + python3 \ + python3-dev \ + python3-setuptools \ + python3-cffi \ + libffi-dev \ + python3-packaging \ + git \ + libcurl4-openssl-dev - **Ansible**: @@ -68,12 +85,7 @@ Getting the sources Clone from GitHub ^^^^^^^^^^^^^^^^^ -Start by cloning the ZFS repository from GitHub. The repository has a -**master** branch for development and a series of **\*-release** -branches for tagged releases. After checking out the repository your -clone will default to the master branch. Tagged releases may be built -by checking out zfs-x.y.z tags with matching version numbers or -matching release branches. +Start by cloning the ZFS repository from GitHub. The repository has a **master** branch for development and a series of **\*-release** branches for tagged releases. After checking out the repository your clone will default to the master branch. Tagged releases may be built by checking out zfs-x.y.z tags with matching version numbers or matching release branches. - **Debian, Ubuntu**: @@ -103,11 +115,7 @@ matching release branches. Preparing the rest of the system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Now pay attention to how your distribution handles kernel modules. On Ubuntu, -for example, the modules from this repository install in the ``extra`` kernel -module path, which is not in the standard ``depmod`` search path. Therefore, -for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add -``extra`` to the beginning of the search path. +Now pay attention to how your distribution handles kernel modules. On Ubuntu, for example, the modules from this repository install in the ``extra`` kernel module path, which is not in the standard ``depmod`` search path. Therefore, for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add ``extra`` to the beginning of the search path. - **Debian, Ubuntu**: @@ -129,11 +137,7 @@ for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add Building ~~~~~~~~ -The ZFS build system is based on GNU Autoconf and GNU Automake. So the -first step is to run the ``autogen.sh`` script to generate the -``configure`` script. This script is used to configure the build -environment and generate the ``Makefile`` used to build the ZFS -modules. +The ZFS build system is based on GNU Autoconf and GNU Automake. So the first step is to run the ``autogen.sh`` script to generate the ``configure`` script. This script is used to configure the build environment and generate the ``Makefile`` used to build the ZFS modules. - **Debian, Ubuntu**: @@ -175,12 +179,7 @@ modules. Installing ~~~~~~~~~~ -The ZFS packages are built using the `Debian Package` format. The -packages are built using the ``make deb-utils deb-dkms`` command. The -``deb-utils`` package contains the ``zfs`` and ``zpool`` user space -utilities. The ``deb-dkms`` package contains the ZFS kernel modules -and a DKMS configuration file. DKMS is used to automatically rebuild -and install the kernel modules when a new kernel is installed. +The ZFS packages are built using the ``Debian Package`` format. The packages are built using the ``make deb-utils deb-dkms`` command. The ``deb-utils`` package contains the ``zfs`` and ``zpool`` user space utilities. The ``deb-dkms`` package contains the ZFS kernel modules and a DKMS configuration file. DKMS is used to automatically rebuild and install the kernel modules when a new kernel is installed. - **Debian, Ubuntu**: @@ -203,24 +202,28 @@ and install the kernel modules when a new kernel is installed. Post Install ~~~~~~~~~~~~ -After installing the ZFS packages, the ZFS services must be enabled -and started. The ``zfs-import-cache`` service is responsible for -importing the ZFS pools during system boot. The ``zfs-mount`` -service is responsible for mounting all filesystems in the system's -root pool. The ``zfs-zed`` service is responsible for monitoring the -system for events and taking appropriate actions. The ``zfs-share`` -service is responsible for automatically sharing any ZFS filesystems -marked as shareable. The ``zfs.target`` is a convenience target that -will start all of the ZFS services. The ``zfs-import.target`` is a -convenience target that will start the ``zfs-import-cache`` and -``zfs-import-scan`` services. +After installing the ZFS packages, the ZFS services must be enabled and started. The ``zfs-import-cache`` service is responsible for importing the ZFS pools during system boot. The ``zfs-mount`` service is responsible for mounting all filesystems in the system's root pool. The ``zfs-zed`` service is responsible for monitoring the system for events and taking appropriate actions. The ``zfs-share`` service is responsible for automatically sharing any ZFS filesystems marked as shareable. The ``zfs.target`` is a convenience target that will start all of the ZFS services. The ``zfs-import.target`` is a convenience target that will start the ``zfs-import-cache`` and ``zfs-import-scan`` services. - **Debian, Ubuntu**: .. code:: sh - sudo service enable zfs-import-cache zfs-import.target zfs-mount zfs-zed zfs-share zfs-volume-wait zfs.target - sudo service start zfs-import-cache zfs-import.target zfs-mount zfs-zed zfs-share zfs-volume-wait zfs.target + sudo service enable \ + zfs-import-cache \ + zfs-import.target \ + zfs-mount \ + zfs-zed \ + zfs-share \ + zfs-volume-wait \ + zfs.target + sudo service start \ + zfs-import-cache \ + zfs-import.target \ + zfs-mount \ + zfs-zed \ + zfs-share \ + zfs-volume-wait \ + zfs.target - **Ansible**: From 1feeb500ad6689e4d647063939c30b2aeaf6cf23 Mon Sep 17 00:00:00 2001 From: Amadeusz Kryze Date: Fri, 17 Mar 2023 10:36:11 +0100 Subject: [PATCH 4/4] Adjusted, bash - ansible to 1:1 --- ...zero to hero, build from source till working guide.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst index b3bf36c3c..5d2a4d1be 100644 --- a/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst +++ b/docs/Developer Resources/User friendly, zero to hero, build from source till working guide.rst @@ -91,6 +91,7 @@ Start by cloning the ZFS repository from GitHub. The repository has a **master** .. code:: sh + sudo chmod 777 /opt git clone https://github.com/openzfs/zfs - **Ansible**: @@ -102,15 +103,10 @@ Start by cloning the ZFS repository from GitHub. The repository has a **master** mode: '0777' become: true - - ansible.builtin.uri: - url: https://api.github.com/repos/openzfs/zfs/releases/latest - return_content: true - register: json_reponse - - ansible.builtin.git: repo: https://github.com/openzfs/zfs.git dest: /opt/zfs - version: "{{ json_reponse.json.tag_name | default('master') }}" + version: master Preparing the rest of the system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~