A server to monitor sequencer run status and host a GUI for run-level QA and variant interpretation.
Acquired a Dell Precision 3640 tower workstation in early 2021 with the following specs.
- Intel Xeon W-1290P (support for ECC memory; best single-thread performance)
- 128GB DDR4-2666 ECC Memory (ECC reduces odds of data corruption)
- 2x 512GB Toshiba Kioxia XG6 PCIe NVMe SSD (nvme0n1, nvme1n1)
- 2x 8TB 7200rpm SATA 3.5" HDD (sda, sdb)
We'll choose to install Ubuntu 20.04 for backward compatibility and for security updates supported till 2025.
- On a Windows PC, download the Ubuntu Desktop 20.04.4 ISO image.
- Boot into BIOS on the server (press F2 during Dell logo), and ensure that:
SATA Operation
is set to AHCI (disables hardware RAID)Secure Boot
is enabled (works with Ubuntu for now)AC Recovery
is set toLast Power State
POST Behaviour
is set toContinue on Warnings
andKeyboard Error Detection
is disabledVirtualization
,VT for Direct I/O
, andTrusted Execution
are enabled
- Follow these instructions to create a bootable USB stick.
- Plug in the USB stick and boot into it (press F12 during Dell logo).
- Choose to
Install Ubuntu
and selectMinimal installation
when prompted. - Under
Installation Type
chooseSomething else
, and partition the first SSD (nvme0n1) as follows:- Two partitions, 128MB for EFI and remainder
ext4
mounted as/
- Make sure the boot loader is installed on this SSD (nvme0n1)
- Two partitions, 128MB for EFI and remainder
- Follow on-screen instructions for remaining steps (includes setting a hostname) until reboot.
- After booting into the installed OS, start Terminal and install the following:
sudo apt update sudo apt install -y openssh-server zfsutils-linux samba git vim screen parallel tree curl
- Create a unix group for your team (E.g.
dx
below) and make it your primary group:sudo groupadd dx sudo usermod -g dx $USER sudo groupdel $USER
- Create a mirrored ZFS disk pool on the two HDDs, with an L2ARC cache on the second SSD, and make it writable for your team:
sudo zpool create hot mirror /dev/sda /dev/sdb sudo chown $USER:dx /hot sudo chmod 775 /hot sudo zpool add hot cache nvme1n1
- Share the folder
/hot
over the network via samba:echo -e "[hot]\n path=/hot\n writeable=yes" | sudo tee -a /etc/samba/smb.conf sudo service smbd restart
- Set a samba password for your username using
sudo smbpasswd -a $USER
- Allow OpenSSH/Samba/Web connections through the firewall and then enable it:
sudo ufw allow OpenSSH sudo ufw allow Samba sudo ufw allow 'Nginx Full' sudo ufw enable
Since we set up /hot
as network-attached storage (NAS) on the server, we can create a folder /hot/runs
and specify \\dx-var\hot\runs
as the run output folder on each sequencer. This is sufficient for small clinical labs with at least one engineer who can be "on call" if anything breaks. However, Illumina strongly recommends an enterprise NAS solution hosted by your organization's IT team to ensure redundancy, reliability, and tech support. Our IT team configured a 10TB NAS (\\aipnasvm01\pathnovaseq
) co-located with our Dragen server in a nearby data center, so we went with that. Below is how we mounted this NAS under /mnt/pns
on our server.
- Install dependencies, create the mount point, and a file to store NAS credentials:
sudo apt install -y cifs-utils sudo mkdir /mnt/pns sudo chmod 775 /mnt/pns sudo touch /etc/cifs-utils/aipnasvm01-creds sudo chmod 600 /etc/cifs-utils/aipnasvm01-creds
- Edit
/etc/cifs-utils/aipnasvm01-creds
and enter credentials in the following format:username=ckandoth password=blahblah domain=ad
- Create
/etc/systemd/system/mnt-pns.mount
with content as follows:[Unit] Description=pathnovaseq share on aipnasvm01 Requires=systemd-networkd.service After=network-online.target Wants=network-online.target [Mount] What=//aipnasvm01.ad.medctr.ucla.edu/pathnovaseq Where=/mnt/pns Options=credentials=/etc/cifs-utils/aipnasvm01-creds,iocharset=utf8,rw,x-systemd.automount,uid=1000,gid=1001,dir_mode=0775,file_mode=0664 Type=cifs TimeoutSec=30 [Install] WantedBy=multi-user.target
- Start and enable the service, then run
df -ht cifs
to make sure the NAS is mounted:sudo systemctl start mnt-pns.mount sudo systemctl enable mnt-pns.mount
- Create
/etc/systemd/system/mnt-pns.automount
with content as follows:[Unit] Description=pathnovaseq share on aipnasvm01 [Automount] Where=/mnt/pns TimeoutIdleSec=0 [Install] WantedBy=multi-user.target
- Enable this service to ensure that the NAS is automounted on demand:
sudo systemctl enable mnt-pns.automount
Click here to read about Run Scanner. These are the installation steps summarized:
- Install dependencies, open a port, and create a config file:
sudo apt install -y tomcat9 maven clang cmake libjsoncpp-dev autoconf libtool build-essential sudo ufw allow 8080/tcp echo -e '<Context>\n <Parameter name="runscanner.configFile" value="/etc/runscanner.json" override="false"/>\n</Context>' | sudo tee /var/lib/tomcat9/conf/Catalina/localhost/runscanner.xml
- Create
/etc/runscanner.json
and use it to describe paths to each run folder. For example:[ { "path": "/hot/runs", "platformType": "ILLUMINA", "name": "default", "timeZone": "America/Los_Angeles", "parameters": {} }, { "path": "/mnt/pns/runs", "platformType": "ILLUMINA", "name": "default", "timeZone": "America/Los_Angeles", "parameters": {} } ]
- Download, build, and deploy runscanner with support for parsing Illumina InterOp files:
git clone --recursive [email protected]:miso-lims/runscanner.git cd runscanner/runscanner-illumina export CXX=$(which clang++) && ./build-illumina-interop && autoreconf -i && ./configure && make && sudo make install cd .. mvn clean package sudo cp scanner/target/scanner-*.war /var/lib/tomcat9/webapps/runscanner.war
- Open
localhost:8080/runscanner
in a browser to make sure the graphical interface works.
Click here to see some tutorials on this suite of highly configurable tools. VSPipeline helps automate most of the steps from variant calling to a clinical report, and VSClinical provides a graphical interface for variant interpretation before sign-out by a molecular pathologist. These are the installation steps summarized:
- Download the version of VarSeq we want to deploy, and install it:
sudo mkdir -m 775 /opt/VarSeq-2.2.5 sudo chown $USER:dx /opt/VarSeq-2.2.5 curl -L https://www.goldenhelix.com/download/VarSeq/VarSeq-Lin64-2.2.5.tar.gz -o /opt/VarSeq-2.2.5/VarSeq-Lin64-2.2.5.tar.gz tar -zxf /opt/VarSeq-2.2.5/VarSeq-Lin64-2.2.5.tar.gz -C /opt/VarSeq-2.2.5 --strip-components 1
- Create
/hot/varseq
for app data and symlink to it from the installation folder:mkdir /hot/varseq chmod 775 /hot/varseq cd /opt/VarSeq-2.2.5 ln -s /hot/varseq AppData
- Create a system user named
vsw-user
and use it to download VSWarehouse:sudo adduser --system --ingroup dx vsw-user sudo usermod -a -G sudo vsw-user sudo mkdir -m 775 /opt/VSWarehouse-1.3.22 sudo chown vsw-user:dx /opt/VSWarehouse-1.3.22 sudo -u vsw-user -s /bin/bash git clone --branch v1.3.22 --recursive https://goldenhelix.kilnhg.com/Code/Public/Warehouse/vswarehouse.git /opt/VSWarehouse-1.3.22
- Configure and install VSWarehouse per instructions here:
cd /opt/VSWarehouse-1.3.22 ./install_setup_warehouse_ini.sh sudo ./install_dependencies.sh ./install_create_db.sh ./install.sh sudo ./install_services.sh ./install_jenkins_cli.sh vsapp/vspipeline -c download_annotations reference GRCh37 vsapp/vspipeline -c download_annotations reference GRCh38