Add transformers dependency and update notebooks (#486)

Added missing dependencies and updated broken notebooks.

Authors:
  - https://github.com/bsuryadevara
  - https://github.com/bsuryadev

Approvers:
  - Tad ZeMicheal (https://github.com/tzemicheal)
  - Eli Fajardo (https://github.com/efajardo-nv)
  - https://github.com/gbatmaz
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #486

Author: bsuryadevara

CONTRIBUTING.md

@@ -115,16 +115,16 @@ run each time you commit changes.
 
 ### Build from Source
 
-The following instructions are for developers and contributors to CLX OSS development. These instructions are tested on Linux Ubuntu 16.04 & 18.04. Use these instructions to build CLX from source and contribute to its development.  Other operating systems may be compatible, but are not currently tested.
+The following instructions are for developers and contributors to CLX OSS development. These instructions are tested on Linux Ubuntu 18.04 & 20.04. Use these instructions to build CLX from source and contribute to its development.  Other operating systems may be compatible, but are not currently tested.
 
 The following instructions are tested on Linux systems.
 
 #### Prerequisites
 
 CUDA requirement:
 
-* CUDA 11.0
-* NVIDIA driver 396.44+
+* CUDA 11.5
+* NVIDIA driver 470.82+
 * Pascal architecture or better
 
 You can obtain CUDA from [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads).
@@ -146,8 +146,7 @@ Create the conda development environment:
 ```bash
 # create the conda environment (assuming in base `clx` directory)
 
-conda env create --name clx_dev --file conda/environments/clx_dev_cuda11.0.yml
-
+mamba env create --name clx_dev --file conda/environments/clx_dev_cuda11.5.yml
 
 # activate the environment
 conda activate clx_dev
@@ -159,7 +158,7 @@ conda deactivate
 The environment can be updated as development includes/changes the dependencies. To do so, run:
 
 ```bash
-conda env update --name clx_dev --file conda/environments/clx_dev_cuda11.0.yml
+mamba env update --name clx_dev --file conda/environments/clx_dev_cuda11.5.yml
 
 conda activate clx_dev
 ```
@@ -176,4 +175,4 @@ $ python setup.py install
 Python API documentation can be generated from [docs](docs) directory.
 
 ## Attribution
-Portions adopted from https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md
+Portions adopted from https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md

Dockerfile

@@ -1,29 +1,36 @@
 # An integration test & dev container which builds and installs CLX from default branch
-ARG RAPIDS_VERSION=21.06
-ARG CUDA_VERSION=10.1
+ARG RAPIDS_VERSION=22.06
+ARG CUDA_VERSION=11.5
 ARG CUDA_SHORT_VERSION=${CUDA_VERSION}
 ARG LINUX_VERSION=ubuntu18.04
-ARG PYTHON_VERSION=3.7
+ARG PYTHON_VERSION=3.8
 FROM rapidsai/rapidsai-dev-nightly:${RAPIDS_VERSION}-cuda${CUDA_VERSION}-devel-${LINUX_VERSION}-py${PYTHON_VERSION}
 
 # Add everything from the local build context
 ADD . /rapids/clx/
 RUN chmod -R ugo+w /rapids/clx/
 
-RUN apt update -y --fix-missing && \
-    apt upgrade -y && \
-    apt install -y krb5-user
-
 RUN source activate rapids && \
-    conda install -c pytorch "pytorch=1.7.1" torchvision "cudf_kafka=${RAPIDS_VERSION}" "custreamz=${RAPIDS_VERSION}" "scikit-learn>=0.21" "nodejs>=12" ipywidgets python-confluent-kafka "transformers=4.*" "seqeval=0.0.12" python-whois seaborn requests matplotlib pytest jupyterlab && \
+    gpuci_mamba_retry install -y -n rapids \
+        "cudf_kafka=${RAPIDS_VER}" \
+        "custreamz=${RAPIDS_VER}" \
+        scikit-learn>=0.21 \
+        nodejs>=12 \
+        ipywidgets \
+        python-confluent-kafka \
+        seqeval \
+        python-whois \
+        seaborn \
+        requests \
+        matplotlib \
+        pytest \
+        jupyterlab=3.0 \
+        faker && \
+    pip install -U torch==1.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html && \
     pip install "git+https://github.com/rapidsai/cudatashader.git" && \
     pip install mockito && \
     pip install wget && \
     pip install "git+https://github.com/slashnext/SlashNext-URL-Analysis-and-Enrichment.git#egg=slashnext-phishing-ir&subdirectory=Python SDK/src"
-    
-RUN source activate rapids \
-  && conda install -n rapids jupyterlab-nvdashboard \
-  && jupyter labextension install @jupyter-widgets/jupyterlab-manager dask-labextension jupyterlab-nvdashboard
 
 # clx build/install
 RUN source activate rapids && \

README.md

@@ -33,7 +33,7 @@ Please see the [Demo Docker Repository](https://hub.docker.com/r/rapidsai/rapids
 Pull image:
 
 ```sh
-docker pull rapidsai/rapidsai-clx:21.06-cuda11.0-runtime-ubuntu18.04-py3.7
+docker pull rapidsai/rapidsai-clx:22.04-cuda11.5-runtime-ubuntu18.04-py3.8
 ```
 
 Nightly images for current development version can be pulled from https://hub.docker.com/r/rapidsai/rapidsai-clx-nightly.
@@ -46,7 +46,7 @@ docker run -it --gpus '"device=0"' \
   -p 8888:8888 \
   -p 8787:8787 \
   -p 8686:8686 \
-  rapidsai/rapidsai-clx:21.06-cuda11.0-runtime-ubuntu18.04-py3.7
+  rapidsai/rapidsai-clx:22.04-cuda11.5-runtime-ubuntu18.04-py3.8
 ```
 
 ##### Legacy - Docker CE v18 and nvidia-docker2
@@ -56,7 +56,7 @@ docker run -it --runtime=nvidia \
   -p 8888:8888 \
   -p 8787:8787 \
   -p 8686:8686 \
-  rapidsai/rapidsai-clx:21.06-cuda11.0-runtime-ubuntu18.04-py3.7
+  rapidsai/rapidsai-clx:22.04-cuda11.5-runtime-ubuntu18.04-py3.8
 ```
 
 #### Container Ports
@@ -73,15 +73,15 @@ The following ports are used by the **runtime containers only** (not base contai
 Prerequisites
 
 * NVIDIA Pascal™ GPU architecture or better
-* CUDA 11.0 compatible NVIDIA driver
-* Ubuntu 16.04/18.04 or CentOS 7
+* CUDA 11.5+ compatible NVIDIA driver
+* Ubuntu 18.04/20.04 or CentOS 7
 * Docker CE v18+
 * nvidia-docker v2+
 
 Pull the RAPIDS image suitable to your environment and build CLX image. Please see the [rapidsai-dev](https://hub.docker.com/r/rapidsai/rapidsai-dev) or [rapidsai-dev-nightly](https://hub.docker.com/r/rapidsai/rapidsai-dev-nightly) Docker repositories, choosing a tag based on the NVIDIA CUDA version you’re running. More information on getting started with RAPIDS can be found [here](https://rapids.ai/start.html).
 
 ```sh
-docker pull rapidsai/rapidsai-dev:21.06-cuda11.0-devel-ubuntu18.04-py3.7
+docker pull rapidsai/rapidsai-dev:22.04-cuda11.5-devel-ubuntu18.04-py3.8
 docker build -t clx:latest .
 ```
 

ci/gpu/build.sh

@@ -39,7 +39,7 @@ gpuci_logger "Activate conda env"
 . /opt/conda/etc/profile.d/conda.sh
 conda activate rapids
 
-gpuci_logger "Install tests dependencies"
+gpuci_logger "Install conda dependencies"
 gpuci_mamba_retry install -y \
     "cuxfilter=${MINOR_VERSION}" \
     "faker" \

conda/environments/clx_dev_cuda11.0.yml conda/environments/clx_dev_cuda11.5.yml

@@ -2,12 +2,10 @@ name: clx_dev
 channels:
 - rapidsai
 - rapidsai-nightly
-- pytorch
 - conda-forge
 dependencies:
-- cudatoolkit=11.0
+- cudatoolkit=11.5
 - python>=3.6,<3.9
-- pytorch=1.7.1
 - cugraph=22.06.*
 - cuml=22.06.*
 - cuxfilter=22.06.*
@@ -35,3 +33,4 @@ dependencies:
   - "git+https://github.com/slashnext/SlashNext-URL-Analysis-and-Enrichment.git#egg=slashnext-phishing-ir&subdirectory=Python SDK/src"
   - wget
   - mockito
+  - torch==1.11.0

notebooks/anomalous_behavior_profiling/anomalous_behavior_profiling_supervised.ipynb

@@ -41,13 +41,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
     "import xgboost as xgb\n",
     "import cudf\n",
-    "from cuml.preprocessing.model_selection import train_test_split\n",
+    "from cuml.model_selection import train_test_split\n",
     "from cuml import ForestInference\n",
     "import sklearn.datasets\n",
     "import cupy"
@@ -379,7 +379,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -393,7 +393,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.8"
+   "version": "3.8.13"
   }
  },
  "nbformat": 4,

notebooks/ids_detection/IDS_using_LODA.ipynb

@@ -40,7 +40,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Load and preprocess dataset"
+    "#### Load and preprocess dataset\n",
+    "\n",
+    "Download `GeneratedLabelledFlows.zip` from the above link and extract all files in a selected path."
    ]
   },
   {
@@ -481,7 +483,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -495,7 +497,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.8"
+   "version": "3.8.13"
   },
   "toc-autonumbering": false,
   "toc-showcode": true,