How to set up an environment to train and analyze PCAP data on a Ubuntu machine in this guide version 20.04 (Focal Fossa) is used.

2 - Install LibPCAP and Wireshark for Capture
3 - Download, Compile and Install Argus and RA
4 - Start Packet Capture on Endpoints of Interest
5 - Generate CSV from Capture Files
6 - Copy Anomaly Detection Notebook
7 - Run the code and the get the results using Jupyter Notebook
8 - (Optional) Use NTFLlowLyzer to get the features from PCAP data
9 - (Optional) Install OpenWebUI to Create a GUI for the Output

1. Install Anaconda

To download and install Anaconda on Ubuntu via the command line, follow these steps:
1. Update your package list and install dependencies:
  sudo apt update sudo apt install wget
2. Download the Anaconda installer:
  Visit the Anaconda distribution page to get the link to the latest version. As of now, you can use the following command to download Anaconda:
  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  Note: Check the website for the latest version and update the URL accordingly
3. Install
  chmod +x Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda3
4. Initialize Miniconda:
  If the installer does not automatically initialize Miniconda, you can do it manually:
  ~/miniconda3/bin/conda init
  This command configures your shell to use Conda. You may need to restart your terminal or run source ~/.bashrc for the changes to take effect.
5. Verify and Update
  conda --version conda update conda

2. Create a Python and Developing Environment:

Run the following command to create a new conda environment with Python and JupyterLab:

conda create -n myenv python=3.10

Replace myenv with the desired name of your environment.

Run the following command to refresh the shell and activate the conda environment:

source ~/.bashrc
conda activate myenv

Replace myenv with the name of your conda environment.

conda install keras pandas scikit-learn matplotlib -y

conda install  seaborn  -y
conda install numba cudatoolkit -y
conda install scikit-learn xgboost lightgbm -y

conda install tensorflow-gpu -y
export TF_ENABLE_ONEDNN_OPTS=0
conda install csv -y

conda install -c conda-forge cupy -y
#conda install jupyter

#if kernel isnt recognized
conda install ipykernel
python -m ipykernel install --user --name ai-ml-env --display-name "Python (your_env_name)"

Install JupyterLab:

With the conda environment activated, run the following command to install JupyterLab:

conda install -c conda-forge jupyterlab

Start JupyterLab:

With the conda environment activated, run the following command to start JupyterLab:

jupyter lab

This command will open JupyterLab in your default web browser.
jupyter lab --ip=0.0.0.0 --port=81 --allow-root &

3. Install NVIDIA Python package

With the conda environment still activated, install the NVIDIA Python package using pip:

pip install nvidia

Direct install if conda installation of CUDA or cuDNN does not work

- Install NVIDIA CUDA Toolkit

NVIDIA CUDA Toolkit - Free Tools and Training .
1. Add the NVIDIA package repository:
  First, add the repository to your system:
  sudo apt update sudo apt install wget gnupg wget <https://developer.download.nvidia.com/compute/cuda/repos/ubuntu$(lsb_release> -rs)/x86_64/cuda-ubuntu$(lsb_release -rs).pin sudo mv cuda-ubuntu$(lsb_release -rs).pin /etc/apt/preferences.d/cuda-repository-pin-600
2. Add the CUDA repository public key:
  sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7fa2af80
3. Add the CUDA repository:
  sudo add-apt-repository "deb <http://developer.download.nvidia.com/compute/cuda/repos/ubuntu$(lsb_release> -rs)/x86_64/ /"
4. Update the package list and install the CUDA Toolkit:
  sudo apt update sudo apt install cuda
5. Set up environment variables:
  Add CUDA to your PATH and LD_LIBRARY_PATH. You can do this by editing your .bashrc or .zshrc file:
  nano ~/.bashrc
  Add the following lines at the end of the file:
  export PATH=/usr/local/cuda-<version>/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda-<version>/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  Replace <version> with the version of CUDA you installed, such as 11.8.
  Save the file and reload it:
  source ~/.bashrc
6. Verify the installation:
  You can check if CUDA is installed correctly by running:
  nvcc --version
  You should see the version of CUDA installed.

- Install NVIDIA cuDNN

To install cuDNN (CUDA Deep Neural Network library) on Ubuntu, follow these steps:
1. Download cuDNN:
  Go to the NVIDIA cuDNN download page and sign in with your NVIDIA Developer account (create one if you don’t have it). Select the version of cuDNN that matches your CUDA version and download the appropriate tarball file for Linux:
  wget https://developer.download.nvidia.com/compute/redist/cudnn/v8.7.1/cudnn-8.7.1-linux-x64-v8.7.1.23.tgz curl -O https://developer.download.nvidia.com/compute/redist/cudnn/v8.7.1/cudnn-8.7.1-linux-x64-v8.7.1.23.tgz
2. Extract the cuDNN tarball:
  Once downloaded, extract the files. For example, if you downloaded cudnn-xx.x-linux-x64-vx.x.x.tgz:
  tar -xzvf cudnn-xx.x-linux-x64-vx.x.x.tgz
  Replace xx.x and vx.x.x with the specific version numbers of the file you downloaded.
3. Copy cuDNN files to CUDA directory:
  Assuming you have CUDA installed in /usr/local/cuda, you need to copy the cuDNN files to the appropriate directories:
  sudo cp cuda/include/cudnn*.h /usr/local/cuda/include sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
4. Set permissions:
  To ensure that the libraries are accessible, set the appropriate permissions:
  sudo chmod a+r /usr/local/cuda/include/cudnn*.h sudo chmod a+r /usr/local/cuda/lib64/libcudnn*
5. Verify the installation:
  Check that the cuDNN library is correctly installed and linked:
  cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
  This should output the version of cuDNN installed.

- Install LibPCAP and Wireshark for Capture

LibPCAP and Wireshark are necessary to perform packet capture and packet decoding.
It is also optional to use tcpdump to do packet capture as well
sudo apt install libpcap-dev sudo apt install wireshark

- Download, Compile and Install Argus and RA

Argus is a network flow monitor that will generate the features for the machine learning dataset
After Argus is installed
sudo apt install flex bison rrdtool wget https://qosient.com/argus/src/argus-3.0.8.tar.gz tar -xzvf argus-3.0.8.tar.gz cd argus-3.0.8 ./configure make && make install cd .. wget https://qosient.com/argus/src/argus-clients-3.0.8.tar.gz tar -xzvf argus-clients-3.0.8.tar.gz cd argus-clients-3.0.8 ./configure make && make install

- Start Packet Capture on Endpoints of Interest

Go to endpoints that have Wireshark or tcpdump installed and run a capture either on a wireless interface or cabled ethernet interface
Pick either tcpdump or tshark methods to capture packets
Run the capture passively on the device using tcpdump
nohup tcpdump -i <interface> -w capture.pcap > /dev/null &
Run capture using tshark (Wireshark toolkit)
nohup tshark -i <interface> -w capture.pcap -F libpcap > /dev/null &
Replace <interface> with the actual device name from ‘ip -s link’ or ‘ip addr show’
Repeat on all the endpoints of interest
Wait for a couple of hours or a couple of days to gather packets, the more the better

- Generate CSV from Capture Files

Using Argus and RA create a CSV file that is digestable by the machine learning code
argus -f capture.pcap -w arg.out
ra -r arg.out -w traffic.csv

- Copy Anomaly Detection Notebook

Copy the Anomaly Detection Notebook onto local Jupyter Notebook

Anomaly Detection Notebook - Kaggle

Change the input file in the python code to use traffic.csv instead of the default csv file.
Add the full path of the CSV file is necessary
df = pd.read_csv('traffic.csv')

- Run the code and the get the results using Jupyter Notebook

- (Optional) Use NTFLlowLyzer to get the features from PCAP data

GitHub - ahlashkari/NTLFlowLyzer

- (Optional) Install OpenWebUI to Create a GUI for the Output

Quantea Knowledge Base - External

Creating an AI Server - Environment Setup