# Notebooks Directory

This directory contains Jupyter notebooks for data exploration, preprocessing, and machine learning tasks.

## Structure

- `README.md`: This file explains the purpose and contents of the notebooks.
- `section0_data_preprocessing_and_cleaning.ipynb`: Data decoding and sampling.
- `section1_data_exploration_and_preprocessing.ipynb`: Data exploration and preprocessing.
- `section2_supervised_learning_classification.ipynb`: Supervised learning for intent classification.
- `section3_unsupervised_learning_clustering.ipynb`: Clustering SSH attack sessions.
- `section4_language_model_exploration.ipynb`: Exploring advanced NLP techniques for the dataset.

## Notebook Descriptions

### Section 0: Data Preprocessing and Cleaning

- Decode Base64-encoded SSH sessions.
- Tokenize sessions based on a predefined vocabulary.
- Save the processed dataset for further use.

### Section 1: Data Exploration and Preprocessing

- Visualize dataset distributions.
- Clean and preprocess data for modeling.
- Perform temporal analysis and session analysis.

### Section 2: Supervised Learning

- Train classifiers for intent prediction.
- Evaluate performance using metrics like accuracy and F1-score.
- Perform hyperparameter tuning and feature experimentation.

### Section 3: Unsupervised Learning

- Cluster sessions to uncover hidden patterns.
- Use algorithms like k-means and Gaussian Mixture Models.
- Evaluate clusters using methods like the Elbow Method and Silhouette Analysis.

### Section 4: Language Models

- Experiment with models like BERT and Doc2Vec.
- Fine-tune pre-trained models for intent classification.
- Plot learning curves and evaluate model performance.

## How to Run

1. Install dependencies using `requirements.txt`.
2. Open the notebooks in Jupyter or JupyterLab.
3. Follow the instructions provided within each notebook.

## Detailed Steps in Each Section

### Section 0: Data Preprocessing and Cleaning

- **Install Dependencies**: Install necessary packages.
- **Helper Functions**: Functions to decode Base64 strings, clean and tokenize sessions.
- **Load and Process Data**: Load raw data, decode sessions, clean and tokenize, save processed data.

### Section 1: Data Exploration and Preprocessing

- **Install Dependencies**: Install necessary packages.
- **Dataset Preparation**: Load and inspect the dataset, check for missing values and duplicates.
- **Temporal Analysis**: Analyze attack frequencies over time (hourly, daily, monthly trends).
- **Session Analysis**: Analyze the content and structure of attack sessions.
- **Intent Distribution**: Visualize the distribution of different attack intents.
- **Text Representation**: Represent text data using techniques like TF-IDF.
- **Error Reporting**: Report any errors encountered during preprocessing.

### Section 2: Supervised Learning

- **Install Dependencies**: Install necessary packages.
- **Data Splitting**: Split the dataset into training and testing sets.
- **Baseline Model Implementation**: Implement baseline classifiers like Logistic Regression, Random Forest, and SVM.
- **Hyperparameter Tuning**: Tune hyperparameters using GridSearchCV.
- **Result Analysis**: Analyze the performance of the models.
- **Feature Experimentation**: Experiment with different feature representations and preprocessing techniques.

### Section 3: Unsupervised Learning

- **Install Dependencies**: Install necessary packages.
- **Determine the Number of Clusters**: Use Elbow Method and Silhouette Analysis to find the optimal number of clusters.
- **Tune Other Hyperparameters**: Tune hyperparameters for clustering algorithms.
- **Visualize the Clusters**: Visualize the resulting clusters.
- **Cluster Analysis**: Analyze the characteristics of each cluster.
- **Assess Homogeneity and Intent Reflection**: Assess how well the clusters reflect different attack intents.
- **Associate Clusters with Specific Attack Categories**: Link clusters to specific categories of attacks.

### Section 4: Language Models

- **Install Dependencies**: Install necessary packages.
- **Model Selection**: Choose between BERT and Doc2Vec for intent classification.
- **Add a Dense Layer**: Extend the model with a custom dense layer for classification.
- **Fine-Tuning**: Fine-tune the model on the training dataset.
- **Plot Learning Curves**: Visualize training and validation loss curves to identify the optimal stopping point.