Splunk’s Machine Learning Toolkit (MLTK) allows you to apply machine learning algorithms to your Splunk data for predictive analytics, anomaly detection, clustering, and more. Splunk ML works by integrating machine learning models into the Splunk search language, allowing you to use standard Splunk searches alongside machine learning commands to train, apply, and evaluate models.

Key Steps for Machine Learning in Splunk:

1. Data Preparation: Select and clean the data you want to use for training and testing the machine learning model.
2. Model Training: Use machine learning algorithms to train a model on your historical data.
3. Model Application: Apply the trained model to predict or detect patterns in new data.
4. Model Evaluation: Measure the model's performance using various metrics.

Splunk ML Workflow Overview:

• Training the Model: Use historical data to train a machine learning model using commands like fit.
• Applying the Model: Once the model is trained, apply it to new data using the apply command.
• Evaluating Performance: Use metrics like accuracy, precision, recall, and other statistical measures to evaluate the model's effectiveness.

Example: Machine Learning with Access Logs (Tutorial Data)

We’ll use the Splunk tutorial data (access logs) to demonstrate a basic example of predicting high response times using machine learning.

Data Setup

If you don’t have the tutorial data, you can download and add it by following these steps: 1. Go to Settings > Add Data. 2. Upload the tutorial data file (access.log) from Splunk Tutorial Files.

Once you have the data indexed (assumed to be in index=web and sourcetype=access_combined), you can follow the example below.

Objective:

We will build a machine learning model to predict high response times based on various features in the access logs, such as status, bytes, and ip.

Step 1: Data Exploration

First, let’s explore the data and understand the fields available for machine learning.

Example Search (Exploring the Data):

index=web sourcetype=access_combined | table _time, ip, status, bytes, response_time

• Description: This command displays the key fields (ip, status, bytes, and response_time) from the access logs that we will use in our machine learning model.

Step 2: Data Preprocessing

Before training the model, we need to clean and preprocess the data.

Example Search (Data Cleaning and Preprocessing):

index=web sourcetype=access_combined 
| eval high_response_time=if(response_time > 1000, 1, 0)
| table ip, status, bytes, response_time, high_response_time

• Description: This search creates a new field high_response_time to label response times greater than 1000ms as "high" (1), and response times less than or equal to 1000ms as "low" (0).

Step 3: Model Training

We’ll use the decision tree classifier to predict whether a request will have a high response time (high_response_time=1) based on other fields like status, bytes, and ip.

Example Search (Training the Model):

index=web sourcetype=access_combined 
| eval high_response_time=if(response_time > 1000, 1, 0)
| fields status, bytes, ip, high_response_time
| fit DecisionTreeClassifier "high_response_time" from "status" "bytes" "ip" as "model"

• Description: This command uses the fit command to train a decision tree classifier (DecisionTreeClassifier). It uses status, bytes, and ip as features to predict the target field high_response_time. The model is saved as model.

Step 4: Apply the Model

Now that we’ve trained the model, we can apply it to new data to predict whether the response time will be high or low.

Example Search (Applying the Model):

index=web sourcetype=access_combined 
| fields status, bytes, ip, response_time
| apply "model"
| table ip, status, bytes, response_time, predicted("high_response_time")

• Description: This search applies the trained model (model) to new data and predicts whether each request will have a high response time (predicted("high_response_time")).

Step 5: Model Evaluation

You can evaluate the performance of the model by comparing the predictions with the actual results. In this case, we will compare the predicted high_response_time values with the actual high_response_time.

Example Search (Model Evaluation):

index=web sourcetype=access_combined 
| eval high_response_time=if(response_time > 1000, 1, 0)
| fields status, bytes, ip, high_response_time
| apply "model"
| eval is_correct=if('high_response_time' == predicted("high_response_time"), 1, 0)
| stats count, sum(is_correct) as correct_predictions, (sum(is_correct)/count)*100 as accuracy

• Description: This command calculates the accuracy of the model by comparing the predicted values (predicted("high_response_time")) with the actual high_response_time. The is_correct field indicates whether the prediction was accurate, and the overall accuracy is calculated.

Explanation of Commands:

1. fit:

   • Trains a machine learning model based on historical data.
   • Syntax: fit <algorithm> <target_field> from <input_fields> as <model_name>

2. apply:

   • Applies a trained machine learning model to new data.
   • Syntax: apply "<model_name>"

3. eval:

   • Used to create new fields, such as high_response_time, by setting conditions or transforming data.
   • Syntax: eval new_field = if(condition, true_value, false_value)

4. stats:

   • Aggregates and calculates statistics (e.g., accuracy).
   • Syntax: stats <stat_function> by <field>

Machine Learning Algorithms in Splunk:

Splunk’s MLTK supports a wide range of algorithms, including: - Classification: DecisionTreeClassifier, LogisticRegression, SVM, etc. - Clustering: KMeans, DBSCAN - Regression: LinearRegression, RandomForestRegressor - Anomaly Detection: DensityFunction, PCA - Time Series Forecasting: StateSpaceForecast, ARIMA

Conclusion:

In this example, we demonstrated how to use Splunk’s machine learning capabilities to predict high response times in access logs. The workflow included: 1. Data exploration and preprocessing. 2. Training a decision tree model using the fit command. 3. Applying the model to new data using the apply command. 4. Evaluating the model's accuracy by comparing predictions to actual values.

Splunk MLTK makes it easy to integrate machine learning into your existing log and event data analysis, enabling predictive analytics and more advanced insights.

LINEAR REGRESSION DEMO

Here’s a step-by-step guide to demonstrate Linear Regression in Splunk using sample events data. Linear regression is useful when you want to predict a numeric value based on other numeric or categorical fields.

Steps to follow:

1. Generate Sample Data: Create a sample dataset with a dependent variable (e.g., sales) that you want to predict based on independent variables (e.g., advertising, temperature).
2. Index the Data: Add the data to Splunk as events.
3. Train the Linear Regression Model: Use historical data to train a linear regression model.
4. Test the Model: Apply the model to new data to predict the dependent variable.
5. Evaluate the Model: Assess the model’s accuracy using statistical measures.

SREE STEPS FINAL

data1.csv

date,sales,advertising_budget,temperature
2024-10-01,1000,200,75
2024-10-02,1100,250,80
2024-10-03,1200,300,85
2024-10-04,1300,350,88
2024-10-05,1400,400,90
2024-10-06,1500,450,95
2024-10-07,1600,500,100
2024-10-08,1700,550,105
2024-10-09,1800,600,110
2024-10-10,1900,650,115

Settings->Add Data and Create linear_regression_index
Input Type
Uploaded File
File Name
data1.csv
Source Type
csv
Host
data1
Index
linear_regression_index


source="data1.csv" index="linear_regression_index"

TRY

index=linear_regression_index
| fields sales, advertising_budget, temperature
| fit LinearRegression "sales" from "advertising_budget" "temperature" as my_lr_model

Unknown search command 'fit'.

Manage Apps > Browse More > Splunk Machine Learning Toolkit login

Splunk Machine Learning Toolkit was successfully installed.

Python for Scientific Computing (for Linux 64-bit)

Python for Scientific Computing (for Linux 64-bit) is being downloaded and installed.

AGAIN

index=linear_regression_index
| fields sales, advertising_budget, temperature
| fit LinearRegression "sales" from "advertising_budget" "temperature" as my_lr_model

| listmodels

#### TRY AGAIN AND AGAIN

WAIT TILL THE RESULTS

TRY AGAIN 4 to 5 TIMES

index=linear_regression_index
| fields sales, advertising_budget, temperature
| fit LinearRegression "sales" from "advertising_budget" "temperature" into "my_lr_model"

| listmodels

#### TRY AGAIN AND AGAIN

SEE THE MODEL

| listmodels

#### YES

|summary my_lr_model

WORKED!! FAILES FIRST

Error in 'apply' command: Failed to load model "my_lr_model": Model does not exist.

index=linear_regression_index
| fields advertising_budget, temperature
| apply my_lr_model
| table advertising_budget, temperature, predicted("sales")

SHOW THE ACTUAL SALES AND PREDICTED SALES

index=linear_regression_index
| fields advertising_budget, temperature, sales
| apply my_lr_model
| table advertising_budget, temperature, , sales, predicted("sales")


ABOVE WORKS!!

index=linear_regression_index
| fields sales, advertising_budget, temperature
| apply my_lr_model
| rename predicted("sales") as PSales
| eval residual = sales - PSales
| eval residual_squared = residual * residual
| stats avg(abs(residual)) as MAE, avg(residual_squared) as MSE




NEW DATA

date,advertising_budget,temperature
2024-10-11,700,120
2024-10-12,750,125
2024-10-13,800,130
2024-10-14,850,135
2024-10-15,900,140


Review
Input Type
Uploaded File
File Name
data1-new.csv
Source Type
csv
Host
data1-new
Index
linear_regression_index

PREDICTED WORKS!!!

index=linear_regression_index host="data1-new"
| fields advertising_budget, temperature
| apply my_lr_model
| table advertising_budget, temperature, predicted("sales")

Step 1: Create Sample Data for Linear Regression

Here’s an example of a small sample dataset that simulates a scenario where sales are influenced by advertising_budget and temperature. You can copy this data into a CSV file or add it directly to Splunk as events.

Sample Data:

date,sales,advertising_budget,temperature
2024-10-01,1000,200,75
2024-10-02,1100,250,80
2024-10-03,1200,300,85
2024-10-04,1300,350,88
2024-10-05,1400,400,90
2024-10-06,1500,450,95
2024-10-07,1600,500,100
2024-10-08,1700,550,105
2024-10-09,1800,600,110
2024-10-10,1900,650,115

Each row contains: - date: The date of the record. - sales: The sales value (dependent variable). - advertising_budget: The advertising budget for that day (independent variable). - temperature: The temperature on that day (independent variable).

Step 2: Index the Data in Splunk

To load the data into Splunk: 1. Go to Settings > Add Data. 2. Choose Upload and upload the CSV file containing the sample data. 3. Choose an index to store the data (e.g., linear_regression_index).

Once the data is indexed, you can begin working with it.

Step 3: Train the Linear Regression Model

Now that the data is indexed in Splunk, we can train a linear regression model using the fit command from the Machine Learning Toolkit (MLTK). The goal is to predict the sales based on advertising_budget and temperature.

Example Search to Train the Model:

WAIT TILL THE RESULTS

index=linear_regression_index
| fields sales, advertising_budget, temperature
| fit LinearRegression "sales" from "advertising_budget" "temperature" into "my_lr_model"

• Description: This command trains a Linear Regression model to predict sales based on advertising_budget and temperature. The model is named my_lr_model.

Step 4: Test and Apply the Model to New Data

Now, let’s assume you have new data (without the sales value), and you want to predict sales using the model we just trained.

Example Search to Apply the Model:

index=linear_regression_index
| fields advertising_budget, temperature
| apply my_lr_model
| table advertising_budget, temperature, predicted("sales")

• Description: This search applies the trained model (my_lr_model) to the new data and predicts the sales values based on the provided advertising_budget and temperature.

Step 5: Evaluate the Model

To evaluate the performance of the model, we compare the predicted sales values to the actual sales values in the test data. We can calculate the error metrics like Mean Squared Error (MSE) or Mean Absolute Error (MAE) to measure the accuracy of the predictions.

Example Search for Evaluation:

index=linear_regression_index
| fields sales, advertising_budget, temperature
| apply my_lr_model
| rename predicted("sales") as PSales
| eval residual = sales - PSales
| eval residual_squared = residual * residual
| stats avg(abs(residual)) as MAE, avg(residual_squared) as MSE

• Description:
  • This search applies the model and compares the predicted sales with the actual sales.
  • It calculates two error metrics:
  • MAE (Mean Absolute Error): The average of the absolute errors between the predicted and actual values.
  • MSE (Mean Squared Error): The average of the squared errors between the predicted and actual values.

Summary of Steps:

1. Create Sample Data: We created a sample dataset with fields sales, advertising_budget, and temperature.
2. Index the Data in Splunk: Load the data into Splunk so it can be queried and used.
3. Train the Model: Use the fit command to train a Linear Regression model on the sales data using advertising_budget and temperature as predictors.
4. Apply the Model: Use the apply command to predict sales based on new values of advertising_budget and temperature.
5. Evaluate the Model: Compare the predicted sales with actual values and calculate error metrics like MAE and MSE.

Key Commands:

• fit: Trains a machine learning model (e.g., LinearRegression) based on the specified target and feature fields.
• apply: Applies the trained model to new data to make predictions.
• eval: Used for calculating the residual (error) between actual and predicted values.
• stats: Aggregates results, here used to calculate MAE and MSE for model evaluation.

By following these steps, you can demonstrate linear regression in Splunk, predict values based on new data, and evaluate the performance of the model with accuracy metrics.

NEW DATA

You're right! In the example I provided, we need new data for testing and applying the model. To clarify, the new data is usually similar to the original training data, but it doesn’t contain the target variable (in this case, sales). The goal is to use the trained model to predict this missing value.

Let’s walk through how to generate and apply new data for testing the model.

Step 1: Create New Data for Testing (Without the sales Field)

Here’s an example of new data that contains only the independent variables (advertising_budget and temperature), but not the dependent variable (sales), because we want to predict it:

New Test Data (Example):

date,advertising_budget,temperature
2024-10-11,700,120
2024-10-12,750,125
2024-10-13,800,130
2024-10-14,850,135
2024-10-15,900,140

Step 2: Index the New Data into Splunk

Just like before, index the new data into Splunk: 1. Go to Settings > Add Data. 2. Upload the new CSV file containing only the advertising_budget and temperature fields. 3. Store this data in the same index (e.g., linear_regression_index).

Step 3: Apply the Trained Model to the New Data

Now that we have the new data indexed in Splunk (without the sales field), we can apply the trained linear regression model to predict the missing sales values.

Here’s how you can apply the model to the new data:

Example Search to Apply the Model to New Data:

index=linear_regression_index 
| fields advertising_budget, temperature
| apply my_lr_model
| table advertising_budget, temperature, predicted("sales")

• Description:
  • This search fetches the advertising_budget and temperature fields from the new data.
  • It applies the trained linear regression model (my_lr_model) to predict the missing sales values.
  • The predicted sales values are shown in the predicted("sales") field.

Step 4: View the Predictions

After running the search, you should see output similar to the following:

This table shows the predicted sales based on the advertising_budget and temperature from the new data.

Step 5: Evaluate Predictions (Optional)

If you eventually receive the actual sales data for the test period, you can evaluate the model’s predictions by comparing the predicted sales to the actual sales using the same error metrics we used earlier (e.g., MAE, MSE).

Example (Compare Predicted and Actual Sales):

index=linear_regression_index
| fields sales, advertising_budget, temperature
| apply my_lr_model
| rename predicted("sales") as PSales
| eval residual = sales - PSales
| eval residual_squared = residual * residual
| stats avg(abs(residual)) as MAE, avg(residual_squared) as MSE

• Description: This evaluates the performance of the model by calculating Mean Absolute Error (MAE) and Mean Squared Error (MSE) between the predicted and actual sales values.

Summary of New Data Application:

1. Create New Data: The new data includes only the features (advertising_budget, temperature), without the sales field (which you want to predict).
2. Index the New Data: Upload the new data to Splunk and store it in the same index.
3. Apply the Model: Use the apply command to predict sales based on the new data using the trained model.
4. Evaluate (Optional): If you receive actual sales data later, you can compare the predictions to the actual results.

By following these steps, you can effectively use the trained model to predict future outcomes (in this case, sales) using new data.

CLASSIFICATION

Here’s a complete tutorial on how to use Splunk Machine Learning Toolkit (MLTK) for a classification problem using the provided dataset and index structure. This tutorial demonstrates how to upload, train, and predict customer purchases.

Tutorial: Predicting Customer Purchases Using Splunk MLTK

In this tutorial, we will: 1. Load and preprocess data. 2. Train a Logistic Regression classification model to predict customer purchases. 3. Apply the trained model to new data for predictions.

Step 1: Initial Setup and Data Upload

1.1 Upload the Training Data (customer_data.csv)

• Go to Settings > Add Data in Splunk.
• Select Upload and add the file customer_data.csv.
• Configure the file upload:
  • Source Type: csv
  • Host: old-data
  • Index: customer_data

After uploading, the data should be accessible in the customer_data index with the source set as customer_data.csv.

1.2 Verify the Data

Run the following SPL command to verify that the data is correctly indexed:

index="customer_data" | table age, income, account_activity, purchase

Expected Output: A table showing fields: - age: Customer's age. - income: Customer's annual income. - account_activity: Level of account activity (e.g., low, medium, high). - purchase: Binary target variable indicating whether the customer made a purchase (1 for purchase, 0 for no purchase).

Step 2: Data Preprocessing

Since account_activity is a categorical variable, we need to convert it to a numeric format for the model.

Use the following SPL command to create a new field called activity_level that encodes account_activity:

index="customer_data" 
| eval activity_level=case(account_activity=="low", 0, account_activity=="medium", 1, account_activity=="high", 2) 
| table age, income, activity_level, purchase

Explanation: - eval activity_level=case(...): Converts account_activity into a numeric field activity_level, where: - low = 0 - medium = 1 - high = 2 - This prepares the data for model training by transforming categorical values into numeric codes.

Step 3: Train the Classification Model

3.1 Train a Logistic Regression Model

Run the following SPL command to train a Logistic Regression model:

age,income,account_activity,purchase
25,45000,low,0
32,54000,medium,1
45,62000,high,1
22,30000,low,0
34,70000,high,1
41,50000,medium,0
29,56000,low,1
39,67000,high,1


index="customer_data"
| eval activity_level=case(account_activity=="low", 0, account_activity=="medium", 1, account_activity=="high", 2)
| fit LogisticRegression purchase from age income activity_level into "Customer_Purchase_Prediction"

Explanation: - fit LogisticRegression purchase from age income activity_level into "Customer_Purchase_Prediction": - fit: Command used to train the model. - LogisticRegression: Specifies the algorithm type. - purchase: Target field to predict. - from age income activity_level: Predictor fields used to train the model. - into "Customer_Purchase_Prediction": Saves the trained model as Customer_Purchase_Prediction.

3.2 Verify Model Creation

To confirm that the model was successfully created, list all models with the following command:

| listmodels

You should see Customer_Purchase_Prediction in the output, indicating that the model is now saved and ready for use.

Step 4: Upload New Data for Prediction

4.1 Upload the New Data (customer_data-new.csv)

Upload a new dataset for prediction, containing only age, income, and account_activity fields, without the purchase field.

• Go to Settings > Add Data in Splunk.
• Upload customer_data-new.csv with the following settings:
  • Source Type: csv
  • Host: new
  • Index: customer_data

After uploading, the new data will be indexed in customer_data with the source customer_data-new.csv.

Sample New Data (customer_data-new.csv):

age,income,account_activity
30,48000,low
36,59000,medium
50,65000,high
28,41000,low
40,72000,high
33,55000,medium
27,57000,low
45,68000,high

Step 5: Predict Purchase Outcomes on New Data

Now that the model is trained and the new data is indexed, we can apply the model to make predictions.

Run the following SPL command to apply the trained model:

index="customer_data" host="new"
| eval activity_level=case(account_activity=="low", 0, account_activity=="medium", 1, account_activity=="high", 2)
| apply "Customer_Purchase_Prediction"
| table age, income, account_activity, activity_level, predicted(purchase)

Explanation: - index="customer_data" host="new": Loads the new data where the host is new. - eval activity_level=case(...): Encodes account_activity into activity_level, matching the format used during training. - apply "Customer_Purchase_Prediction": Applies the trained model to predict purchase. - table age, income, account_activity, activity_level, predicted(purchase): Displays the fields along with the predicted(purchase) output.

Interpreting the Prediction Results

The predicted(purchase) field shows the model’s prediction: - 1: Indicates the model predicts that the customer is likely to make a purchase. - 0: Indicates the model predicts that the customer is unlikely to make a purchase.

Final Notes

• Fine-tuning: If the model’s accuracy is low, consider adding more predictor fields or using different algorithms (available in MLTK).
• Real-Time Monitoring: This model can be integrated into a real-time dashboard, allowing Splunk to continuously predict purchase likelihoods as new customer data is added.

This completes the step-by-step process for building, training, and applying a classification model in Splunk MLTK using an indexed dataset. The model can now predict purchase likelihoods for new customer records, offering valuable insights into customer behavior.

Review
Input Type
Uploaded File
File Name
customer_data.csv
Source Type
csv
Host
old-data
Index
customer_data

source="customer_data.csv" host="old-data" index="customer_data" sourcetype="csv"
index="customer_data" 

index="customer_data" | table age, income, account_activity, purchase


index="customer_data" 
| eval activity_level=case(account_activity=="low", 0, account_activity=="medium", 1, account_activity=="high", 2) 
| table age, income, activity_level, purchase


index="customer_data"
| eval activity_level=case(account_activity=="low", 0, account_activity=="medium", 1, account_activity=="high", 2)
| fit LogisticRegression purchase from age income activity_level into "Customer_Purchase_Prediction"

| listmodels


NEW DATA

Review
Input Type
Uploaded File
File Name
customer_data-new.csv
Source Type
csv
Host
new
Index
customer_data

PREDICT PURCHASE

index="customer_data" host="new"
| eval activity_level=case(account_activity=="low", 0, account_activity=="medium", 1, account_activity=="high", 2)
| apply "Customer_Purchase_Prediction"
| table age income account_activity activity_level predicted(purchase)