Splunk Machine Learning Toolkit (MLTK) commands along with practical examples

Here's a summarized breakdown of the Splunk Machine Learning Toolkit (MLTK) commands along with practical examples:

1. fit Command

• Description: Fits and applies a machine learning model to search results (both supervised and unsupervised learning).

• Syntax: spl fit <algorithm> [option_name]=[option_value]... <response-field> from <explanatory-field> [into <model_name>]

• Examples:

  • Fit a Linear Regression model to predict errors based on time: spl ... | fit LinearRegression errors from _time
  • Fit a model and save it as errors_over_time: spl ... | fit LinearRegression errors from _time into errors_over_time
  • Fit a Logistic Regression model to predict species: spl ... | fit LogisticRegression species from petal_length petal_width sepal_length sepal_width

2. apply Command

• Description: Applies a previously trained model to search results to make predictions.

• Syntax: spl apply <model_name> [as <output_field>]

• Examples:

  • Apply a trained model to predict future results: spl ... | apply errors_over_time
  • Rename the output to predicted_errors: spl ... | apply errors_over_time as predicted_errors

3. summary Command

• Description: Returns a summary of the learned machine learning model (e.g., model coefficients).

• Syntax: spl summary <model_name>

• Examples:

  • Summarize the coefficients of a LinearRegression model: spl | summary errors_over_time

4. listmodels Command

• Description: Lists all machine learning models that were learned using the fit command.

• Syntax: spl listmodels

• Example:

  • List all the models in Splunk: spl | listmodels

5. deletemodel Command

• Description: Deletes a machine learning model.

• Syntax: spl deletemodel <model_name>

• Examples:

  • Delete the model named errors_over_time: spl | deletemodel errors_over_time

6. sample Command

• Description: Randomly samples or partitions events. It has several modes (ratio, count, proportional, and partition).

• Syntax: spl sample [ratio=<float>] [count=<int>] [proportional=<field>] [partitions=<int>] [seed=<number>]

• Examples:

  • Sample approximately 1% of all events: spl ... | sample ratio=0.01
  • Retrieve exactly 20 events at random: spl ... | sample count=20
  • Sample 20 events from each host: spl ... | sample count=20 by host
  • Partition events into 7 groups: spl ... | sample partitions=7 fieldname="partition_number"

7. score Command

• Description: Runs statistical tests to validate model outcomes (e.g., confusion matrix for classification).

• Syntax: spl score <scoring-method-name> <field1> against <field2>

• Examples:

  • Validate a classification model using a confusion matrix: spl ... | score confusion_matrix true="species" pred="predicted(species)"

Practical Example: Using Machine Learning on Data

Step-by-Step Workflow for Predicting Sales Using fit, apply, and score Commands

1. Train a Linear Regression Model:

   • Assume you're predicting sales based on advertising budget and temperature: spl index=your_index | fit LinearRegression sales from advertising_budget temperature into sales_model

2. Apply the Model to Make Predictions:

   • Apply the trained sales_model to predict sales: spl index=your_index | apply sales_model as predicted_sales

3. Evaluate the Model's Performance:

   • Calculate residuals (actual - predicted) and performance metrics: spl index=your_index | apply sales_model as predicted_sales | eval residual = sales - predicted_sales | eval residual_squared = residual * residual | stats avg(abs(residual)) as MAE, avg(residual_squared) as MSE

4. Validate the Model (Classification):

   • If using a classification model (e.g., LogisticRegression), you can validate it using a confusion matrix: spl ... | score confusion_matrix true="actual_field" pred="predicted_field"

Key Takeaways

• fit: Use to train models.
• apply: Use to apply models for predictions.
• summary: Get the details of a trained model (e.g., coefficients).
• listmodels: List all trained models.
• deletemodel: Delete a specific model.
• sample: Randomly sample or partition data for training/testing.
• score: Validate model outcomes with statistical tests.

These commands provide powerful tools for integrating machine learning into Splunk searches, allowing you to apply predictive analytics and statistical models to your data directly within the Splunk environment.