Here’s a detailed exercise for creating a Data Model in Splunk using the access log data. This exercise will guide you through the process of setting up a Data Model, defining datasets, and working with the data in an accelerated way.

Objective:

To create a Data Model from access logs (e.g., access_combined sourcetype), which can be used to generate optimized reports and pivots for web traffic analysis, including page requests, user sessions, and response times.

Step 1: Load the Access Log Data into Splunk

If you haven’t already uploaded the access log data, follow these steps: 1. Go to Settings > Add Data. 2. Choose Upload and select the access log file (access_combined.log or similar format). 3. Assign the data to an index (e.g., index=web) and sourcetype (access_combined). 4. Complete the data ingestion process.

Step 2: Create a New Data Model

1. Navigate to the Data Models section:

   • Go to Settings > Data Models.
   • Click on New Data Model.

2. Define the Data Model:

   • Title: Enter a name for your Data Model (e.g., "Web Access Logs Model").
   • ID: This will auto-generate based on the title, or you can define it.
   • Description: Optionally add a description (e.g., "Data model for analyzing web access logs").
   • Permissions: Set to Private for now, or make it shared as needed.

3. Click Create to proceed to the next step.

Step 3: Define a Root Dataset

A root dataset is the base dataset from which other datasets can be derived. It forms the foundation of your data model.

1. Create a Root Dataset:
   • In the Data Model editor, click Add Dataset and choose Root Event Dataset.
   • Title: Name this root dataset (e.g., "Web Events").
   • Sourcetype Selection: Choose access_combined or whichever sourcetype your access log data is associated with.
   • Constraints: Apply a search filter to define the events included in this root dataset. For access logs, use:

     # sourcetype=access_combined
     
     Sree Root Web Events
     index=tutorial_data sourcetype=access_combined_wcookie
     

• Description: Add a description for your root dataset (e.g., "This dataset contains all web access events").

1. Click Save to create the root dataset.

Step 4: Add Fields to the Root Dataset

Now, define which fields are important to include in your root dataset. These fields will allow you to extract meaningful insights from the data.

1. Add Fields:

   • In the dataset editor, click Add Field and select Auto-Extracted.
   • Field Name: Choose fields that are useful for analysis. Some common fields in access logs include:
     • ip: Client IP address.
     • status: HTTP status code (e.g., 200, 404).
     • bytes: Number of bytes returned in the response.
     • response_time: Time taken to respond to the request.
     • url: The requested URL.

2. Add Calculated Fields:

   • You can also create calculated fields. For example, to convert bytes to kilobytes:

     • Click Add Field and choose Calculated Field.
     • Name it response_size_kb.
     • Eval Expression: Enter the following expression to convert bytes to kilobytes: ```spl bytes / 1024

     round(bytes/1024,1)

     - **Field Type**: Choose `Number`.

3. Add Aliases (Optional):

   • If you want to rename fields or create field aliases, use the Add Field > Alias option.
   • For example, alias ip to client_ip for consistency:
     • Original Field: ip.
     • Alias: client_ip.

4. Save your root dataset after adding fields and aliases.

Step 5: Create Child Datasets

Child datasets are derived from the root dataset. You can create specialized child datasets to focus on specific aspects of your access logs, such as error logs or successful page loads.

1. Add a Child Dataset for Error Logs:

   • In your root dataset, click Add Dataset and select Search Dataset.
   • Title: Name it "Error Logs".
   • Constraint: Define the constraint to filter only error events. For HTTP errors (status codes 4xx or 5xx), you can use: spl status>=400
   • Save the child dataset.

2. Add a Child Dataset for Successful Requests:

   • Add another Search Dataset.
   • Title: Name it "Successful Requests".
   • Constraint: Define the constraint to filter successful requests. For HTTP success codes (status 200): spl status=200
   • Save the dataset.

3. You can create more child datasets based on other criteria, such as response time thresholds or specific IP addresses.

Step 6: Enable Data Model Acceleration (Optional)

Data models can be accelerated to speed up searches and reports, especially when working with large datasets.

1. Enable Acceleration:
   • After creating the data model, click on Acceleration.
   • Set Acceleration to Enabled.
   • Define the acceleration period (e.g., last 7 days, last 30 days).
   • Save the settings.

This will pre-compute summaries of the data and significantly improve search performance on the accelerated data.

Step 7: Use the Data Model in Searches and Pivots

Once your data model is created, you can use it in Pivots or Searches to generate reports and visualizations.

1. Search Using the Data Model:
   • Use the datamodel command to search within the data model:

| datamodel "Web_Access_Logs_Model"

| from datamodel:"Web_Access_Logs_Model"."Error_Logs"

| from datamodel:"Web_Access_Logs_Model"."Error_Logs" | search status=500
| table _time, error_code, host, uri

| datamodel "Web_Access_Logs_Model" "Error_Logs" search
| stats count by status

| datamodel "Web_Access_Logs_Model" "Error_Logs" search
| stats count by clientip, status

| datamodel "Web_Access_Logs_Model" "Error_Logs" search
| where status >= 500
| stats count by clientip, status

| datamodel "Web_Access_Logs_Model" "Error_Logs" search
| timechart count by status





     | datamodel "Web Access Logs Model" "Web Events" search
     
     | datamodel "SreeDataModal" search
     |datamodel "SreeDataModal" "Sree_Root_Web_Events" search
     |datamodel "SreeDataModal" "Sree_Child_Error_Logs" search
 

• This search retrieves all the data from the Web Events dataset.

1. Pivot Interface:
   • Go to the Pivot interface and select your new data model ("Web Access Logs Model").
   • You can now use drag-and-drop functionality to build charts, tables, and reports using the fields you defined (e.g., group data by status or ip, calculate averages for response_time, etc.).

Step 8: Visualize the Data

Use the Pivot interface to visualize the data model, or run searches based on the datasets you created to generate insights. For example:

Example: Count of Events by Status Code

| datamodel "Web Access Logs Model" "Web Events" search
| stats count by status

Example: Average Response Time by Status Code

| datamodel "Web Access Logs Model" "Web Events" search
| stats avg(response_time) by status

Example: Error Rate by Status Code

| datamodel "Web Access Logs Model" "Error Logs" search
| stats count by status

Step 9: Create Dashboards Using the Data Model

Once you have created visualizations or saved searches using the data model, you can add them to dashboards for real-time monitoring and reporting.

1. Run your search (e.g., count of events by status).
2. Click Save As > Dashboard Panel.
3. Create a new dashboard or add to an existing one.
4. Customize the layout and add filters for interactivity.

Conclusion:

This exercise walks you through creating a data model based on access log data in Splunk. You defined a root dataset and added fields like ip, status, and response_time. You then created child datasets to focus on specific aspects like error logs and successful requests. By enabling acceleration, you improved the performance of searches and reports based on this data model. Finally, you used pivots and searches to analyze the data and visualize it in dashboards.

Data models are a powerful way to structure your data in Splunk, making complex searches and reporting much faster and easier to manage.

You can use child datasets from a Data Model in searches using the datamodel command in Splunk. Child datasets, which are subsets of a parent (root) dataset, allow you to run more specific, focused queries. Below is a step-by-step guide on how to use a child dataset in a search, along with examples.

Step-by-Step Guide to Using Child Datasets in a Search

1. Identify the Data Model and Child Dataset:

   • You must know the name of the data model and the name of the child dataset you created.
   • For this example, we’ll assume you’ve created a data model named "Web Access Logs Model" and a child dataset named "Error Logs".

2. Use the datamodel Command:

   • The datamodel command is used to search within a data model and its datasets.
   • The syntax to access a child dataset is: spl | datamodel "<Data Model Name>" "<Child Dataset Name>" search

Example Search for the Child Dataset (Error Logs):

Let’s assume you have a child dataset called "Error Logs" that filters events where status >= 400 (HTTP error codes). Here’s how you would query it:

Example Search:

| datamodel "Web Access Logs Model" "Error Logs" search
| stats count by status

• Explanation:
  • datamodel "Web Access Logs Model" "Error Logs" search: This command accesses the "Error Logs" child dataset within the "Web Access Logs Model".
  • The search keyword tells Splunk to retrieve all events from this dataset.
  • The stats count by status command counts how many events exist for each HTTP status code.

Working with Fields in Child Datasets

You can also work with the fields defined in the child dataset. For example, if your "Error Logs" dataset includes fields like ip, status, and response_time, you can access these fields in your search:

Example: Group Errors by IP and Status

| datamodel "Web Access Logs Model" "Error Logs" search
| stats count by ip, status

• Explanation: This search counts the number of error events (status >= 400) grouped by ip and status.

Filtering Results within a Child Dataset

You can add more filtering criteria when working with child datasets. For example, if you want to look for error events that took more than 500ms to respond, you can use additional where or eval filters.

Example: Filter Errors with High Response Times

| datamodel "Web Access Logs Model" "Error Logs" search
| where response_time > 500
| stats count by status, ip

• Explanation: This search retrieves events from the "Error Logs" dataset and filters them to only include those where response_time is greater than 500ms. The results are grouped by status and ip.

Using Other Commands with Child Datasets

You can combine child datasets with other Splunk commands like timechart, chart, eval, etc., to further manipulate the data.

Example: Timechart of Error Counts Over Time

| datamodel "Web Access Logs Model" "Error Logs" search
| timechart count by status

• Explanation: This search retrieves error events from the "Error Logs" child dataset and creates a time-based chart that shows the count of events over time, grouped by status.

Listing Available Fields from a Child Dataset

To see what fields are available in the child dataset, you can use the table command to list them.

Example: List Available Fields in "Error Logs" Dataset

| datamodel "Web Access Logs Model" "Error Logs" search
| table _time, ip, status, response_time, bytes

• Explanation: This lists key fields (_time, ip, status, response_time, bytes) from the "Error Logs" dataset.

Recap of Key Concepts:

• datamodel Command: This command is used to search within a data model. The structure is | datamodel "<Data Model Name>" "<Dataset Name>" search.
• Child Dataset: When you search within a child dataset, you’re querying a pre-filtered subset of the data, which allows for faster, more specific analysis.
• Combining Commands: You can use all standard Splunk commands (stats, timechart, where, eval, etc.) on the child dataset for further analysis.

Final Example: Complete Analysis Using a Child Dataset

Here’s a complete example using a child dataset (Error Logs), where we: - Query the dataset. - Filter events where response_time is high. - Calculate the average response time by status code.

| datamodel "Web Access Logs Model" "Error Logs" search
| where response_time > 500
| stats avg(response_time) as avg_response_time by status

• Explanation: This search retrieves all error events where the response_time is greater than 500ms and calculates the average response time (avg_response_time) for each HTTP status code.

Conclusion:

You can use the datamodel command to query and analyze child datasets created within a data model in Splunk. Child datasets allow you to work with filtered and pre-defined event sets, making it easier to focus on specific types of data (e.g., error logs, successful requests, or specific user sessions). You can apply any Splunk command (e.g., stats, timechart, eval) to the child dataset just like you would in a regular search, giving you powerful insights into your data.