Splunk Buckets

Restoring Index Data in Splunk: Methods and Commands

Restoring index data in Splunk can be essential when recovering from accidental data loss, migrating data to a new environment, or restoring from backups. Below are the primary methods for restoring index data in Splunk, along with relevant commands and steps for each approach.

1. Restoring from File System Backups

The most reliable way to restore data in Splunk is to use file system-level backups of the index directories. Splunk stores its indexed data in the directory specified for each index, usually located at $SPLUNK_HOME/var/lib/splunk/<index_name>.

Steps:

1. Stop Splunk: Before restoring, always stop the Splunk instance to ensure data consistency. $SPLUNK_HOME/bin/splunk stop

2. Restore Backup Files: Copy the backed-up data files to the respective index directory. Ensure that you place the files in the correct path ($SPLUNK_HOME/var/lib/splunk/<index_name>).

3. Start Splunk: After restoring, start the Splunk instance. $SPLUNK_HOME/bin/splunk start

Commands Recap:

• Stop Splunk: splunk stop
• Restore files (example for Linux): cp -r /backup/path/* $SPLUNK_HOME/var/lib/splunk/
• Start Splunk: splunk start

Note: Ensure permissions are correct on the restored files. Typically, files should be owned by the Splunk user.

2. Restoring from Archived Buckets

Splunk organizes index data into buckets, which are directories within each index. If you have archived buckets (directories containing raw data files and metadata), you can restore them to the live index.

Steps:

1. Identify Bucket Locations: Find where your archived buckets are stored (e.g., on external storage).

2. Copy Buckets to the Index Directory: Move or copy the archived buckets to the index’s main directory ($SPLUNK_HOME/var/lib/splunk/<index_name>/db/).

3. Rebuild Metadata: To ensure Splunk recognizes the restored data, use the splunk rebuild command on each bucket directory. $SPLUNK_HOME/bin/splunk rebuild <bucket_directory>

Commands Recap:

• Copy archived bucket: cp -r /path/to/archived/bucket $SPLUNK_HOME/var/lib/splunk/<index_name>/db/
• Rebuild bucket metadata: splunk rebuild <bucket_directory>

Note: Always perform this action while Splunk is stopped to prevent inconsistencies.

3. Using the restore Command with Data from Hadoop (Hunk/Splunk Analytics for Hadoop)

If you are using Splunk’s Hadoop integration, you can archive and restore data from Hadoop. This is useful if you’ve configured data archival policies that move data from Splunk to Hadoop for long-term storage.

Steps:

1. Access the Hadoop Environment: Ensure your Hadoop environment is connected to Splunk and accessible.

2. Initiate the Restore Command: Run a search to restore data from Hadoop using the restore command: | restore hdfs://<path_to_data> | collect index=<index_name>

3. Verify Data in Splunk: After restoring, check the index in Splunk to confirm the data is available.

Commands Recap:

• Restore data from Hadoop: | restore hdfs://<path_to_data> | collect index=<index_name>

Note: The restore command requires configuration with Splunk’s HDFS and may need additional permissions for Hadoop access.

4. Restoring Data Using Splunk Cloud Backups

If you are using Splunk Cloud, data restoration involves working with Splunk Support to access backups and restore data, as Splunk Cloud does not give direct access to the backend filesystem.

Steps:

1. Contact Splunk Support: Open a support ticket to request data restoration. Specify the index, time range, and any other relevant details.

2. Splunk Cloud Support Restores Data: Support will handle the restoration process from their managed backups.

Note: You may need to wait for support assistance, as Splunk Cloud does not offer direct commands for data restoration.

5. Re-indexing Data from Original Sources

If you have access to the original log files or data sources, you can re-ingest the data into Splunk by re-indexing it. This is useful if the original data is accessible and you can easily resend it.

Steps:

1. Prepare Data Files: Place the original data files in an accessible directory.

2. Use oneshot Command to Index Data: The oneshot command ingests a file once without monitoring it. $SPLUNK_HOME/bin/splunk add oneshot /path/to/datafile.log -index <index_name> -sourcetype <sourcetype>

3. Verify Data in Index: After re-indexing, check the index to ensure data is restored.

Commands Recap:

• Re-index data: splunk add oneshot /path/to/datafile.log -index <index_name> -sourcetype <sourcetype>

Note: Use this method only if the original data source is available and easily accessible.

Additional Tips and Best Practices

• Ensure Consistency: Always stop Splunk before performing filesystem-level restores.
• Backup Regularly: Set up routine backups of index data to avoid data loss scenarios.
• Verify Permissions: Restored files should have correct permissions and be owned by the Splunk user to avoid read/write issues.
• Use Data Integrity Checks: After restoration, use searches or reports to validate that all expected data is present and accurate.

Summary of Commands

• Stop Splunk: $SPLUNK_HOME/bin/splunk stop
• Start Splunk: $SPLUNK_HOME/bin/splunk start
• Rebuild Bucket Metadata: $SPLUNK_HOME/bin/splunk rebuild <bucket_directory>
• Copy Files: cp -r /backup/path/* $SPLUNK_HOME/var/lib/splunk/<index_name>/db/
• Restore from Hadoop: | restore hdfs://<path_to_data> | collect index=<index_name>
• Re-index Original Data: $SPLUNK_HOME/bin/splunk add oneshot /path/to/datafile.log -index <index_name> -sourcetype <sourcetype>

These methods provide comprehensive ways to restore index data in Splunk, covering various scenarios from local backups to cloud solutions.

Bucket Details

To find the bucket paths for indexes in Splunk, you can use the following SPL (Search Processing Language) command, which leverages the dbinspect command to inspect index data and retrieve information about bucket locations.

Using dbinspect to Find Bucket Paths

The dbinspect command provides details about each bucket within a specified index, including the bucket paths, start times, end times, and bucket sizes.

Example Command:

| dbinspect index=<index_name>

Replace <index_name> with the name of the index you want to inspect. If you want to inspect all indexes, simply omit the index parameter.

Example Command to Show Bucket Paths for All Indexes:

| dbinspect | table index, path, bucketId, startEpoch, endEpoch, sizeOnDiskMB

Explanation of Key Fields:

• index: The name of the index the bucket belongs to.
• path: The full file system path to the bucket directory.
• bucketId: The unique identifier of the bucket.
• startEpoch and endEpoch: The time range for the events in the bucket (in epoch format).
• sizeOnDiskMB: The size of the bucket on disk, in megabytes.

Example Output Interpretation

This command lists each bucket’s path, along with other relevant information, making it easy to locate specific buckets on disk. You can use this information for backup or restoration purposes.

Filtering for Specific Index

To narrow down results to a specific index, for example, tutorial_data: spl | dbinspect index="tutorial_data" | table path, bucketId, startEpoch, endEpoch, sizeOnDiskMB

This command provides a concise view of the bucket paths and related details for the tutorial_data index, helping you locate and manage the index's storage efficiently.

Note: The path field output by dbinspect contains the full directory path to each bucket, which is useful for backup and restore operations.

In Splunk, setting the bucket policy allows you to manage the lifecycle of indexed data by defining how long data is retained in different stages (hot, warm, cold, and frozen). Each index in Splunk has its own bucket settings, which determine data storage, retention, and deletion policies.

Steps to Set Bucket Policy in Splunk

1. Access indexes.conf Configuration File: The bucket policy is configured in the indexes.conf file, located in $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/.

2. Define Bucket Parameters for Each Stage: Configure parameters for hot, warm, cold, and frozen stages within the index section in indexes.conf. Below are key settings for each stage.

   Example Configuration:

   [<index_name>]
   homePath = $SPLUNK_DB/<index_name>/db
   coldPath = $SPLUNK_DB/<index_name>/colddb
   thawedPath = $SPLUNK_DB/<index_name>/thaweddb

3. Setting Specific Bucket Policy Parameters

   • Hot and Warm Buckets:
     • maxDataSize: Maximum size of each hot bucket (e.g., auto, 100MB).
     • maxHotBuckets: Maximum number of hot buckets before rolling to warm.
     • maxWarmDBCount: Maximum number of warm buckets; after this limit, the oldest warm buckets roll to cold.
   • Cold Buckets:
     • coldPath: Specifies the directory where cold buckets are stored.
     • maxTotalDataSizeMB: Maximum total size of data in the index across hot, warm, and cold buckets (in MB).
   • Frozen Buckets:
     • frozenTimePeriodInSecs: The time (in seconds) after which data moves to frozen (and is deleted if no coldToFrozenDir is set).
     • coldToFrozenDir: Specifies a directory to archive frozen data instead of deleting it.

   Example:

   [tutorial_data]
   homePath = $SPLUNK_DB/tutorial_data/db
   coldPath = $SPLUNK_DB/tutorial_data/colddb
   thawedPath = $SPLUNK_DB/tutorial_data/thaweddb
   maxHotBuckets = 3
   maxWarmDBCount = 10
   maxDataSize = auto
   frozenTimePeriodInSecs = 31536000   ; 1 year
   coldToFrozenDir = /path/to/frozen   ; Move to frozen storage instead of deletion

4. Reload the Configuration: After editing indexes.conf, restart Splunk to apply the changes: bash $SPLUNK_HOME/bin/splunk restart

Key Parameters in Bucket Policy

• maxDataSize: Controls hot bucket size to manage disk usage.
• maxHotBuckets and maxWarmDBCount: Manage the number of hot and warm buckets.
• frozenTimePeriodInSecs: Sets data retention duration before data moves to frozen or is deleted.
• coldToFrozenDir: Specifies a storage location for frozen data, useful for archiving.

By configuring these parameters, you can optimize data retention, manage storage, and implement a bucket policy tailored to your organization’s needs.

micro@ip-172-31-46-125:~/splunk-lab$ docker compose exec box bash

root@4a83624585fe:/# cd /opt/splunk/
root@4a83624585fe:/opt/splunk# find var | grep tutorial
var/lib/splunk/tutorial_data.dat
var/lib/splunk/tutorial_data
var/lib/splunk/tutorial_data/colddb
var/lib/splunk/tutorial_data/db
var/lib/splunk/tutorial_data/db/hot_v1_0
var/lib/splunk/tutorial_data/db/hot_v1_0/1730658242-1730004110-5039443361484883503.tsidx
var/lib/splunk/tutorial_data/db/hot_v1_0/SourceTypes.data
var/lib/splunk/tutorial_data/db/hot_v1_0/splunk-autogen-params.dat
var/lib/splunk/tutorial_data/db/hot_v1_0/Hosts.data
var/lib/splunk/tutorial_data/db/hot_v1_0/bucket_info.csv
var/lib/splunk/tutorial_data/db/hot_v1_0/rawdata
var/lib/splunk/tutorial_data/db/hot_v1_0/rawdata/slicesv2.dat
var/lib/splunk/tutorial_data/db/hot_v1_0/rawdata/24927904
var/lib/splunk/tutorial_data/db/hot_v1_0/rawdata/journal.zst
var/lib/splunk/tutorial_data/db/hot_v1_0/.rawSize
var/lib/splunk/tutorial_data/db/hot_v1_0/Sources.data
var/lib/splunk/tutorial_data/db/hot_v1_0/Strings.data
var/lib/splunk/tutorial_data/db/CreationTime
var/lib/splunk/tutorial_data/db/.bucketManifest
var/lib/splunk/tutorial_data/datamodel_summary
var/lib/splunk/tutorial_data/thaweddb

DATA RETENSION

Data retention is set on a per-index basis rather than per-bucket. Retention settings apply only to hot, warm, and cold buckets. Splunk does not manage frozen or thawed buckets.

You can find the default retention settings in $SPLUNK_HOME/etc/system/default/indexes.conf, but those settings can be overridden by another indexes.conf file. Use btool to see the current (on-disk) config:

micro@ip-172-31-46-125:~/splunk-lab$ docker-compose exec box bash
root@4a83624585fe:/# cd /opt/splunk/
root@4a83624585fe:/opt/splunk# ls -l etc
total 196
drwxr-xr-x  2 41812 41812  4096 Sep  5 17:54 anonymizer
drwxr-xr-x 35 41812 41812  4096 Nov  5 16:00 apps
drwx------  8 41812 41812  4096 Nov  3 17:01 auth
-r--r--r--  1 41812 41812    57 Sep  5 17:25 copyright.txt
-r--r--r--  1 41812 41812  8182 Sep  5 17:25 datetime.xml
drwxr-xr-x  4 41812 41812  4096 Nov  3 17:01 deployment-apps
drwxr-xr-x  2 41812 41812  4096 Sep  5 17:54 disabled-apps
-rw-r--r--  1 41812 41812  2759 Sep  5 17:29 findlogs.ini
drwxr-xr-x  2 41812 41812  4096 Sep  5 17:54 init.d
-rw-------  1 41812 41812    54 Nov  3 17:01 instance.cfg
drwx--x---  3 41812 41812  4096 Nov  3 17:01 licenses
-r--r--r--  1 41812 41812   498 Sep  5 17:29 log-btool-debug.cfg
-r--r--r--  1 41812 41812   526 Sep  5 17:29 log-btool.cfg
-r--r--r--  1 41812 41812   256 Sep  5 17:29 log-cmdline-debug.cfg
-r--r--r--  1 41812 41812  2643 Sep  5 17:29 log-cmdline.cfg
-r--r--r--  1 41812 41812  6946 Sep  5 17:29 log-debug.cfg
-r--r--r--  1 41812 41812  2812 Sep  5 17:29 log-searchprocess.cfg
-r--r--r--  1 41812 41812  1288 Sep  5 17:29 log-utility.cfg
-rw-r--r--  1 41812 41812 28365 Sep  5 17:29 log.cfg
-rw-------  1 root  root    145 Nov  5 14:53 login-info.cfg
drwxr-xr-x  3 41812 41812  4096 Sep  5 17:54 manager-apps
drwxr-xr-x  3 41812 41812  4096 Sep  5 17:54 master-apps
drwxr-xr-x  7 41812 41812  4096 Sep  5 17:54 modules
drwxr-xr-x  2 41812 41812  4096 Nov  3 17:01 myinstall
drwxr-xr-x  2 41812 41812  4096 Nov  3 17:01 openldap
drwxr-xr-x  2 41812 41812  4096 Sep  5 17:54 packages
-rw-------  1 41812 41812   164 Nov  3 17:01 passwd
-r--r--r--  1 41812 41812  1836 Sep  5 17:25 prettyprint.xsl
-r--r--r--  1 41812 41812 21719 Sep  5 17:25 searchLanguage.xml
drwxr-xr-x  4 41812 41812  4096 Sep  5 17:54 shcluster
-rw-r--r--  1 41812 41812  1725 Sep  5 17:54 splunk-enttrial.lic
-rw-r--r--  1 41812 41812   787 Nov  3 17:01 splunk-launch.conf
-rw-r--r--  1 41812 41812   787 Sep  5 17:54 splunk-launch.conf.default
-r--r--r--  1 41812 41812    70 Sep  5 17:29 splunk.version
drwxr-xr-x  9 41812 41812  4096 Sep  5 17:54 system
drwxr-xr-x  3 41812 41812  4096 Nov  3 17:02 users
root@4a83624585fe:/opt/splunk# ls -l etc/system/
total 28
drwxr-xr-x 2 41812 41812 4096 Sep  5 17:54 README
drwxr-xr-x 3 41812 41812 4096 Nov  4 16:16 bin
drwxr-xr-x 3 41812 41812 4096 Sep  5 17:54 default
drwxr-xr-x 2 41812 41812 4096 Nov  3 17:01 local
drwxr-xr-x 2 41812 41812 4096 Sep  5 17:54 lookups
drwxr-xr-x 2 41812 41812 4096 Nov  3 17:01 metadata
drwxr-xr-x 2 41812 41812 4096 Sep  5 17:54 static
root@4a83624585fe:/opt/splunk# ls -l etc/system/default/
total 1960
-r--r--r-- 1 41812 41812   7963 Sep  5 17:29 alert_actions.conf
-r--r--r-- 1 41812 41812   4246 Sep  5 17:29 app.conf
-r--r--r-- 1 41812 41812    466 Sep  5 17:29 audit.conf
-rw------- 1 41812 41812   1385 Sep  5 17:29 authentication.conf
-r--r--r-- 1 41812 41812  13646 Sep  5 17:29 authorize.conf
-r--r--r-- 1 41812 41812   1157 Sep  5 17:29 collections.conf
-r--r--r-- 1 41812 41812   2374 Sep  5 17:29 commands.conf
-r--r--r-- 1 41812 41812   7163 Sep  5 17:29 conf.conf
drwxr-xr-x 3 41812 41812   4096 Sep  5 17:54 data
-r--r--r-- 1 41812 41812   1765 Sep  5 17:29 datamodels.conf
-r--r--r-- 1 41812 41812   1207 Sep  5 17:29 datatypesbnf.conf
-r--r--r-- 1 41812 41812    866 Sep  5 17:29 default-mode.conf
-r--r--r-- 1 41812 41812   3208 Sep  5 17:29 distsearch.conf
-r--r--r-- 1 41812 41812    620 Sep  5 17:29 event_renderers.conf
-r--r--r-- 1 41812 41812   4777 Sep  5 17:29 eventdiscoverer.conf
-r--r--r-- 1 41812 41812   1634 Sep  5 17:29 eventtypes.conf
-r--r--r-- 1 41812 41812   1606 Sep  5 17:29 federated.conf
-r--r--r-- 1 41812 41812    682 Sep  5 17:29 field_filters.conf
-r--r--r-- 1 41812 41812   2235 Sep  5 17:29 fields.conf
-r--r--r-- 1 41812 41812    247 Sep  5 17:29 global-banner.conf
-r--r--r-- 1 41812 41812  39620 Sep  5 17:29 health.conf
-r--r--r-- 1 41812 41812   9705 Sep  5 17:29 indexes.conf
-r--r--r-- 1 41812 41812   3570 Sep  5 17:29 inputs.conf
-r--r--r-- 1 41812 41812  51980 Sep  5 17:29 limits.conf
-r--r--r-- 1 41812 41812    207 Sep  5 17:29 literals.conf
-r--r--r-- 1 41812 41812 298514 Sep  5 17:29 livetail.conf
-r--r--r-- 1 41812 41812 297673 Sep  5 17:29 messages.conf
-r--r--r-- 1 41812 41812    996 Sep  5 17:29 metric_alerts.conf
-r--r--r-- 1 41812 41812    208 Sep  5 17:29 metric_rollups.conf
-r--r--r-- 1 41812 41812    317 Sep  5 17:29 multikv.conf
-r--r--r-- 1 41812 41812   2514 Sep  5 17:29 outputs.conf
-r--r--r-- 1 41812 41812    854 Sep  5 17:29 procmon-filters.conf
-r--r--r-- 1 41812 41812  32738 Sep  5 17:29 props.conf
-r--r--r-- 1 41812 41812  80683 Sep  5 17:29 restmap.conf
-r--r--r-- 1 41812 41812  10315 Sep  5 17:29 savedsearches.conf
-r--r--r-- 1 41812 41812 385240 Sep  5 17:29 searchbnf.conf
-r--r--r-- 1 41812 41812   1764 Sep  5 17:29 segmenters.conf
-r--r--r-- 1 41812 41812  28386 Sep  5 17:29 server.conf
-r--r--r-- 1 41812 41812   1324 Sep  5 17:29 serverclass.conf
-r--r--r-- 1 41812 41812  38104 Sep  5 17:29 source-classifier.conf
-r--r--r-- 1 41812 41812 416176 Sep  5 17:54 sourcetypes.conf
-r--r--r-- 1 41812 41812    492 Sep  5 17:29 telemetry.conf
-r--r--r-- 1 41812 41812   3496 Sep  5 17:29 times.conf
-r--r--r-- 1 41812 41812    697 Sep  5 17:29 transactiontypes.conf
-r--r--r-- 1 41812 41812  13540 Sep  5 17:29 transforms.conf
-r--r--r-- 1 41812 41812   2598 Sep  5 17:29 ui-prefs.conf
-r--r--r-- 1 41812 41812  11901 Sep  5 17:29 ui-tour.conf
-r--r--r-- 1 41812 41812    188 Sep  5 17:29 viewstates.conf
-r--r--r-- 1 41812 41812  12383 Sep  5 17:29 visualizations.conf
-r--r--r-- 1 41812 41812   8029 Sep  5 17:29 web-features.conf
-r--r--r-- 1 41812 41812  60216 Sep  5 17:29 web.conf
-r--r--r-- 1 41812 41812    488 Sep  5 17:29 workflow_actions.conf
-r--r--r-- 1 41812 41812     55 Sep  5 17:29 workload_policy.conf
-r--r--r-- 1 41812 41812    589 Sep  5 17:29 workload_pools.conf
-r--r--r-- 1 41812 41812     75 Sep  5 17:29 workload_rules.conf
root@4a83624585fe:/opt/splunk# cat etc/system/default/indexes.conf 
#Version 9.3.1
#DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
#
# This file configures Splunk's indexes and their properties.
#

################################################################################
# "global" params (not specific to individual indexes)
################################################################################
sync = 0
indexThreads = auto
memPoolMB = auto
defaultDatabase = main
enableRealtimeSearch = true
suppressBannerList =
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
bucketRebuildMemoryHint = auto
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
serviceInactiveIndexesPeriod = 60
maxBucketSizeCacheEntries = 0
processTrackerServiceInterval = 1
hotBucketTimeRefreshInterval = 10
rtRouterThreads = 0
rtRouterQueueSize = 10000
selfStorageThreads = 2
fileSystemExecutorWorkers = 5
hotBucketStreaming.extraBucketBuildingCmdlineArgs =

################################################################################
# index specific defaults
################################################################################
maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxGlobalRawDataSizeMB = 0
maxGlobalDataSizeMB = 0
maxMemMB = 5
maxConcurrentOptimizes = 6
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = auto
metric.maxHotBuckets = auto
minHotIdleSecsBeforeForceRoll = auto
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
enableDataIntegrityControl = false
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000
journalCompression = zstd
enableTsidxReduction = false
suspendHotRollByDeleteQuery = false
tsidxReductionCheckPeriodInSec = 600
timePeriodInSecBeforeTsidxReduction = 604800
datatype = event
splitByIndexKeys =
metric.splitByIndexKeys =
tsidxWritingLevel = 3
hotBucketStreaming.sendSlices = false
hotBucketStreaming.removeRemoteSlicesOnRoll = false
hotBucketStreaming.reportStatus = false
hotBucketStreaming.deleteHotsAfterRestart = false
tsidxDedupPostingsListMaxTermsLimit = 8388608
tsidxTargetSizeMB = 1500
metric.tsidxTargetSizeMB = 1500
metric.enableFloatingPointCompression = true
metric.compressionBlockSize = 1024
metric.stubOutRawdataJournal = true
metric.timestampResolution = s
waitPeriodInSecsForManifestWrite = 60
bucketMerging = false
bucketMerge.minMergeSizeMB = 750
bucketMerge.maxMergeSizeMB = 1000
bucketMerge.maxMergeTimeSpanSecs = 7776000

#
# By default none of the indexes are replicated.
#
repFactor = 0

# Splunk to Splunk federated index
federated.provider =
federated.dataset =

[volume:_splunk_summaries]
path = $SPLUNK_DB

[provider-family:hadoop]
vix.mode          = report
vix.command       = $SPLUNK_HOME/bin/jars/sudobash
vix.command.arg.1 = $HADOOP_HOME/bin/hadoop
vix.command.arg.2 = jar
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-h1.jar
vix.command.arg.4 = com.splunk.mr.SplunkMR
vix.env.MAPREDUCE_USER             =
vix.env.HADOOP_HEAPSIZE            = 512
vix.env.HADOOP_CLIENT_OPTS         = -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.env.HUNK_THIRDPARTY_JARS       = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.9.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.9.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.21.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.10.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-exec-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-metastore-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-serde-0.12.0.jar
vix.mapred.job.reuse.jvm.num.tasks = 100
vix.mapred.child.java.opts         = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapred.reduce.tasks            = 0
vix.mapred.job.map.memory.mb       = 2048
vix.mapred.job.reduce.memory.mb    = 512
vix.mapred.job.queue.name          = default
vix.mapreduce.job.jvm.numtasks     = 100
vix.mapreduce.map.java.opts        = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.reduce.java.opts     = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.job.reduces          = 0
vix.mapreduce.map.memory.mb        = 2048
vix.mapreduce.reduce.memory.mb     = 512
vix.mapreduce.job.queuename        = default
vix.splunk.search.column.filter    = 1
vix.splunk.search.mixedmode        = 1
vix.splunk.search.debug            = 0
vix.splunk.search.mr.maxsplits     = 10000
vix.splunk.search.mr.minsplits     = 100
vix.splunk.search.mr.splits.multiplier = 10
vix.splunk.search.mr.poll          = 2000
vix.splunk.search.recordreader     = SplunkJournalRecordReader,ValueAvroRecordReader,SimpleCSVRecordReader,SequenceFileRecordReader
vix.splunk.search.recordreader.avro.regex     = \.avro$
vix.splunk.search.recordreader.csv.regex      = \.([tc]sv)(?:\.(?:gz|bz2|snappy))?$
vix.splunk.search.recordreader.sequence.regex = \.seq$
vix.splunk.home.datanode           = /tmp/splunk/$SPLUNK_SERVER_NAME/
vix.splunk.heartbeat               = 1
vix.splunk.heartbeat.threshold     = 60
vix.splunk.heartbeat.interval      = 1000
vix.splunk.setup.onsearch          = 1
vix.splunk.setup.package           = current

################################################################################
# index definitions
################################################################################

[main]
homePath   = $SPLUNK_DB/defaultdb/db
coldPath   = $SPLUNK_DB/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

[history]
homePath   = $SPLUNK_DB/historydb/db
coldPath   = $SPLUNK_DB/historydb/colddb
thawedPath = $SPLUNK_DB/historydb/thaweddb
tstatsHomePath = volume:_splunk_summaries/historydb/datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800

[summary]
homePath   = $SPLUNK_DB/summarydb/db
coldPath   = $SPLUNK_DB/summarydb/colddb
thawedPath = $SPLUNK_DB/summarydb/thaweddb
tstatsHomePath = volume:_splunk_summaries/summarydb/datamodel_summary

[_internal]
homePath   = $SPLUNK_DB/_internaldb/db
coldPath   = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[_audit]
homePath   = $SPLUNK_DB/audit/db
coldPath   = $SPLUNK_DB/audit/colddb
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary

[_thefishbucket]
homePath   = $SPLUNK_DB/fishbucket/db
coldPath   = $SPLUNK_DB/fishbucket/colddb
thawedPath = $SPLUNK_DB/fishbucket/thaweddb
tstatsHomePath = volume:_splunk_summaries/fishbucket/datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200

# this index has been removed in the 4.1 series, but this stanza must be
# preserved to avoid displaying errors for users that have tweaked the index's
# size/etc parameters in local/indexes.conf.
#
[splunklogger]
homePath   = $SPLUNK_DB/splunklogger/db
coldPath   = $SPLUNK_DB/splunklogger/colddb
thawedPath = $SPLUNK_DB/splunklogger/thaweddb
disabled = true

[_introspection]
homePath   = $SPLUNK_DB/_introspection/db
coldPath   = $SPLUNK_DB/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600

[_telemetry]
homePath   = $SPLUNK_DB/_telemetry/db
coldPath   = $SPLUNK_DB/_telemetry/colddb
thawedPath = $SPLUNK_DB/_telemetry/thaweddb
maxDataSize = 256
frozenTimePeriodInSecs = 63072000

[_metrics]
homePath   = $SPLUNK_DB/_metrics/db
coldPath   = $SPLUNK_DB/_metrics/colddb
thawedPath = $SPLUNK_DB/_metrics/thaweddb
datatype = metric
#14 day retention
frozenTimePeriodInSecs = 1209600
metric.splitByIndexKeys = metric_name

# Internal Use Only: rollup data from the _metrics index.
[_metrics_rollup]
homePath   = $SPLUNK_DB/_metrics_rollup/db
coldPath   = $SPLUNK_DB/_metrics_rollup/colddb
thawedPath = $SPLUNK_DB/_metrics_rollup/thaweddb
datatype = metric
# 2 year retention
frozenTimePeriodInSecs = 63072000
metric.splitByIndexKeys = metric_name

[_configtracker]
homePath   = $SPLUNK_DB/_configtracker/db
coldPath   = $SPLUNK_DB/_configtracker/colddb
thawedPath = $SPLUNK_DB/_configtracker/thaweddb
frozenTimePeriodInSecs = 2592000

# NOTE: When adding a new index, please also add an entry in cfg/bundles/cluster/default/indexes.conf.in
# with repFactor=0, homePath, coldPath, and thawedPath
root@4a83624585fe:/opt/splunk# 

https://docs.splunk.com/Documentation/Splunk/9.3.1/Indexer/Setaretirementandarchivingpolicy

https://docs.splunk.com/Documentation/Splunk/9.3.1/Indexer/Automatearchiving

https://docs.splunk.com/Documentation/Splunk/9.3.1/Indexer/Restorearchiveddata

SCRIPTS ON MANAGING INDEX AND BUCKETS

Here’s a comprehensive guide on deleting, archiving, and restoring Splunk index data across the various storage stages: warm, cold, frozen, thawed, and back.

1. Deleting an Index in Splunk

Deleting an index will permanently remove all data in that index.

Steps to Delete an Index:

1. Log in to Splunk as an admin user.
2. Navigate to Indexes:
   • Go to Settings > Indexes.
   • Locate the index you want to delete.
3. Delete the Index:
   • Click on Delete next to the index name.
   • Confirm the deletion when prompted.

Note: This process removes the index and all associated data. If you only want to delete the data and keep the index, use the clean command.

Delete Index Data Without Deleting the Index:

To delete only the data from an index without deleting the index configuration itself, use the following CLI command:

$SPLUNK_HOME/bin/splunk stop
$SPLUNK_HOME/bin/splunk clean eventdata -index <index_name>
$SPLUNK_HOME/bin/splunk start

2. Archiving Index Data

Splunk stores index data in buckets (hot, warm, cold, frozen). Moving data to frozen (archiving) helps manage data retention and storage limits.

Configuring Frozen Buckets for Archiving

1. Edit indexes.conf:

   • Find indexes.conf in $SPLUNK_HOME/etc/system/local or in an app directory.
   • Configure the frozen bucket location:

   [<index_name>]
   coldToFrozenDir = /path/to/frozen_storage
   frozenTimePeriodInSecs = 31536000   # Retention time (in seconds) before data moves to frozen

   • coldToFrozenDir specifies the directory for frozen (archived) data.
   • frozenTimePeriodInSecs determines how long data is retained before moving to frozen.

2. Restart Splunk:

   • Apply the changes by restarting Splunk:

   $SPLUNK_HOME/bin/splunk restart

3. Archiving Script Example:

   Here’s a simple script to move data from cold to frozen using the coldToFrozenDir setting.

   #!/bin/bash
   INDEX_NAME="<index_name>"
   FROZEN_DIR="/path/to/frozen_storage/$INDEX_NAME"
   
   # Ensure the frozen directory exists
   mkdir -p $FROZEN_DIR
   
   # Move old cold buckets to frozen directory
   mv $SPLUNK_HOME/var/lib/splunk/$INDEX_NAME/colddb/* $FROZEN_DIR/
   
   echo "Data moved to frozen directory for archiving."

   Run this script to periodically archive cold buckets to the frozen directory.

3. Restoring Index Data from Frozen to Thawed

To view data in the frozen state, you can move it into the thawed directory. This makes it temporarily searchable.

Steps to Move Frozen to Thawed

1. Locate Frozen Buckets:

   • Identify the frozen bucket location (coldToFrozenDir).

2. Copy Data to Thawed Directory:

   • Move frozen data back to the thaweddb directory for the index.

   # Copy frozen data to thaweddb directory
   INDEX_NAME="<index_name>"
   FROZEN_DIR="/path/to/frozen_storage/$INDEX_NAME"
   THAWED_DIR="$SPLUNK_HOME/var/lib/splunk/$INDEX_NAME/thaweddb"
   
   # Copy frozen data to thawed
   cp -r $FROZEN_DIR/* $THAWED_DIR/

3. Restart Splunk (optional):

   • Splunk automatically indexes data in thaweddb for searching, but a restart can ensure immediate indexing.

   $SPLUNK_HOME/bin/splunk restart

4. Search Thawed Data:

   • Data in thaweddb is now searchable in Splunk.

Note: After analysis, you may delete the thawed data to free up space.

4. Moving Thawed Data Back to Warm

Directly moving data from thawed to warm is not typically supported in Splunk. However, to re-ingest data:

1. Extract Data from Thawed:

   • Export thawed data as raw files or CSV.

2. Re-ingest Data:

   • Use the oneshot command to re-index data into warm buckets.

   $SPLUNK_HOME/bin/splunk add oneshot /path/to/extracted_data.log -index <index_name> -sourcetype <sourcetype>

5. Moving Warm Data to Cold

Warm data automatically rolls to cold when warm bucket limits are reached. You can configure this in indexes.conf:

[index_name]
maxWarmDBCount = 300   # Max number of warm buckets

When this threshold is reached, the oldest warm bucket moves to the cold storage.

Summary of Bash Scripts

Here’s a collection of the scripts mentioned:

Script 1: Delete Index Data

#!/bin/bash
INDEX_NAME="<index_name>"
$SPLUNK_HOME/bin/splunk stop
$SPLUNK_HOME/bin/splunk clean eventdata -index $INDEX_NAME
$SPLUNK_HOME/bin/splunk start

Script 2: Archive Cold Data to Frozen

#!/bin/bash
INDEX_NAME="<index_name>"
FROZEN_DIR="/path/to/frozen_storage/$INDEX_NAME"

mkdir -p $FROZEN_DIR
mv $SPLUNK_HOME/var/lib/splunk/$INDEX_NAME/colddb/* $FROZEN_DIR/

echo "Data moved to frozen directory for archiving."

Script 3: Restore Frozen Data to Thawed

#!/bin/bash
INDEX_NAME="<index_name>"
FROZEN_DIR="/path/to/frozen_storage/$INDEX_NAME"
THAWED_DIR="$SPLUNK_HOME/var/lib/splunk/$INDEX_NAME/thaweddb"

cp -r $FROZEN_DIR/* $THAWED_DIR/
$SPLUNK_HOME/bin/splunk restart

Important Considerations

• Back Up Data: Before deleting or archiving, ensure you have backups.
• Permissions: Ensure all directories are accessible by the Splunk user.
• Monitor Disk Usage: Archiving and thawing require extra disk space.
• Test on Non-Production: Perform these operations on a test environment first to avoid accidental data loss.

These scripts and instructions provide a robust approach to managing index data lifecycle, archiving for long-term storage, and restoring data for analysis.

BUCKET POLICY

Certainly! Let's delve into how Splunk manages data movement between the different bucket stages—hot, warm, cold, and frozen—using the settings specified in your provided indexes.conf file. Understanding these settings is crucial for effective data lifecycle management in Splunk.

Understanding Splunk's Bucket Lifecycle

Splunk organizes indexed data into directories called buckets, which represent different stages in the data lifecycle:

1. Hot Buckets: Where incoming data is initially stored. Hot buckets are writable.
2. Warm Buckets: When hot buckets roll over, they become warm. Warm buckets are read-only.
3. Cold Buckets: Older data moves to cold buckets, which are stored in a different directory.
4. Frozen Buckets: Data that has aged beyond retention policies is moved to frozen buckets. By default, frozen data is deleted unless you specify an archive location.
5. Thawed Buckets: If you need to access frozen data, you can manually move it to thawed buckets for Splunk to read.

Key Settings in indexes.conf

The indexes.conf file contains settings that control when and how data moves between these stages. Below, we'll explore the relevant parameters and explain their impact on data movement.

1. Transition from Hot to Warm Buckets

Parameters:

• maxHotBuckets: Maximum number of hot buckets allowed.
• maxHotSpanSecs: Maximum time span in seconds for data in a hot bucket.
• maxDataSize: Maximum size of a hot bucket.
• maxHotIdleSecs: Maximum idle time in seconds before a hot bucket rolls to warm.

Explanation:

• maxHotBuckets = auto:

  • Splunk automatically manages the number of hot buckets. The default is typically 3.
  • When the number of hot buckets reaches this limit, the oldest hot bucket rolls to warm.

• maxHotSpanSecs = 7776000:

  • Hot buckets with data older than this time span (in seconds) will roll to warm.
  • 7776000 seconds = 90 days.

• maxDataSize = auto:

  • Controls the size of hot buckets.
  • Splunk uses predefined sizes based on maxDataSize, such as auto, auto_high_volume, etc.

• maxHotIdleSecs = 0:

  • If set to a value greater than 0, hot buckets idle for more than this time will roll to warm.
  • maxHotIdleSecs = 0 disables this setting.

Transition Logic:

• A hot bucket rolls to warm when:
  • The number of hot buckets exceeds maxHotBuckets.
  • The time span of the data exceeds maxHotSpanSecs.
  • The size of the bucket exceeds maxDataSize.
  • The bucket has been idle for longer than maxHotIdleSecs (if set).

2. Transition from Warm to Cold Buckets

Parameters:

• maxWarmDBCount: Maximum number of warm buckets allowed.
• homePath.maxDataSizeMB: Maximum size of data in hot and warm paths combined.

Explanation:

• maxWarmDBCount = 300:

  • Sets the maximum number of warm buckets.
  • When this limit is exceeded, the oldest warm bucket rolls to cold.

• homePath.maxDataSizeMB = 0:

  • If set, limits the total size of the homePath (hot and warm buckets).
  • 0 means unlimited size.

Transition Logic:

• A warm bucket rolls to cold when:
  • The number of warm buckets exceeds maxWarmDBCount.
  • The combined size of hot and warm buckets exceeds homePath.maxDataSizeMB (if set).

3. Transition from Cold to Frozen Buckets

Parameters:

• frozenTimePeriodInSecs: Time in seconds before data is moved to frozen.
• maxTotalDataSizeMB: Maximum total size of data in hot, warm, and cold paths.
• coldPath.maxDataSizeMB: Maximum size of data in the cold path.

Explanation:

• frozenTimePeriodInSecs = 188697600:

  • Data older than this value will be moved to frozen.
  • 188697600 seconds = approximately 6 years.

• maxTotalDataSizeMB = 500000:

  • Limits the total size of the index (hot, warm, cold).
  • When exceeded, Splunk deletes the oldest data (from cold buckets) to free space.

• coldPath.maxDataSizeMB = 0:

  • If set, limits the size of the cold path.
  • 0 means unlimited size.

Transition Logic:

• A cold bucket moves to frozen when:
  • The data in the bucket is older than frozenTimePeriodInSecs.
  • The total size of the index exceeds maxTotalDataSizeMB.
  • The size of the cold path exceeds coldPath.maxDataSizeMB (if set).

4. Handling of Frozen Data

Parameters:

• coldToFrozenDir: Directory where frozen data is archived.
• coldToFrozenScript: Script to execute when data moves to frozen.

Explanation:

• coldToFrozenDir =:

  • If set, Splunk moves frozen data to this directory instead of deleting it.
  • Empty value means frozen data will be deleted.

• coldToFrozenScript =:

  • Specifies a script to run when data moves to frozen.
  • Can be used to archive data externally.

Transition Logic:

• When data moves to frozen:
  • If coldToFrozenDir is specified, data is moved there.
  • If not, and coldToFrozenScript is specified, the script is executed.
  • If neither is set, data is deleted.

Example of Data Movement Through Buckets

Let's illustrate how data moves through the bucket lifecycle based on the default settings:

1. Hot to Warm:

   • The index starts ingesting data, storing it in hot buckets.
   • When the number of hot buckets exceeds maxHotBuckets (auto/default is 3), the oldest hot bucket rolls to warm.
   • If a hot bucket's data spans over maxHotSpanSecs (90 days), it rolls to warm.
   • If a hot bucket exceeds maxDataSize, it rolls to warm.

2. Warm to Cold:

   • Warm buckets accumulate until the number reaches maxWarmDBCount (300).
   • When this limit is exceeded, the oldest warm bucket rolls to cold.
   • If homePath.maxDataSizeMB is set and exceeded, warm buckets roll to cold.

3. Cold to Frozen:

   • Cold buckets remain until data exceeds frozenTimePeriodInSecs (6 years).
   • When data in a bucket is older than this period, the bucket moves to frozen.
   • If maxTotalDataSizeMB (500,000 MB) is exceeded, oldest cold buckets are frozen.

4. Frozen Data Handling:

   • Since coldToFrozenDir is empty, frozen data is deleted by default.
   • To archive frozen data, specify a coldToFrozenDir or a coldToFrozenScript.

Customizing the Bucket Policy

To adjust how data moves between buckets, you can modify the relevant parameters in indexes.conf:

• Decrease Data Retention:

  • Lower frozenTimePeriodInSecs to move data to frozen sooner.
  • Reduce maxTotalDataSizeMB to limit index size.

• Adjust Bucket Sizes:

  • Change maxDataSize to control hot bucket sizes (auto, auto_high_volume, or a specific size in MB).
  • Modify maxHotBuckets and maxWarmDBCount to control the number of buckets.

• Archive Frozen Data:

  • Set coldToFrozenDir to a directory where you want to archive frozen data.
  • Alternatively, use coldToFrozenScript to execute a custom archival process.

Example Configuration to Archive Frozen Data:

[index_name]
frozenTimePeriodInSecs = 15552000        # 180 days
coldToFrozenDir = /mnt/archived_splunk_data/index_name

Understanding Specific Index Definitions

Your indexes.conf also contains definitions for specific indexes like main, _internal, _audit, etc. Each index can have its own settings overriding the defaults.

Example:

[main]
homePath = $SPLUNK_DB/defaultdb/db
coldPath = $SPLUNK_DB/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
maxDataSize = auto_high_volume

• maxDataSize = auto_high_volume:
  • Adjusts the hot bucket size for high-volume data ingestion.
  • Hot buckets are larger, reducing the frequency of bucket rolling.

Key Takeaways

• Hot Buckets: Write data; roll to warm based on size, time span, or count.
• Warm Buckets: Read-only; roll to cold when warm bucket count exceeds maxWarmDBCount.
• Cold Buckets: Stored in a different directory; data ages here until it reaches frozenTimePeriodInSecs.
• Frozen Buckets: Data is deleted by default unless archived via coldToFrozenDir or coldToFrozenScript.
• Thawed Buckets: Used to re-ingest frozen data for searching.

Recommendations

• Monitor Index Sizes: Keep an eye on your index sizes to prevent unexpected data loss due to size limits.
• Set Retention Policies: Adjust frozenTimePeriodInSecs according to your data retention requirements.
• Archive Data: If you need to keep data beyond the frozen stage, configure coldToFrozenDir or use a script.
• Optimize Performance: Adjust maxHotBuckets and maxDataSize to optimize indexing performance based on your ingestion rate.

By understanding and configuring these settings, you can effectively manage how Splunk handles data aging, retention, and storage, ensuring that your system performs optimally while retaining necessary data for your compliance and analysis needs.

FINAL

Certainly! Let's revisit the explanation and include the actual values from your indexes.conf file to show exactly when data moves between hot, warm, cold, and frozen stages in Splunk.

The Lifecycle of Data in Splunk with Actual Values

Using the settings from your provided indexes.conf, we'll explain how data moves through the stages with the specific values applied.

1. Transition from Hot to Warm Buckets

Parameters from indexes.conf:

• maxHotBuckets = auto
  • Defaults to 3 hot buckets.
• maxHotSpanSecs = 7776000
  • 7776000 seconds = 90 days.
• maxDataSize = auto
  • Splunk manages hot bucket sizes automatically, typically around 750 MB per bucket.
• maxHotIdleSecs = 0
  • 0 means this setting is disabled (no idle time limit).

Explanation with Actual Values:

• Number of Hot Buckets (maxHotBuckets):

  • Splunk allows up to 3 hot buckets by default. When this limit is reached, the oldest hot bucket rolls to warm to make room for new data.

• Maximum Time Span (maxHotSpanSecs):

  • A hot bucket containing data that spans more than 90 days will be rolled to warm.
  • Example: If a hot bucket has events from January 1 to April 2 (over 90 days), it will transition to warm.

• Bucket Size (maxDataSize):

  • Hot buckets grow until they reach the size determined by maxDataSize, which is managed automatically (approximately 750 MB).
  • When a hot bucket exceeds this size, it rolls to warm.

• Idle Time (maxHotIdleSecs):

  • Since this is set to 0, Splunk does not roll hot buckets based on idle time.

**So, data moves from hot to warm when:**

• There are more than 3 hot buckets.
• A hot bucket's data spans over 90 days.
• A hot bucket exceeds the size limit (around 750 MB).

2. Transition from Warm to Cold Buckets

Parameters from indexes.conf:

• maxWarmDBCount = 300
• homePath.maxDataSizeMB = 0
  • 0 means no size limit on the hot and warm buckets combined.

Explanation with Actual Values:

• Maximum Warm Buckets (maxWarmDBCount):

  • Splunk allows up to 300 warm buckets.
  • When this limit is exceeded, the oldest warm bucket rolls to cold.

• Home Path Size Limit (homePath.maxDataSizeMB):

  • Since it's set to 0, there is no size limit on the combined hot and warm buckets based on storage size.

**So, data moves from warm to cold when:**

• There are more than 300 warm buckets in the index.

3. Transition from Cold to Frozen Buckets

Parameters from indexes.conf:

• frozenTimePeriodInSecs = 188697600
  • 188,697,600 seconds ≈ 6 years.
• maxTotalDataSizeMB = 500000
  • 500,000 MB = 500 GB.
• coldPath.maxDataSizeMB = 0
  • 0 means no size limit on cold buckets.

Explanation with Actual Values:

• Data Age (frozenTimePeriodInSecs):

  • Data older than 6 years is moved from cold to frozen.

• Total Index Size (maxTotalDataSizeMB):

  • If the total size of hot, warm, and cold buckets exceeds 500 GB, Splunk deletes the oldest data (from cold buckets) to stay within the limit.

• Cold Path Size Limit (coldPath.maxDataSizeMB):

  • Since it's set to 0, there is no size limit specifically for cold buckets.

**So, data moves from cold to frozen when:**

• The data is older than 6 years.
• The total index size exceeds 500 GB, triggering deletion of the oldest data to free up space.

4. Handling of Frozen Data

Parameters from indexes.conf:

• coldToFrozenDir =
  • Not set (empty), so no directory specified for frozen data.
• coldToFrozenScript =
  • Not set (empty), so no script specified.

Explanation:

• Deletion of Frozen Data:
  • Since neither coldToFrozenDir nor coldToFrozenScript is specified, Splunk deletes data when it moves to frozen.
  • This means data older than 6 years or data causing the index to exceed 500 GB will be permanently deleted.

Putting It All Together

Example Scenario:

1. Hot Buckets:

   • New data is ingested and stored in up to 3 hot buckets.
   • When there are more than 3 hot buckets, the oldest hot bucket rolls to warm.
   • If a hot bucket exceeds 90 days of data or grows beyond approximately 750 MB, it rolls to warm.

2. Warm Buckets:

   • Warm buckets accumulate up to 300 buckets.
   • When there are more than 300 warm buckets, the oldest warm bucket rolls to cold.

3. Cold Buckets:

   • Cold buckets store data until it becomes older than 6 years.
   • If the total size of hot + warm + cold exceeds 500 GB, the oldest cold buckets are deleted to maintain the size limit.

4. Frozen Data:

   • Data moved to frozen is deleted since no archive directory or script is specified.

Key Takeaways with Actual Values

• Hot to Warm Transition:

  • Occurs when:
  • More than 3 hot buckets exist.
  • A hot bucket contains data spanning over 90 days.
  • A hot bucket exceeds approximately 750 MB.

• Warm to Cold Transition:

  • Occurs when:
  • There are more than 300 warm buckets.

• Cold to Frozen Transition:

  • Occurs when:
  • Data is older than 6 years.
  • The total index size exceeds 500 GB.

• Frozen Data Handling:

  • Data is deleted upon reaching frozen stage unless an archive path or script is configured.

What You Should Do

• Monitor Data Retention:

  • Be aware that your data will be retained for up to 6 years or until the index reaches 500 GB.

• Plan for Archiving:

  • If you need to keep data beyond the frozen stage, consider setting coldToFrozenDir to specify an archive location.

• Adjust Settings if Necessary:

  • Modify indexes.conf in the $SPLUNK_HOME/etc/system/local/ directory to change retention periods or size limits to suit your requirements.

• Regular Backups:

  • Implement regular backups if you need to preserve data that Splunk might delete due to these policies.

By including the actual values from your indexes.conf, we've detailed exactly when and how data moves through the stages in Splunk. This information helps you manage your data retention policies effectively and ensures you are aware of when data might be deleted.

root@619f220eaaae:/# cat /opt/splunk/etc/system/default/indexes.conf
#Version 9.3.1
#DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
#
# This file configures Splunk's indexes and their properties.
#

################################################################################
# "global" params (not specific to individual indexes)
################################################################################
sync = 0
indexThreads = auto
memPoolMB = auto
defaultDatabase = main
enableRealtimeSearch = true
suppressBannerList =
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
bucketRebuildMemoryHint = auto
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
serviceInactiveIndexesPeriod = 60
maxBucketSizeCacheEntries = 0
processTrackerServiceInterval = 1
hotBucketTimeRefreshInterval = 10
rtRouterThreads = 0
rtRouterQueueSize = 10000
selfStorageThreads = 2
fileSystemExecutorWorkers = 5
hotBucketStreaming.extraBucketBuildingCmdlineArgs =

################################################################################
# index specific defaults
################################################################################
maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxGlobalRawDataSizeMB = 0
maxGlobalDataSizeMB = 0
maxMemMB = 5
maxConcurrentOptimizes = 6
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = auto
metric.maxHotBuckets = auto
minHotIdleSecsBeforeForceRoll = auto
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
enableDataIntegrityControl = false
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000
journalCompression = zstd
enableTsidxReduction = false
suspendHotRollByDeleteQuery = false
tsidxReductionCheckPeriodInSec = 600
timePeriodInSecBeforeTsidxReduction = 604800
datatype = event
splitByIndexKeys =
metric.splitByIndexKeys =
tsidxWritingLevel = 3
hotBucketStreaming.sendSlices = false
hotBucketStreaming.removeRemoteSlicesOnRoll = false
hotBucketStreaming.reportStatus = false
hotBucketStreaming.deleteHotsAfterRestart = false
tsidxDedupPostingsListMaxTermsLimit = 8388608
tsidxTargetSizeMB = 1500
metric.tsidxTargetSizeMB = 1500
metric.enableFloatingPointCompression = true
metric.compressionBlockSize = 1024
metric.stubOutRawdataJournal = true
metric.timestampResolution = s
waitPeriodInSecsForManifestWrite = 60
bucketMerging = false
bucketMerge.minMergeSizeMB = 750
bucketMerge.maxMergeSizeMB = 1000
bucketMerge.maxMergeTimeSpanSecs = 7776000

#
# By default none of the indexes are replicated.
#
repFactor = 0

# Splunk to Splunk federated index
federated.provider =
federated.dataset =

[volume:_splunk_summaries]
path = $SPLUNK_DB

[provider-family:hadoop]
vix.mode          = report
vix.command       = $SPLUNK_HOME/bin/jars/sudobash
vix.command.arg.1 = $HADOOP_HOME/bin/hadoop
vix.command.arg.2 = jar
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-h1.jar
vix.command.arg.4 = com.splunk.mr.SplunkMR
vix.env.MAPREDUCE_USER             =
vix.env.HADOOP_HEAPSIZE            = 512
vix.env.HADOOP_CLIENT_OPTS         = -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.env.HUNK_THIRDPARTY_JARS       = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.9.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.9.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.21.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.10.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-exec-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-metastore-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-serde-0.12.0.jar
vix.mapred.job.reuse.jvm.num.tasks = 100
vix.mapred.child.java.opts         = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapred.reduce.tasks            = 0
vix.mapred.job.map.memory.mb       = 2048
vix.mapred.job.reduce.memory.mb    = 512
vix.mapred.job.queue.name          = default
vix.mapreduce.job.jvm.numtasks     = 100
vix.mapreduce.map.java.opts        = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.reduce.java.opts     = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.job.reduces          = 0
vix.mapreduce.map.memory.mb        = 2048
vix.mapreduce.reduce.memory.mb     = 512
vix.mapreduce.job.queuename        = default
vix.splunk.search.column.filter    = 1
vix.splunk.search.mixedmode        = 1
vix.splunk.search.debug            = 0
vix.splunk.search.mr.maxsplits     = 10000
vix.splunk.search.mr.minsplits     = 100
vix.splunk.search.mr.splits.multiplier = 10
vix.splunk.search.mr.poll          = 2000
vix.splunk.search.recordreader     = SplunkJournalRecordReader,ValueAvroRecordReader,SimpleCSVRecordReader,SequenceFileRecordReader
vix.splunk.search.recordreader.avro.regex     = \.avro$
vix.splunk.search.recordreader.csv.regex      = \.([tc]sv)(?:\.(?:gz|bz2|snappy))?$
vix.splunk.search.recordreader.sequence.regex = \.seq$
vix.splunk.home.datanode           = /tmp/splunk/$SPLUNK_SERVER_NAME/
vix.splunk.heartbeat               = 1
vix.splunk.heartbeat.threshold     = 60
vix.splunk.heartbeat.interval      = 1000
vix.splunk.setup.onsearch          = 1
vix.splunk.setup.package           = current

################################################################################
# index definitions
################################################################################

[main]
homePath   = $SPLUNK_DB/defaultdb/db
coldPath   = $SPLUNK_DB/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

[history]
homePath   = $SPLUNK_DB/historydb/db
coldPath   = $SPLUNK_DB/historydb/colddb
thawedPath = $SPLUNK_DB/historydb/thaweddb
tstatsHomePath = volume:_splunk_summaries/historydb/datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800

[summary]
homePath   = $SPLUNK_DB/summarydb/db
coldPath   = $SPLUNK_DB/summarydb/colddb
thawedPath = $SPLUNK_DB/summarydb/thaweddb
tstatsHomePath = volume:_splunk_summaries/summarydb/datamodel_summary

[_internal]
homePath   = $SPLUNK_DB/_internaldb/db
coldPath   = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[_audit]
homePath   = $SPLUNK_DB/audit/db
coldPath   = $SPLUNK_DB/audit/colddb
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary

[_thefishbucket]
homePath   = $SPLUNK_DB/fishbucket/db
coldPath   = $SPLUNK_DB/fishbucket/colddb
thawedPath = $SPLUNK_DB/fishbucket/thaweddb
tstatsHomePath = volume:_splunk_summaries/fishbucket/datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200

# this index has been removed in the 4.1 series, but this stanza must be
# preserved to avoid displaying errors for users that have tweaked the index's
# size/etc parameters in local/indexes.conf.
#
[splunklogger]
homePath   = $SPLUNK_DB/splunklogger/db
coldPath   = $SPLUNK_DB/splunklogger/colddb
thawedPath = $SPLUNK_DB/splunklogger/thaweddb
disabled = true

[_introspection]
homePath   = $SPLUNK_DB/_introspection/db
coldPath   = $SPLUNK_DB/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600

[_telemetry]
homePath   = $SPLUNK_DB/_telemetry/db
coldPath   = $SPLUNK_DB/_telemetry/colddb
thawedPath = $SPLUNK_DB/_telemetry/thaweddb
maxDataSize = 256
frozenTimePeriodInSecs = 63072000

[_metrics]
homePath   = $SPLUNK_DB/_metrics/db
coldPath   = $SPLUNK_DB/_metrics/colddb
thawedPath = $SPLUNK_DB/_metrics/thaweddb
datatype = metric
#14 day retention
frozenTimePeriodInSecs = 1209600
metric.splitByIndexKeys = metric_name

# Internal Use Only: rollup data from the _metrics index.
[_metrics_rollup]
homePath   = $SPLUNK_DB/_metrics_rollup/db
coldPath   = $SPLUNK_DB/_metrics_rollup/colddb
thawedPath = $SPLUNK_DB/_metrics_rollup/thaweddb
datatype = metric
# 2 year retention
frozenTimePeriodInSecs = 63072000
metric.splitByIndexKeys = metric_name

[_configtracker]
homePath   = $SPLUNK_DB/_configtracker/db
coldPath   = $SPLUNK_DB/_configtracker/colddb
thawedPath = $SPLUNK_DB/_configtracker/thaweddb
frozenTimePeriodInSecs = 2592000

# NOTE: When adding a new index, please also add an entry in cfg/bundles/cluster/default/indexes.conf.in
# with repFactor=0, homePath, coldPath, and thawedPath