Study notes for Splunk Administration Part I.

• Example deployement:
• 1 searchhead server,  16 core CPU, 16GB memory;
• 8 index servers,  8 core CPU, 24GB memory each;
• 1 deployment server:  8 core CPU, 32GB memory;
• 2 syslog servers:  2 core CPU, 12GB memory each;
• 1 job server:  8 core CPU, 16GB memory;
• File system layout example:
• /opt/splunk   10GB local filesystem, for splunk binaries;
• /usr/local/splunk  5TB  for splunk data (hot/cold/warm), not shared;
• /usr/local/splunk/frozen  5TB NFS volume for archived (frozen) data, shared;
• Example Splunk package names:
• splunk
• splunkapp-pdf
• splunk-unix-forwarder
• splunkapp-unix-splunk
• splunkforwarder
• Splunk ports needed:
• 443:  splunk indexers web;
• 8089: 
• splunk search head - splunk indexer;  
• Splunk deployment server; 
• Splunk License server
• 9997: Splunk forwarder - splunk indexer
• Example Config/Settings:
• SPLUNK_DB=/splunk/splunkserver/    On Indexer servers;
• SPLUNK_HOME=/opt/splunk
• SPLUNK_SERVER_NAME=<splunk-server>
• SPLUNK_WEB_NAME=<splunk-web>
• MOGOC_DISABLE_SHM=1
• Splunk search head server config:  /opt/splunk/etc/system/local/distsearch.conf
• [DistributedSearch]  servers=https://<IP_INDEXER1>:8089,https://<IP_INDEXER2>:8089,......
• Splunk agent install examples:
• Create splunk user and group, a home directory need be created by splunk user;
• Create directory /opt/splunkforwarder with 0755 permission;
• /etc/init.d/splunk  should be included in the RPM and get installed;
• rpm -ivh splunkforwarder-......;
• /opt/splunkforwarder/splunk start --accept-license   (accept license )
• Create two folders and set owner to splunk user:
• /opt/splunkforwarder/etc/apps/<my_deployment_client>
• /opt/splunkforwarder/etc/apps/<my_deployment_client>/local
• create empty deploymentclient.conf with proper splunk user permission
• Modify above file with following contents:
• Deployment Server
• [target-broker:deploymentServer]
• targetUri = <splunkdeployserver>:8089
• phoneHomeIntervalInSecs = 600
• Create app.conf unser the above .../local folder with followign contents:
• [install]
• state = enabled
• Configure inputs.conf file with following content in it:  /opt/splunkforwarder/etc/system/local/inputs.conf
• [default]
• _meta=servername:XXX
• Splunk configruation directories:  default, local, app,  each has different scope;
• default directory:  $SPLUNK_HOME/etc/system/default,  has pre-configured versions  of the configuration files;  Should never change manually, they will be overwritten during upgrades;
• local directory:  $SPLUNK_HOME/etc/system/local,  won't be overwritten during upgrades; Most changes are made here, they are site wide, shared by all apps;
• app directory:  $SPLUNK_HOME/etc/apps
• such as time settings:  $SPLUNK_HOME/etc/apps/search/local/
• users directory:  $SPLUNK_HOME/etc/users:  specific configurations for users;
• reference documents:  $SPLUNK_HOME/etc/system/README:
• .spec:  such as inputs.conf.spec,  specifies syntax, list of available attributes and variables;
• .example:  such as inputs.conf.example:  has examples to reference;