However, manually installing Splunk Enterprise Server and configuring forwarders on several client machines can become time-consuming. In this blog post, we will automate the process from end-to-end.
Understanding the Components
Splunk Enterprise Server
This is the main Splunk system that stores, indexes, and searches all logs. It provides:
- Web UI
- Indexing database
- Search head
- User management
- Dashboard visualization
This is a lightweight agent installed on client machines. It:
- Sends logs to the Splunk server
- Runs silently as a background service
- Consumes minimal CPU & memory
Server Requirements
- OS: Linux (Ubuntu/RHEL/CentOS/Amazon Linux)
- 4+ GB RAM
- 20+ GB Disk
- Port 8000 (Web), 8089 (mgmt), 9997 (data input) open
- Linux-based client machines
- sudo access
- Network reachability to server port: 9997
Component Download URL
Splunk Enterprise https://www.splunk.com/en_us/download/splunk-enterprise.html
Splunk Forwarder https://www.splunk.com/en_us/download/universal-forwarder.html
SPLUNK ENTERPRISE SERVER INSTALLATION
Step 1: Update system
# dnf update -y
Step 2: Create Splunk OS User
Splunk should never run as root.
# useradd -m splunk
Verify:
# id splunk
uid=1001(splunk) gid=1001(splunk) groups=1001(splunk)
Step 3: Download Splunk Enterprise
Download Splunk from the official Splunk website and copy it to your server (example path used below):
/root/splunk-9.0.2-17e00c557dc1-linux-2.6-x86_64.rpm
Step 4: Install Splunk Enterprise
Install the RPM package:
# rpm -ivh splunk-9.0.2-17e00c557dc1-linux-2.6-x86_64.rpm
By default, Splunk installs to:
# ls -ld /opt/splunk/
drwx------ 12 splunk splunk 4096 Dec 20 03:04 /opt/splunk/
[root@inddcpspn01 ~]#
Give ownership of Splunk files to the splunk user:
# chown -R splunk:splunk /opt/splunk
Step 6: First Start of Splunk (Create Admin User)
This step is critical
The admin user is created only on the first successful start.
Run the following command as the splunk user:
# sudo -u splunk /opt/splunk/bin/splunk start
Type q
Do you agree with this license? [y/n]: y
Please enter an administrator username:admin
Please enter a new password:Welcome@123
Please confirm new password:Welcome@123
Step 7: Verify Admin User Creation
Check the password file:
# ls -l /opt/splunk/etc/passwd
# cat /opt/splunk/etc/passwd
You should see:
:admin:$6$5SYFmoISyswPtUPt$AXKb2n0RD7mL8UAz1wyZkgTdHkHWFIes/9DMz.4gw3.xnVyLyxpzj1mADGt8HTVJ.ky7f8tay1.bg.7osl7ci1::Administrator:admin:changeme@example.com:::20441
If this file exists, the admin user is created successfully.
Step 8: Enable Splunk at Boot
Install chkconfig if your system complains it's missing (e.g., on RHEL/CentOS 9)
$ dnf install chkconfig
$ sudo -u splunk /opt/splunk/bin/splunk stop
$ /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
$ sudo -u splunk /opt/splunk/bin/splunk start
Step 9: Start / Stop Splunk
Start Splunk
$ sudo -u splunk /opt/splunk/bin/splunk start
Stop Splunk
$ sudo -u splunk /opt/splunk/bin/splunk stop
Check Status
$ sudo -u splunk /opt/splunk/bin/splunk status
Step 10: Access Splunk Web UI
Open a browser and go to:
http://<server-ip>:8000 or http://<Server FQDN>:8000
Login with:
Username: admin
Password: Welcome@123
Step 11: (Optional) Firewall Configuration
Allow Splunk Web port:
# firewall-cmd --add-port=8000/tcp --permanent
# firewall-cmd --reload
Common Issues & Fixes
Admin password not working
Splunk was started before --seed-passwd
/opt/splunk/etc/passwd not created
Wrong user used to start Splunk
Fix: Stop Splunk, remove init files, and start again with --seed-passwd.
ENABLE SPLUNK DATA INPUT (TCP 9997)
Log into Splunk Web UI:URL:
http://<server-ip>:8000 or http://<Server FQDN>:8000
Then:
Enable Receiving Port
Go to Settings → Forwarding and Receiving
Click Configure Receiving
Click New Receiving Port
Enter:
Port: 9997
Verify Receiving Port
# netstat -tulnp | grep 9997
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 42402/splunkd
# ss -tulnp | grep 9997
tcp LISTEN 0 128 0.0.0.0:9997 0.0.0.0:* users:(("splunkd",pid=42402,fd=197))
Step 1: Create Splunk User on Client
Splunk services should not run as root.
# useradd -m splunk
Verify:
# id splunk
uid=1001(splunk) gid=1001(splunk) groups=1001(splunk)
Step 2: Download Splunk Universal Forwarder
Download the Universal Forwarder package from the Splunk website and copy it to the client server.
Example RPM file:
splunkforwarder-9.0.2-17e00c557dc1-linux-2.6-x86_64.rpm
Step 3: Install Splunk Universal Forwarder
Install the RPM:
# rpm -ivh splunkforwarder-9.0.2-17e00c557dc1-linux-2.6-x86_64.rpm
Default installation path:
# ls -ld /opt/splunkforwarder
drwxr-xr-x 9 splunk splunk 4096 Dec 19 22:02 /opt/splunkforwarder
# chown -R splunk:splunk /opt/splunkforwarder
Step 5: First Start of Splunk Forwarder
Start the Universal Forwarder for the first time:
$ sudo -u splunk /opt/splunkforwarder/bin/splunk start
Type q
Do you agree with this license? [y/n]: y
Do you agree with this license? [y/n]: y
Please enter an administrator username: admin
Please enter a new password: Welcome@123
Please confirm new password: Welcome@123
Step 6: Enable Forwarder to Start at Boot
Install chkconfig using dnf or yum.
# dnf install chkconfig
$ sudo -u splunk /opt/splunkforwarder/bin/splunk stop
# /opt/splunkforwarder/bin/splunk enable boot-start -user splunkSystemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.
$ sudo -u splunk /opt/splunkforwarder/bin/splunk start
Add Splunk Indexer as Receiving Destination
$ sudo -u splunk /opt/splunkforwarder/bin/splunk add forward-server 192.168.10.109:9997
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Splunk username: admin
Password:
Added forwarding to: 192.168.10.109:9997.
$ sudo -u splunk /opt/splunkforwarder/bin/splunk list forward-server
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Active forwards:
192.168.10.109:9997
Configured but inactive forwards:
None
Step 8: Add Log Files to Monitor
Example: Monitor Linux system logs$ sudo -u splunk /opt/splunkforwarder/bin/splunk add monitor /var/log/messages
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Added monitor of '/var/log/messages'.
For Ubuntu:
$ sudo -u splunk /opt/splunkforwarder/bin/splunk add monitor /var/log/syslogStep 9: Restart Splunk Forwarder
$ sudo -u splunk /opt/splunkforwarder/bin/splunk restart
Step 10: Verify Data on Splunk Server
On the Splunk Enterprise server:
Login to Splunk Web : http://<server-ip>:8000 or http://<Server FQDN>:8000
Go to Search & Reporting
Run:
index=_internal | stats count by host
You should see the client hostname.
Firewall Configuration (Optional)
Allow outgoing traffic to indexer:
# firewall-cmd --add-port=9997/tcp --permanent
# firewall-cmd --reload
Common Issues & Troubleshooting
- Forwarder not sending data
- Indexer port 9997 not enabled
- Firewall blocking traffic
- Incorrect indexer IP
$ sudo -u splunk /opt/splunkforwarder/bin/splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
egrep: warning: egrep is obsolescent; using grep -E
splunkd is running (PID: 1768).
splunk helpers are running (PIDs: 1794).
egrep: warning: egrep is obsolescent; using grep -E
$ tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
Conclusion
You have successfully installed and configured the Splunk Enterprise Server and the Splunk Universal Forwarder on the Splunk server and Splunk client machine. The Splunk client is now actively forwarding log data to the Splunk Enterprise server, enabling centralized log collection, monitoring, and analysis across the environment.
This setup provides better visibility into system activity, faster troubleshooting, and a scalable foundation for enterprise-level monitoring and observability.
