Monitoring Nginx Logs with Splunk Universal Forwarder in Docker

What is the Splunk Universal Forwarder?

The Splunk Universal Forwarder (UF) is a lightweight agent that, once installed, can be configured to collect data (e.g., log data) from various sources (like a server) and forward it to a Splunk instance. The Splunk UF is a minimal-footprint agent designed solely to collect and forward raw data. In contrast, the Splunk Heavy Forwarder (HF) runs a full Splunk instance capable of processing, filtering, and routing data before sending it on.

Architecture Overview

In our setup, Nginx generates access logs. Splunk UF collects and forwards those logs to the main Splunk instance.

Code Walkthrough

The full code is available here, so now let’s walk through how it works step by step.

Docker Compose Explained

Here’s the full docker-compose.yml:

version: '3.8'

services:
  splunk:
    image: splunk/splunk:9.2.0
    container_name: splunk
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD=password123
    ports:
      - "8000:8000"  # Splunk UI
      - "9997:9997"  # Receiving port for Universal Forwarder
    volumes:
      - splunk-data:/opt/splunk/var

  nginx:
    image: nginx:latest
    container_name: nginx
    ports:
      - "8080:80"
    volumes:
      - ./nginx_logs:/var/log/nginx
    logging:
      driver: "none"

  splunk-uf:
    image: splunk/universalforwarder:9.2.0
    container_name: splunk-uf
    depends_on:
      - splunk
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD=password123
    volumes:
      - ./nginx_logs:/nginx_logs:ro
      - ./config:/opt/splunkforwarder/etc/system/local
    entrypoint: >
      /bin/bash -c "
      /sbin/entrypoint.sh start;
      tail -f /dev/null
      "

volumes:
  splunk-data:

• Splunk - runs in a container with UI access on port 8000 and listens for data on port 9997.
• Nginx - serves content on port 8080 and stores logs in ./nginx_logs.
• Splunk UF - mounts the Nginx log directory read-only and uses configuration files from the config/ directory to know what to forward and where.

Splunk UF Config Files

Splunk UF uses config files to manage data collection and forwarding.

• inputs.conf defines what data sources to monitor.
• outputs.conf specifies where to send the collected data.

inputs.conf

[monitor:///nginx_logs/access.log]
disabled = false
sourcetype = nginx
index = main

• monitor:///nginx_logs/access.log – This tells UF to watch the Nginx access log.
• sourcetype = nginx – Helps categorize the log data.
• index = main – Sends the data to the default main index in Splunk.

outputs.conf

[tcpout]
defaultGroup = primary-indexers

[tcpout:primary-indexers]
server = splunk:9997

[tcpout-server://splunk:9997]

This config directs Splunk UF to forward logs to the Splunk instance at splunk:9997

How to Test It

Run:

docker-compose up -d

Access the services:

• Splunk UI: http://localhost:8000
• Nginx: http://localhost:8080

Generate traffic:

curl http://localhost:8080

Go to Splunk and search:

index="main" sourcetype="nginx"

You should see the forwarded Nginx logs: