Implementing Comprehensive Logging and Monitoring with Elasticsearch, Logstash, and Kibana (ELK Stack)

Introduction to the ELK Stack

The ELK stack combines three powerful tools to create a robust logging and monitoring solution that has become essential for modern DevOps and IT operations. As of July 2025, the ELK stack, also known as the Elastic Stack, remains a cornerstone for monitoring applications, infrastructure, and blockchain systems.

Key Capabilities

Why Use the ELK Stack?

The ELK stack has become the go-to solution for logging and monitoring due to its comprehensive feature set and proven reliability in production environments.

Blockchain Development Applications

For blockchain developers, the ELK stack can monitor node health, track transaction latencies, and analyze smart contract logs, ensuring robust and secure decentralized applications (dApps). It's particularly valuable for:

Setting Up the ELK Stack on Ubuntu

This section provides step-by-step instructions to install and configure Elasticsearch, Logstash, and Kibana on an Ubuntu 22.04 server. Ensure you have a server with at least 4GB RAM and 2 CPUs for optimal performance.

Prerequisites

sudo apt update
sudo apt install openjdk-11-jdk

Step 1: Install and Configure Elasticsearch

Elasticsearch stores and indexes log data for fast search and analysis.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

sudo apt install elasticsearch

cluster.name: my_cluster
node.name: node-1
network.host: 0.0.0.0
discovery.type: single-node

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

curl -X GET "localhost:9200"

Expected output:

{
  "name": "node-1",
  "cluster_name": "my_cluster",
  "version": {
    "number": "8.0.0",
    ...
  },
  "tagline": "You Know, for Search"
}

Step 2: Install and Configure Logstash

Logstash collects, processes, and sends logs to Elasticsearch.

sudo apt install logstash

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
  }
}

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

sudo systemctl start logstash
sudo systemctl enable logstash

Step 3: Install and Configure Kibana

Kibana provides a web interface for visualizing and analyzing logs.

sudo apt install kibana

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

sudo systemctl start kibana
sudo systemctl enable kibana

Configuring Logstash for Different Log Sources

Logstash's flexibility allows it to collect logs from various sources using input plugins, process them with filters, and send them to outputs like Elasticsearch. Below are examples of common configurations.

Example 1: System Logs with Parsing

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{DATA:program}: %{GREEDYDATA:message}" }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
  }
}

Example 2: Apache Web Server Logs

input {
  file {
    path => "/var/log/apache2/access.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "apache-%{+YYYY.MM.dd}"
  }
}

Example 3: Python Application Logs

For Python applications, use the python-logstash library to send logs to Logstash.

pip install python-logstash

import logging
from logstash import TCPLogstashHandler

logger = logging.getLogger('my_app')
logger.setLevel(logging.INFO)

handler = TCPLogstashHandler('localhost', 5000, version=1)
logger.addHandler(handler)

logger.info('This is a log message')

input {
  tcp {
    port => 5000
    codec => json
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "python-logs-%{+YYYY.MM.dd}"
  }
}

Creating Dashboards in Kibana

Kibana allows you to create visualizations and dashboards to monitor your logs effectively.

Step 1: Create an Index Pattern

Step 2: Create Visualizations

Step 3: Create Dashboards

Best Practices for the ELK Stack

Security

Performance

Scalability

Backup and Recovery

Use Cases in Blockchain Development

The ELK stack is highly relevant for blockchain development, providing essential monitoring capabilities for decentralized systems.

For example, a Logstash pipeline can collect logs from an Ethereum node's log file, parse transaction details, and visualize them in Kibana to monitor gas usage trends and network performance.

Challenges and Considerations

Future Trends (2025)

As of July 2025, the ELK stack continues to evolve with new features and capabilities:

Key Takeaways

Next Steps

Ready to implement the ELK stack? Here's your learning path:

Further Reading