The Elastic Stack, commonly referred to as ELK (Elasticsearch, Logstash, Kibana), has established itself as the definitive standard for centralized log management and data visualization. Among its components, Kibana serves as the primary interface for interacting with the data stored in Elasticsearch, offering sophisticated tools for monitoring logs, application performance, and operational intelligence. While the platform is inherently intuitive, unlocking its full potential requires a methodical approach to data ingestion, index pattern configuration, and visualization design. Effective dashboarding is not merely about displaying data; it is about structuring information to reveal anomalies, track key metrics, and provide actionable insights. This article explores the technical procedures for setting up a local ELK environment, configuring application logging via Serilog in a .NET Web API, and constructing specialized Kibana dashboards for web server traffic, database operations, and server uptime.
Environment Setup and Data Ingestion
Before constructing visualizations, a functional data pipeline must be established. This involves deploying the Elasticsearch and Kibana services and configuring an application to emit structured logs that Elasticsearch can index. For development and testing environments, Docker provides a streamlined method for spinning up these services without complex system-level installations.
To create a local ELK instance, a docker-compose.yml file is utilized to define the services. This configuration specifies the use of Elasticsearch version 7.12.0 with security disabled (xpack.security.enabled=false) and single-node discovery (discovery.type=single-node). Kibana is configured to connect to the Elasticsearch container via a shared network (es-net) and exposes port 5601 for web access, while Elasticsearch exposes port 9200.
yaml
version: "3.0"
services:
elasticsearch:
container_name: es-container
image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
environment:
- xpack.security.enabled=false
- "discovery.type=single-node"
networks:
- es-net
ports:
- 9200:9200
kibana:
container_name: kb-container
image: docker.elastic.co/kibana/kibana:7.12.0
environment:
- ELASTICSEARCH_HOSTS=http://es-container:9200
networks:
- es-net
depends_on:
- elasticsearch
ports:
- 5601:5601
networks:
es-net:
driver: bridge
Executing the command docker-compose up initiates the containers, making Elasticsearch accessible at http://localhost:9200/ and Kibana at http://localhost:5601/. With the infrastructure in place, the next step is to generate log data. In a .NET Web API context, this is achieved by integrating Serilog, a popular logging library. The Program.cs file is modified to include the UseSerilog extension method, allowing the application to read logging configuration from the appsettings.json file.
csharp
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
})
.UseSerilog((hostingContext, loggerConfiguration) =>
loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration));
The default Microsoft logging configuration in appsettings.json is replaced with a Serilog configuration that routes logs to Elasticsearch. This configuration sets the minimum log level, enriches logs with context, and defines the Elasticsearch sink. The indexFormat is set to demo-api-{0:yyyy.MM} to create time-based indices, and autoRegisterTemplate is enabled to ensure the schema is automatically registered with Elasticsearch version 7 compatibility.
json
"Serilog": {
"MinimumLevel": {
"Default": "Debug",
"Override": {
"Microsoft": "Information",
"System": "Information"
}
},
"Enrich": [ "FromLogContext" ],
"WriteTo": [
{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9200",
"indexFormat": "demo-api-{0:yyyy.MM}",
"autoRegisterTemplate": true,
"autoRegisterTemplateVersion": "ESv7"
}
}
]
}
In the Startup class, the Configure method is updated to include app.UseSerilogRequestLogging(), which captures HTTP request and response details. After running the API and executing several calls via Swagger, logs are generated and stored in Elasticsearch.
Index Pattern Creation and Visualization Foundations
Once logs are ingested, Kibana must be informed of the data structure. This is done by creating an index pattern. In the Kibana interface, navigating to the index pattern creation screen allows the user to define the pattern for the stored data. For the Serilog-generated logs, the pattern demo-api-* is entered. Clicking Next validates the pattern and maps the fields available for visualization, such as timestamp, level, and message.
Creating a dashboard requires building individual visualizations first. Kibana offers various visualization types, each suited for specific data analysis tasks. The choice of visualization depends on the nature of the data and the insights sought. For instance, area charts are ideal for time series data with split lines, while heat maps visualize the magnitude of phenomena through color intensity. Pie charts represent parts of a whole, and bar charts compare discrete fields. A critical principle in dashboard design is simplicity; overwhelming users with too many visualization types can obscure the core metrics.
To create a visualization in Kibana, users navigate to the Analytics menu, select Visualize library, and click Create new visualization. While Lens is often the quickest way to create visualizers, understanding aggregation-based visualizations provides greater control and is easier to replicate for multiple panels.
Constructing Application Log Visualizations
For application logs, such as those from the demo-api index, key metrics often include the total volume of logs and the distribution of log levels (e.g., Info, Warning, Error).
To create a Total Logs metric visualizer:
- Select the Metric visualizer type.
- Choose the demo-api-* index pattern.
- Save the visualization as Total logs.
To create a Log Levels Count visualizer:
- Create a new aggregation-based visualizer.
- In the Buckets section, click Add and select Split group.
- From the Aggregation dropdown, select Terms.
- In the Field dropdown, select level.raw.
- Click Update to apply the aggregation.
- Save the visualization as Log levels count.
To create a Log Levels Percentage visualizer:
- Create a new visualizer and choose the Pie type.
- In the Buckets section, click Add and select Split slices.
- In the Aggregation dropdown, select Terms.
- In the Field dropdown, select level.raw.
- Click Update.
- Save the visualization as Log levels percentage.
- Users can further refine this by clicking Options to adjust display properties.
These visualizations can be combined into a single dashboard to provide a quick health check of the application, showing overall activity and error distribution.
Analyzing Web Server Traffic
For web server monitoring, such as Nginx reverse proxy logs or general website traffic, the focus shifts to user behavior, traffic sources, and performance metrics. Kibana provides sample web logs data that can be used to learn these techniques, which are applicable to any web log data source.
To set up a web traffic dashboard:
- Install the web logs sample data set if not already available.
- Navigate to Dashboards and click Create dashboard.
- Set the time filter to Last 90 days to capture a sufficient historical window.
Key fields for web log analysis include records, timestamp, bytes, clientip, and referer. Understanding the distribution of these fields is crucial. For example, clicking on clientip reveals top values and distributions, which can help identify unique visitors.
To visualize data transfer over time:
- Select Add > Visualization from the application menu.
- Click Create visualization in the dashboard toolbar.
- Ensure the Kibana Sample Data Logs data view is selected.
- Drag the bytes field to the workspace. This automatically creates a bar chart showing the median bytes transferred over time.
- To change the visualization type to a line chart for better trend analysis, open the Layer visualization type menu in the layer pane and select Line.
- To adjust the granularity, click timestamp in the layer pane and change the Minimum interval to 1d (one day). Note that intervals cannot be decreased below the configured Advanced Settings.
- To conserve dashboard space, axis labels can be hidden by opening the Left axis and Bottom axis menus and selecting None from the Axis title dropdown.
- Save the visualization and add a panel title, such as Median of bytes, via the Settings flyout.
To analyze traffic sources and popular content:
- Create a visualization that displays the most frequent values of request.keyword ranked by unique visitors. This helps identify the most popular endpoints and the volume of traffic they receive.
- Horizontal bar charts are effective for visualizing relationships between two fields, such as social media platform referrers and specific web pages.
Advanced Dashboard Scenarios: Nginx, PostgreSQL, and Uptime
Beyond basic web traffic, specialized dashboards provide deeper operational insights.
Nginx Reverse Proxy Dashboard
For a corporate website using Nginx as a reverse proxy, a comprehensive dashboard might include:
- Heat Map: Breaks down activity by hour of the day, revealing peak usage times.
- Geo-IP Mapping: Visualizes the geographic distribution of requests and bytes transferred.
- Geo-IP Source and Destination: Provides a detailed view of where traffic originates and where it is being routed.
PostgreSQL Database Dashboard
For database monitoring, the dashboard focuses on log volume and severity:
- Total Log Type Count: Aggregates the number of logs by specific database event types.
- Logs by Level Over Time: Tracks the frequency of Information, Warning, and Error logs over a period to detect degrading performance or failures.
- Recent Logs List: A data table providing a raw view of the most recent log entries for immediate troubleshooting.
Server Uptime Dashboard
Availability monitoring is critical for infrastructure health. An ICMP-based uptime dashboard displays the availability status of multiple servers, often using simple metric panels or status indicators to show which servers are responding and which are down.
Best Practices and Visualization Selection
The effectiveness of a Kibana dashboard hinges on the appropriate selection of visualization types. While Kibana offers a vast array of options, choosing the wrong chart can obscure data trends. The following table outlines common visualization types and their optimal use cases based on the provided reference materials.
| Visualization Type | Function | Example Use Case |
|---|---|---|
| Area Chart | Visualizes time series data with split lines | Sign-ups over time |
| Horizontal Bar Chart | Visualizes relationships between two fields | Social media referrer vs. web page |
| Line Chart | Visualizes time series data; highlights anomalies | Memory usage over time by server |
| Pie Chart | Visualizes parts of a total figure | Top 10 referral websites |
| Heat Map | Visualizes magnitude of a phenomenon via color | Clicks on a web page |
| Vertical Bar Chart | Compares one or more fields | Users on page over time |
| Data Table | Creates a basic table representing fields | Sign-ups over time |
When creating visualizations, it is advisable to start with aggregation-based methods for precision, even though Lens offers a quicker alternative. For time-series data, adjusting the minimum time interval ensures that the graph is readable and not cluttered with excessive data points. Hiding axis labels can save space, but panel titles must be clearly defined to maintain context.
Conclusion
Constructing effective Kibana dashboards requires a blend of technical setup and analytical design. By establishing a robust logging pipeline with tools like Serilog and Docker, administrators ensure that high-quality data is fed into Elasticsearch. From there, the creation of index patterns and the strategic selection of visualizations—whether metrics for total logs, pie charts for distribution, or line charts for trends—enable the transformation of raw logs into actionable operational intelligence. Whether monitoring web server traffic, database health, or server uptime, the principles of simplicity, clarity, and purposeful visualization remain constant. Mastering these techniques allows teams to move beyond basic monitoring to proactive infrastructure management and detailed application performance analysis.