Architecting Advanced Cyber Threat Intelligence Observability with CrowdSec, VictoriaMetrics, and Grafana

The implementation of a robust security monitoring stack is a foundational requirement for any modern infrastructure, particularly when managing distributed environments such as Kubernetes clusters or large-scale Docker deployments. While CrowdSec provides an unparalleled engine for detecting and remediating malicious activity through its community-driven threat intelligence and local API, the raw visibility provided by its default Prometheus HTTP endpoint is often insufficient for deep forensic analysis or long-term trend identification. The default metrics available through the CrowdSec Local API (LAPI) provide a high-level summary of specific API routes, access methods, and request volumes, which are critical for understanding load distribution across machines handling requests and evaluating the performance of Remediation Components (bouncers). However, these metrics are ephemeral; running the cscli metrics command only reveals data accumulated since the Security Engine's last restart. For Security Operations Center (SOC) professionals, this lack of persistence represents a significant gap in historical auditing and incident reconstruction.

To bridge this gap, a sophisticated observability pipeline must be constructed using VictoriaMetrics as a high-performance, long-term storage broker and Grafana as the visualization layer. This architecture moves beyond simple metric monitoring into the realm of true Cyber Threat Intelligence (CTI) dashboards. By leveraging VictoriaMetrics, organizations can ingest JSON-formatted alert data from CrowdSec via a specialized notification handler, transforming transient security events into persistent, queryable time-series data. This allows for the creation of interactive world maps, detailed IP reputation lookups via external services like Shodan and Censys, and a comprehensive overview of attack scenarios, blocked pods, and banned IP addresses. The resulting ecosystem provides not just a view of what is happening now, but a historical ledger of security decisions, enabling a proactive stance against evolving digital threats.

The Architecture of Persistent Threat Intelligence

A functional CTI pipeline relies on the seamless integration of three distinct technological layers: the detection engine (CrowdSec), the data broker and storage engine (VictoriaMetrics), and the visualization frontend (Grafanam). In a standard deployment, CrowdSec acts as the sensor and decision-maker, identifying patterns of malicious behavior and triggering bouncers to block traffic. However, to achieve the level of detail required for advanced threat hunting, the data must be exported to a medium that supports high-cardinality data and long-term retention.

VictoriaMetrics serves as the critical intermediary in this architecture. Unlike the standard Prometheus-compatible endpoint provided by CrowdSec—which is limited in the depth of detail it can provide compared to the proprietary CrowdSec Console or specific cscli command outputs—VictoriaMetrics is designed to act as a data broker. It receives JSON alerts from the CrowdSec instance and makes these metrics available via a Prometheus-compatible API. This allows Grafana to query complex datasets that include not just simple counters, but detailed attributes of each security event.

The infrastructure configuration requires that VictoriaMetrics resides within the same Docker network as the existing Grafana and CrowdSec instances. This is a critical security design pattern; by utilizing a shared network, such as crowdsec_grafana, the VictoriaMetrics service can remain completely isolated from the underlying host server. No ports from the VictoriaMetrics container need to be mapped or exposed to the host machine, significantly reducing the attack surface of the monitoring stack itself.

Deploying VictoriaMetrics via Docker Compose

The deployment of the VictoriaMetrics service must be handled with precision to ensure it can receive the incoming feed from CrowdSec. The process involves defining a service in a Docker Compose configuration that points to the correct image and establishes the necessary volume mounts for data persistence.

The following configuration represents a production-ready implementation for spawning a VictoriaMetrics container:

```yaml
services:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.138.0
expose:
- 8428 # VM UI
volumes:
- vmdata:/storage
command:
- "--storageDataPath=/storage"
- "--httpListenAddr=:8428"
restart: always
networks:
- crowdsec_grafana

networks:
crowdsec_grafana:
external: true

volumes:
vmdata: {}
```

In this configuration, several technical decisions are made to ensure stability and security:

The use of expose rather than ports ensures that the VictoriaMetrics UI and API are only reachable by other containers on the crowdsec_grafana network.
The restart: always policy ensures that the monitoring pipeline recovers automatically from system reboots or container crashes.
The command flags explicitly define the storage path and the HTTP listening address, ensuring the service operates predictably within the containerized environment.
The external: true designation for the network ensures that the new service attaches to the existing, pre-established security network rather than creating a new, isolated one.

Once the configuration is finalized, the service can be initialized using the following command:

bash docker compose up -d

After the container is running, it will wait at the http://victoriametrics:8428/api/v1/import URL for the incoming alert feed from the CrowdSec nodes.

Configuring CrowdSec for Alert Ingestion

Simply running VictoriaMetrics is insufficient; the CrowdSec instance must be explicitly instructed to forward its security decisions to the new storage endpoint. This requires the creation of a new notification handler and the modification of the existing profiles configuration. The exact file paths for these configuration files depend heavily on whether the deployment is running as a Docker container or as a baremetal service.

For users running CrowdSec within a Docker environment, the configuration files are typically found within the volume bind mounts defined in the original CrowdSec Docker Compose project. The paths are generally structured as follows:

/mnt/docker-uploads/crowdsec/etc/notifications/victoriametrics.yaml
/mnt/docker-uploads/crowdsec/etc/profiles.yaml

For baremetal installations, the paths follow the standard Linux filesystem hierarchy:

/etc/crowdsec/notifications/victoriametrics.yaml
/etc/crowdsec/profiles.yaml

The first critical step is the creation of the victoriametrics.yaml file within the CrowdSec notifications directory. This file defines the destination for the alerts. The content must be configured to point to the VictoriaMetrics service name within the Docker network.

The second step involves editing profiles.yaml. This file acts as the logic engine for CrowdSec, determining which events trigger which actions. To implement the new observability layer, the victoriametrics notification handler must be added to the active profiles. This ensures that every time a security decision is made (such as a ban or a block), the event is not only processed by the bouncer but also transmitted to the VictoriaMetrics import API.

Dashboard Integration and Visualization

Once the data pipeline is flowing—meaning CrowdSec is sending alerts to VictoriaMetrics and VictoriaMetrics is storing them—the final stage is the implementation of the Grafana visualization layer. This process involves importing a pre-configured dashboard that is specifically designed to parse the schema of the data being sent to VictoriaMetrics.

The implementation steps for the Crowdsec Cyber Threat Intelligence dashboard are as quite precise:

Log into the Grafana instance using administrative credentials.
Navigate to the dashboards management area via the /dashboards URL.
Select the "New" option and then click on "Import".
Enter the dashboard ID 21689 (sourced from GrafanaLabs) and click "Load".
Provide a descriptive name for the dashboard, such as "Crowdsec Cyber Threat Insight".
Select the victoriametrics data source that was previously configured in the Grafana environment.
Click "Import" to finalize the deployment.

The resulting dashboard provides an exhaustive view of the security landscape. It is not merely a collection of graphs but a multi-layered analytical tool. The dashboard capabilities include:

Block Total for Pods: This metric tracks the number of blocked attempts on a per-pod basis. In a Kubernetes context, this is vital for identifying which specific microservices are being targeted by automated scanners or exploit attempts.
Attacks by Scenarios: This visualization displays the distribution of attacks based on CrowdSec detection scenarios. It allows security engineers to see if the infrastructure is facing a surge in SSH brute-force attempts, web application exploits, or specific protocol-level attacks.
Total Ban by IP: This section lists the specific IP addresses that have been blacklisted, paired with the scenario that triggered the ban and the total frequency of occurrences. This enables rapid identification of persistent malicious actors.
Decisions Log: This provides the highest level of granularity, capturing detailed logs with precise timestamps and request data. This transparency is essential for post-incident forensics and auditing.
External Intelligence Integration: The dashboard can be configured to include direct links to external Cyber Threat Intelligence (CTI) services. By clicking on an IP in the dashboard, an analyst can immediately launch a lookup in Shodan, Censys, or other IP reputation databases to gather more context on the attacker.

Advanced Observability and Ecosystem Convergence

The true power of this configuration is realized when the CrowdSec dashboard is integrated into a broader, unified observability strategy. An advanced setup does not view security in a vacuum; rather, it correlates security events with system performance and application logs.

Expert-level implementations often combine the CrowdSec CTI dashboard with other specialized monitoring tools to create a "Single Pane of Glass." A comprehensive homelab or enterprise dashboard might simultaneously monitor:

Proxmox or Ubuntu VM resources (CPU, memory, disk usage, and uptime).
Docker container health (resource consumption, image counts, and restart loops).
Application and System Logs: Utilizing the Grafana Loki stack (Promtail and Loki) to parse Traefik or OS-level logs (HTTP, SSH, etc.).
Security Intelligence: The CrowdSec Threat Intelligence feed, providing the context of which of the aforementioned logs are actually part of an ongoing attack.

By correlating a spike in CPU usage on a Docker container with a simultaneous surge in "Attacks by Scenario" in the CrowdSec dashboard, an administrator can instantly determine if a resource exhaustion issue is the result of a Denial of Service (DoS) attack or a legitimate workload increase.

Analytical Conclusion

The transition from reactive security to proactive threat intelligence requires a fundamental shift in how telemetry is handled. The standard approach of monitoring "up/down" status or simple error counts is insufficient for defending against modern, automated adversaries. By constructing a pipeline that utilizes CrowdSec for detection, VictoriaMetrics for high-performance persistence, and Grafana for multi-dimensional visualization, organizations move from a state of ignorance to a state of informed defense.

The architecture described herein solves the critical problem of metric volatility. By moving away from the ephemeral metrics of the Local API and toward a structured, long-term storage model, security teams gain the ability to perform longitudinal studies of threat patterns. This allows for the identification of slow-moving, low-frequency attacks that might otherwise bypass threshold-based detection. Furthermore, the integration of external CTI services directly into the visualization workflow minimizes the "time-to-context," enabling analysts to move from detection to investigation in seconds. Ultimately, this ecosystem creates a dense web of information where every blocked packet, every banned IP, and every anomalous system metric contributes to a singular, coherent understanding of the infrastructure's security posture.