Architecting Persistent Time-Series Data Pipelines from Homematic CCU to Grafana

The architectural challenge of implementing long-term trending and data visualization within a smart home ecosystem necessitates a profound distinction between real-time monitoring and historical archiving. In industrial automation, specifically within the realm of Siemens PLC programming, the ability to visualize process values over extended durations—ranging from days and weeks to months or even years—is critical for evaluating controller optimization and assessing environmental process conditions. For the home automation enthusiast, this requirement translates into the need for a system that can track variables such as underfloor heating flow/return temperatures, boiler states, and radiator valve positions over vast timescales. While lightweight solutions like the Node-RED ui_chart function provide excellent short-term trending, they lack the fundamental persistence required to survive system reboots or hardware restarts. Achieving true "long-term trending" requires a decoupled architecture where data is not merely captured but is written to a persistent, queryable storage engine capable of managing large datasets through compression and scheduled maintenance.

The Fundamentals of Historical Data Persistence

True historical logging is defined by its ability to maintain data integrity across system power cycles. In a robust implementation, the data must survive any kind of reboot or restart of the primary processing unit, such as a Raspberry Pi running Node-RED. A significant hurdle in DIY automation is the confusion between logging, archiving, and trending. Logging refers to the act of recording a value at a specific timestamp, while archiving involves the long-term storage and periodic maintenance (deletion or compression) of these records to prevent storage exhaustion. Trending is the subsequent visualization of these archived values as curves on a graph.

The Homematic CCU ecosystem offers a native approach via the CCU-Historian add-on. This specific implementation utilizes a backend structure that functions effectively as a database, though it may physically manifest as structured text files. The primary advantages of this native approach include:

• Persistence across reboots: The data remains intact regardless of the state of the CC/CCU software or the underlying hardware, provided the storage medium remains uncorrupted.
• Dynamic Chart Generation: Users can create dynamic graphs that represent various process conditions.
• URL-based State Recovery: The configuration of specific graphs, including the selected curves and the time ranges for trends, can be encapsulated within a URL string. This allows for the creation of bookmarks or dashboards that instantly recall specific visualization parameters.
• Managed Lifecycle: The system allows for the maintenance of the "database" through manual or automated deletion and compression of old records, which is essential for preventing the degradation of performance on SD-card-based systems like the Raspberry Pi.

For users migrating away from proprietary add-ons toward open-source stacks, the challenge shifts to replicating this persistence using external tools like Telegraf and Grafana.

Data Ingestion via MQTT and the JSON Payload Structure

Transitioning from a closed system like CCU-Historian to a modern observability stack requires a reliable data ingestion pipeline. A common method involves utilizing the MQTT protocol to broadcast state changes from the Homematic CCU to a broker, which is then consumed by a processing agent. When observing the data stream through a subscriber, such as mosquitto_sub, the payload typically arrives as a complex JSON object.

The structure of this JSON payload is vital for downstream parsing. A single message represents a snapshot of a device's state, containing not only the current value but also high-precision timestamps and metadata. Consider the following structure of a captured MQTT message:

json {"val":1,"ts":1602431336562,"lc":1602430409000,"hm":{"ccu":"localhost","iface":"BidCos-RF","device":"PEQ1311478","deviceName":"HT_Gang","deviceType":"HM-CC-RT-DN","channel":"PEQ1311478:4","channelName":"HM-CC-RT-DN PEQ1311478:4","channelType":"CLIMATECONTROL_RT_TRANSCEIVER","channelIndex":4,"datapoint":"CONTROL_MODE","datapointName":"BidCos-RF.PEQ1311478:4.CONTROL_MODE","datapointType":"ENUM","datapointMin":0,"datapointMax":3,"datapointDefault":0,"datapointControl":"HEATING_CONTROL.CONTROL_MODE","valuePrevious":1,"valueStable":1,"rooms":["Gang"],"room":"Gang","functions":["Heizung","Homekit"],"function":"HT_Gang","ts":1602431336562,"tsPrevious":1602431161293,"lc":1602430409000,"change":false,"cache":false,"uncertain":false,"stable":true}}

To understand the implications of this data for a visualization engineer, we must dissect the individual components:

• val: The primary measurement value (e.g., temperature or state).
• ts: The epoch timestamp in milliseconds, representing the exact moment of the event.
• lc: The Last Change timestamp, which is crucial for calculating the duration of a state.
• hm object: A nested metadata object containing device identifiers (device), interface information (iface), and human-readable names (deviceName).
• datapoint: The specific attribute being reported, such as CONTROL_MODE.
• stable: A boolean flag indicating if the value has remained unchanged for a sufficient period.

This level of detail allows for advanced multidimensional analysis, such as correlating the "on/off" status of a boiler with the rising temperature in a specific room. However, the complexity of this nested JSON structure presents significant hurdles for automated ingestion engines.

Troubleshooting Telegraf MQTT Consumer Errors

The most common failure point in the pipeline between the Homematic CCU and Grafana is the Telegraf mqtt_consumer plugin. Telegraf acts as the intermediary, subscribing to the MQTT broker and attempting to parse the JSON payloads into a format suitable for time-series databases (like InfluxDB). When the parser encounters a structure it does not expect, it triggers a metric parse error.

A frequent error log entry in the Telegraf service looks like this:

text Okt 11 17:50:17 hqmain telegraf[8843]: 2020-10-11T15:50:17Z E! [inputs.mqtt_consumer] Error in plugin: metric parse error: expected tag at 1:29: "{\"val\":23,\"ts\":1602431417182...

The "expected tag" error indicates that the Telegraf configuration is failing to correctly map the JSON keys to InfluxDB tags or fields. In the context of the Homematic JSON, the parser is stumbling over the deeply nested hm object. Because the payload contains a hierarchy of information (e.g., hm.device, hm.deviceName), the mqtt_consumer must be explicitly configured to flatten this structure or specifically target the desired paths. If the parser expects a flat tag-value pair but encounters the start of a nested object at a specific character position (e.g., 1:29), the metric is dropped, leading to gaps in the historical timeline.

The impact of these parsing errors is catastrophic for long-term trending. If the parser fails to process messages containing vital state changes—such as a radiator valve moving from 0% to 100%—the resulting Grafana graphs will show "ghost" data or missing segments, rendering the entire historical analysis unreliable for process optimization.

Optimizing Grafana for Large-Scale Visualizations

Once data is successfully ingested and stored, the final stage is visualization in Grafana. While Grafana is a world-class tool for observability, it faces physical limitations when dealing with extremely high-density datasets. A common issue arises when attempting to use panels like the Status History panel or the Time Series panel over very long time ranges (months or years).

The core problem is the "Too many points to visualize" error. This occurs when the number of data points returned by the underlying query exceeds the rendering capacity of the browser or the configured resolution of the panel. If a query attempts to pull every single second of data for a three-month period, the browser will likely crash or the visualization will become an unreadable blur of pixels.

To resolve this, one must move away from the "magic" v.windowperiod approach and implement controlled aggregation. The following strategies are essential for professional-grade dashboards:

• Custom Interval Variables: Instead of relying on the automatic window period, create a custom dashboard variable (e.s., a dropdown menu) that allows the user to select a specific aggregation step.
• Fixed Step Count: Configure the query to return a deterministic number of points. For instance, setting a variable so that Grafana always attempts to create an aggregation window that returns exactly 30 values for the selected time period.
• Aggregation Functions: Use SQL or Flux/InfluxQL functions (such as aggregateWindow or GROUP BY time(1h)) to downsample the data at the database level before it ever reaches the Grafana frontend.
• Auto-Value Variables: Implement an "auto" value in your custom variable that can be hidden from the end-user but used within the query logic to dynamically adjust the granularity of the data based on the selected time range.

By implementing these optimization techniques, the transition from the simple, localized CCU-Historian to a high-performance, distributed Grafana stack becomes possible, enabling true industrial-grade monitoring of the smart home environment.

Comparative Analysis of Data Management Strategies

The following table compares the different approaches to handling home automation trends, highlighting the trade-offs in complexity and capability.

Analysis of Architectural Requirements

The transition from a localized, single-app logging system to a decoupled, multi-component observability stack represents a significant leap in technical maturity. For the user, the implications are twofold. On one hand, the move to a Grafana-centric architecture introduces a "moving parts" problem, where the health of the MQTT broker, the Telegraf agent, and the time-series database must all be monitored simultaneously. A failure in the mqtt_consumer plugin, as evidenced by the parsing errors, can silently degrade the historical record, making the system's "truth" suspect.

On the other hand, the benefits of this architecture are unparalleled. The ability to perform complex correlations—such as comparing outdoor ambient temperature against the internal heating valve positions over a six-month period—is only possible when using a system designed for high-cardinality data. The key to success lies in the rigorous management of data ingestion (ensuring JSON payloads are correctly flattened) and the intelligent use of aggregation variables in Grafana to prevent browser-side computational exhaustion. Ultimately, the goal is to replicate the persistence and reliability of the CCU-Historian while gaining the analytical depth of a professional DevOps monitoring stack.

Sources

1. Node-RED Discourse - Long-term trending and visualization
2. Raspberry Pi Forum - Homematic to Grafana via Telegraf
3. Grafana Community - Too many points to visualize properly