MQTT data not received

[bandi22]

Issue:

No MQTT data is received in Grafana via TCP, despite the connection test being successful ("MQTT Connected") and the Grafana client showing up as 'Connected' in EMQX.

Messages are broadcasted in the broker:

"Explore data" option and dashboards show no data, time range is set to <1 minute. Subscription and the topic name is set correctly in the queries:

Explore data:

Dashboard:

Grafana logs show no apparent errors.

Environment:

• MQTT Datasource v1.1.0-beta.1 (installed from archive)
• Grafana v11.1.0 (Docker container)
• EMQX 5.7.1 (Docker container)
• MQTT protocol used: MQTT v3.1.1
• Connecting to MQTT via tcp/1883, no TLS, password only authentication
• Docker host is running on Oracle Linux 8.10

[jtowe1]

I am having the same problem

[nosovk]

The same problem :( I see connection in EMQX dashboard, but it seems that there is no subscription from Grafana.

[ArchiZ0]

I always got issue with this MQTT plugin and the "Explore data" feature. Try creating a dashboard and add a panel using your MQTT connection. Then enter the topic and switch to table view and you should see your data. From there, you will need to play with the visualization you want for a state change representation.

[nosovk]

We ended by running container with mosquito, that collects required topics and sends them to loki. Works like a charm.

[bandi22]

@ArchiZ0
Sadly, no luck. Table view shows no data either.

[ArchiZ0]

@bandi22 Sorry to hear that...

What is your use-case? How often data are sent in the topic? This MQTT plugin is for streaming data only as far as I know. There's no storage of any sort and small change like modifying the time range will reset the content.

We had to use a trick on the collector side to always keep a small buffer of data and when connection to topic are detected, we start by dumping that buffered data in the topic and then continue the live streaming.

In my case, we end up using this plugin as a real-time view for equipment metrics and we use it as a live debugging tools. I use variables to select an equipment dynamically and connect to the related topics.

For periodic / more historical oriented use-cases, we use a different datasource.

[nosovk]

Actually it's not very clear from documentation, that there is no storage inside. We moved to mosquito + Loki after realizing that.

[ArchiZ0]

@nosovk Well, is there any "client-side local storage" at all to any data sources in Grafana?.

I'm no Grafana expert but with all the data sources we've used so far, when you reload the page or change the time range, the data is re-read from the data source, like in your case, Loki... but it seems to be the case with others too, like InfluxDB or BigQuery, which are other data sources we're using.

What's different with MQTT is once the data is read by the client from a topic, the data is gone, by design... Even with the broker feature "retained messages", only the last one received in a specific topic is persisted at any time:

• EMQX : https://docs.emqx.com/en/emqx/latest/messaging/mqtt-retained-message.html
• HiveMQ : https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages
• etc.

Reconnecting to the topic will simply read the last one and any subsequent data published.

[nosovk]

Yep, you have correctly described all. The main point is that users mostly want to connect their house temperature measurement to grafana. And its a bit unclear that it wouldn't work with that datasource. That's why there are so manu issues where people complain that something is not working, but its just not intended to work like they think.

[bandi22]

@ArchiZ0

Data is sent every few seconds or so from a machine, so I guess we could say it is live data, not historical. When I'm subscribed to the same topic on the broker with an MQTTX client, I can see the messages appearing, one after the other, live, but no data is shown in Grafana.

[ArchiZ0]

@bandi22 can you share the data you sent to the topic again please? This is very odd you don't see anything.

[wavelov3r]

so, i've read all the topic. My scope is to have a fixed value to multiply some data with it. No need for graphs or any other cool representation.
I've installed the plugin, and correctly configured it (as it said). I am using MOSQUITTO, and publish the mqtt topic (retain ON) with an homeassistant automation.
In MQTT explorer it's shown, it does not work in grafana. i've tried to fire an update every 5 seconds (manually), and still does not work.

logger=cleanup t=2024-09-03T17:17:00.664211895+02:00 level=info msg="Completed cleanup jobs" duration=4.937017ms
logger=plugins.update.checker t=2024-09-03T17:17:02.131359573+02:00 level=info msg="Update check succeeded" duration=189.260171ms
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:13.159095224+02:00 level=info msg="MQTT Connecting"
logger=context userId=1 orgId=1 uname=admin t=2024-09-03T17:18:13.223930642+02:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdwp89k9t5vk0e/health status=400 remote_addr=10.1.1.4 time_ms=66 duration=66.892441ms size=94 referer=https://homeassistant.hass:8123/api/hassio_ingress/CWPu3tOxQqyjvp3N6rrJnI88Mql3Xcs1ObNOIvRIn80/connections/datasources/edit/fdwp89k9t5vk0e handler=/api/datasources/uid/:uid/health status_source=server
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:18.159859594+02:00 level=info msg="MQTT Disconnecting"
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:29.817031439+02:00 level=info msg="MQTT Connecting"
logger=context userId=1 orgId=1 uname=admin t=2024-09-03T17:18:29.881905105+02:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdwp89k9t5vk0e/health status=400 remote_addr=10.1.1.4 time_ms=66 duration=66.864805ms size=83 referer=https://homeassistant.hass:8123/api/hassio_ingress/CWPu3tOxQqyjvp3N6rrJnI88Mql3Xcs1ObNOIvRIn80/connections/datasources/edit/fdwp89k9t5vk0e handler=/api/datasources/uid/:uid/health status_source=server
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:34.661198983+02:00 level=info msg="MQTT Connecting"
logger=context userId=1 orgId=1 uname=admin t=2024-09-03T17:18:34.663235978+02:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdwp89k9t5vk0e/health status=400 remote_addr=10.1.1.4 time_ms=5 duration=5.139493ms size=83 referer=https://homeassistant.hass:8123/api/hassio_ingress/CWPu3tOxQqyjvp3N6rrJnI88Mql3Xcs1ObNOIvRIn80/connections/datasources/edit/fdwp89k9t5vk0e handler=/api/datasources/uid/:uid/health status_source=server
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:34.817789052+02:00 level=info msg="MQTT Disconnecting"
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:35.561030993+02:00 level=info msg="MQTT Connecting"
logger=context userId=1 orgId=1 uname=admin t=2024-09-03T17:18:35.638354422+02:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdwp89k9t5vk0e/health status=400 remote_addr=10.1.1.4 time_ms=79 duration=79.522796ms size=83 referer=https://homeassistant.hass:8123/api/hassio_ingress/CWPu3tOxQqyjvp3N6rrJnI88Mql3Xcs1ObNOIvRIn80/connections/datasources/edit/fdwp89k9t5vk0e handler=/api/datasources/uid/:uid/health status_source=server
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:39.664512656+02:00 level=info msg="MQTT Disconnecting"
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:40.561412548+02:00 level=info msg="MQTT Disconnecting"
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:42.169489788+02:00 level=info msg="MQTT Connecting"
logger=plugin.grafana-mqtt-datasource t=2024-09-03T17:18:47.173638179+02:00 level=info msg="MQTT Disconnecting"
logger=cleanup t=2024-09-03T17:27:00.664007824+02:00 level=info msg="Completed cleanup jobs" duration=5.135499ms
logger=plugins.update.checker t=2024-09-03T17:27:02.130258621+02:00 level=info msg="Update check succeeded" duration=188.226194ms
logger=context userId=1 orgId=1 uname=admin t=2024-09-03T17:32:34.274538091+02:00 level=error msg="Error writing to response" err="write tcp 127.0.0.1:3000->127.0.0.1:36060: write: broken pipe"

MQTT Explorer
https://github.com/user-attachments/assets/bef19e5a-e0b4-4f27-8d2a-d56bcc918030
Grafana
https://github.com/user-attachments/assets/1e638145-c4ac-4f17-a82a-20cc1caecd17
HAssistant
https://github.com/user-attachments/assets/3e34840a-4155-4ba2-b9fa-9e8878f030cd

[ArchiZ0]

@wavelov3r Sorry but what is the expectation here? MQTT is not about data persistence, this is not a time-series data store... it will only retain the last value by design, and when subscribed using that plugin, it will continue adding until you refresh the page... which I believe you understand.

Your data seems to be a simple float sent in a topic without structure (named field using JSON) or timestamp...

...Still, here's an example using a "simple" topic that dump a "simple" float every 3 second:

and here's the data in MQTTX client:
(https://github.com/user-attachments/assets/72da74b4-87c7-4231-8dcf-340034b571aa)

I'm using the community version of EMQX and did nothing special to the broker.

[wavelov3r]

@ArchiZ0 the expectation is clear, i wrote it. to have a (one) number for multiplications (it's a price per kW).
I don't need the history, i need only to update it when necessary (by republishing the topic).

In the documentation i read: This plugin automatically supports topics publishing numbers, strings, booleans, and JSON formatted values. , so i tought "raw numbers are ok".
Also, FYI
Retained Messages is an MQTT feature that allows storing the “last known good” message for a particular topic on the broker and delivering the message to a client whenever the client subscribes to a matching topic.
So they are ALWAYS available on MQTT, until you don't delete it.

Ending the message with other unsuccessful json tries:

MQTT EXPLORER
https://github.com/user-attachments/assets/38e47eee-be7b-4051-84e4-1bbffa290ed1
GRAFANA
https://github.com/user-attachments/assets/fc7af89d-0997-4451-9d62-3c3496d3bfbf

[teefixx]

same here, see screen shots.
Data in MQTT Explorer available, MQTT Plugin states connected and green check mark
Grafana panel with MQTT plugin: no data

Plugin version: 11.0 beta 2
MQTT broker: flashMQ version 1.11.0, release build without SSE support

MQTT Explorer
(https://github.com/user-attachments/assets/96e731db-06f1-4bf8-99d0-8e429437a631)

Grafana 11.1.4
(https://github.com/user-attachments/assets/0ff3286b-9909-4def-aeca-7a43c9a94c32)

Any hint would be appreciated.
thank you
teefixx

[ArchiZ0]

@wavelov3r My bad, I miss the part you want to display the retained message too.

I replicate your use-case and I believe I understand what's happening at least for the retained message... new messages in the topic always worked for me so I'll give you a better view of my setup.

Specifically about the retained message not displaying, I believe this has to do with the MQTT plugin optimizing / reusing existing connection to the broker and subscription to a topic vs. re-connecting and re-subscribing with a new client-id on each dashboard reload. It is my understanding a MQTT client will get the retained message only once when it subscribed to the topic... if its already subscribed, it will only receive the subsequent update when it arrive.

Here's my test to confirm:

• EMQX as broker
• MQTTX as client
• Grafana 11.2.0
• Grafana MQTT plugin version 1.1.0-beta.2

In MQTTX Client:

• I publish a retained message in the test topic (red box):
  (https://github.com/user-attachments/assets/e8deab20-45db-4877-b730-db20b3084526)

• I subscribed to the test topic and the EMQX system topic for the clients events $SYS/brokers/+/clients/#
  (same screenshot as above, green box)

In EMQX management dashboard:

• I confirmed that the topic test is created and is indeed in retained message topic list in EMQX:
  (https://github.com/user-attachments/assets/5dbfdafa-6887-4682-95e7-b6652eceeb41)

• and here's the initial retained value:
  (https://github.com/user-attachments/assets/22d38d37-6ff4-4bfd-8fa5-62ec8a11d660)

In Grafana:

• Simple dashboard with two panels:

  • a Table (left)
  • a Stat (right) that display the latest value
    (https://github.com/user-attachments/assets/c9801300-f8cc-40f2-bd77-6ca610f57616)

• Both panel share the same datasource config using the MQTT plugin and test as topic
  (https://github.com/user-attachments/assets/1343f45a-d63a-4e46-b51c-85f0303ca835)

• The only customisation I ajusted for the dashboard itself is:

  • Default time-range of 1 hour... but you'll need to adjust so it cover the time the retained message is refreshed in your topic
  • In settings "Refresh Live Dashboard" (https://github.com/user-attachments/assets/98bb2f19-498b-412f-a149-7ac5a9967091)

From there, I restarted Grafana to make sure I start fresh (no MQTT connection or subscription) then I got to the dashboard I just created.

In MQTTX client, I clearly see the Grafana MQTT plugin connecting and the subscribing to the test topic:
(https://github.com/user-attachments/assets/ee18be0d-86b0-4c9e-b086-7010ebf612f1)

...and after ~4 or 5 seconds, the retained message appear in both panels:
(https://github.com/user-attachments/assets/44ca4f29-5f22-4ae6-967e-f927da6ce0fc)

Using the MQTTX client, I published a new message with the number 12 as example... the panels display the new value
(https://github.com/user-attachments/assets/b3d2db17-c601-4fe5-b1ef-87979d9f8b40)

So far, all normal behavior... but now the non-intuitive behavior start:

If you refresh the Grafana dashboard in the browser (normal or forced refresh), the panels will show no data. Looking at the client events in the MQTTX client, I clearly see NO new subscription... which mean the MQTT plugin retain the current subscription to test and therefore, it won't re-read the last retained message from the topic. This is normal behavior from a MQTT standpoint but not the expected Grafana behavior... which seems to be your problem?

From there, if you leave the Grafana dashboard, you'll eventually see the plugin detecting nobody is looking at that dashboard anymore and unsubscribed from the topic.
(https://github.com/user-attachments/assets/df5b0199-a5e9-4843-84ba-579bca75de87)

I do see sometime, when you do something like changing the time range, the plugin seems to unsubscribed and subscribed again immediately... in that case, the retained message will show after 4-5 seconds.

But, from time to time, normally after quick test back to back, the retained message is not displayed... which can probably be a bug but I don't see any log that can confirm this.

In any case, this is probably more a behavior that doesn't match the expectation. We can probably asked the contributor of this plugin to add an option to use different client-id to subscribed to topic on all event in the dashboard, similar to the variable refresh on dashboard reload.

Let me know if you're able to replicate my test. Hope this help.

[javiBajoCero]

same issue here , topics are being updated (checked with mosquitto_sub) and graphana says connected to mosquitto broker no problem.

But the queries are not receiving a n y t h i ng, dry

[ncoghlan]

Similar problem here - there is something very strange going on with the topic mapping between the "query" layer and the MQTT layer.

I have a broker which is receiving messages on 5 topics:

• /sensors/temp/C
• /sensors/temp/F
• /sensors/hidx/C
• /sensors/hidx/F
• /sensors/humidity

If I set the query topic as #, then all 5 signals show up as values in the exploration query view (I'm monitoring them separately via mosquitto_sub, so I can identify them by value).

As soon as I try to filter the query down to a specific topic (with or without the leading slash), I stop getting any data at all.

[javiBajoCero]

this is very annoying i think there is an issue with / started topic names

…

On Mon, 2 Dec 2024, 09:10 Alyssa Coghlan, ***@***.***> wrote: Similar problem here - there is something *very* strange going on with the topic mapping between the "query" layer and the MQTT layer. I have a broker which is receiving messages on 5 topics: - /sensors/temp/C - /sensors/temp/F - /sensors/hidx/C - /sensors/hidx/F - /sensors/humidity If I set the query topic as #, then all 5 signals show up as values in the exploration query view (I'm monitoring them separately via mosquitto_sub, so I can identify them by value). As soon as I try to filter the query down to a specific topic (with or without the leading slash), I stop getting any data at all. — Reply to this email directly, view it on GitHub <#107 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGD36N5NAXACG7L5KI4KEEL2DQIXZAVCNFSM6AAAAABMANN4D6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQHA2DAOJUGU> . You are receiving this because you commented.Message ID: ***@***.***>

[Weicocu]

I'm just popping in to say that I'm experiencing the same issue originally being discussed here -- the MQTT plugin connects and gets a green checkmark at the "save & test" stage but then does not receive any data under any circumstances, even when I confirm that data is being sent (e.g. using MQTT Explorer). The issue isn't about data persistence -- nothing ever shows up at all. I've tried specific topics where I knew data was coming in, and I've also tried the wildcard topic # and still got nothing.

Regarding topic names starting with a slash, they are discouraged in MQTT (see here ) but they ought to work. But that's not the issue I'm having.

• Mosquitto 2.0.18
• Grafana 11.4.0
• MQTT plugin 1.1.0-beta.2