1

I am trying to set up a Grafana dashboard, pulling information from Prometheus, that is scraping data from a Kubernetes cluster. I'm running into issues that seem related to dashboard level variables.

Dashboard variable query syntax

There is very little information regarding how to build queries to populate the available values for a given dashboard variable. I've based what I've done so far on the few examples I've found in other dashboards. While this is helpful, it gives a very limited view into how this works. As an example, there is a function (label_values) that appears to get the available values for a given label in the scraped data. I cannot find documentation for this function anywhere. I also do not know what other functions may exist. Where is there documentation that explains how to use all of this? I think this is probably the main issue.

Dashboard variable filtering

I've been reading the Grafana documentation and am under the impression that dashboard level variables can be filtered so that, for example, if I select a node, the other variable values can be filtered to only show the values available on that selected node. I can't get this to work - if I select a node, the other variable available values are not being filtered. I assume I'm not writing the variable queries correctly, but I can't tell for sure. The Grafana documentation is a little vague or too high-level in some areas.

Example dashboard variable definitions:

type: query
name: node
query: label_values(node)

type: query
name: nemspace
query: label_values(namespace)

Note: I also tried setting the query for namespace to label_values(node="$node", namespace) but this did not work either. I have not been able to find good information on the correct query syntax, functions, etc. Again, I refer back to the first bullet item.

Dashboard variable "include all" option

I'm under the impression that if you configure a variable to allow the "all" option, that this essentially turns off filtering for this variable. However, if I do this, I get no data at all in the view. I've tried setting the associated "custom all value" to "All" and wildcard ("*"), but neither makes any difference. What am I doing wrong?

Conclusion

I think of all of these issues stem from the fact that I do not have information on how to properly configure Grafana dashboard variables. It would be so helpful if there was a detailed comprehensive guide - alas, I have not been able to find one.

1 Answer 1

1

Where is there documentation that explains how to use all of this?

I don't know if there is any documentation on label_values, but it is pretty simple: you supply metric selector and label to extract, and id extracts distinct values of said label.

For example

label_values(up{instance=~".*prod.*"}, job)

Executes query up{instance=~".*prod.*"} and extracts distinct values of job metric.

I'm not sure on the case of label_values without metric selector, but I assume it is semantically equivalent to something like label_values({__name__=~".+"}, job).

Notice, that label_values function doesn’t support queries, if you need to extract labels from some more complex query, use query_result with appropriate regex, per this example.

I assume I'm not writing the variable queries correctly, but I can't tell for sure.

Based on your description I assume, you need query like this for your second variable:

label_values({node=~"$node"}, namespace)

It selects all metrics with value of label node equal to currently selected value of your variable with name node. But if it is possible, I'd recommend to consider using some specific metric to lower performance impact of this query. For example, something like label_values(up{node=~"$node"}, namespace) should e way faster. Same applies to query of first variable: consider something like label_values(up, node).

I'm under the impression that if you configure a variable to allow the "all" option, that this essentially turns off filtering for this variable.

That is not entirely correct. "Include all" basically substitutes variable with all possible values of variable. How substitution happens depends on data source.

For example, if your value node has three possible values: node1.com, node2.com, node3.com, then on substitution of this value to Prometheus query value (node1.com|node2.com|node3.com) will be used.

Note: If you plan to use variables with either options "multiple values" or "include all", it's expected that you will use regex selectors, for example up{node=~"$node"}, to accommodate multiple values supplied to query through variable.

It would be so helpful if there was a detailed comprehensive guide

You can raise said concern in grafana's issues on GitHub. Folks around there are quite responsive, and if you'll convey to them need of more detailed description on how to use label_values, there is pretty good chance that it will be added to official documentation of Grafana.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.