We only have link-recommendation task pool in Grafana. This is configured in two different places:listTaskCounts.php has a condition in reportTaskCounts to only report task counts to statsd for link-recommendation task type and puppet then explicitly passes --tasktype link-recommendation (so it doesn't run for other task types in the first place).

The script code has the following reasoning around the topic limitation:

// Limit to link recommendations to avoid excessive use of statsd metrics as we don't
// care too much about the others. Maybe there will be a nicer way to handle this in
// the future with Prometheus.

That's fair, as currently reporting to statsd for each task type means 1 metric per topic plus 1 global metric. I think it makes sense to stop producing the per-topic metrics, as they make very little sense for small-ish wikis, while the global task count is still useful even on small-ish wikis.

I'm in favour of removing the per-topic metrics for everything (incl. link recommendation), but it is currently used in Grafana (albeit in a broken dashboard) and it might be relied on by other engineers. I'll fill a separate task about removing the per-topic metrics. In the meantime, let's produce per-topic metrics only for link-recommendation (which are already included) and not for other tasks.

Once the task counts for all task types are in statsd, we can follow https://wikitech.wikimedia.org/wiki/Alertmanager#Grafana_alerts to implement the alerting.

Change 953343 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] listTaskCounts: Push total task counts to statsd for all tasks

https://gerrit.wikimedia.org/r/953343

Change 953344 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/puppet@production] growthexperiments: Run listTaskCounts for all task types

https://gerrit.wikimedia.org/r/953344

Change 953347 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/puppet@production] alertmanager: route Growth team alerts

https://gerrit.wikimedia.org/r/953347

Moving to radar, please let me know if there's anything we can help with.

In T345204#9146561, @lmata wrote:

Moving to radar, please let me know if there's anything we can help with.

Thanks! Review of the puppet patch would be helpful :). It is by no means urgent, given I still need to get the data I want to base the alerting on from MediaWiki into graphite, but AFAIK those two parts should be largely independent.

In T345204#9146804, @Urbanecm_WMF wrote:

In T345204#9146561, @lmata wrote:

Moving to radar, please let me know if there's anything we can help with.

Thanks! Review of the puppet patch would be helpful :). It is by no means urgent, given I still need to get the data I want to base the alerting on from MediaWiki into graphite, but AFAIK those two parts should be largely independent.

You are correct that the two parts are independent, please feel free to add me as a reviewer of the relevant changes and I'm happy to help

In T345204#9149207, @fgiunchedi wrote:

[...]
You are correct that the two parts are independent, please feel free to add me as a reviewer of the relevant changes and I'm happy to help

Thanks for the offer! Done. It's https://gerrit.wikimedia.org/r/c/operations/puppet/+/953347/ for the Puppet part and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/953343/ for the pushing data part.

Change 953347 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: route Growth team alerts

https://gerrit.wikimedia.org/r/953347

Change 953343 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] listTaskCounts: Push total task counts to statsd for all tasks

https://gerrit.wikimedia.org/r/953343

Change 957396 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.26] listTaskCounts: Push total task counts to statsd for all tasks

https://gerrit.wikimedia.org/r/957396

Change 957396 merged by Urbanecm:

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.26] listTaskCounts: Push total task counts to statsd for all tasks

https://gerrit.wikimedia.org/r/957396

Mentioned in SAL (#wikimedia-operations) [2023-09-14T15:47:52Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:957396|listTaskCounts: Push total task counts to statsd for all tasks (T345204)]], [[gerrit:957758|linkTaskCounts: Stop producing per-topic statsd data (T345210)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-14T15:55:29Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:957396|listTaskCounts: Push total task counts to statsd for all tasks (T345204)]], [[gerrit:957758|linkTaskCounts: Stop producing per-topic statsd data (T345210)]] (duration: 07m 37s)

Change 953344 merged by Clément Goubert:

[operations/puppet@production] growthexperiments: Run listTaskCounts for all task types

https://gerrit.wikimedia.org/r/953344

Mentioned in SAL (#wikimedia-operations) [2023-09-19T16:28:40Z] <claime> Deployed https://gerrit.wikimedia.org/r/953344 - T345204

I'm untagging o11y here since I don't think there's any immediate actionable, please reach out if that's not the case