I've noticed something curious, I wonder if anyone can explain it to me.

When I look at the data for these two metrics for the last 6 days, they are exactly the same:

	https://grafana-rw.wikimedia.org/explore?orgId=1&left={"queries":[{"target":"summarize(groupByNodes(MediaWiki.centralauth.centrallogin_errors.*.rate,'sum',2),'1h','avg',false)"},{"target":"summarize(groupByNodes(MediaWiki.authmanager.centrallogin.*.failure.*.rate,'sum',2,4),'1h','avg',false)"}],"range":{"from":"now-6d","to":"now"}}

But if I use the same query to view the last 8 days, they are slightly different:

https://grafana-rw.wikimedia.org/explore?orgId=1&left={"queries":[{"target":"summarize(groupByNodes(MediaWiki.centralauth.centrallogin_errors.*.rate,'sum',2),'1h','avg',false)"},{"target":"summarize(groupByNodes(MediaWiki.authmanager.centrallogin.*.failure.*.rate,'sum',2,4),'1h','avg',false)"}],"range":{"from":"now-8d","to":"now"}}

https://wikitech.wikimedia.org/wiki/Graphite#Extended_properties

Last 7 days, we store one datapoint per 1 minute, as flushed by Statsd.
Last 14 days, we keep one datapoint per 5 minutes, aggregated at ingestion time (avg for most properties, except sum for some).

Each data source is only queried once. Depending on date time range, you'll query either Graphite's 7d sequences, or it's longer term sequences.

Older data thus is naturally flatter, with fewer spikes and dips.

The average should remain accurate, eg if used to derive a multiple or total over the same period. However generating another average will produce unstable results, and does not have an (obvious) meaning to me.

I see, but that doesn't really explain it to me. How come that identical datapoints-per-1-minute result in different datapoints-per-5-minutes when aggregated? (Are the aggregation periods not aligned?)

They don't even average out to the same value. One of the metrics consistently has higher values. (link)

Well, at least the 'sum' is consistent. (link)

Change 977781 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@master] Remove logging to "centralauth.centrallogin_errors.*"

https://gerrit.wikimedia.org/r/977781

In T351948#9357367, @matmarex wrote:

I see, but that doesn't really explain it to me. How come that identical datapoints-per-1-minute result in different datapoints-per-5-minutes when aggregated?

The authevents metrics are split into api/web, so even though the individual reported values are the same, there are twice as many buckets and there is less aggregation happening.

I never quite understood why avg is inaccurate (for percentile metrics, it's impossible to get an exact result by aggregating aggregates, but for sum/avg it should be fine) but whatever mechanism causes avg aggregation to lose fidelity is used more for the old metric.

Change 977781 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Remove logging to "centralauth.centrallogin_errors.*"

https://gerrit.wikimedia.org/r/977781

Ah, that sounds plausible enough, thanks.

Change 978679 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/puppet@production] Update CentralAuth login failures metric

https://gerrit.wikimedia.org/r/978679

I updated both https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1&viewPanel=41 and https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?orgId=1&viewPanel=3 to show both the old and the new data sources. Currently they're shown in a slightly ugly overlapped way, but the old one will stop being logged next week.

Change 978679 merged by Filippo Giunchedi:

[operations/puppet@production] Update CentralAuth login failures metric

https://gerrit.wikimedia.org/r/978679