They look ok to me …

Let me demonstrate how it looks on my end.

Here is the image that is generated:

Here is a recording of how it looks like when I do the steps above in Firefox 123.0.1 after clearing the browser cache:

Note that I get the same weird image even if I use the following desktop browsers:

• Chrome 122.0.6261.129
• Safari 17.3.1

I also get the same result when I use the following mobile browsers that uses a different ISP:

• Chrome 122.0.6261.119 on Android
• Firefox 123.1.0 on Android

for d in codfw drmrs eqiad eqsin esams ulsfo; do echo -n $d: ; curl  -sI --resolve upload.wikimedia.org:443:$(host -t a upload-lb.$d.wikimedia.org | awk '{print $NF}') https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg/3888px-San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg | grep last-modified
done;
codfw:last-modified: Tue, 03 Oct 2023 04:13:36 GMT
drmrs:last-modified: Sat, 02 Mar 2024 09:44:36 GMT
eqiad:last-modified: Sat, 02 Mar 2024 09:44:36 GMT
eqsin:last-modified: Tue, 03 Oct 2023 04:13:36 GMT
esams:last-modified: Sat, 02 Mar 2024 09:44:36 GMT
ulsfo:last-modified: Tue, 03 Oct 2023 04:13:36 GMT

so codfw, eqsin and ulsfo are out of date

K. I included the 3888px width size in my own page, then purged the image with action=purge. Now most centers seems up to date except for codfw

for d in codfw drmrs eqiad eqsin esams ulsfo; do echo -n $d: ; curl  -sI --resolve upload.wikimedia.org:443:$(host -t a upload-lb.$d.wikimedia.org | awk '{print $NF}') https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg/3888px-San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg | grep content-length; done
codfw:content-length: 2897900
drmrs:content-length: 4201111
eqiad:content-length: 4201111
eqsin:content-length: 4201111
esams:content-length: 4201111
ulsfo:content-length: 4201111

Interestingly enough after i did the purge, eqsin and esams are now without last-modified header.

for d in codfw drmrs eqiad eqsin esams ulsfo; do echo -n $d: ; curl  -sI --resolve upload.wikimedia.org:443:$(host -t a upload-lb.$d.wikimedia.org | awk '{print $NF}') https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg/3888px-San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg | grep last-modified
done;
codfw:last-modified: Tue, 03 Oct 2023 04:13:36 GMT
drmrs:last-modified: Mon, 18 Mar 2024 18:02:45 GMT
eqiad:last-modified: Mon, 18 Mar 2024 18:02:45 GMT
eqsin:
esams:
ulsfo:last-modified: Mon, 18 Mar 2024 18:03:07 GMT

ping @akosiaris Ideas on why codfw is out of date and won't correct ? Is it out of rotation or something ?

It seems codfw is now also in sync. Still kinda curious why there is no last-modified header on some of these responses now.

@TheDJ, thanks for confirming the issue. Is there a way to use your shell trick to determine which images would need purging? (For instance, the next image in the sequence also has the same issue.) I think it's a UX issue if people who don't know the purging workaround will get broken images when they click on the perfectly okay-looking thumbnail links on Commons.

Additional note: @tstarling said on Mastodon that this is most probably just T344233 manifesting for old images that haven't yet been purged.

In T358738#9640913, @seav wrote:

Is there a way to use your shell trick to determine which images would need purging?

No, you can only use this to confirm when you have a suspicion that there is a multiple DC issue.

I think it's a UX issue if people who don't know the purging workaround will get broken images when they click on the perfectly okay-looking thumbnail links on Commons.

Of course, this is not supposed to happen. Unfortunately sometimes it does and when it happens the recovery is a bit more complex than a normal purge.

@MatthewVernon can we check if this (remaining example: https://commons.wikimedia.org/wiki/File:San_Isidro_Town_Plaza_(Nueva_Ecija)_22.jpg ) is a reoccurrence of T327253 ?

In T358738#9639319, @TheDJ wrote:

ping @akosiaris Ideas on why codfw is out of date and won't correct ? Is it out of rotation or something ?

Not only it is not out of rotation, but it is currently (and until tomorrow) the primary datacenter.

For what is worth, this now appears to have been fixed

for d in codfw drmrs eqiad eqsin esams ulsfo; do echo -n $d: ; curl  -sI --resolve upload.wikimedia.org:443:$(host -t a upload-lb.$d.wikimedia.org | awk '{print $NF}') https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg/3888px-San_Isidro_Town_Plaza_%28Nueva_Ecija%29_21.jpg | grep content-length; done
codfw:content-length: 4201111
drmrs:content-length: 4201111
eqiad:content-length: 4201111
eqsin:content-length: 4201111
esams:content-length: 4201111
ulsfo:content-length: 4201111

So, my (unproven) theory right now would be some lost or delayed event.

I thought there was no cross-DC replication of thumbnails. T299125#8221206 seems to support that. So it's expected that a bad file created by T344233 would only affect one swift DC.

Yes, we don't replicate thumbnails between DCs any more (and this has been the case since July 2022 cf. T313102)

In T358738#9644283, @tstarling wrote:

I thought there was no cross-DC replication of thumbnails. T299125#8221206 seems to support that. So it's expected that a bad file created by T344233 would only affect one swift DC.

There is no cross-DC replication of thumbnails indeed. However, IIRC, purges do go to both swift clusters and my comment regarding delayed or lost event was for

K. I included the 3888px width size in my own page, then purged the image with action=purge. Now most centers seems up to date except for codfw