Convert eqiad imagescalers to HHVM, Trusty
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Joe
	Dec 18 2014, 8:18 AM

Referenced Files

None

Subscribers

View All 24 Subscribers

Tokens

"Yellow Medal" token, awarded by ori.

Description

In our quest to convert the whole cluster to trusty and HHVM, we've got to the imagescalers.

From what I can see from puppet/the current image scalers is that we have the following packages custom-built at the moment:

ffmpeg2theora (disabled libav multithreading)
librsvg (external resources loading protection)
libvips31 (backport)
libvpx1 (backport)

We also need to evaluate the performance of HHVM under the peculiar imagescalers load, which is probably going to be non-optimal.

Finally, testing should be very accurate to assess any rendering differences on trusty vs precise.

Details

Subject	Repo	Branch	Lines +/-
imagescalers: convert the last two servers to HAT	operations/puppet	production	+2 -8
Add Content-Length header to thumb.php redirects	mediawiki/core	master	+1 -0
Add Content-Length header to thumb.php redirects	mediawiki/core	wmf/1.26wmf16	+1 -0
Add Content-Length header to thumb.php redirects	mediawiki/core	wmf/1.26wmf15	+1 -0
Re-introduce ProxyPass rule for thumb_handler.php, with corrected docroots	operations/puppet	production	+12 -0
Follow-up for Ie17cb06: add thumb_handler.php ProxyPass rule to all vhosts	operations/puppet	production	+11 -0
Add ProxyPass rule for thumb_handler.php	operations/puppet	production	+1 -0
imagescalers: re-image mw115[6-8] to trusty, HHVM	operations/puppet	production	+4 -4
imagescalers: reimage mw1154, mw1155 to HAT	operations/puppet	production	+4 -4
imagescalers: reimage mw1153 with HAT	operations/puppet	production	+4 -4
mediawiki: Touch /etc/wikimedia-image-scaler on image scalers	operations/puppet	production	+9 -0
mediawiki: install fonts metric-compatible with Calibri and Cambria	operations/puppet	production	+6 -0
imagescaling: remove legacy option for HHVM imagescalers	operations/mediawiki-config	master	+7 -3

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	None	T86081 Complete the use of HHVM over Zend PHP on the Wikimedia cluster
Resolved	Joe	T86096 Switch HAT appservers to trusty's ICU (or newer)
Resolved	None	T65907 Tiff image with jpeg compressed data has incorrect colours (image magick needs upgrading?)
Declined	None	T65899 Upgrade Wikimedia servers to Ubuntu Trusty (14.04)
Resolved	• brooke	T55863 Support VP9 in TMH (Unable to decode)
Resolved	Joe	T84842 Convert eqiad imagescalers to HHVM, Trusty
Resolved	tstarling	T91468 HHVM with FastCGI does not support streaming output
Resolved	Joe	T93194 Create an HHVM 3.6.0 package, adding Tim's streaming patch
		· · ·

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Eloquence moved this task from March 2015: Platform to March 30-April 3 on the Roadmap board.Mar 23 2015, 7:23 AM

Restricted Application added a project: Notice. · View Herald TranscriptMar 27 2015, 11:17 PM

Legoktm mentioned this in T94149: Get rid of Zend 5.5 tests for wmf branches.Mar 29 2015, 8:02 AM

Change 196173 merged by Giuseppe Lavagetto:
mediawiki: install fonts metric-compatible with Calibri and Cambria

https://gerrit.wikimedia.org/r/196173

Joe mentioned this in rOPUP139c36fc44af: mediawiki: install fonts metric-compatible with Calibri and Cambria.Mar 30 2015, 6:21 AM

• gpaumier moved this task from Backlog to Triaged on the Notice board.Mar 30 2015, 8:16 PM

• gpaumier moved this task from Triaged to Archive on the Notice board.Apr 2 2015, 7:01 PM

@Joe, do we need to push this to next week?

In T84842#1175265, @Eloquence wrote:

@Joe, do we need to push this to next week?

At minimum. Giuseppe ran into segfaults with Tim's output buffer patch and that's blocking the rollout of a new package, which the scalers need.

OK, let's schedule to a specific week once we have confidence on the ETA.

Eloquence moved this task from April 27-May 1 to April 2015: Platform on the Roadmap board.Apr 2 2015, 7:58 PM

Tgr mentioned this in T94982: Add file/thumbnail test suite to multimedia vagrant role.Apr 3 2015, 6:56 AM

• MZMcBride subscribed.Apr 9 2015, 4:44 PM

Eloquence moved this task from April 2015: Platform to April 27-May 1 on the Roadmap board.Apr 21 2015, 5:43 AM

Joe closed subtask T93194: Create an HHVM 3.6.0 package, adding Tim's streaming patch as Resolved.May 6 2015, 3:10 PM

Ok so, after a lot of battles with HHVM 3.6.1, the mysterious 503s on the imagescalers continued.

One possible cause is the fact that apparently some images require more memory to be converted on trusty:

curl -H 'Host: commons.wikimedia.org' 'http://mw1152/w/thumb_handler.php/1/17/HawkridgeBarton_Chittlehampton_NorthDevon.PNG/1000px-HawkridgeBarton_Chittlehampton_NorthDevon.PNG' -I
HTTP/1.1 500 Internal server error
Date: Wed, 06 May 2015 15:13:17 GMT
Server: Apache
X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
Cache-control: no-cache
X-MW-Thumbnail-Renderer: mw1152
Connection: close
Content-Type: text/html; charset=utf-8

which results in the kernel recording

May  6 15:13:20 mw1152 kernel: [4775538.759268] memory: usage 307200kB, limit 307200kB, failcnt 50
May  6 15:13:20 mw1152 kernel: [4775538.759269] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
May  6 15:13:20 mw1152 kernel: [4775538.759270] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
May  6 15:13:20 mw1152 kernel: [4775538.759271] Memory cgroup stats for /mediawiki/job/15592: cache:0KB rss:307200KB rss_huge:202752KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:307132KB inactive_file:0KB active_file:0KB unevictable:0KB
May  6 15:13:20 mw1152 kernel: [4775538.759282] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
May  6 15:13:20 mw1152 kernel: [4775538.759334] [15592]    33 15592     5222      426      16        0             0 bash
May  6 15:13:20 mw1152 kernel: [4775538.759336] [15594]    33 15594     2867      167      11        0             0 timeout
May  6 15:13:20 mw1152 kernel: [4775538.759337] [15595]    33 15595    93743    77414     186        0             0 convert
May  6 15:13:20 mw1152 kernel: [4775538.759339] Memory cgroup out of memory: Kill process 15595 (convert) score 1010 or sacrifice child
May  6 15:13:20 mw1152 kernel: [4775538.768545] Killed process 15595 (convert) total-vm:374972kB, anon-rss:307004kB, file-rss:2652kB
May  6 15:15:01 mw1152 CRON[15698]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Also, I tested that at smaller resulting sizes this doesn't cause the 500 to be emitted.

This is the mediawiki error log: P612

I'm still unsure this is really the cause of the larger number of 503s I see.

greg moved this task from April 27-May 1 to This week: May 4-8 on the Roadmap board.May 6 2015, 9:17 PM

greg moved this task from This week: May 4-8 to This week: May 11-15 on the Roadmap board.May 11 2015, 2:19 PM

greg moved this task from This week: May 11-15 to May 18-22 on the Roadmap board.May 16 2015, 5:32 AM

greg moved this task to This week: May 4-8 on the Roadmap workboard.
greg moved this task to This week: May 11-15 on the Roadmap workboard.
greg moved this task to May 18-22 on the Roadmap workboard.

What ya'll think now? :)

In T84842#1306421, @greg wrote:

What ya'll think now? :)

We should probably not schedule this until we resolve the issue Giuseppe spotted in T84842#1265055.

greg moved this task from May 18-22 to Unscheduled on the Roadmap board.May 24 2015, 10:09 AM

Change 213228 had a related patch set uploaded (by Ori.livneh):
mediawiki: Touch /etc/wikimedia-image-scaler on image scalers

https://gerrit.wikimedia.org/r/213228

Change 213228 merged by Ori.livneh:
mediawiki: Touch /etc/wikimedia-image-scaler on image scalers

https://gerrit.wikimedia.org/r/213228

ori mentioned this in rOPUP655b837b0ac4: mediawiki: Touch /etc/wikimedia-image-scaler on image scalers.May 24 2015, 10:27 AM

Krenair added a parent task: T65899: Upgrade Wikimedia servers to Ubuntu Trusty (14.04).May 30 2015, 1:40 PM

Since the latest test I did after ori found we needed that file was unsuccessful, and I have zero time to work on this at the moment, I'll release this ticket so that somebody else can grab it.

As matanya clearly stated "We've been waiting forever for this".

Joe removed Joe as the assignee of this task.Jun 2 2015, 1:42 PM

Matanya added a parent task: T55863: Support VP9 in TMH (Unable to decode).Jun 16 2015, 3:53 PM

ori assigned this task to Joe.Jun 29 2015, 11:12 PM

Bah, edit conflicts.

Krinkle moved this task from Inbox, needs triage to Backlog: Maintenance, non-prioritized on the Performance-Team board.Jul 2 2015, 5:21 AM

For the past several weeks, this task was blocked on the fact that the HHVM renderer (mw1152) would cause 5xx spikes whenever it was pooled. The issue appears to have been a corruption in the cached byte-code for /w/404.php which caused Varnish to be unable to parse 404 responses generated by HHVM on mw1152. These responses were transformed by varnish into 503s. Touching the file fixed it.

The remaining task is to re-image the other scalers.

(codfw imagescalers are all HHVM+Trusty already, so we're just left with mw1153-1160)

Krinkle moved this task from Backlog: Maintenance, non-prioritized to To-do: Goals prioritized current Quarter on the Performance-Team board.Jul 6 2015, 7:00 PM

Change 223331 had a related patch set uploaded (by Giuseppe Lavagetto):
imagescalers: reimage mw1153 with HAT

https://gerrit.wikimedia.org/r/223331

gerritbot added a project: Patch-For-Review.Jul 7 2015, 3:49 PM

Change 223331 merged by Giuseppe Lavagetto:
imagescalers: reimage mw1153 with HAT

https://gerrit.wikimedia.org/r/223331

Joe mentioned this in rOPUP5af0ba4851f0: imagescalers: reimage mw1153 with HAT.Jul 8 2015, 9:20 AM

Change 224594 had a related patch set uploaded (by Giuseppe Lavagetto):
imagescalers: reimage mw1154, mw1155 to HAT

https://gerrit.wikimedia.org/r/224594

Change 224594 merged by Giuseppe Lavagetto:
imagescalers: reimage mw1154, mw1155 to HAT

https://gerrit.wikimedia.org/r/224594

Joe mentioned this in rOPUP74256e7b1395: imagescalers: reimage mw1154, mw1155 to HAT.Jul 14 2015, 10:04 AM

Change 225285 had a related patch set uploaded (by Giuseppe Lavagetto):
imagescalers: re-image mw115[6-8] to trusty, HHVM

https://gerrit.wikimedia.org/r/225285

Change 225285 merged by Giuseppe Lavagetto:
imagescalers: re-image mw115[6-8] to trusty, HHVM

https://gerrit.wikimedia.org/r/225285

Joe mentioned this in rOPUP5d0d9dc9301b: imagescalers: re-image mw115[6-8] to trusty, HHVM.Jul 17 2015, 6:27 AM

Krenair changed the task status from Stalled to Open.Jul 17 2015, 7:24 AM

FTR, I am going to depool all remaining Zend imagescaler today to test any outstanding problems with those. If none arise, I'm going to upgrade the last three tomorrow.

In T84842#1466863, @Joe wrote:

FTR, I am going to depool all remaining Zend imagescaler today to test any outstanding problems with those. If none arise, I'm going to upgrade the last three tomorrow.

Great. Sounds like a good plan. Thanks for pushing forward.

New imagescalers fun found today:

an high level of 5xx were going on today, so I intercepted a few urls that were returning a 503 - for example, I requested

https://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Casquette-IMG_0922.jpg/600px-Casquette-IMG_0922.jpg

from my browser and got a 503:

...
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: GET http://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Casquette-IMG_0922.jpg/600px-Casquette-IMG_0922.jpg, from 10.64.32.81 via cp1048 cp1048 ([10.64.32.100]:3128), Varnish XID 3696382267
Forwarded for: 79.58.168.196, 10.64.32.81, 10.64.32.81
Error: 503, Service Unavailable at Wed, 22 Jul 2015 16:21:18 GMT

while if I do a plain curl I get the correct behaviour, a 302:

curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Casquette-IMG_0922.jpg/600px-Casquette-IMG_0922.jpg
HTTP/1.1 302 Found
Server: nginx/1.9.3
Date: Wed, 22 Jul 2015 16:30:29 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive
X-Content-Type-Options: nosniff
Expires: Thu, 23 Jul 2015 04:22:53 GMT
Vary: X-Forwarded-Proto
Location: https://upload.wikimedia.org/wikipedia/commons/thumb/9/99/Casquette_d%27administrateur_colonial_-IMG_0922.jpg/600px-Casquette_d%27administrateur_colonial_-IMG_0922.jpg
Cache-Control: no-cache

So I tought this could have to do with compression. But subsequent requests were successful from the browser as well, so no idea what is going on here - maybe one imagescaler doing something wrong?

2015-07-23 00:06:38 mw1153 commonswiki exception ERROR: [d68c4bb1] /w/thumb_handler.php/2/2e/Mirage_III_A_01_Mus%0Aee_du_Bourget_P1020118.JPG/424px-%0AMirage_III_A_01_Musee_du_Bourget_P1020118.JPG   MWException from line 171 of /srv/mediawiki/php-1.26wmf15/includes/objectcache/ObjectCache.php: CACHE_ACCEL requested but no suitable object cache is present. You may want to install APC. {"exception":"[object] (MWException(code: 0): CACHE_ACCEL requested but no suitable object cache is present. You may want to install APC. at /srv/mediawiki/php-1.26wmf15/includes/objectcache/ObjectCache.php:171)"}

Seen from mw115[2-8]

In T84842#1473663, @Krenair wrote:

2015-07-23 00:06:38 mw1153 commonswiki exception ERROR: [d68c4bb1] /w/thumb_handler.php/2/2e/Mirage_III_A_01_Mus%0Aee_du_Bourget_P1020118.JPG/424px-%0AMirage_III_A_01_Musee_du_Bourget_P1020118.JPG   MWException from line 171 of /srv/mediawiki/php-1.26wmf15/includes/objectcache/ObjectCache.php: CACHE_ACCEL requested but no suitable object cache is present. You may want to install APC. {"exception":"[object] (MWException(code: 0): CACHE_ACCEL requested but no suitable object cache is present. You may want to install APC. at /srv/mediawiki/php-1.26wmf15/includes/objectcache/ObjectCache.php:171)"}

Seen from mw115[2-8]

Ummm, why/how do we have HHVM boxes without APC enabled?

In T84842#1473663, @Krenair wrote:

2015-07-23 00:06:38 mw1153 commonswiki exception ERROR: [d68c4bb1] /w/thumb_handler.php/2/2e/Mirage_III_A_01_Mus%0Aee_du_Bourget_P1020118.JPG/424px-%0AMirage_III_A_01_Musee_du_Bourget_P1020118.JPG   MWException from line 171 of /srv/mediawiki/php-1.26wmf15/includes/objectcache/ObjectCache.php: CACHE_ACCEL requested but no suitable object cache is present. You may want to install APC. {"exception":"[object] (MWException(code: 0): CACHE_ACCEL requested but no suitable object cache is present. You may want to install APC. at /srv/mediawiki/php-1.26wmf15/includes/objectcache/ObjectCache.php:171)"}

Seen from mw115[2-8]

See T106743: Exceptions due to APC missing on some scalers due to Gadgets caching

Change 226658 had a related patch set uploaded (by Ori.livneh):
Add ProxyPass rule for thumb_handler.php

https://gerrit.wikimedia.org/r/226658

Change 226658 merged by Ori.livneh:
Add ProxyPass rule for thumb_handler.php

https://gerrit.wikimedia.org/r/226658

ori mentioned this in rOPUP9d45102200e7: Add ProxyPass rule for thumb_handler.php.Jul 24 2015, 12:17 AM

Change 226738 had a related patch set uploaded (by Ori.livneh):
Follow-up for Ie17cb06: add thumb_handler.php ProxyPass rule to all vhosts

https://gerrit.wikimedia.org/r/226738

Change 226738 merged by Ori.livneh:
Follow-up for Ie17cb06: add thumb_handler.php ProxyPass rule to all vhosts

https://gerrit.wikimedia.org/r/226738

ori mentioned this in rOPUP3bfb0b5d4e87: Follow-up for Ie17cb06: add thumb_handler.php ProxyPass rule to all vhosts.Jul 24 2015, 4:48 PM

faidon mentioned this in T106895: Thumbnails no longer rendering for recent local uploads.Jul 25 2015, 10:33 AM

Change 227160 had a related patch set uploaded (by Ori.livneh):
Re-introduce ProxyPass rule for thumb_handler.php, with corrected docroots

https://gerrit.wikimedia.org/r/227160

Change 227160 merged by Ori.livneh:
Re-introduce ProxyPass rule for thumb_handler.php, with corrected docroots

https://gerrit.wikimedia.org/r/227160

ori mentioned this in rOPUP5cd1ccadc215: Re-introduce ProxyPass rule for thumb_handler.php, with corrected docroots.Jul 26 2015, 11:14 PM

I think I found what is the latest gotcha with imagescalers:

curl -H 'X-Forwarded-Proto: https' -H 'Host: commons.wikimedia.org' 'http://mw1152/w/thumb_handler.php/7/75/John_Bauer-Tyr_and_Fenrir.jpg/180px-John_Bauer-Tyr_and_Fenrir.jpg' -v
* About to connect() to mw1152 port 80 (#0)
*   Trying 10.64.16.132... connected
> GET /w/thumb_handler.php/7/75/John_Bauer-Tyr_and_Fenrir.jpg/180px-John_Bauer-Tyr_and_Fenrir.jpg HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> X-Forwarded-Proto: https
> Host: commons.wikimedia.org
> 
< HTTP/1.1 302 Found
< Date: Wed, 29 Jul 2015 09:56:29 GMT
< Server: Apache
< X-Powered-By: HHVM/3.6.1
< X-Content-Type-Options: nosniff
< Cache-control: no-cache
< Expires: Wed, 29 Jul 2015 21:56:29 GMT
< Vary: X-Forwarded-Proto
< Location: https://upload.wikimedia.org/wikipedia/commons/thumb/1/18/Tyr_and_Fenrir-John_Bauer.jpg/180px-Tyr_and_Fenrir-John_Bauer.jpg
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8

while on non-HHVM ones

curl -H 'X-Forwarded-Proto: https' -H 'Host: commons.wikimedia.org' 'http://mw1159/w/thumb_handler.php/7/75/John_Bauer-Tyr_and_Fenrir.jpg/180px-John_Bauer-Tyr_and_Fenrir.jpg' -v
* About to connect() to mw1159 port 80 (#0)
*   Trying 10.64.16.139... connected
> GET /w/thumb_handler.php/7/75/John_Bauer-Tyr_and_Fenrir.jpg/180px-John_Bauer-Tyr_and_Fenrir.jpg HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> X-Forwarded-Proto: https
> Host: commons.wikimedia.org
> 
< HTTP/1.1 302 Found
< Date: Wed, 29 Jul 2015 09:58:24 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< Cache-control: no-cache
< Location: https://upload.wikimedia.org/wikipedia/commons/thumb/1/18/Tyr_and_Fenrir-John_Bauer.jpg/180px-Tyr_and_Fenrir-John_Bauer.jpg
< Expires: Wed, 29 Jul 2015 21:58:24 GMT
< Vary: X-Forwarded-Proto
< Content-Length: 0
< Content-Type: text/html
<

In this case we have the same problem we had with 404s: thumb.php doesn't set the content-length, which makes apache assume the transfer-encoding is chunked, which for a 0 sized body makes varnish barf.

Change 227676 had a related patch set uploaded (by Giuseppe Lavagetto):
Add Content-Length header to thumb.php redirects

https://gerrit.wikimedia.org/r/227676

Change 227732 had a related patch set uploaded (by BryanDavis):
Add Content-Length header to thumb.php redirects

https://gerrit.wikimedia.org/r/227732

Change 227733 had a related patch set uploaded (by BryanDavis):
Add Content-Length header to thumb.php redirects

https://gerrit.wikimedia.org/r/227733

Change 227732 merged by jenkins-bot:
Add Content-Length header to thumb.php redirects

https://gerrit.wikimedia.org/r/227732

Change 227733 merged by jenkins-bot:
Add Content-Length header to thumb.php redirects

https://gerrit.wikimedia.org/r/227733

Change 227676 merged by jenkins-bot:
Add Content-Length header to thumb.php redirects

https://gerrit.wikimedia.org/r/227676

• Forrestbot added projects: MW-1.26-release, WMF-deploy-2015-08-04_(1.26wmf17), WMF-deploy-2015-07-21_(1.26wmf15), WMF-deploy-2015-07-28_(1.26wmf16).Jul 29 2015, 5:00 PM

Change 227967 had a related patch set uploaded (by Giuseppe Lavagetto):
imagescalers: convert the last two servers to HAT

https://gerrit.wikimedia.org/r/227967

Krenair mentioned this in rMW304e95a3337b: Add Content-Length header to thumb.php redirects.Jul 30 2015, 10:14 PM

bd808 mentioned this in rMW2c9518ed78b0: Add Content-Length header to thumb.php redirects.Jul 30 2015, 10:15 PM

bd808 mentioned this in rMWacb2a9294683: Add Content-Length header to thumb.php redirects.

Change 227967 merged by Giuseppe Lavagetto:
imagescalers: convert the last two servers to HAT

https://gerrit.wikimedia.org/r/227967

Joe mentioned this in rOPUPdbe0b64bbf90: imagescalers: convert the last two servers to HAT.Aug 4 2015, 11:05 AM

Joe closed this task as Resolved.Aug 4 2015, 1:46 PM

ori awarded a token.Aug 4 2015, 3:59 PM

Liuxinyu970226 unsubscribed.Aug 4 2015, 11:35 PM

McZusatz mentioned this in T106516: Greyscale pngs without gAMA chunk rendered with incorrect contrast [or setting the gamma in GIMP exports to PNG].Aug 5 2015, 8:34 AM

McZusatz added a parent task: T106516: Greyscale pngs without gAMA chunk rendered with incorrect contrast [or setting the gamma in GIMP exports to PNG].Aug 9 2015, 8:45 AM

Joe closed subtask T91468: HHVM with FastCGI does not support streaming output as Resolved.Aug 24 2015, 12:57 PM

Just for cross-reference, a regression was found with rsvg - svgs > 10 MB (there's about 9000 such files) don't render. We'd need to upgrade to a later version of rsvg to fix, see T111815

AntiCompositeNumber removed a parent task: T106516: Greyscale pngs without gAMA chunk rendered with incorrect contrast [or setting the gamma in GIMP exports to PNG].May 21 2020, 1:25 AM

Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:54 PM

Convert eqiad imagescalers to HHVM, TrustyClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Convert eqiad imagescalers to HHVM, Trusty
Closed, ResolvedPublic
Actions

Related Objects
Search...