Http errors 502 Bad Gateway during scale up on Heroku

Question

After I solvied H13 errors during scale down events I realized I am getting also buch of 502 Bad Gateway HTTP errors during scale up events.

I am running container with Docker image on Heroku containing Django application running with supervisord/nginx/gunicorn/pgbouncer

I am getting these few (e.g. 9) errors during every scale up:

This can be found in Logentries when I search for status>=500:

13 Nov 2022 12:06:49.796373 <158>1 2022-11-13T11:06:49.573611+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=327c9a22-b5ee-4118-90f2-283bbdae5124 fwd="xxxxxxxx,172.68.254.103" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:49.828483 <158>1 2022-11-13T11:06:49.574134+00:00 heroku router - - at=info method=GET path="/api/v1/search/?query=+asset_type:material+order:-last_upload&dict_parameters=1&page_size=15&addon_version=3.2.0&blender_version=3.3.1&scene_uuid=8c4fa678-e970-44df-bf3d-c4db2f76da33" host=www.blenderkit.com request_id=aff26f77-00c0-4ae3-92a0-47c6caee0cb4 fwd="xxxxxxxx,172.68.210.10" dyno=web.2 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:49.903304 <158>1 2022-11-13T11:06:49.672778+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=bad94daa-967d-4fc1-89f1-abf90e935b99 fwd="xxxxxxxx,172.70.82.185" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:49.958305 <158>1 2022-11-13T11:06:49.748174+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=d73be07c-a9e8-4d9b-8895-5449b7c12376 fwd="xxxxxxx,162.158.170.37" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:50.083388 <158>1 2022-11-13T11:06:49.931239+00:00 heroku router - - at=info method=GET path="/comments/api/assets-uuidasset/xxxxxxx/count/?_=1668337010" host=www.blenderkit.com request_id=c20cf157-19c5-43cd-95e9-e55b31c1c6d4 fwd="xxxx,172.70.111.52" dyno=web.2 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:50.091396 <158>1 2022-11-13T11:06:49.740483+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=84ac1bee-7bdb-43c0-91a6-60a6f901bf64 fwd="xxxxxx,172.70.242.110" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:50.099396 <158>1 2022-11-13T11:06:49.868807+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=1374d6d0-ca09-4f38-a45d-1d82dbb60a86 fwd="2003:c7:7f21:5f75:c0c9:bbbd:73f1:4975,172.70.242.110" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:50.127304 <158>1 2022-11-13T11:06:49.961583+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=9a174072-f93f-4220-afb7-8378c84c0937 fwd="xxxxxx,172.71.178.99" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:50.133395 <158>1 2022-11-13T11:06:49.99942+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=abb84ce4-f4c8-4226-a23f-32d7310007c8 fwd="xxxxx,172.70.242.110" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:50.151358 <158>1 2022-11-13T11:06:49.783907+00:00 heroku router - - at=info method=GET path="/avatar-redirect/xxxxxxxxx/128/" host=www.blenderkit.com request_id=44e69bb9-2ae4-4d20-a03d-eb71a370f364 fwd="xxxxxxxxxx,172.68.66.87" dyno=web.2 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:50.402302 <158>1 2022-11-13T11:06:50.033443+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=810db70f-8b56-404b-befb-456424aacdce fwd="xxxxxxxx,172.70.91.43" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:55.828359 <158>1 2022-11-13T11:06:55.672729+00:00 heroku router - - at=info method=GET path="/avatar-redirect/xxxxxxxxx/128/" host=www.blenderkit.com request_id=8e090a61-7aae-466a-a22a-d2ed5947a260 fwd="xxxxx,172.71.130.55" dyno=web.4 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:55.994399 <158>1 2022-11-13T11:06:55.618393+00:00 heroku router - - at=info method=GET path="/comments/api/assets-uuidasset/xxxxxxxx/count/?_=1668337016" host=www.blenderkit.com request_id=41af736c-c990-42fb-b39b-681a139cce9a fwd="xxxx,172.70.242.61" dyno=web.4 connect=1ms service=0ms status=502 bytes=295 protocol=httpsException

Here is how it looks in Heroku Metrics:

Answer 1

I realized that the problem is exact opposite of the problem with H13 errors during scaledown. The problem is, that Nginx started sooner than rest of the applications.

The solution here is simple, delaying Nginx startup through Supervisord configuration:

[program:nginx]
command=bash -c "sleep 5 && /usr/local/nginx/sbin/nginx -c /etc/nginx/nginx.conf"