As far as I can tell from the UEFI specification, the firmware is supposed to support FAT16 or FAT32:

13.3.1.1 File System Format

The EFI firmware must support the FAT32, FAT16, and FAT12 variants of the EFI file system. What variant
of EFI FAT to use is defined by the size of the media. The rules defining the relationship between media
size and FAT variants is defined in the specification for the EFI file system.

We seem to not be specifying the actual FAT size when running mkfs.fat, which looks like it makes it also auto-detect the right one based on size:

-F FAT-SIZE
           Specifies  the type of file allocation tables used (12, 16 or 32 bit).  If nothing is speci‐
           fied, mkfs.fat will automatically select between 12, 16 and 32 bit, whatever fits better for
           the filesystem size.

which seems to fit with the UEFI definition.

AFAIK, we've had other users successfully boot FCOS on the Raspberry Pi 4. I suspect this may be related to the fact that you're booting via USB instead of SD card? @dustymabe Have you tried booting via USB?

My understanding is that the ESP on a non-removable disk must be FAT32. UEFI spec 2.9 section 13.3 says:

EFI encompasses the use of FAT32 for a system partition, and FAT12 or FAT16 for removable media.

We had the same bug in Container Linux (coreos/bugs#2246).

Thanks for the information about the specification and updates. I should spell check before I submit! I recall that during my initial efforts to install and run FCOS on RPi4 hardware last February (see https://forums.raspberrypi.com/viewtopic.php?t=304318 for a write up if interested), this was not an issue I ran into. The USB disk I uses is a KINGSTON SA400S37120G (120 GB). Please let me know if I can provide any more information and/or testing.

@dustymabe Have you tried booting via USB?

I've been recently testing things via both uboot and UEFI (working on documentation to add to our docs site). I have to admit I have seen some cases where the pi4 will loop on trying partition 0 like @buckaroogeek suggested, but I've also taken that exact USB out and plugged it back in and it worked the next time, which was odd.

Maybe use of FAT16 makes the issue more likely to happen? Maybe I'm misremembering? I definitely know things are working on an SD card just fine the way it is today.

I checked the eeprom firmware page on github (https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/release-notes.md) and see this note: "Update the FAT detection to support FAT16 for EFI/ESD paritions with GPT instead of assuming FAT32. The latest firmware is also required for a similar update." This is for a 4 Oct update.

This is a corner case that likely affects more than just the Pi, so we should fix it regardless. Fix in coreos/coreos-assembler#2491.

And thanks for the report!

Glad to help.

Going to re-open until the change lands

There's a complication. Microsoft's EFI FAT32 specification says:

The FAT type—one of FAT12, FAT16, or FAT32—is determined by the count of clusters on the volume and nothing else.

FAT32 is defined to have at least 65525 clusters, and with one 4 KiB sector/cluster, that means the ESP on a 4Kn disk needs to be at least 257 MiB. Otherwise EDK II (and presumably other UEFI firmware) won't accept it as a valid filesystem. Our ESP is 127 MiB, and the partition layout is difficult to change because it's also wired into Butane.

We could switch to FAT32 only in the 512b image, which would improve the situation somewhat, but would create a weird inconsistency between our image types. Or we could change the partition layout again, or stick with FAT16.

If we switched the ESP to FAT32, we should probably update the Butane RAID template to do the same. Butane doesn't have any way to know the boot disk's sector size (the user may not know either), so effectively it can't use different templates for 512b and 4Kn disks.

discuss at meeting?

Yeah. Added meeting label.

Was curious about what Anaconda did here. It uses blivet, which looks like it currently always creates a FAT16 filesystem: https://github.com/storaged-project/blivet/blob/314675891c86ba185deabc52f77ca6389973d9a1/blivet/formats/fs.py#L925-L926