Hah, race condition indeed. That reply (call it B) was to another reply (A) by the same author that was still processing when B came in. BF saw that A wasn't bridged, so it didn't bridge B either, due to the logic that replies to non-bridged post aren't bridged.

Ugh. I can think of ways to solve this, but they're all complicated. I don't see an easy fix yet.

(The log message was unrelated, we got two inbox deliveries of this reply and that log message was for the second one. Our logic for finding log messages is not the best 🤷)

Let me know if you want me to file a different issue for this, but I'm pretty sure this is related to this issue

https://fed.brid.gy/ap/@[email protected] <- this bot auto reposts any post from https://fed.brid.gy/ap/@[email protected] if certain conditions are met and it boosts them basically immediately after the post is created. It looks like Bridgy is seeing the boost but doesn't do anything about it (I'm guessing because Bridgy hasn't processed the original post yet)

Ah, interesting, true! That does sound likely.

Got another report of this, #1393 ^, for deletes. Probably the same issue.

One way to fix this could be to error out if we get a reply, repost, quote, delete, or undo of an object that either we haven't seen yet or isn't bridged. We'd then retry the receive task 5m later, and then again 10m after that.

I'm reluctant to do that though, since we get a lot of these, and the vast majority of the time, the object in question will never be bridged. We'd end up tripling our load for those incoming activities to fix a rare edge case. 😕

FWIW, I ended up fixing my issue by just delaying the boost by 30 seconds. Though I'm not sure how useful that info is for fixing the actual issue

One complication is that I don't think there's any guarantee an e.g. Create won't be sent after its matching Delete in ActivityPub (unless the instance cancels relevant outbox items (synchronously), which Mastodon doesn't appear to do at all).

Backoff due to connectivity issues can sporadically (and individually explicitly somewhat randomly!) reorder activities across a long time span, at least in Mastodon.

Mastodon fixes this problem with a database lock on the post id and by permanently tombstoning e.g. statuses it receives a Delete for. That creates significant 'garbage' of course, which I don't think exists as such in ATProto.
(Sidenote: That technically means tombstones have to be keyed on object id and owner in order to prevent denial of service attacks via tombstone spam against Actors whose object ids are predictable.)

I think it also asynchronously fetches the target of an Announce if it doesn't have it already, but I think in bridgy's case it would probably be better to create a trigger with a timeout somehow that resumes processing in case the post is in fact bridged, unless it's a self-boost. (Can that even reliably be distinguished without fetching the Announced post, though?)

Related: #1338