Should mangrove forests be within the coastline?

I doubt there is a convention concerning coastlines and mangrove areas. Ideally, the coastline is based on tidal extremes, officially the Mean High Water Level. When I'm mapping in Alaska or Thailand those levels simply are not known so my coastline alignments are somewhat arbitrary. That's perhaps why sometimes a mangrove area will be found outside (or inside) the coastline.

Reiterating, in my experience, coastlines are often drawn without actual knowledge of their Mean High Water Level and are therefore only approximations. In other words, I think it's mostly the coastlines that are the culprit here.

I hope this helped. Other mappers may have more to offer on your issue.

Mangroves are a class of trees made up of many species.

What they all have in common is that mangroves will only grow if their feet are in water. Or in some cases only if their feet are wetted regularly by water at high tides. If this is not the case, no mangroves will grow.

Coastlines are conventionally demarcated by high water on a average spring tide. Your country might be different - make sure to check. Some high spring tides are exceptional, some are underwhelming, but the coastline will normally be placed to align with the average high spring tide. Sometimes the line is called HWOST (High Water Ordinary Spring Tide).

So, the logic is that mangroves want to be permanently wet, or at minimum want get wet on high tides, so they always grow on the seaward side of the normal high tide mark, and certainly on the seaward side of the high spring tide mark.

In some cases the high spring tide is not good enough a wetting for the mangroves and they will leave a little sand flat or mud flat between their landward edge and the shoreline.

So, I always put mangroves on the seaward side of the shoreline. If you see a little sand or mud flat area inland of the mangroves then that too is probably seaward of the shoreline.