A little surprise hidden in OS Terrain50 open elevation data

The reporting and discussion of a mountain rescue on the the minor Lakeland peak of Barf continues to generate a mass of things to investigate and follow-up. I've already written two OSM diary entries on assigning slope angle to paths because of it: here and here.

To recap, recently the Keswick Mountain Rescue were called out to help 3 people who were stuck ("cragfast") on steep ground on Barf. They didn't feel confident to retrace their steps back down very steep (30 degrees or over) scree and had reached a small crag with no obvious visible route through it. They had been using one of the mobile phone outdoor hiking apps which uses OpenStreetMap data for suggesting walks.

The event was reported in the national press (The Guardian) and, of most relevance, by a specialist magazine, The Great Outdoors. The latter did a great job of thoroughly researching a news article, getting comments from the OpenStreetMap Foundation. Their specialist advisor Alex Roddie decided to publish the content of his initial response to Carey Davies, the author of the article. Both articles are well worth reading for an in-depth initial response to the situation and the issues arising. There have also been lively exchanges both on Twitter and Mastodon, again containing useful comments from people with a lot of experience of the outdoor scene in the UK.

I won't comment here on the whys and wherefores of what kinds of paths and hiking routes should get mapped on OSM. The subject is complicated enough in Great Britain as all this discussion shows. One of the side distractions this did prompt for me was looking at how we could show contour data for OSM in the British Isles (and specifically on Andy Townsend's rural hiking map).

A toot by Nigel Parish did highlight that typically maps based on OpenStreetMap have contours which are greatly smoothed compared with the actual terrain. His example was from Andy Allan's Thunderforest Outdoor style which is incorporated into a number of hiking apps. Like most regular OSM contributors, I'm used to OSM-based maps using SRTM data to create contours: both because it's free and because it has more-or-less worldwide coverage. However, we are also aware of the deficiencies of this data: it has a relatively low resolution of around 1 second of arc (so roughly 30 metres resolution) which smooths contour lines and hillshade overlays.

I had recently downloaded the Ordnance Survey's own open data elevation model Terrain50, which I used for the first of my diary entries. This has a grid interval of 50 m, so although undoubtedly more accurate than SRTM, it is of somewhat lower resolution. I therefore expected contours generated from the DTM to be comparable with those of SRTM.

I'd forgotten that I'd used the vector dataset, so my first step was to generate contours from the raster tiles provided by Terrain50.

 

Barf with contours derived from the raster DTM of Terrain50.

After I'd done this, I remembered that the Geopackage was easy to use, so I added that. To my surprise the contours had rather more detail than those derived from ostensibly the same dataset.

With contours from Terrain50 vector data

Lastly, I created 10 m contours from the 1m Lidar data available from the Environment Agency for NW22NW. This, as expected shown both high resolution and accuracy.

With 10m contours derived from Environment Agency 1m Lidar.

To compare the three types of 10 m contours I changed colours and their background. Here are all three at a scale of 1:1000 on the steep SW slope of Barf between the Bishop of Barf and the crag which balks some walkers. In the image below it can be seen that the vector Terrain50 data is of much higher resolution than the raster data from the same source. In most places vector Terrain50 data is much closer in alignment with the contours derived from 1m Lidar data. As Nigel pointed out the more detailed contours give a significantly different impression of the shape of the hillside.

All three types of contours shown together. Red derived from OS Terrain50 raster; Orange: OS Terrain50 vector; and Green derived from Environment Agency 1m Lidar DTM. Scale 1:1000.

Nigel also pointed out that lots of relevant terrain features, shown on printed maps (Harvey & OS), and available as online tiles, were missing. This type of detail is often difficult to generate automatically using software, and even then is unlikely to match the output of skilled draughtsman working to detailed composition rules. I tried adding a hillshade layer from EA data, but it didn't really pick out this detail. Then I remembered that Luke Smith of Grough had used a layer provided with OS VectorMap District called "Ornament" when he prototyped maps based on open data. I therefore added this layer. It works quite well at low zooms, but is rather ugly at anything higher than 1:5000.

Added hillshade from Environment Agency data and Ornament from OS VectorMap District open data.
Hillshade is less effective than I expected at high zooms, and the ornaments only really work in a range 1:5k (as here) to 1;25k or thereabouts.

Lastly, and more for fun than anything else, I regenerated contours at 15 m intervals and clipped these by scree layers mapped on OpenStreetMap and changed the colour of the contours to a shade of grey to replicate how this information is shown on the Harvey Map. The end result is a good illustration of these differences: the Harvey Map generalises features such as cliffs and places them to enhance the map users' impression of the terrain, whereas the OS ornament looks nice, but is nothing like as good a guide for the walker.

With 15 m contours (from EA 1m DTM) and with contours on scree coloured grey rather than brown (as done by Harvey Maps), plus OS VMD ornament.

The bottom line from this is quite simple: don't bother with Environment Agency Lidar data if you just want contours: Terrain50 vector data is much easier to get up and running. The data is also packaged as mbtiles, but I couldn't find a way to style this in QGIS.

Lastly a big thank you to Nigel Parrish who caused to to look at the data, and thus discover the difference between the vector & raster versions.

Bristol (& New Brighton) Buildings from Lidar

West Front, Bristol Cathedral
One of the buildings where we could use Lidar data to enhance its representation in OSM

At OpenData Camp 3 over the weekend I asked John Murray if he could give me a set of polylines extracted from the Environment Agency Lidar Open Data. Do read John's amazing post about the tools he has built for doing nifty things with Lidar data: Turning Lidar Data into Actionable Insight.

I thought if might be a bit of fun to actually show directly how opendata produced by one of the ODCamp sponsors might end up in OpenStreetMap.

In practice John got keen over the lunch break and wrote a bit of code to turn his polylines into polygons. So just before the last session of the day kicked off I had been sent a shape file for the 1km Ordnance Survey gird square where the meeting was taking place.

We had one initial teething problem that the data was out by 1 km, but notwithstanding that it was very simple to perform some simple manipulations in the off-line OpenStreetMap editor JOSM.

JOSM can read shapefles (and some other geo-formats such as geojson too) and automatically transforms these into OSM elements projected in WGS84. Therefore the additional data manipulation steps were pretty trivial:

• Select all way elements (using a type:way search)
• Add a source=EA Lidar Open Data tag
• Select all type:relation elements to find multipolygons
• Add building=yes tag
• Select all way elements which were not part of multipolgons (type:way and not child type:relation)
• Add building=yes tag
• Select all way elements again and simplify them.

 The image below shows how this looks in the editor

Now this is all pretty amazing, and if you read John's blog post there's lot more info which can be gleaned. However a close inspection of the data still shows a sizeable number of artefacts which would need cleaning up. Some John has dealt with in the intervening few days, but turning any automatically extracted feature into something of the sort of quality which can be one in OSM is another matter: and is not too dissimilar to the points I made some time ago about OpenMap Local.

For me the real advantage is that it's a major step in making if more feasible to use Lidar data to enrich OSM data. For instance data on roof orientations could be combined with the algorithms & crowd-sourced validation methods from OpenSolarMap. I hadn't realised until listening to John's talk just how valuable gable or eaves heights are in building datasets. It certainly persuaded me that they belong in OSM.

Another downside is that it takes a whiz like John to create this software and it makes use of a powerful machine, powerful algorithms, optimised hardware & proprietary storage. I have therefore spent a little time this week looking again at what is available in QGIS to do similar (but much less powerful) manipulation of the Lidar data.

Basic transformations of Lidar have been described elsewhere (for instance see Chris Hill's posts) so I won't dwell on them here. Suffice it to say I presume that the following have been created for a given area:

• Combined Digital Surface Model (DSM). I usually do this as a virtual time set (can be done directly in QGIS)
• Combined Digital Terrain Model (DTM). As above.
• Delta of the two. DSM-DTM. This gives things (buildings, cars, trees etc) which are elevated above ground level.

To get somewhere near what John's approach involves ideally requires:

• Filtering out shorter objects (mainly cars, garages & some street furniture)
• Filtering out smaller objects (mainly trees)
• Edge detection
• Polygonisation

In practice I found it relatively easy to do the first & last and did not find a simple way of doing the other 2 in QGIS (although in part that might be because I'm short of disk right now).

The other two can be achieved easily:

• Filtering by Height: this is merely another raster calculation using the QGIS Raster Calculator. In my test area (New Brighton on the Wirral, OS grid ref SJ3093) most houses are Edwardian and much higher than 3 metres, whereas garages are usually a touch over 2 metres. I therefore used 3 metres as a cut-off.
• Polygonisation. I used the height filtered data directly with the Raster...Conversion...Polygonize option in QGIS. This is a much cruder and more naive method than I was hoping to use, but there it is.

I show the results of these steps below (in separate images to allow easier inspection & then combined).

Lidar Height data (DSM-DTM) filtered for >3m

Extracted & OSM Building polygons compared
(garages are deliberately excluded from OSM data)

Height data combined with Polygons

Firstly it's worth noticing a few features from the raw height data:

• Most buildings are tall, usually in excess of 8 metres (and probably at least that height at the gables).
• There are a limited number of lower height buildings. The most obvious ones are near the top of the image and include two small factory premises N of the railway & the platform canopy of the railway station. S of these the road bridge over the railway is obvious; and immediately to the SE there are apparently two largish buildings of low height, albeit quite a bit of noise in the height profile. (These are, in fact, Victoria View a development of flats which halted for several years). Further S still there are a small number of bungalows.
• Terraces with a lower rear service area are obvious.
• There are a significant number of linear features above 3 m in height. Most look to be walls, and indeed garden walls in the area tend to be high as most gardens are small & given building heights would tend to be overlooked.
• Isolated trees are obvious in one or two back gardens
• Larger groups of trees are equally obvious along the railway line (& elsewhere)
• Swirly patterns in the 3-4 metre range occur in a number of places. These are mainly scrub (mainly gorse) or shrubberies.
• There are still parked vehicles giving returns in the 3-4 metre height range. 

Edwardian Streets S of Mount Road New Brighton (Dovedale & Langdale Rds)

I include a couple of photos of streets in the area to help with context. I would recommend strongly Russ Oakes' work documenting suburban streets all over Merseyside for a much broader perspective.

Junction of Dudley & Hamilton Roads, New Brighton

Comparing the extracted polygons with OSM (and ignoring some OSM data which is missing) shows:

• There is a fairly constant offset of OSM data (presumably inherited from the Bing imagery).
• Building footprints are broadly comparable
• Small gaps in terraces are resolve much better by tracing.
• Some detail has not been added to some of the terraces in OSM which are still drawn as plain rectangles.
• It's certainly possible to spot missing features & use Lidar data as an aid to add them in (notably Victoria View, but the W part of the development was started after the Lidar data.

Now as for deriving data to enhance OSM there's a fair more bit of processing needed.

Absolute building height is relatively easy, one just needs to find the maximum height within a (location corrected) OSM polygon. Generating the other more useful Simple 3D building (S3DB) tags is rather more involved, and certainly I have the impression that QGIS would be a fairly clunky way to do things. I really hope that some more technically-minded OSM folk can take inspiration from John's ideas and start thinking about tools to mainpulate Lidar data specifically for OSM.

There is no doubt that the Environment Agency Lidar data was one of the most significant open data releases last year. Furthermore it is likely that other agencies & local government bodies will make Lidar available more widely in the near future. For instance I believe much data from Kanton Zurich is open, including Lidar. This example shows the extensive slumping caused by peri- and post-glacial phenomena in the woods near Bergietikon: so this is a reminder that it's not just buildings which are of interest.

One last thing to note is that there's lots one can do with this data immediately (the subject of John's original article). Working how to add this data to OSM begins to look not dissimilar to creating authoritative datasets. It's of course worth spending time working out how to do this because once in OSM the data is potentially available for a multitude of purposes.

UK Open Data and Buildings in OpenStreetMap

I've finally (after 8 months) got around to looking at the OpenMap Local buildings. This new dataset was launched at the first OpenDataCamp, and I've had the SU 100 kilometre square data on the PC since then (it's contains Southampton, where Ordnance Survey are based). I use Meridian 2 OS Open Data regularly and extensively, but these days don't make much use of the larger scale vector data.

Comparison of Building polygons for Central Nottingham
OSM has more detail and does not merge discrete buildings.
Contains Ordnance Survey data (c) copyright and database right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Image CC-BY-SA, the author.

I needed them for something else which caused me to download the SK data. Co-incidentally Christian Ledermann had asked on talk-gb about using this data to add buildings to OpenStreetMap for Newark-on-Trent. A little earlier the Environment Agency had released Lidar data for England, and this is also useful as input for mapping buildings.

OpenMap Local

Apart from the area I originally needed which were in SK41 (no buildings in OSM), I've also looked at areas which I know much better & compared some selected areas where we have good building coverage around Nottingham. The comparisons I made are shown visually, with my main observations summarised at the end. Note that comparisons have not been made on any systematic basis.

University Park, University of Nottingham
an area of predominantly large academic buildings.
OpenStreetMap and OS OpenMap are largely in agreement: the minor differences applying to newer buildings which post-date the Bing imagery.
Contains Ordnance Survey data (c) copyright and database right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Aerial Imagery via Bing, (c) as in image. Image CC-BY-SA, the author.

The Science City part of the University Park campus.
A new large lecture theatre block is not present in OpenMap data, and the outline of the building top centre (Tower Building) is over-simplified.
Contains Ordnance Survey data (c) copyright and database right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Image CC-BY-SA, the author.

Central Newark. OpenMap vs Lidar.
Many instances of building merging & over-simplification are apparent here, notably with the outline of the parish church.
Contains Ordnance Survey data (c) copyright and database right 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Image CC-BY-SA, the author.

Newark-on-Trent, residential areas, Showing inconistency in size for similar houses, and merging of terraced housing.
Contains Ordnance Survey data (c) copyright and database right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Image CC-BY-SA, the author.

I have not made systematic comparisons, but these are my main observations (in brackets the 1km grid square where I've noted any particular issue):

• Best for larger buildings. The data seem much more reliable (actually matching building footprints fairly well) for larger buildings. Even for large detached houses I would regard the data as unreliable: on our road of 40 detached houses, at least 16 are represented as terraces (SK5439). Similar artefacts occur in other areas with detached houses: apparently caused when a garage is close to both houses. Smaller houses are inherently simplified: no better than drawing one in JOSM and then copying the outline in fact.
• Building fusion. This is particularly clearly seen in the city centre image, where a whole block of buildings has been simplified to a single building (centre of image), but also occurs in suburban housing (see above).
• Inconsistency in geometry simplification. This is most noticeable in the city centre. (SK5739). For instance compare the OSM and the OpenMap Local outlines for St Peter's Church (bottom right in map above). In OpenMap Local the church is just shown as a rectangle, whereas in practice it is more complex. Modern buildings on the Jubilee Campus of Nottingham University are generally shown with more detail.
• Inconsistency in building size. In SK5439 there are a very large number of houses which were identical when built. However, in the OpenMap Local they are often of different sizes. (This is also probably true of OSM, if buildings have not been created by duplication).
• Voids. Gaps between closely packed buildings in the city centre appear slightly arbitrary in both placing and whether such a void exists or not.
• Some selection inconsistency with small size buildings. Only 2 garages are shown in an area of around 500 houses. With OSM the figure is nearer 200+. (SK5439)
• Demolished buildings. Whilst I would not expect the data to show the building demolished in the past month, I would expect it to not show one demolished 2 years ago, and I would certainly expect it not to show one demolished in 1970 (although MasterMap shows this too). (SK5439)
• Better locational accuracy. If using the full transform it may be useful to take advantage of the better locational accuracy of this data. In the main OSM buildings are rarely more than 3 m displaced from the OS OpenMap Local. (SK5439) In general the more recently mapped buildings in Nottingham city centre have better locational accuracy than this (SK5739).

Taken together, my use of this directly within OSM would be along the following lines :

• Selective transfer of larger buildings (schools, offices, public buildings, factories, warehouses, larger shops) on a case-by-case basis from a shapefile to a JOSM editing layer, or to Potlatch 2. Some minor refinement will probably be needed (for instance a university building here has long narrow courtyards which act as light wells which are not shown in OpenMap Local.
• Only use it for houses and similar when shapes are very simple and everything has been double checked, at the very least, against aerial imagery. For simple shapes it's as quick to draw & copy in JOSM anyway. A similar principle holds for more complex building shapes on modern estates, where one building can be cloned.
• Watch out for demolished buildings. This requires not just checking against Bing/MapBox imagery, but some local knowledge for sense checking.

Environment Agency Lidar Data

Another source of building data is the recently released Environment Agency Lidar data. This does not cover the whole country, and in many places may only be at 1 or 2 m resolution. It may also be quite old. However, because it does not suffer from parallax artefacts it can be used in conjunction with both aerial imagery (whether from Bing, MapBox or more local sources) and OS OpenData. I have provided examples from Nottingham, Newark, and Melton Mowbray of this data, combined with one or more of OSM buildings data, OS OpenMap or Bing aerial imagery.

Melton Mowbray. EA Lidar DSM (1m) overlaid on OSM.
The Lidar data was used to refine the OSM building outlines
which originally were traced from OS StreetView as block-sized polygons.
(see commentary)

Melton Mowbray illustrates many of the benefits of Lidar data. It is a fairly typical country town, with many of the buildings in the town centre ranging in age from 10 to 500 years old. Many extend back from the street in a series of outbuildings (e.g., stables) which have eventually been incorporated into the main building, but this process leaves lots of small courtyards, service yards, etc which are more or less impossible to discern on aerial imagery.

Butter Cross in Market Place, Melton Mowbray
Despite the different styles & ages of the buildings, several have long ranges at the rear.

By doing a street-level ground survey one can identify which buildings are distinct on the street front. Lidar than helps to construct a building outline which is consistent with this. I surveyed the cetre of Melton in September, and this was the first place where I used Lidar data to aid in the interpretation of aerial imagery. In this case I find it essential to have adequate street level pictures to be able to relate to the aerial imagery: most useful are the presence and distribution of chimneys: because they throw shadows they are often visible even on poor quality imagery.

The Lidar data also allows one to do some other things: notably find building heights. I've done this for a 1980's estate on the edge of Maidenhead: particularly easy as the residential buildings fall into a small number of categories: bungalows, two-storey-houses & maisonettes (purpose built flats in a house-like structure.

A 1980s housing estate with building heights mapped from English Environment Agency LIDAR Open Data. Buildings fall into 3 height categories: bungalows (green: approx 4m high), 2-storey houses of various kinds (blue: approx 6 m high), and maisonettes (condominiums) which are about 7 m high (red). Heights were calculated in m, so the values represent minimum heights of the highest part of the building, which is nearly always the gable line.
Outpur via Overpass Turbo, styled with MapCSS.

There are many other useful blog posts about using this Lidar data, both specifically for OSM, but also generally. See posts by Chris Hill ("More Lidar Goodness" and "Building Heights") and Ed Loach for some of the specifics, and the write-up on the wiki. A nice post and map (v. slow in my browser) showing building heights in London on OpenMap Local may also be of interest. HousePrices has processed all the Lidar data from EA and Natural Resources Wales  as a hillshaded slippy map which is useful to look at what is available. Slightly unfortunately the map is in OSGB projection (ESPG:27700) and is not shown with other slippy maps which would make it a bit easier to locate oneself.

What kind of building data should be added to OSM?

From past experience single building outlines traced from OS StreetView, turn out to represent tens of buildings on the ground. Such simplified outlines just makes the work of splitting the buildings properly quite a lot harder. This can be particularly bad in town/city centres.

Usually if adding detail of POIs and addresses it is important to have individual buildings mapped: this makes it much easier to correlate photos to roofline features such as chimneys, gables etc. A single very simple outline may be OK, because for more detailed mapping it should just be a question of deleting the original outline. However, the question must be asked, as to what purpose such an outline fulfils on OSM, when the source data can be readily combined with OSM data for downstream consumption.

Granby Street, Leicester.
The multiple buildings shown here are represented in OSM as single buildings for each block
(imported from OS StreetView Open Data).
CC-BY-SA   © Copyright Malc McDonald and licensed for reuse under this Creative Commons Licence.

I think the fundamental question about straight imports into OpenStreetMap should be "Will it make life easier or harder for subsequent mappers?".

If the work involved refining a building outline takes longer than re-drawing the building then I doubt if its worth importing the building at all. This is particularly true if the outline is actually of multiple buildings. This is why large building outlines are most valuable: they are generally pretty good compared with what an initial hand-traced outline might look like, and they lend themselves better to stepwise refinement. One group of buildings I find particularly tedious to do well are schools which tend to be a sprawling mass of interconnected buildings. Starting with a decent polygon with orthogonalised angles make adding such detail much easier. The current quarterly project for UK-based mappers might be the time to test this.

Of course it may be that adding buildings assists in some other mapping goal. I've already mentioned that details of buildings are very useful for addresses. However OpenMap Local lacks the detail in precisely the areas where it would be most useful (city & town centres). For suburban or inner-city housing similar polygons can be created as quickly in OSM editors (notably in JOSM, by duplicating existing buildings or using the Terracer (or even UberTerracer) plugins.

The other thing which many people want is rendered maps largely derived from OSM, but showing more buildings. In practice, because many mappers do not have the know-how, wherewithal or time to create such a rendered view, they tend to want to import buildings. Historically, OSM tools for importing data are often much easier to use than ways to incorporate the same data and OSM data to  render maps and make them accessible on the web. Perhaps we need to do more to help people in the latter task: which is now getting more complicated again with the move to vector tiles (at least outwith use of MapBox Studio), and TileMill's effective status of being a legacy application.

Summary 

Sadly, although the new building outlines are better than what preceded them, in most cases they don't offer a decent route for iterative refinement with OpenStreetMap.

This absence of a simple way to improve building outlines means that ideally people wishing to use this data would merge it with OSM data outside of OSM. I do recognise this is often too much work, or too big a learning curve for many, and consequently there will always be a desire to add buildings to OSM because many people are much more comfortable with consuming only OSM data for their purposes.

Existing tools for drawing buildings in OSM are pretty powerful & getting more powerful all the time. Many of us, and I include myself in this group, are unaware of the full extent of these utilities. See bdiscoe's diary post about mass adjustment of circular buildings (huts) for some insights.