Help:Extension:Kartographer/OSM

How does Kartographer relate to OpenStreetMap ?

Kartographer is a MediaWiki extension that allows Wikitext editors to include a map in a page. This map is generated by Wikimedia's map server using OpenStreetMap data.

The graphical design of this (base layer) map is called the style and determines what OSM data is shown in the map tiles (like a filter), but also the colors and graphics used to render this data. While the style of the map is similar to the one by OpenStreetMap, it is specific to Wikimedia but based on what is called OSM Bright.

In addition to the base map, Kartographer allows editors to draw highlight shapes and markers on top of the map. These shapes can be from several sources, but are generally retrieved from OpenStreetMap. This makes use of the Wikidata information that OpenStreetMap keeps track of.

If you want to correct a problem in a map, you might have to become an OpenStreetMap editor or add a Wikidata id to the OpenStreetMap data. Keep in mind that OpenStreetMap generally only accepts existing and therefore no historical objects in its maps. This data must be provided by other means. Then you will have to wait for that data to synchronise back to Wikimedia (generally within a day). Lastly, it might be that some part of the system has cached old data. Because of this it can sometimes take quite a while before a change in OpenStreetMap is reflected in Wikimedia.

How to use objects from OpenStreetMap

You can draw polygon and way objects (regions and lines), such as countries, roads, or parks from OpenStreetMap directly on maps. To do that, the OSM object must be tagged with the Wikidata ID. Wikipedia maps do not support the points (nodes) yet.

Bryant Park, New York

For example, to highlight Bryant Park in New York, you need to find its Wikidata ID number. The easiest is to go to the corresponding wiki article, Bryant Park, and click on "Wikidata item" in the toolbox on the left. In some rare cases, there may not be a corresponding Wikidata item, in which case you should create it and add the Wikipedia article to it.

Once added (see "how to" below if it's missing), you will be able to use the "geoline" service for lines, "geoshape" service for polygons, and "geomask" service to mask-out everything except the needed polygon.

<mapframe text="Bryant Park, New York" width="300" height="300" latitude="40.75333" longitude="-73.9827" zoom="15">
{
  "type": "ExternalData",
  "service": "geoshape",
  "ids": "Q995174",
  "properties": {
    "title": "Bryant Park",
    "description": "[[File:2011-NYC-Bryant-Park-Library.JPG|300px]]",
    "stroke": "#125123",
    "stroke-width": 3,
    "fill": "#2aaf4d"
  }
}
</mapframe>

How to add Wikidata ID to OSM

Go to osm.org
If you don't have an account, click "login" (upper right corner), and click the Wikipedia icon - this will allow OSM to use your Wikipedia account.
Using the search box, find the object you need in the left pane, and click on it.
Once the object is shown in a bright orange line, check if the "wikipedia" and/or "wikidata" tag is there. If both are there, you are done.
Zoom in onto any part of the orange line all the way to the ground.
Click "edit" button at the top. Be careful not to click anything in the map.
The left side of the screen should now show "Edit feature" box. Scroll down to "Add field:" box, which is right above the "All tags (..)".
If there was no wikipedia and no wikidata tags, click "add field" box and select Wikipedia. Select the language and copy/paste the article name. This will automatically add the Wikidata tag as well (you can check them in the "all tags" section).
If there was a wikipedia tag, but no wikidata, you can either delete and re-add the wikipedia box, and simply add the "wikidata" tag by hand in the "all tags" section.
Click Save, and enter a good change description.
The data should become available for Wikipedia in about 1 day (we are still working on making this quicker)

How to get a single object from multiple lines

Often, a group of lines correspond to a single item – such as a road that is made up of multiple ways. Each "wikidata" tag should have a unique value, so therefore the individual lines need to be grouped into a single object. The following instructions assume you want to create (or find) a single object for a road, but should work (with minor variations) for other types of lines.

Terminology

node: A node is a single point on the map
way: A way is a line that connects two or more nodes
relation: A relation is a group of ways which make up a route

Limitation

Tracked in Phabricator
Task T156433

Currently, only relations with type=multipolygon, type=route, type=boundary, type=waterway will display on maps.

Process overview

In OSM, find or create a relation that covers the entire road
Add a Wikidata item ID (Q-number) to using the wikidata tag (if not already present)
Wait 1 or 2 days for the database to be processed

Process details

Go to osm.org
Move the map so you can see the road you want, and enter the road name into search box
Results (ways) will come up in a pane of the left of the screen. Find a way which is part of the road you want, and click it. The pane will then load details for the selected way, and it will be highlighted on the map.
Scroll down the pane to the bottom, and look at the section "Part of". If there are any relations listed there, click on them to check if they cover the whole road. If you find one you want, good; otherwise you will need to create one (see 'Create a new relation' below)
Click "edit" button at the top. Be careful not to click anything in the map. The left side of the screen should now show "Edit feature" box.
Scroll down to "Add field:" box, which is right above the "All tags (..)".
- If there was no wikipedia and no wikidata tags, click "add field" box and select Wikipedia. Select the language and copy/paste the article name. This will automatically add the Wikidata tag as well (you can check them in the "all tags" section).
- If there was a wikipedia tag, but no wikidata, you can either delete and re-add the wikipedia box, and simply add the "wikidata" tag by hand in the "all tags" section.
Click Save, and enter a good change description. The data should be available to Wikipedia in about 1 day.

Create a new relation

If there is no existing relation for the road you want, you'll have to create a new one. There are multiple ways do this: Potlatch 2 editor is good for shorter roads, JOSM editor is better for longer roads. Instructions for the iD online editor, see https://community.openstreetmap.org/t/succinct-steps-to-add-a-relation-in-id-editor/87669

Using Potlatch

Zoom in to the area around the road, then click the dropdown arrow next to "Edit", select "Edit with Potlatch 2"
Once Potlatch has loaded data onto the screen, use Ctrl+click to select the first way (road segment). It will show up highlighted on the map. Continue to Ctrl+click additional ways until the whole road is highlighted.
In the pane on the left, there is a table down the bottom listing any existing relations for the ways that have been selected. Below this table, click on "Add to". A window titled "Select relation" will pop-up, down the bottom of this window click the "New relation..." button.
At the bottom of the window, click advanced. A blank table of Keys and Values is shown – add the following keys/values to the table (click "Add" just under the table)
- name : name of the road
- type : route
- route : road
- wikidata : Q-number (the wikidata ID)
Close the "Edit relation" pop-up window, and save the edit. You can go back to https://www.openstreetmap.org and repeat Step 1 to verify the new relation you just created.

Using JOSM

Make sure your account has an OSM password associated with it: login with Wikipedia account, then go to your profile (click your name box in the top-right corner of screen), go to Settings, fill in Password and Confirm password fields, scroll down and click Save changes.
Download or launch JOSM from https://josm.openstreetmap.de/
Now, in JOSM, you need to download the ways that make up the road you want. There are two ways to do this (and you can also use a combination of both methods):
- Download existing relations which overlap with the road you want (e.g. Relation: 94 (317411) overlaps with most of w:Great Eastern Highway. Use the method described in 'Process details' above to find any overlapping ways, and note their object id (the number in brackets, also in the url following /relation/ once clicked on)
  Go to File -> Download object... -> enter object id (or ids), click Download objects
- Download areas containing the road. Be aware that the bounding boxes can't be too large, or the OSM server will not allow the download.
  Go to File -> Download from OSM... -> select area with mouse, click Download. Repeat if necessary.
Once downloaded, you should see lines representing all the ways along the route (as well as other data if you downloaded areas). Now you need to select all the ways that make up the road. There are two ways to do this:
- Search for items to add to or remove from the selection
  Go to Edit -> Search... -> Enter the name of the road in the search box. This can be repeated as many times as necessary, if multiple names are used along the route, or non-road items are selected (e.g. buildings along the road) – for the first search select "Replace selection", for subsequent searches, select "Add to selection" or "Remove from selection". Click Start search.
  
  Shift+click on items to add them to the selection, or Ctrl+click on items to remove them from the selection. Multiple items can be added or removed by holding down the click, and drawing a box that completely contains the items.
The selected items will be highlighted on the screen, and will be listed in a box on the right. Look through this list for items to remove with another search. Also look at the route on the main part of the screen for any missed ways to add with another search.
Once you're happy with the selection, go to Presets -> Relations -> Routes -> Road route. Enter the road name in the name field, click New relation. A more detailed window will pop up. In the Key/Value table, click into the blank space in the Key column, type "wikidata" (lowercase). In the Value field to the right, type the Wikidata ID (Q-number) for the road. Click Okay button.
Now upload the data to OSM:
- Go to File -> Upload data. A warning box may pop up - per the instructions there, "When in doubt ignore them" (or if you know what you are doing, you can fix them); click Continue upload
- In the next window, check that you're not accidentally deleting things, changing things, or adding anything other than the new relation you just made. Enter a meaning comment for the changes, for the data source you can specify "OSM data", then click Upload changes.
- Login with your username and OSM password when requested.
Verify that the new relation has been uploaded (follow the steps in 'Process details' above and the new relation should come up). Minor changes can be made using Potlatch if required.

Synchronization between OSM and Wikimedia wikis

Kartographer server structure in a nutshell

For performance reasons, Wikimedia wikis are not requesting OSM data live from OSM. Instead, the data from OSM is regularly copied to Wikimedia servers.

The map server
- contains all the logic and delivers the features needed to draw maps: e.g. definition of how Kartographer maps are styled, which parts of OSM data are not shown on Kartographer maps, how data and map features are put together, etc.
- contains a database with a set of geoshapes and geolines imported from the OSM data that has been linked with Wikidata items.
Server copies - The map data is copied to 5 servers, because one server alone would not be able to handle the amount of requests from the wiki pages. When a visitor opens a wiki page with a Kartographer map on it, the data for that map is being fetched from one of these server copies. A load balancer decides which server to pick based on server load and where the request is coming from.

This whole server structure is duplicated on two data centers: eqiad and codfw. This way, there is always a fallback when one data center needs to undergo maintenance, is broken etc.

Process for syncing between OSM and Wikimedia wikis

The most current copy of relevant OSM objects worldwide is imported once to the main map servers. What’s considered relevant is pre-configured, so as to not overwhelm the database.
This export is copied to all server copies.
These big exports are not happening regularly. What happens instead is that Wikimedia servers are fetching diffs from the OSM server every day, containing all the changes that were made since the day before.
With these diffs, the data on both Wikimedia data centers, and from there on all server copies, are updated daily.
If everything works as it should, the data on Wikimedia servers is in sync with OSM once a day. That means: If you make a change to an OSM map or connect an OSM shape to a Wikidata item, it can take up to one day until the change is visible on Wikimedia wikis.

Problems and their solutions	How to find out if there is a synchronization issue
If a diff can not be imported, all the changes from OSM from that day will not be synchronized to the Wikimedia wikis. In this case, maps are out of synch for everyone. This won’t be rectified with the next diff on the next day because any diff only ever contains changes from OSM within the last day. In the past, diffs were failing at times, so OSM and Wikimedia servers were sometimes inconsistent for several weeks. To solve this issue, teams at the WMF have synchronized OSM and Wikimedia servers anew (doing a copy from OSM as described above). Furthermore, they worked on to eliminating some issues causing the imports to fail.	see if there is a synchronization lag on the main map server on eqiad see if there is a synchronization lag on the main map server on codfw
Sometimes, data is not copied from the main servers to all server copies. This means that the data between server copies is inconsistent: From a user perspective, one person looks at a map and sees different data on it than another person, depending on which server copy was used to generate the maps. Similarly, people might see different data for the same map, depending on whether they are looking at a static map on the wiki page or the dynamic version in full size view. This is because when you add or edit a Kartographer map, this map is usually rendered and cached (on most wikis). This way, everyone sees the same map when they look at the wiki page, and the page loads faster for everyone. Full size maps, on the other hand, are drawn live every time someone opens the map. Depending on which server is sending the data to your browser, one might see different data on the full size map than on the wiki page with the cached result, and also different data than another person looking at the full size map. To solve these issues, all map servers were reset with the most current data. Also, updates and improvements were rolled out to make sure the synchronization process is more stable.	see if there is a synchronization lag on a server copy on eqiad see if there is a synchronization lag on a server copy on codfw

Synchronization errors can never be fully prevented, and a bit of a latency is considered normal. If you notice a latency bigger than 2 weeks, we encourage you to file a bug report on Phabricator.