Grants talk:IEG/Wikimaps Atlas

Congratulations! Your proposal has been selected for an Individual Engagement Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, $12,500

Comments regarding this decision:
We look forward to seeing the base maps and a workflow that makes it easier for Wikimedia’s cartographer’s to create and update these important visual resources.

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Questions? Contact us.

Great start

This looks like a great start on a really good idea for an IEG, guys! Let us know if you have questions about creating the rest of the proposal before the deadline of 30 September. If you click the button on your page it will walk you through making a project plan, but I'm happy to discuss if you get stuck anywhere. Cheers, Siko (WMF) (talk) 16:01, 20 September 2013 (UTC)

Thanks Siko. We have been talking about that in real world. Need to write down now. Cheer ! Yug (talk) 18:49, 24 September 2013 (UTC)

Where do you get basic map data?

Can you point out where you explain where you get basic map data? What data is freely available, what differences are there in free sources, how do you commit to a source, are all sources equal, and is there any particular controversy associated with choosing a source?

I am having trouble understanding the scope of this proposal and do not know where to begin, but I think I want to know the source of all this data and am not following how available or authenticated any of this may be. Blue Rasberry (talk) 15:46, 1 October 2013 (UTC)

en:Wikipedia:Graphic_Lab/Resources/Gis_sources_and_palettes.

For relief, this project will mainly reuse the NASA topographic rasters. ETOPO1 (2009, Public domain), SRTM3 (2000, Public domain).

For administrative areas, this project will reuse the naturalearth (2009, Public domain) and/or gadm.org data (CC-by-sa-3.0 for SVG representation).

Imperfect demos have been done with ETOPO1 and NaturalEarth's data, see http://bl.ocks.org/hugolpz .

I think these sources were cited within the proposal. Seems we should add more details about the in the proposal. Yug (talk) 20:04, 2 October 2013 (UTC)

gdam data is somewhat generalised, making it unsuitable for larger-scale mapping. This is noticeable when you drill down to 1:1,000,000 scale - which will affect many level 2 areas. In some cases, the level 2 area is approximately that of a city (c. 1:100,000 scale?) then gdam is garbage.

Far superior sources are available in some regions (eg if the national mapping agency supplies data freely) and OSM generally incorporates these. IMO its better to create large-scale maps only for those regions with the high-quality data that can support it.

This will necessarily affect the scope of project, as some areas just will not have sufficiently good data to produced desired output.--Nilfanion 11:04, 19 October 2013 (UTC)

I just checked gadm.org for Germany. Even http://gadm.org/problems is a problem. Instead of one Land that is said to be wrong (since 2007! although it is quite simple to get correct data for Germany) it is four Länder with incorrect data, so 1/4 of all Länder. And checking the kmz file of German boundaries with Google Earth (which has got quite good information about that it) showed several differences: a really strong generalization and lines that are shifted several kilometers. For larger scale maps this is a problem. I agree with Nilfanion: garbage for detailed maps. NNW (talk) 11:05, 28 October 2013 (UTC)

Basic informational unit

What is the basic unit of information with which you work to map a region? Once you have a region defined, how do you map that anywhere? How is mapping on Wikipedia different from mapping anywhere else? Blue Rasberry (talk) 15:46, 1 October 2013 (UTC)

We don't need roadmaps, we need encyclopedic maps : historical maps (typically: historic states, state expansion, wars, battles), descriptive maps (typically: geology, migrations, animal distribution), gradiant maps (typically:vote). These maps need solid base maps such relief_location_map backgrunds -which often explain history, biology-, or like administrative location_map background. Yug (talk) 19:53, 2 October 2013 (UTC)

Possible improvement to the proposal

1. Grants:IEG/Wikimaps_Atlas#Budget_breakdown: we may let appear the data cleaning (GIS to Geojson to optimized Topojson) vs visualisation system development (Topojson to D3js to SVG).
   • The right GIS files, layers, should be used, with the right filters to keep only the right bytes of data. The question of Topojson optimization will be important as well to generate right-sized SVG.
   • When the data is structured properly, the D3js part is "simply" to project the data according to a style. SVG optimization include: data to svg fit, granicule (grid of parallels/meridians), reprojection ?, color-ramp / legend, scale,
2. #Scope section: I make a full background explanation about map concentions. Maybe a short bullet list would be better, no need to details further than "we will use the lasted best map styles and best practices available there (en), there (de), there (fr)."

Yug (talk) 10:18, 3 October 2013 (UTC)

Eligibility confirmed, round 2 2013

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for round 2 review. Please feel free to ask questions here on the talk page and make changes to your proposal as discussions continue during this community comments period.

The committee's formal review for round 2 begins on 23 October 2013, and grants will be announced in December. See the schedule for more details.

Questions? Contact us.

Siko (WMF) (talk) 05:53, 4 October 2013 (UTC)

Minor wording feedback

I understand the project timeline is out of date. You may want to update the months names to fit the current time of year. Gryllida 09:58, 13 October 2013 (UTC)

We have planned the work over the months of March-May 2014 based on our personal availability. This seems to be within the stipulated grant period --Planemad (talk) 15:02, 18 October 2013 (UTC)

Some explanation

I have corrected some of the spelling mistakes in the text. One more question. What is meant by the phrase:" Project management and status: etherpad and meta" Wereldburger758 (talk) 09:32, 14 October 2013 (UTC)

I'll remove that. Meant to say that we would keep updating the status of work using ether pad, but I guess it doesn't need to be in the proposal. --Planemad (talk) 14:39, 18 October 2013 (UTC)

Scope and activities

@Planemad & others willing to help: I explain in this Scope&activities section the current situation of maps within Wikipedia. However, the hiden comment in the code say :

 == Scope and activities ==
 < !-- What will you spend your time doing if this project is funded? What will be completed by the end of the project?-->

Seems we clean up the section to put a kind of to do list, no ? Yug (talk) 21:26, 20 October 2013 (UTC)

The Grants:IEG/Wikimaps_Atlas#Timeline section covers how we will be spending our time and the final deliverables. Overall the structure looks good as seems to cover all aspects --Planemad (talk) 18:12, 21 October 2013 (UTC)

Weird "Mo, Go, ko" units.

Single GIS files are between 50Mo and 5Go... a final SVG background of about 100~500Ko each

I've never heard of these units. Do you mean bytes? in which case the unit is B and these should be 50MB and 5GB, and 100~500KB. -- S Page (WMF) (talk) 09:00, 21 October 2013 (UTC)

S Page (WMF) : Yes, en:Bytes. French speakers (as I am) name them "octets" to differentiate from non-8-binaries based bytes (001001: is a 6 binary byte, not an octet). We thus use Ko, Mo, Go in France. I corrected it. Yug (talk) 21:22, 21 October 2013 (UTC)

Where is i18n / l10n ?

m:Wikimaps talked a lot about translating place names. Your proposal doesn't, except to say "The gadm.org GIS administrative data provide the English name of each item." So how do I make a Russian or Chinese version of a map with labels, like m:File:Hawaii_Island_stratigraphic_map-fr.svg? 1) Wikidata is one place to look for place names in other languages, e.g. https://www.wikidata.org/wiki/Q159762#sitelinks-wikipedia will tell me "Mauna Loa" in Cyrillic; but I don't think Wikidata is envisioned to have names of every geographic feature on Earth (e.g. it lacks the caldera "Moku'Aweoweo"), and who/how will code every place name in GIS datasets to Wikidata items? 2) An alternative is a TranslateWiki workflow to translate strings in particular GIS datasets between languages, but at what level would this operate? 3) That SVG on commons links to Commons:Translation_possible/Learn_more which explains how someone can edit an SVG to create a translated version, but manual translation of automatically-generated maps doesn't seem sensible. -- S Page (WMF) (talk) 09:30, 21 October 2013 (UTC)

P.S. when you write "bidden" I think you mean "bound to data" :)

We are not particularly focusing on the l10n aspect with this project, as we see the translation in the future to be an automatic process like you mentioned. You can see a prototype of such a possibility with this hack I made using Wikidata [1]. As far as i18n goes, the SVG outputs will be standards compliant and we can embed useful metadata into the element id's so that it can be localized easily in the future --Planemad (talk) 18:06, 21 October 2013 (UTC)

We do have automatic translation in mind as a later possibility and objective. We don't develop it further because it may actually be tricky, so the workload is not clear. What we can say is that given an English administrative name provided by GIS resources, we can request Wikidata API a translation which it likely have. Example : Florida (admin entity level -1), Plam_Beach_County, Florida (admin entity level -2), or even my lost home place Montpouillan (France, admin entity level -5). Each providing about 20 interwikis translations that we can grasp and reinject within a map, as for Planemad's wiki-atlas. Technically, we need :

1. well coded SVG maps with administrative entities names in English
2. the administrative entity needs an a English wikipedia entry (preferably)
3. edit a bit Planemad's code.

We have point 2 for most major administrative entities such countries (L 0), provinces (L -1), and subentities ( L -2), but the data may present variability between gdam and wikipedia, typically "Palm_Beach_County" VS "Palm_Beach_County,_Florida,_United-States_of_America", and this with irregular pattern, so the workload is unclear for now. Point 3 is easy to do, something like one day of dev / test. Point 1 is the objective of this grant request. Yug (talk) 21:35, 21 October 2013 (UTC)

I've been involve with the {translation possible}, the graphic lab and myself requested the tool which help to translate SVG maps. We also noticed this approach to be a poor answer, since it duplicate the background image of a map, it duplicate the file, the making the whole difficult to maintain. One well coded SVG with suitable anchors (id), and a system such Planemad's wiki-atlas / wikidata would be better. Yug (talk) 22:06, 21 October 2013 (UTC)

Rather than importing the names of equivalent wikipedia articles in other languages from Wikidata you would probably be better importing the Labels from the wikidata items as these will have been cleaned up a little to delete the disambiguation additions some labels have. These multilingual labels are not obvious when you visit a wikidata item, as it only shows you the label in your own language, but there are slots for labels in 280 different languages for each wikidata item. Filceolaire (talk) 13:12, 23 October 2013 (UTC)

Filceolaire, this will be an useful trick. I was looking for a way to clean up the sitelinks from these disambiguations. :) Yug (talk) 13:21, 24 October 2013 (UTC)

I would totally support this proposal if the multi-lingual aspects could be taken into account. It would be awesome to have maps that could be translated like svg files. I was sort of hoping we would have a collaboration with open street maps, but I guess that is pretty much impossible. Jane023 (talk) 22:32, 28 October 2013 (UTC)

As I said upper, our SVGs will be witty, well structured XML. Each given shape will contain a data element such data-english-name="London City". The final syntax is not know yet, we could need names such data-english-name="England, London", but this English entry should allows semi-automatic translations / multilingual maps for most objects (major places / administrative divisions / rivers). For something more solid at deeper zoom, it would be need to synchronize these English names with OSM items, which is a deeper project. Yug (talk) 18:13, 29 October 2013 (UTC)

Most maps will correspond to an item on wikidata with a unique item number and a bunch of labels in different languages. Where there isn't an item it will probably be appropriate to create one. Maybe these item numbers can be used. In some cases however one wikidata item will have more than one map - borders and river courses and animal extent maps change over time so the wikidata item number on it's own won't be enough. Long term the solution is the proposal for each Commons file to have a wikidata type data page, with statements describing the properties of the file and queries replacing the Commons Categories. I'm assuming the SVG files for the maps will be stored on Commons. Filceolaire (talk) 01:19, 2 November 2013 (UTC)

Community Notifications

Hi Arun and Hugo,
As you know, the IEG committee will begin their review of round 2 proposals on 23 October 2013. To expedite proposal review, I'm looking over the community notifications section of each eligible application. Your engagement targets include Wikipedia, Commons, and Wikivoyage. If you contacted those communities directly, please paste links to your notices. This will help the committee follow up more easily. Thanks!
Best of luck!
Anna Koval (WMF) (talk) 20:31, 22 October 2013 (UTC)

Anna Koval (WMF), our project engagement targets include Wikipedia, Commons, Wikidata and Wikivoyage. What do you call "your notices" ?~I think all is there Grants:IEG/Wikimaps_Atlas#Community_Notification:. Yug (talk) 20:37, 22 October 2013 (UTC)

Yug, my apologies if this was unclear. By "your notices" I meant any additional communication you had with other projects that might be impacted by your proposal. You did notify Wikidata; that I saw. And you linked English, French, and German Wikipedia notifications, too. But in the infobox for your proposal, there are 3 2 other engagement targets: Commons and Wikivoyage. Did you post anywhere on these projects about your proposal? If so, please include links. Thanks! And if this still doesn't make sense, let me know. :)

--Anna (WMF) 00:33, 23 October 2013 (UTC)

As far as I know we didn't noticed them for the following reasons:

Commons (I've been admin there) is a repository place, people affected by our project are mainly wikicartographers, who serves the community of article editors. While these cartographers store their derivate maps on commons, and while our project will store its maps on commons, our users are more easily found on wikipedia's map workshops.

Wikivoyage is a peripheric project for now. While it will use our maps, there is no cartographer nor developers communities over there like we have on Wikipedia or Wikidata. Yug (talk) 13:18, 24 October 2013 (UTC)

Thanks for explaining that, Yug; makes sense to me. I'm sure the committee will appreciate knowing why, too. --Anna (WMF) 15:49, 25 October 2013 (UTC)

Updates

If you take a map with a point on it (say, Australia with Canberra on it), and then the continent shrinks half a year later, would there be means for automatic update of that fact on the map? Or only manually? Thanks. Gryllida 09:46, 30 October 2013 (UTC)

Hello, it will depend on our GIS data / resources. If the NASA/Gadm/NaturalEarth/OSM GIS data is updated, the +6 month rerun of the script and generated SVG will be updated. We already had talks about this 2 sequences process. The wikimap Atlas project focus on "Taking GIS resources in, generating Wikipedia / Wikidata complient SVG basemaps out" (2nd sequence). The GIS data should be keept up to date, and this is an other issue / project, where Sharemap and OSM are the right actors to rely on. Yug (talk) 10:31, 30 October 2013 (UTC)

Advisor

Dear all, I joined myself as advisor. I am a lot interested in this project and I would monitor the development and the results. I am in the IEG Committee and the IEG Committee encourage members to be advisors. May be in future I will propose myself with a more active role. --Ilario (talk) 08:45, 7 November 2013 (UTC)

Hello Ilario, we indeed welcome / will welcome ideas, advices, GIS/dev tricks, peripheric dev push, etc. We will naturally listen at experienced GIS graphists for designs, experienced programmers for devs, and experienced administrators for project/resources/grant managements. As far as I know, your profile seems more as an experienced administrator related to the WMF CH & WMF IT. If you have other strong competences which may potentially be helpful to strengthen Wikipedia map design, please let me know, so I keep it in mind for the futur. Yug (talk) 12:16, 7 November 2013 (UTC)

Aggregated feedback from the committee for Wikimaps Atlas

Scoring criteria (see the rubric for background)	Score 1=weakest 5=strongest
Potential for impact
(A) The project fits with the Wikimedia movement's strategic priorities	4.5
(B) The project has the potential to lead to significant online impact.	4.5
(C) The impact of the project can be sustained after the grant ends.	4
(D) The project has potential to be scaled or adapted for other languages or projects.	4
Ability to execute
(E) The project has demonstrated interest from a community it aims to serve.	5
(F) The project can be completed as scoped within 6 months with the requested funds.	4
(G) The budget is reasonable and an efficient use of funds.	4
(H) The individual(s) proposing the project have the required skills and experience needed to complete it.	4.5
Fostering innovation and learning
(I) The project has innovative potential to add new strategies and knowledge for solving important issues in the movement.	3.5
(J) The risk involved in the project's size and approach is appropriately balanced with its potential gain in terms of impact.	4
(K) The proposed measures of success are useful for evaluating whether or not the project was successful.	4
(L) The project supports or grows the diversity of the Wikimedia movement.	4
Comments from the committee: A strong idea for improving maps content - it would be great to have the ability to make quality maps using these tools. Promotes consistency, flexibility, and efficiency in map development. Addresses a clear need of the community with a reasonable cost to do so. The grant request is detailed and well-explained, and we appreciate the thought that went into creating this proposal. It identifies the problem, project scope, and project plan quite well. Very significant amount of community endorsement.

Thank you for submitting this proposal. The committee is now deliberating based on these scoring results.

Funding decisions will be announced by December 16. — ΛΧΣ²¹ 00:21, 14 November 2013 (UTC)

Thanks for the notification. Yug (talk) 22:44, 14 November 2013 (UTC)

Video interview with WMF IEG manager

Abstract of questions raised and answers provided for the video interview of December 6th.
Q: (questions)

1. About the 3 parts of your project: Workflow & script (on labs, yes?), Output maps by team, Output maps on demand:
   1. Do you expect many others to use a workflow and script from toolslabs? Or will main impact be from you creating the 5,000+ base maps yourselves?
   2. How will you coordinate/encourage volunteers to upload the maps you generate to commons? Where will the maps live until they do?
   3. Thoughts on what’s needed to make on-demand really user friendly? Gadget or tool linked from the Graphics Lab? I think this part of your project is pretty interesting, am curious to what extent it feels like a key focus for you, and to what extent it seems feasible for the given time period.
2. Hosting on Labs - do you have any concerns about limitations? We’d like you to make all source code etc freely available, GitHub etc, obviously. (I'm sure you have a plan for this already!)
3. Time period: IEGs are so far always for 6 months, and this seems like it could be an ambitious scope for just 3. Are you thinking 3 months because with work and school that's all the time you can put towards it? Or have you thought perhaps 3 months intensive focused work, with 3 more for buffer/followup? Based on what I've seen from past projects, your April and May scope of work I would guess could easily take double the time - even if your code works perfectly the first time, consensus and feedback/usability testing always takes time, waiting for input and conversations to move forward, and then iterating.
4. Would you think about a start date of March 1 for 6 months, or a start date before then, with the first months being non-intensive and remote working? You have flexibility on the start date, could start as early as Jan 1, as late as March 1, I'd just like to better understand what works best for you and your project.
5. Do you have plans to usability test if you're serious about creating the on-demand setup? Plan has enough time/space to usability test the front end w/ volunteers?
6. Do you imagine online on demand requests are on their own website? or integrated on wikipedia, or? (ie, what might "Connecting our system with a website UI for online custom request" look like?)
7. Budget overall looks quite reasonable. Please tell more about the books, & lets talk about what you plan to do with them after the grant ends.
8. How many Wikipedia cartographers are there right now (I don't know if you know this, and I know the easy answer is "not enough", but I'm curious if you've got any ballpark guesses)?
9. Measures of success - can we put some targets into you measures, beyond product outcomes? Usage targets? (ie, you plan to make 6k maps, but we want to see how many are used by editors/articles. And what about Wikipedia cartographer activity, do you have any interest in that?) I'm also curious how you think you might demonstrate/measure smooth workflow at the end?

--Siko (WMF) (talk) 19:06, 6 December 2013 (UTC)

A: (Draft answers)

1. Map dump and online map generation:

1a. The main contribution will be the thousand basemaps based on current administrative areas. This will satisfy a large need (most of them). Derivative files will be done. Monitoring of this will likely be done simply using commons templating system. Similar to the Map workshops monitoring approach.

There is other cases such historical empires, kingdoms, cultural regions, geographic regions, international groups (ASEAN, etc.), where users will have to use the online tool. We naturally want to store meta data about these requests : parameters, number of such requests, username. -- Yug

1b. Our core competence and project focus on buiding a map generating system. For upload to commons, either we rely on users themselves. Either we may plugin the lower end of our workflow to some system such Upload_Wizard. en:User:Jarry1250 has create the SVGtranslate tool, which first translate commons svg, to then upload the derivate file to commons. Similarly, we could request a skillful user to add to our map generator the code to upload the generated file to commons. -- Yug

1c. User experience: we expect this part to be a webpage, with relevant fields to fill, then our serverside scripts doing the work need : 1. generating the map (for sure!), 2. uploading to commons (if possible). -- Yug

2a. Technical limitations: an email exchange with Ryan Kaldari (WMF) confirmed us that Wikimedia lab satisfy our technical requirements for storage size (data), environment (linux server), so our data, script, and webpage could be hosted.

b. We plan to share our code on github, indeed. -- Yug

7. Books: (surprise for some users, full answer in Siko emails :) ). --Yug

I: (Recently emerging informations)

• https://commons.wikimedia.org/wiki/Template_talk:Translate#Update_to_mention_support_of_.3Cswitch.3E_statements -- switch statement now managed by MW & ImageMagic!

Yug (talk) 19:06, 6 December 2013 (UTC)

Hi Yug, thanks again for all of your updates so far. The updated targets etc look good! As discussed, please don't forget to update your proposal from the 3 months scope to 6 (I see that there are still a few places that you mention the 3 months of development, but don't include a note about an additional 3 for the community uptake - provided that is still the plan, I'd like to see it reflected on your page consistently). And please also double check your budget numbers to make sure they all match as well, ok (I still see an error in external freelancers - 60 hours or 90 hours)? Best wishes, Siko (WMF) (talk) 23:58, 16 December 2013 (UTC)

uMap

Hi, there exists a tool on the OSM side, uMap, aimed at creating maps (I’m not a mapper or catographier myself, but it seems quite useful -- credit to Otourly for speaking about this tool [2] (private archives, require free subscription)). Perhaps this tool can be used in a way or another in this IEG, although at first sight and with my limited knowledge of OSM some features need to be added, like managing collections of maps and simple export to Commons as SVG. ~ Seb35 [^_^] 11:26, 19 December 2013 (UTC)

Thanks Seb35, this is quite good and modern. I contacted the author and wait his feedback. Surprise : he actually may be French and in Paris as well. :) Yug (talk) 10:35, 17 May 2014 (UTC) (French Open Source Rocks !)

Tool Labs help

Let me know if you want any help with labs / toollabs stuff :) Yuvipanda (talk) 21:20, 22 April 2014 (UTC)

Thanks for the notice ! Yug (talk) 10:36, 17 May 2014 (UTC)

Wikidata queries

Is there any chance that Wikidata queries can be displayed in the WikiAtlas by the end of this project? I think that WikiAtlas would be a great independent project for people to create and share maps with specific queries to Wikidata. --Tobias1984 (talk) 20:03, 19 July 2014 (UTC)

user:Tobias1984, You will be able to create JS script (module). Module should take as input a data attibute within our maps xml (svg), such as administrative area's english name, zip-code, use this string to make a Wikidata request, extract the information of intereste from Wikidata JSON, and inject it back into the svg. You will need to know Wikidata API, JS, and some basic D3js. Yug (talk) 23:23, 21 August 2014 (UTC)

@Yug and Planemad: Thanks for the answer. Did you take a look at the geoshape feature request on Wikidata? https://bugzilla.wikimedia.org/show_bug.cgi?id=55549 --Tobias1984 (talk) 15:13, 27 August 2014 (UTC)