Grants talk:IEG/LaTeX Export

Server power

Hi. Is the software expected to run on Wikimedia servers — as an extension of some sort — or will it need an external server for processing the format? Thanks. Gryllida 07:04, 24 August 2013 (UTC)

Hi, we are quite free in the choice of the operating system of the server. Currently binaries are available for Linux as well as Windows. So yes we can run it on a windows server. But from the technical point of view I think it is most promissing to use a virtual machiene running debian inside wikimedias server farm. Because there you can produce the results most quickly and you got a high level of reliablility. Still yes we will also be able to produce good results if we just put it on a server somewhere in the internet.Dirk Hünniger (talk) 07:24, 24 August 2013 (UTC)

Hi, thanks for the reply. Did you consider just running the tool on the Wikimedia's server farm — I think the service is free of charge? Gryllida 09:27, 24 August 2013 (UTC)

«Any code or other materials produced must be published and released as free and open-source.»

Are the particular binaries you are referring to open-source, or planned to become such at a point? I think the grants are only given for open-source tools. --Gryllida 09:27, 24 August 2013 (UTC)

The sourcecode of the bianries is also available. In particaulr the project was accepted into Debian, which as got particularaly high standards regarding open source requirenments. Dirk Hünniger (talk) 09:51, 24 August 2013 (UTC)

If you want to care about running it on some kind of server I really appricaiate it. The only problem is that I currently got a full time job, so the time I can spend on that is very limited at the moment. But everything you need to get it running is there. You just need a noramlly skilled system adminiatrotor to set up the system. The software commes with a command line interface. So you also need to do the web interface yourself. But it should be doable in a few lines. And you need to design that yourself anyways since it dependes on the server technology you are going to use. Dirk Hünniger (talk) 10:11, 24 August 2013 (UTC)

What do you actually need? A sysadmin? Labs is already maintained by sysadmins, you don't need to take care of updates; only of your own tool.

I got the impression you can do everything for free and request things <somewhere where integration of tools into wikimedia projects sites is requested, with community consent (I don't know where exactly, possibly village pumps and technical mailing lists)>. Gryllida 08:51, 27 August 2013 (UTC)

To be a bit more clear, there are a few things that need to be done.

1) Develop the tool. (This is mostly done.)

2) Run the tool somewhere and check community feedback, set up a bug tracker somewhere. Maintain it for a few weeks.

3) Have it run on Wikimedia Labs, for free. You will only put your binaries or code there, and run them. Sysadmins will maintain the boxes.

4) Check that the tool runs stably, and collect community feedback at places such as Technical village pump, Wikitech-l mailing list.

5) Write an extension, if not already, so the wiki users can see your tool in the interface. Again ask for community feedback on it.

6) Ask at a technical village pump whether a project (such as English Wikibooks, French Wikipedia) wants to install the tool for it to be visible to users of the wiki. Have community discussion, vote, and one of them will file a request for you.

As far as I could see, these steps do not require a grant, but only some patience going through required communication before the tool goes live on all projects. Gryllida 09:28, 27 August 2013 (UTC)

I'm seeing the same as Gryllida here, I think, not sure how a grant would help you accomplish any of your project. If, on the other hand, what you are looking for is assistance from WMF Engineering staff to make any of this happen, that would not be eligible either. IEGrants don't offer special access to WMF staff time, servers, etc. Please let us know if we're misunderstanding what you're looking for, though! Siko (WMF) (talk) 22:58, 30 August 2013 (UTC)

Well I already stated that I don't want money. Still this is the idea lab. And this is my idea. So I posted it here so the community can discuss it. And from what I can see here the community is interested in this project. So filing this request was a such useful. I also tried to file similar requests elsewhere and did not receive any reply. And here I received some. We are currently tracking bugs here. https://de.wikibooks.org/wiki/Benutzer_Diskussion:Dirk_Huenniger/wb2pdf/Requests . So from what I can understand from here the next step is to run it on wikimedia labs. The question is how to contact them. Wikitech-l was already informed with no feedback here http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071064.html . The wikipedia technical village pump was informed with no feedback here https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29/Archive_108#LaTeX_Export . So from my point of view I am trying to go through the necessary steps of communication and nobody is replying to me. And I don't know how to resolve this problem myself. And since nobody is replying it is hard for me to find out what the next steps should be. Currently you can already run the tool locally and convert any wiki page (especially those on wikibooks and wikipedia) to latex. So the community essentially has got a chance to test it. Also most of the pdf versions of featured books on the English as well as German wikibooks are done with this tool. So the community can essentially comment on it. So the best thing for me would be to find someone who say: "I got a server, I will installed it, I want the software to be packaged in this and that particular way". I will now also contact wikimedia labs now. I just feel a little bit like Asterix in this wonderful movie http://www.youtube.com/watch?v=JtEkUmYecnk . Yours Dirk Hünniger (talk) 06:58, 31 August 2013 (UTC)

So I looked for the right place to file the request on labs. I didn't find it. So could someone please point me to the page where I have to file the request. Dirk Hünniger (talk) 07:06, 31 August 2013 (UTC)

Here, you can read about how the labs work and how to register yourself and your project. --Gryllida 07:37, 1 September 2013 (UTC)

If it complies better with you formalities i can rewrite my proposal to just ask for money to run an external server farm. So you just pay for the servers I send you the bills and I do the administration an web interface all by myself. It isn't the solution I originally intended but it is a workable one too,Dirk Hünniger (talk) 11:30, 31 August 2013 (UTC)

Yes, please, don't ask for a grant if you don't need one. Don't ask for money for external servers — wmlabs is already here. --Gryllida 07:37, 1 September 2013 (UTC)

I will suggest that you write an extension for your work to appear in wiki interface. Install a test wiki instance (also possible on wmlabs) with your extension, and ask community input again? --Gryllida 07:37, 1 September 2013 (UTC)

Note: if you would like funding to hire developers or testers, this would be potentially useful (as a grant idea). You would have to document the plan of development work. What do you want developed? In what time frame? How many people will you hire? An example of a technical grant, which hired developers, here. (If so, I hope you find someone who knows more than me about the Wikimedia community structure to help you draft the plan. Ask German Wikipedia village pump (technical) how to reach technical communities and get feedback, where to start if you want it on all projects?) --Gryllida 07:44, 1 September 2013 (UTC)

So now I really think a grant does not make sense. It is not necessary to hire people for this task. It should be very simple and I should be able to do it myself. Since I got very limited time at the moment it will take a few month. Still I would like to keep this page wiki page, since this is where the community actually started to discuss the project and is still drawing the attention of people interested in it. Dirk Hünniger (talk) 17:42, 1 September 2013 (UTC)

Hey Dirk. If this is very simple, maybe you could submit a feature request that can be tracked in Bugzilla as defined and explained at en:WP:WMF? That way even if you don't get to it in a couple months maybe someone else will? I'm not sure. But I'm very glad you're planning on doing it. Thank you!!! Biosthmors (talk) 12:53, 4 October 2013 (UTC)

Status change for non-grant ideas

Hi Dirk, I'm really glad to hear that you got some good feedback and interest from posting this idea here :) I think that since you've decided an IEGrant doesn't make sense for this project, we should probably mark the IEGrant proposal as ineligible or withdrawn now so that it won't confuse the volunteer committee who will be reviewing grants in the next few weeks. But, we can still keep this page up and it can still appear in the IdeaLab so that you will continue to get feedback on the project, if that is useful. And if in the future you or someone else decides that some sort of grant is necessary or useful for the project, then you would still have this page to build on. What do you think? If you agree, I'll make the changes to your page soon. Best wishes, Siko (WMF) (talk) 20:27, 24 September 2013 (UTC)

Hi again, we're preparing grant proposals for review now that the deadline to proposed has passed. I didn't hear back from you, so I assume that means you're busy with other things right now. I've changed the status of this idea to withdrawn. I hope this meets your expectations and that you'll continue to get help and feedback in IdeaLab! And if you ever decide that this project needs a grant after all, feel free to resubmit it in a future round. Cheers! Siko (WMF) (talk) 22:23, 1 October 2013 (UTC)

Revise for money - even if you don't want it

Just a suggestion: I would revise this to request money for a developer. As everyone has shared, we'd love this feature, but the rules explicitly state that grantees will not (automatically) have access to Wikimedia engineers; you might be hindering yourself simply by applying if you want their help. I'd recommend budgeting money for engineers and a small stipend for yourself (because, yes, it is time-consuming to handle all of this). Just my 2 cents. As it stands, this doesn't seem appropriate for an IEG application, to me; it's more like a (very good, very much desired) project proposal or feature request. --Jackson Peebles (talk) 00:46, 23 September 2013 (UTC)

Hey Jackson, what do you think about my idea of submitting a feature request to Bugzilla, as described above? Biosthmors (talk) 12:55, 4 October 2013 (UTC)

Sure, why not? If the engineers don't want to or can't take it on, they'll just say so. I have found their triage to be a bit lacking (I think due to sheer quantity); you're more likely to get results from doing this yourself, but if you submit the proposal with some background information, then you might see some results. --Jackson Peebles (talk) 13:07, 4 October 2013 (UTC)

Another IEG grant: Can we collaborate?

First of all, I strongly support this project! Secondly, maybe we can find some interesting ways to work together. In the IEG proposal I'm working on, we talk about a LaTeX to Mediawiki export. Can we collaborate to "roundtrip" content? --Arided (talk) 22:53, 27 September 2013 (UTC)

Lack of time

Hello, I currently got a full time position at a German university of applied sciences which is very time consuming. It German sally class E 13. From the economic point of view it is easy to calculate how much you need to pay to hire me for this project. Currently I don't have time to work on this project at all. I am really happy that some people became interested in it and for the the binaries are available for windows as well as there is a command line version which is currently part of the official Ubuntu distribution. These versions do not require to be installed on Wikipedia servers and can be used with full functionality on you private machines. Also the command line version is well suited to be installed on wikipedias servers. The administrative steps to achieve that are very time consuming so I can not do the in the foreseeable future. But if someone reading this has got enough to do so, s/he is welcome to try. I will definitely try to support him/her in any way I can. But it is really up to you now. Yours Dirk Hünniger (talk) 16:50, 26 October 2013 (UTC)

Do you have the time to submit a feature request in Bugzilla? =) Biosthmors (talk) 14:30, 29 October 2013 (UTC)

See mw:How_to_report_a_bug#Reporting_a_new_bug_or_feature_request, just in case. Biosthmors (talk) 14:33, 29 October 2013 (UTC)

Server Version

Hello, I made a webserver version of mediawiki2latex. Its just 61 lines. So the problem is just about finding a websever and deploying it. You can get it here http://sourceforge.net/p/wb2pdf/git/ci/master/tree/src/mediawiki2latex-server.hs Yours Dirk Hünniger (talk) 14:08, 17 November 2013 (UTC)

The server is online http://mediawiki2latex.mooo.com/

Reopening

I will be reopening this issue soon. I discussed with labs. The current program needs to many computational resources to run on labs. I currently put it up on one of my old machines here http://mediawiki2latex.mooo.com/ . But it is to slow an can not handle large documents because the machine is old and slow. Furthermore wikimedia uses one of its employees to develop its own system https://git.wikimedia.org/summary/mediawiki%2Fextensions%2FCollection%2FOfflineContentGenerator%2Flatex_renderer . I think it will take two years until he comes up with a system that works reliably. Until then I would like to run my code on a powerful machine. I would like to rent one. Its that one http://www.hetzner.de/hosting/produkte_rootserver/ex40ssd for tow years . Its 59 EUR per month. So its $1936 in total. Dirk Hünniger (talk) 13:44, 22 December 2013 (UTC)

Cf. AWS instance (Linux, Virginia) costs starting from: m2.2xlarge 796 $ for 3 years, m2.4xlarge 1592 $.[1] --Nemo 13:54, 24 December 2013 (UTC)

Hi. Are you sure you cannot run it on Labs? I think it would be better... PiRSquared17 (talk) 18:15, 25 December 2013 (UTC)

No I can not I tried. I can not grantee its impossible but I surely don't have enough free time to get it to work. I just need to be root of some system then I can install it in 10 Minutes. On labs I can just get CGI and it is by no means clear that I can install my dependencies, nor it is clear their CGI will work with binaries in Haskell nor that it will keep that way for a long time. There are also limitations on computational resources that are hard to work around. So there are just too many problems to run it on labs and I don't have enough time to work it all out. So I am not going to discuss running on labs any further. I will just ask for a root server and if I get it the system get fast and otherwise it stays slow and limited.Dirk Hünniger (talk) 10:07, 26 December 2013 (UTC)

Labs is not only Tools. The largest instance (virtual server) you can create there has 8 cores and 16 GB RAM: not as big as the one you linked and not a permanent solution, but should be enough for some testing. It shouldn't be too hard to get a project created for you; maybe you could even just be added to an existing project (by Mwalker, MaxSem, Jgreen or Anomie) to create your instance there. --Nemo 15:39, 30 December 2013 (UTC)

Dirk, if you want to set it up in the collection-alt-renderer project, I just added you to it. Create yourself a new instance at [2] (add it to the web group so that port 80 is allowed) and then go forth and prosper. Mwalker (WMF) (talk) 20:12, 30 December 2013 (UTC)

Hello, I managed to create the machine. I could log into and compile my code. Unfortunately I got runtime dependencies with are not met in Ubuntu 12.04 so I tried to dist-upgrade to 13.10 the machine rebooted correctly, but I could not log into it anymore. I think you will be allowing 14.04 LTS as soon as it gets released. So I think its currently best to wait for three more month. Yours Dirk Hünniger (talk) 14:55, 31 December 2013 (UTC)

I put the status of this request to withdrawn since I now got access to the machine. I will wait now until ubuntu 14.04 gets released and set up the system then. Please keep this website since it contains the links I am going to use by then. Yours Dirk Hünniger (talk) 16:20, 31 December 2013 (UTC)

And of course thank you very much for giving me access to the machine. Btw. I can run it on 2 GByte single core machines now.Dirk Hünniger (talk) 17:05, 31 December 2013 (UTC)

I was able to set up the machine. I activated 13.10 in sources.list but did not run dist-upgrade. The server is running now in a screen session. The machine is called mediawiki2latex.pmtpa.wmflabs . The only thing that remains to be do is to give it a public ip address and DNS entry. And I don't know how to do that. Dirk Hünniger (talk) 13:52, 1 January 2014 (UTC)