Re: the new [[Special:GadgetUsage]] page, e.g. https://commons.wikimedia.org/wiki/Special:GadgetUsage and the tasks that led to it (T115152: Create a Special:GadgetUsage QueryPage to show Gadget usage stats per wiki and related)
It would be useful to track month-on-month changes for both columns, so that we could:
See T120895#1864312 for the SQL query that is used to generate those columns.
I don't have advice/preferences on how to render the data.
Simpler (e.g. raw csv monthly dumps, for now) is probably better than nothing.
Please boldly edit the description. :)
graphite/grafana is probably the best counts over time visualizer we have right now. We'd want to hack into the update process somehow to send the counts everytime the special page is regenerated.
Sounds good to me, Lego, but then you wouldn't need us. Feel free to add us back if you do.
Another request for this feature at https://en.wikipedia.org/wiki/Wikipedia_talk:Special:GadgetUsage#Does_this_data_exist_over_time%3F%3F
Question for @DAbad and @kzimmerman: where do you think this task should land? It seems to me someone should own instrumenting gadgets, and we'd be happy to support it on our platform.
https://en.wikipedia.org/wiki/User:Alexis_Jazz/GUS2Wiki
Takes 2-5 regulars per project. (someone using it has to visit the wiki when the stats have been updated, a single regular who never misses a day would also suffice) Currently just overwrites the page so for history you'll have to dig through the page history.
You could just write a toolforge cronjob that does this nightly using a bot account. Making it a userscript that edits silently from a human account whenever they log in seems silly.
Sounds like you are volunteering! /sarcasm
I don't have a Toolforge account. I'm currently not interested in creating one either. Toolforge is burdened by a bunch of rules and restrictions and the WMF can suspend accounts "at its sole discretion". Accounts must be approved. While I understand that I also don't particularly like it, it doesn't feel very open. If you need a lot of resources and/or offer some public service (like Web2Cit) it can be very attractive though. Personally, for small stuff, if I needed it I'd rather pay for some private VPS or something.
But also, if such a cronjob breaks and the maintainer isn't around, there's nothing anyone can do. This has happened (and continues to happen) with stuff on Toolforge. A userscript can be forked and fixed by anyone.
Sure, a well-maintained bot that deals with this would be ideal, but in six and a half years nobody has volunteered.
The SQL query uses querycachetwo (to find which users are 'active users'), which is not exposed in toolserver replicas (should it?), nor are 'gadget-%' up_property values (which would reveal the gadgets each one has enabled).
So this would have to run in the cluster.
I think this should be a cron running foreachwiki on a script which produces these statistics, publishing it on https://dumps.wikimedia.org/other/analytics/ so snapshot1008 is probably the appropriate host to do that.
if someone produces a script, I can point them at the puppet manifests needed for getting it running on snapshot1008, and updating the relevant index.html files.
All Wikipedias with >300 active users and all sister projects of those (Wikibooks, Wiktionary, etc) are now tracked on https://meta.wikimedia.org/wiki/User:Alexis_Jazz/GUS2Wiki.json.
I've been running GUS2Wiki since July 2022. I think I'll have to shut this down at some point as I'm trying to phase out my 24/7 boxes. The next two weeks it'll be fine I think. The next month? Not sure. Next two months? Less sure.
You need an account with some particular code in its common.js like https://en.wikipedia.org/wiki/User:Alexis_Jazz/common.js and have some browser visit https://en.wikipedia.org/wiki/Main_Page?autopost=1 (or any other page, only the autopost=1 matters) every 24 hours. It takes about 10 minutes each day. (depending on how many pages actually get updated)
IIRC Firefox can run headless, so a cronjob should theoretically be an option.
@AlexisJazz as someone who helps support Toolforge and other WMCS services, I'd be happy to understand more about how this might be able to hosted in an open and sustainable manner for yourself and others. I share your concerns about sustainability (including open and accessible code, the ability to fork, etc) for critical infrastructure and tools like this. Reach out if you're open to discussing. Either way, thank you your work in hosting and maintaining these statistics!
@AlexisJazz: Did you find time to have a chat with nskaggs? Asking as it would be great to keep your service available in some sustainable way. Thanks!
The machine I used for GUS2wiki been shut down. I try to remember to run gus2wiki on my daily driver every once in a while. It's not ideal.
@Aklapper nope, mostly due to ongoing personal issues.
@nskaggs any particular channel you want to use to communicate? Either here or a talk page on-wiki works for me. Anything in particular you need to know? I've already mostly explained how to do it.
#!/bin/bash firefox -headless "https://en.wikipedia.org/wiki/Main_Page?autopost=1" & sleep 15m && kill $!
Is it pretty? Not particularly, but it serves its purpose. Adding bot:1 to the postWithEditToken instances in https://en.wikipedia.org/wiki/User:Alexis_Jazz/GUS2Wiki.js may be useful if you apply for https://meta.wikimedia.org/wiki/Steward_requests%2FBot_status#Global_bot_status_requests.
@AlexisJazz Phabricator works fine for communicating for now. Thanks for sharing this! I was able to have a look at https://en.wikipedia.org/wiki/User:Alexis_Jazz/GUS2Wiki.js to understand a bit better what's happening. What do you think of the suggestions posted to use either Toolforge and/or publishing to dumps? I have concerns about relying on a human to be the primary runner, as it's not always possible to ensure updates. Your concern of a tool or service breaking that no one can fix is also shared by me. I would think it possible to have an automated job that is running, with published source code, that anyone could also run themselves as a backup solution. Do you agree that would be the ideal scenario? Or if not, what would be?
I'm asking these questions because I'd like to understand how to make creating services like this easier and more sustainable. If you have an idea on how WMCS could better support you, please let me know. A reply on this ticket to start works fine, but feel free to use my Talk page as well.
@nskaggs I agree, I designed GUS2wiki to allow multiple people to run it. (it detects if the page has been updated recently and skips it in that case) I also agree about a human as the primary runner being problematic, but whatcha gonna do.
I don't plan on creating a ToolForge account. GUS2wiki may technically not be allowed on ToolForge (I've discussed this before: https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#4.5_Using_WMCS_as_a_network_proxy says "all network connections must originate from or terminate at WMCS." which is easily failed as Wikipedia etc are not WMCS) I'd just be asking for someone who dislikes me to screw me over which unfortunately wouldn't be a first. As long as that rule is on the books (whether it's effectively a dead letter or not) I could never be comfortable creating a TF account.
What do you mean by "publishing to dumps"? You mean https://dumps.wikimedia.org/ ? That wouldn't be very helpful as it expires.
There are also personal circumstances due to which I have quite limited time for wiki these days.