Page MenuHomePhabricator
Create Task
Maniphest T180777

Move portal deployment artifacts into their own repo
Closed, ResolvedPublic
Actions

Description

Currently the wikimedia-portals repo contains all the source files and build artifacts for the wikimedia portals.
The build artifacts change frequently because of a cache-busting hash on the build file names, and running a build effectively deletes and adds hundreds of files.
https://phabricator.wikimedia.org/rWPOR3862e2f5c2c3196872b960a6e63cc5f1a70e9a2b
Because of this, most Gerrit patches are only submitted with the source files to make it clear what the functional changes are. Then, occasionally, a patch with a build is submitted that changes the build artifacts.

Placing the build artifacts in their own repo would make it easier to track the portal builds and lighten the source repo significantly. The repo with the build artifacts can then be included into the source repo as a submodule.

Steps

  • create a new wikimedia/portals/deploy repository in Gerrit
  • replace the prod folder in the wikimedia-portals repo with a submodule of wikimedia-portals-deploy (393223, 393224)
  • move the files sync-portals and urls-to-purge.txt into the root of the wikimedia-portals-deployrepo
  • change line 5 in sync-portals
scap sync-file portals/prod/wikipedia.org/assets $*
to 
scap sync-file portals/wikipedia.org/assets $*
  • change the portals submodule in mediawiki-config from wikimedia-portals to wikimedia-portals-deploy(393239)
  • change the following symlink in mediawiki-config (393239)
 /mediawiki-config/docroot/wwwportal/portal@ -> ../../portals/prod 
to 
/mediawiki-config/docroot/wwwportal/portal@ -> ../../portals
  • Set the Jenkins job to commit the build assets from portals to the wikimedia/portals/deploy repo (393252)

Details

SubjectRepoBranchLines +/-
Move job from experimental to test and gate-and-submit pipelines for wikimedia/portals/deployintegration/configmaster+3 -2
Replace portals submodule with portals/deploy submoduleoperations/mediawiki-configmaster+3 -3
Setting portals JJ to commit to portals/deployintegration/configmaster+7 -3
Initializing portals/deploy submodulewikimedia/portalsmaster+1 -0
Adding portals/deploy repo as submodulewikimedia/portalsmaster+4 -3 K
Adding scap script and urls-to-purge.txt to repowikimedia/portals/deploymaster+321 -0
Adding selenium testswikimedia/portals/deploymaster+5 K -0
docker: add Chromedriver to npm-browser-testintegration/configmaster+41 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdrewniak created this task.Nov 17 2017, 9:50 AM
Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptNov 17 2017, 9:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
debt triaged this task as Medium priority.Nov 17 2017, 4:47 PM
debt added a parent task: T140159: [epic] Wikipedia.org Portal: automating updates for the portal.
Stashbot subscribed.Nov 21 2017, 1:08 PM

Mentioned in SAL (#wikimedia-releng) [2017-11-21T13:08:31Z] <hashar> gerrit: created wikimedia/portals/deploy https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/portals/deploy for jan_drewniak | T180777

hashar updated the task description. (Show Details)Nov 21 2017, 1:08 PM
debt moved this task from Backlog to In Progress on the Discovery-Portal-Sprint board.Nov 21 2017, 2:32 PM
Jdrewniak added a comment.Nov 23 2017, 7:00 PM

Just to be clear about the submodule setup, as I was discussing with @hashar today, I think it should look like this:

portals-repo/
- data
- build-scripts, etc...
- dev
  |- wikipedia.org <-(source templates)
- prod <-(portals/deploy submodule)
  |- wikipedia.org<-(compiled assets)
  |- wikibooks.org
  |- etc.. 
  |- sync-portals
  |- urls-to-purge.txt

Where just the compiled assets and a few scripts (sync-portals, urls-to-purge.txt) would live in the deploy repository.
Then in mediawiki-config, we replace the portals repository with the portals/deploy repo:

mediawiki-config/ 
- portals <-(portals/deploy submodule)
  |- wikipedia.org <-(compiled assets)
  |- wikibooks.org
  |- etc.. 
  |- sync-portals
  |- urls-to-purge.txt

How this will effect the build process
With this submodule setup, the process for submitting a new build to Gerrit will change slightly.
Instead of submitting the build as a patch to the source repo, it will be submitted as a patch to the deploy repo.
So instead of:
checkout portals -> run build -> commit to portals
it would be
checkout portals -> run build -> cd into submodule -> commit to portals/deploy
For this to work in Jenkins, it would only require a minor modification (cd prod and commit from there).
For this to work on developer machines, users will have to change the git remote origin inside the submodule from https to ssh to push to Gerrit (I'll include a helper npm script that can do this).

gerritbot subscribed.Nov 24 2017, 12:46 PM

Change 393223 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] [WIP] Adding portals/deploy repo as submodule

https://gerrit.wikimedia.org/r/393223

gerritbot added a project: Patch-For-Review.Nov 24 2017, 12:46 PM

Change 393224 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] [WIP] Initializing portals/deploy submodule

https://gerrit.wikimedia.org/r/393224

gerritbot added a comment.Nov 24 2017, 2:47 PM

Change 393239 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[operations/mediawiki-config@master] [WIP] Replace portals submodule with portals/deploy submodule

https://gerrit.wikimedia.org/r/393239

gerritbot added a comment.Nov 24 2017, 4:01 PM

Change 393252 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[integration/config@master] [WIP] Setting portals JJ to commit to portals/deploy

https://gerrit.wikimedia.org/r/393252

Jdrewniak updated the task description. (Show Details)Nov 24 2017, 4:02 PM
Jdrewniak updated the task description. (Show Details)Nov 24 2017, 4:08 PM
gerritbot added a comment.Dec 1 2017, 9:58 AM

Change 394539 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals/deploy@master] Adding selenium tests

https://gerrit.wikimedia.org/r/394539

hashar mentioned this in T179360: Create "npm-browser" docker image with npm, xvfb, chromium, and firefox installed.Dec 1 2017, 12:06 PM
gerritbot added a comment.Dec 1 2017, 1:48 PM

Change 394565 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: add Chromedriver to npm-browser-test

https://gerrit.wikimedia.org/r/394565

gerritbot added a comment.Dec 4 2017, 4:00 PM

Change 394565 merged by jenkins-bot:
[integration/config@master] docker: add Chromedriver to npm-browser-test

https://gerrit.wikimedia.org/r/394565

zeljkofilipin added a project: User-zeljkofilipin.Dec 8 2017, 3:35 PM
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin subscribed.Dec 12 2017, 5:20 PM
zeljkofilipin moved this task from Deep work 🌊 to Watching 📺 on the User-zeljkofilipin board.Dec 12 2017, 5:46 PM
debt mentioned this in T140159: [epic] Wikipedia.org Portal: automating updates for the portal.Dec 12 2017, 5:59 PM
zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.Jan 22 2018, 3:24 PM
zeljkofilipin moved this task from Deep work 🌊 to Watching 📺 on the User-zeljkofilipin board.Feb 20 2018, 3:48 PM
zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.Feb 26 2018, 12:48 PM
gerritbot added a comment.Feb 27 2018, 10:07 AM

Change 414970 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Move job from experimental to test and gate-and-submit pipelines for wikimedia/portals/deploy

https://gerrit.wikimedia.org/r/414970

gerritbot added a comment.Feb 27 2018, 10:12 AM

Change 414970 merged by jenkins-bot:
[integration/config@master] Move job from experimental to test and gate-and-submit pipelines for wikimedia/portals/deploy

https://gerrit.wikimedia.org/r/414970

zeljkofilipin moved this task from Deep work 🌊 to Waiting 🏹 on the User-zeljkofilipin board.Feb 27 2018, 10:29 AM
hashar added a comment.Feb 27 2018, 2:35 PM

@Jdrewniak looks like this task is complete?

zeljkofilipin added a comment.Feb 27 2018, 2:57 PM

@hashar 394539 is not yet merged.

gerritbot added a comment.Feb 28 2018, 1:31 PM

Change 394539 merged by jenkins-bot:
[wikimedia/portals/deploy@master] Adding selenium tests

https://gerrit.wikimedia.org/r/394539

gerritbot added a comment.Feb 28 2018, 1:59 PM

Change 415291 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals/deploy@master] Adding scap script and urls-to-purge.txt to repo

https://gerrit.wikimedia.org/r/415291

Jdrewniak updated the task description. (Show Details)Feb 28 2018, 2:02 PM
gerritbot added a comment.Feb 28 2018, 2:02 PM

Change 415291 merged by jenkins-bot:
[wikimedia/portals/deploy@master] Adding scap script and urls-to-purge.txt to repo

https://gerrit.wikimedia.org/r/415291

gerritbot added a comment.Feb 28 2018, 2:32 PM

Change 393223 merged by jenkins-bot:
[wikimedia/portals@master] Adding portals/deploy repo as submodule

https://gerrit.wikimedia.org/r/393223

Jdrewniak updated the task description. (Show Details)Feb 28 2018, 2:34 PM
gerritbot added a comment.Feb 28 2018, 2:35 PM

Change 393224 merged by jenkins-bot:
[wikimedia/portals@master] Initializing portals/deploy submodule

https://gerrit.wikimedia.org/r/393224

Jdrewniak added a comment.Feb 28 2018, 2:43 PM

ok! I'm on a roll here. The selenium tests are working. The portal/deploy repo has been submodule'd into the src repo, there's just a few outstanding config/jenkins things left.
@hashar I've updated this patch that run the portalsBuilder in the deploy repository https://gerrit.wikimedia.org/r/#/c/393252/.
After that, this config patch will have to be updated to point to the new portal/deploy submodule https://gerrit.wikimedia.org/r/#/c/393239/

zeljkofilipin moved this task from Waiting 🏹 to Watching 📺 on the User-zeljkofilipin board.Mar 1 2018, 9:34 AM
debt added a comment.Mar 9 2018, 5:27 PM

Woohoo - glad to see this progressing (even if I'm late to the party to see it!) :)

hashar mentioned this in T160943: Automate WDQS GUI deployment.Mar 13 2018, 8:16 AM
Jdrewniak added a comment.Mar 13 2018, 8:36 AM

pinging @hashar, do you have time to review https://gerrit.wikimedia.org/r/#/c/393252/ ?
That patch sets the portalBuilder job to push a commit to the portals/deploy repository, instead of portals.

hashar added a comment.Mar 13 2018, 12:16 PM

Eek sorry I have missed it. I have reconfigured the job and triggered a build:

hashar assigned this task to Jdrewniak.Mar 13 2018, 12:17 PM

Jan did all the work. I have just been pushing buttons.

gerritbot added a comment.Mar 13 2018, 12:18 PM

Change 393252 merged by jenkins-bot:
[integration/config@master] Setting portals JJ to commit to portals/deploy

https://gerrit.wikimedia.org/r/393252

Jdrewniak updated the task description. (Show Details)Mar 13 2018, 3:50 PM
gerritbot added a comment.Mar 19 2018, 11:02 AM

Change 393239 merged by jenkins-bot:
[operations/mediawiki-config@master] Replace portals submodule with portals/deploy submodule

https://gerrit.wikimedia.org/r/393239

Stashbot added a comment.Mar 19 2018, 11:13 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-19T11:13:26Z] <jdrewniak@tin> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:393239|Switching portals submodule to portals-deploy (T180777)]] (duration: 00m 58s)

Stashbot added a comment.Mar 19 2018, 11:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-19T11:14:25Z] <jdrewniak@tin> Synchronized portals: Wikimedia Portals Update: [[gerrit:393239|Switching portals submodule to portals-deploy (T180777)]] (duration: 00m 58s)

Jdrewniak updated the task description. (Show Details)Mar 19 2018, 11:43 AM
Jdrewniak closed this task as Resolved.Apr 23 2018, 10:13 AM
Jdrewniak mentioned this in T136185: Establish a merge and deploy workflow for wikimedia/portals repository.Aug 31 2018, 2:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits