Page MenuHomePhabricator

Migrate Database::select usages to SelectQueryBuilder in MediaWiki core
Closed, ResolvedPublic

Description

To split from the parent.

What's left:

FileComplexity
includes/block/DatabaseBlock.phpNeeds to introduce a query builder to replace ::getQueryInfo()
includes/language/MessageCache.phpTakes RevisionStore's query info and unsets pieces
includes/utils/BatchRowIterator.phpDesigned to do ::select()
includes/user/UserEditTracker.phpCalls to deprecated ActorMigration::getWhere (easy)
includes/changes/RecentChange.phpNeeds introducing RecentChangeSelectQueryBuilder to replace ::getQueryInfo
includes/filerepo/file/LocalFile.phpPasses the query array to onLocalFile__getHistory hook
includes/filerepo/file/LocalFileDeleteBatch.phpNone
includes/Permissions/RestrictionStore.phpCalls $this->commentStore->getJoin( 'pt_reason' ); (easy)
includes/export/WikiExporter.phpCalls $this->commentStore->getJoin( 'log_comment' ); (easy)
includes/editpage/EditPage.phpCalls $this->commentStore->getJoin( 'log_comment' ); (easy)
includes/watcheditem/WatchedItemQueryService.phpSends the query array to modifyWatchedItemsWithRCInfoQuery
includes/specialpage/QueryPage.phpThe query object is being built in subclasses via ::getQueryInfo()
includes/specialpage/ChangesListSpecialPage.phpRuns runMainQueryHook on the query array
includes/page/WikiPage.phpHas ::getQueryInfo() and pass it to hooks. getContributors one is easy though.
includes/page/WikiPageFactory.phpGets the query info object from Wikipage::getQueryInfo()
includes/page/DeletePage.phpTakes the query info from revision store and fiddles with the query array
includes/title/Title.phpNone
includes/jobqueue/jobs/RecentChangesUpdateJob.phpTakes RecentChange::getQueryInfo() query array
includes/Revision/RevisionStore.php::getSlotsQueryInfo(), needs a dedicated SQB subclass
includes/specials/SpecialRandomPage.phpPasses the query object into a hook onRandomPageQuery
includes/specials/SpecialExport.php::getLinks() takes the join condition as argument, protected method so (easy)
includes/specials/SpecialRecentChanges.phpPasses the query object in runMainQueryHook
includes/specials/pagers/ActiveUsersPager.phpSubquery
includes/specials/pagers/ImageListPager.phpGets the query object from ::getQueryInfo and ::buildQueryInfo
includes/specials/pagers/ContribsPager.phpditto
tests/phpunit/MediaWikiIntegrationTestCase.phpNone
tests/phpunit/includes/block/DatabaseBlockTest.phpNeeds`DatabaseBlock::getQueryInfo();`
tests/phpunit/includes/CommentStore/CommentStoreTest.phpTesting CommentStore's query building part
tests/phpunit/includes/user/ActorMigrationTest.phpditto
tests/phpunit/includes/Storage/PageUpdaterTest.phpNeeds RecentChange::getQueryInfo()
tests/phpunit/integration/includes/block/DatabaseBlockStoreTest.phpNone
tests/phpunit/unit/includes/Permissions/RestrictionStoreTest.phpHorrifying assert
tests/phpunit/unit/includes/libs/rdbms/database/DatabaseMysqlBaseTest.phpNone
maintenance/namespaceDupes.phpNeeds $linksMigration->getQueryInfo
maintenance/populateIpChanges.phpCalls to deprecated ActorMigration::getWhere (easy)

Details

SubjectRepoBranchLines +/-
mediawiki/coremaster+42 -50
mediawiki/coremaster+69 -53
mediawiki/coremaster+28 -34
mediawiki/coremaster+23 -30
mediawiki/coremaster+95 -60
mediawiki/coremaster+57 -79
mediawiki/coremaster+11 -20
mediawiki/coremaster+26 -36
mediawiki/coremaster+12 -12
mediawiki/coremaster+7 -8
mediawiki/coremaster+6 -2
mediawiki/coremaster+20 -25
mediawiki/coremaster+86 -109
mediawiki/coremaster+61 -51
mediawiki/coremaster+35 -40
mediawiki/coremaster+274 -264
mediawiki/coremaster+2 -1
mediawiki/coreREL1_41+2 -1
mediawiki/coremaster+251 -266
mediawiki/coremaster+12 -48
mediawiki/coremaster+14 -14
mediawiki/coremaster+6 -2
mediawiki/coremaster+66 -98
mediawiki/coremaster+207 -254
mediawiki/coremaster+87 -50
mediawiki/coremaster+36 -23
mediawiki/coremaster+272 -207
mediawiki/coremaster+291 -310
mediawiki/coremaster+8 -9
mediawiki/coremaster+249 -269
mediawiki/coremaster+90 -79
mediawiki/coremaster+48 -54
mediawiki/coremaster+47 -11
mediawiki/coremaster+160 -111
mediawiki/coremaster+36 -48
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 958557 merged by jenkins-bot:

[mediawiki/core@master] tests: Migrate more easy cases to SQB

https://gerrit.wikimedia.org/r/958557

Change 958906 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Migrate another major batch to SelectQueryBuilder

https://gerrit.wikimedia.org/r/958906

Change 958964 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] jobs: Migrate to use SQB in HTMLCacheUpdateJob

https://gerrit.wikimedia.org/r/958964

Change 958906 merged by jenkins-bot:

[mediawiki/core@master] Migrate another major batch to SelectQueryBuilder

https://gerrit.wikimedia.org/r/958906

Change 958964 merged by jenkins-bot:

[mediawiki/core@master] jobs: Migrate to use SQB in HTMLCacheUpdateJob

https://gerrit.wikimedia.org/r/958964

Change 959725 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] maintenance: Migrate another batch to SelectQueryBuilder

https://gerrit.wikimedia.org/r/959725

Change 959725 merged by jenkins-bot:

[mediawiki/core@master] maintenance: Migrate another batch to SelectQueryBuilder

https://gerrit.wikimedia.org/r/959725

Change 959810 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] tests: Migrate more cases to SQB

https://gerrit.wikimedia.org/r/959810

Change 959810 merged by jenkins-bot:

[mediawiki/core@master] tests: Migrate more cases to SQB

https://gerrit.wikimedia.org/r/959810

Change 960026 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] tests: Migrate to SelectQueryBuilder in DbTestPreviewer

https://gerrit.wikimedia.org/r/960026

Change 960026 merged by jenkins-bot:

[mediawiki/core@master] tests: Migrate to SelectQueryBuilder in DbTestPreviewer

https://gerrit.wikimedia.org/r/960026

Change 960594 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Introduce LoggingSelectQueryBuilder

https://gerrit.wikimedia.org/r/960594

Change 960594 merged by jenkins-bot:

[mediawiki/core@master] Introduce LoggingSelectQueryBuilder

https://gerrit.wikimedia.org/r/960594

Change 961073 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Introduce ChangeTagsStore::modifyDisplayQueryBuilder()

https://gerrit.wikimedia.org/r/961073

Change 961185 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/core@master] Convert numerous DB queries to use QueryBuilders

https://gerrit.wikimedia.org/r/961185

Change 961187 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/core@master] includes: Convert some DB queries to use QueryBuilders

https://gerrit.wikimedia.org/r/961187

Change 961073 merged by jenkins-bot:

[mediawiki/core@master] Introduce ChangeTagsStore::modifyDisplayQueryBuilder()

https://gerrit.wikimedia.org/r/961073

Change 961185 merged by jenkins-bot:

[mediawiki/core@master] Convert numerous DB queries to use QueryBuilders

https://gerrit.wikimedia.org/r/961185

Change 970342 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Migrate several subqueries to use SQB

https://gerrit.wikimedia.org/r/970342

Change 970342 merged by jenkins-bot:

[mediawiki/core@master] Migrate several subqueries to use SQB

https://gerrit.wikimedia.org/r/970342

Change 970824 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] RenameUserJob: Migrate to SQB

https://gerrit.wikimedia.org/r/970824

Change 970824 merged by jenkins-bot:

[mediawiki/core@master] RenameUserJob: Migrate to SQB

https://gerrit.wikimedia.org/r/970824

Change 973864 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Migrate two simple cases to SQB

https://gerrit.wikimedia.org/r/973864

Change 973864 merged by jenkins-bot:

[mediawiki/core@master] Migrate two simple cases to SQB

https://gerrit.wikimedia.org/r/973864

Change 975072 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] installer: Migrate to SQB in several places

https://gerrit.wikimedia.org/r/975072

Change 975072 merged by jenkins-bot:

[mediawiki/core@master] installer: Migrate to SQB in several places

https://gerrit.wikimedia.org/r/975072

Change 983534 had a related patch set uploaded (by Winston Sung; author: Winston Sung):

[mediawiki/core@master] Maintenance: Fix RebuildTextIndex

https://gerrit.wikimedia.org/r/983534

I'm also getting this issue after installing a wiki's pages via importDump.php. Possible replication steps:

  1. Install MediaWiki 1.42 (version a398498e7058275d959a8ee8088f58b57cf76e19 in this case) and use the generated LocalSettings.php
    • This dump needs wfLoadExtensions(['TemplateStyles', 'Scribunto']); but I'm not sure that is relevant. No other extensions are loaded.
    • The only skin loaded is Vector.
  2. Download the pages XML from https://archive.org/download/wmau_officialwiki_2023-10-22
  3. Import it with $ php maintenance/importDump.php ~/tmp/officialwiki-pages_2023-10-22.xml.gz --no-updates
  4. Run $ php maintenance/rebuildtextindex.php and see the error:
Clearing searchindex table...Done
Rebuilding index fields for 2802 pages...
InvalidArgumentException from line 181 of mediawiki/includes/CommentStore/CommentStore.php: $row does not contain fields needed for comment rev_comment
#0 mediawiki/includes/CommentStore/CommentStore.php(277): MediaWiki\CommentStore\CommentStore->getCommentInternal()
#1 mediawiki/includes/Revision/RevisionStore.php(1779): MediaWiki\CommentStore\CommentStore->getCommentLegacy()
#2 mediawiki/includes/Revision/RevisionStore.php(1625): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots()
#3 mediawiki/maintenance/rebuildtextindex.php(115): MediaWiki\Revision\RevisionStore->newRevisionFromRow()
#4 mediawiki/maintenance/rebuildtextindex.php(69): RebuildTextIndex->populateSearchIndex()
#5 mediawiki/maintenance/includes/MaintenanceRunner.php(703): RebuildTextIndex->execute()
#6 mediawiki/maintenance/doMaintenance.php(100): MediaWiki\Maintenance\MaintenanceRunner->run()
#7 mediawiki/maintenance/rebuildtextindex.php(167): require_once('...')
#8 {main}

Also getting the same issue after trying to use rebuildall.php with importDump.php with MediaWiki 1.41-beta releases. Is there any immediate hotfix that I can work with to get around the issue and use the rebuild script?

@adrelanos are you able to replicate the problem without FlaggedRevs enabled?

I tried commenting out all lines in my configuration that are loading extensions, and the issue persists. Looking back at T252043, the issue appeared since 1.35, so is it possible to be as T276782#6891983 outlined, having a issue with scripts not updated with the new database structure?

No. I don't have any similar issues without FlaggedRevisions. And I wouldn't know how to reproduce this issue without FlaggedRevisions. Because only by running php reviewAllPages.php --username Patrick this issue happened. And the reviewAllPages.php comes within the FlaggedRevisions folder.

Or do you mean I should test disable the FlaggedRevisions in mediawiki config and then run reviewAllPages.php?

Our current case is that running rebuildall.php also breaks in the same way as they call the same part of CommentStore. Can you reproduce using rebuildall.php without FlaggedRevs? If you can, then this bug is not related to FlaggedRevs but to MediaWiki core itself, and we can rename the task and hand this to the right team to fix.

Change 983534 merged by jenkins-bot:

[mediawiki/core@master] Maintenance: Fix RebuildTextIndex

https://gerrit.wikimedia.org/r/983534

Change 983731 had a related patch set uploaded (by Ladsgroup; author: Winston Sung):

[mediawiki/core@REL1_41] Maintenance: Fix RebuildTextIndex

https://gerrit.wikimedia.org/r/983731

Change 983731 merged by jenkins-bot:

[mediawiki/core@REL1_41] Maintenance: Fix RebuildTextIndex

https://gerrit.wikimedia.org/r/983731

Change #1024818 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] maintenance: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1024818

Change #1024818 merged by jenkins-bot:

[mediawiki/core@master] maintenance: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1024818

Change #1026175 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Revision: Migrate some calls to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1026175

Change #1026675 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] tests: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1026675

Change #1026175 merged by jenkins-bot:

[mediawiki/core@master] Revision: Migrate some calls to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1026175

Change #1026675 merged by jenkins-bot:

[mediawiki/core@master] tests: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1026675

Change #1027018 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] UserEditTracker: Replace deprecated ActorMigration::getWhere

https://gerrit.wikimedia.org/r/1027018

Change #1027018 merged by jenkins-bot:

[mediawiki/core@master] UserEditTracker: Replace deprecated ActorMigration::getWhere

https://gerrit.wikimedia.org/r/1027018

Change #1027080 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] title: Migrate to SelectQueryBuilder in Title::estimateRevisionCount

https://gerrit.wikimedia.org/r/1027080

Change #1027091 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] specials: Migrate to SelectQueryBuilder in Special:Export

https://gerrit.wikimedia.org/r/1027091

Change #1027080 merged by jenkins-bot:

[mediawiki/core@master] title: Migrate to SelectQueryBuilder in Title::estimateRevisionCount

https://gerrit.wikimedia.org/r/1027080

Change #1027095 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] editpage: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027095

Change #1027096 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Permissions: Migrate to SelectQueryBuilder in RestrictionStore

https://gerrit.wikimedia.org/r/1027096

Change #1027097 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Migrate to IReadableDatabase::newSelectQueryBuilder with queryInfo

https://gerrit.wikimedia.org/r/1027097

Change #1027095 merged by jenkins-bot:

[mediawiki/core@master] editpage: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027095

Change #1027096 merged by jenkins-bot:

[mediawiki/core@master] Permissions: Migrate to SelectQueryBuilder in RestrictionStore

https://gerrit.wikimedia.org/r/1027096

Change #1027282 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] watcheditem: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027282

Change #1027314 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] WikiExporter: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027314

Change #1027338 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Title: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027338

Change #1027338 merged by jenkins-bot:

[mediawiki/core@master] Title: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027338

Change #1027314 merged by jenkins-bot:

[mediawiki/core@master] WikiExporter: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027314

Change #1027097 merged by jenkins-bot:

[mediawiki/core@master] Migrate to IReadableDatabase::newSelectQueryBuilder with queryInfo

https://gerrit.wikimedia.org/r/1027097

Change #1027599 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Replace deprecated ActorMigration::getWhere in WikiPage::getContributors

https://gerrit.wikimedia.org/r/1027599

Change #1027605 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027605

Change #1027605 merged by jenkins-bot:

[mediawiki/core@master] Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027605

Change #1027091 merged by jenkins-bot:

[mediawiki/core@master] specials: Migrate to SelectQueryBuilder in Special:Export

https://gerrit.wikimedia.org/r/1027091

Change #1027599 merged by jenkins-bot:

[mediawiki/core@master] Replace deprecated ActorMigration::getWhere in WikiPage::getContributors

https://gerrit.wikimedia.org/r/1027599

Change #961187 abandoned by Umherirrender:

[mediawiki/core@master] includes: Convert some DB queries to use QueryBuilders

Reason:

Already done

https://gerrit.wikimedia.org/r/961187

Change #1027282 merged by jenkins-bot:

[mediawiki/core@master] watcheditem: Migrate to IReadableDatabase::newSelectQueryBuilder

https://gerrit.wikimedia.org/r/1027282

Umherirrender closed this task as Resolved.EditedMay 6 2024, 9:26 PM
Umherirrender removed a project: Patch-For-Review.

Usage in DB-specific Installer classes (MysqlInstaller and PostgresInstaller) and in rdbms classes remain.