Page MenuHomePhabricator

ExternalData is no longer usable when using the new database schema
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Install ExternalData on master
  • Run update.php (or otherwise add the ed_url_cache table used by ExternalData)
  • Try to use ExternalData on an HTTPS source

What happens?

[2391d601f49f8f853d52635b] /wiki/Special:ExpandTemplates?wpInput={{%23external_value%3A+id+|+url%3Dhttps%3A%2F%2Frain.thecomicseries.com%2Fcomics%2F2+|+format%3Dtext+|+regex%3D%40%3Cmeta+property%3D%22og%3Aurl%22+content%3D%22https%3A%2F%2F\w%2B\.thecomicseries\.com%2Fcomics%2Fpl%2F(%3F%27id%27\d%2B)%2F%3F%22%40}} Wikimedia\Rdbms\DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading or after adding a new extension?

Please see https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Upgrading and https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:How_to_debug for more information.

Error 1364: Field 'post_vars' doesn't have a default value
Function: Wikimedia\Rdbms\Database::insert
Query: INSERT INTO `ed_url_cache` (url,result,req_time) VALUES ('https://rain.thecomicseries.com/comics/2:array (\n \'timeout\' => \'default\',\n \'HTTPTimeout\' => 25,\n \'HTTPConnectTimeout\' => 5.0,\n \'postData\' => NULL,\n)','<truncated long html that no one here cares about>',1727839274)

Backtrace:

from /srv/mediawiki/1.42/includes/libs/rdbms/database/Database.php(1203)
#0 /srv/mediawiki/1.42/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/1.42/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/1.42/includes/libs/rdbms/database/Database.php(652): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/1.42/includes/libs/rdbms/database/Database.php(1481): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
#4 /srv/mediawiki/1.42/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->insert(string, array)
#5 /srv/mediawiki/1.42/includes/libs/rdbms/database/DBConnRef.php(407): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/1.42/extensions/ExternalData/includes/connectors/traits/EDConnectorCached.php(164): Wikimedia\Rdbms\DBConnRef->insert(string, array)
#7 /srv/mediawiki/1.42/extensions/ExternalData/includes/connectors/traits/EDConnectorCached.php(92): EDConnectorHttp->cache(string, string, boolean)
#8 /srv/mediawiki/1.42/extensions/ExternalData/includes/connectors/EDConnectorHttp.php(124): EDConnectorHttp->callCached(Closure, string, array)
#9 /srv/mediawiki/1.42/extensions/ExternalData/includes/EDParserFunctions.php(95): EDConnectorHttp->run()
#10 /srv/mediawiki/1.42/extensions/ExternalData/includes/EDParserFunctions.php(118): EDParserFunctions::get(MediaWiki\Title\Title, string, array)
#11 /srv/mediawiki/1.42/extensions/ExternalData/includes/EDParserFunctions.php(202): EDParserFunctions::fetch(MediaWiki\Title\Title, string, array)
#12 /srv/mediawiki/1.42/extensions/ExternalData/includes/EDParserFunctions.php(228): EDParserFunctions::emulateGetExternalData(array, MediaWiki\Title\Title)
#13 /srv/mediawiki/1.42/includes/parser/Parser.php(3451): EDParserFunctions::doExternalValue(MediaWiki\Parser\Parser, string, string, string, string)
#14 /srv/mediawiki/1.42/includes/parser/Parser.php(3136): MediaWiki\Parser\Parser->callParserFunction(PPFrame_Hash, string, array)
#15 /srv/mediawiki/1.42/includes/parser/PPFrame_Hash.php(275): MediaWiki\Parser\Parser->braceSubstitution(array, PPFrame_Hash)
#16 /srv/mediawiki/1.42/includes/parser/Parser.php(2970): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#17 /srv/mediawiki/1.42/includes/parser/Parser.php(958): MediaWiki\Parser\Parser->replaceVariables(string, PPFrame_Hash)
#18 /srv/mediawiki/1.42/includes/specials/SpecialExpandTemplates.php(114): MediaWiki\Parser\Parser->preprocess(string, MediaWiki\Title\Title, ParserOptions)
#19 /srv/mediawiki/1.42/includes/specialpage/SpecialPage.php(719): MediaWiki\Specials\SpecialExpandTemplates->execute(NULL)
#20 /srv/mediawiki/1.42/includes/specialpage/SpecialPageFactory.php(1669): MediaWiki\SpecialPage\SpecialPage->run(NULL)
#21 /srv/mediawiki/1.42/includes/actions/ActionEntryPoint.php(504): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#22 /srv/mediawiki/1.42/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
#23 /srv/mediawiki/1.42/includes/MediaWikiEntryPoint.php(199): MediaWiki\Actions\ActionEntryPoint->execute()
#24 /srv/mediawiki/config/initialise/entrypoints/index.php(98): MediaWiki\MediaWikiEntryPoint->run()
#25 {main}

What should have happened instead?
I don't get an exception, but instead the result of the external data call

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

  • MediaWiki: 1.42.1 (bb2e9db) 06:06, 26 September 2024
  • PHP: 8.2.20 (fpm-fcgi)
  • MariaDB: 10.11.6-MariaDB-1:10.11.6+maria~deb12-log
  • External Data: 3.4-alpha (161575d) 12:39, 24 September 2024

Other information (browser name/version, screenshots, etc.):
Currently, the SQL table requires that post_vars be set: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ExternalData/+/7c6f4143d5fd65c8fa7baf06af93ea6ae1d3b653/sql/mysql/ExternalData.sql#8
However, the code for inserting rows into the table doesn't set it: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ExternalData/+/7c6f4143d5fd65c8fa7baf06af93ea6ae1d3b653/includes/connectors/traits/EDConnectorCached.php#163

Looking back when the cache table was first added, it seems like it doesn't require post_vars: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ExternalData/+/a3c26b8603fa7c15e33716d4faa4574647162996/ExternalData.sql#4
Even back then, post_vars wasn't set when inserting either: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ExternalData/+/a3c26b8603fa7c15e33716d4faa4574647162996/ED_ParserFunctions.php#158

The change in NOT NULL is caused by this commit: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ExternalData/+/56a8fb1b9140435e89478015292c6635edd77dac%5E%21/

Miraheze issue tracker task: https://issue-tracker.miraheze.org/T12679

Event Timeline

Restricted Application added a subscriber: Reception123. · View Herald Transcript

I reckon we should drop the post_vars column, I don't think we ever used it

Change #1077142 had a related patch set uploaded (by BlankEclair; author: BlankEclair):

[mediawiki/extensions/ExternalData@master] Fix new ExternalData installations by dropping post_vars

https://gerrit.wikimedia.org/r/1077142

I reckon we should drop the post_vars column, I don't think we ever used it

Perhaps. The POST vars are a part of HTTP $options array and are included in the cache key.

Change #1077142 merged by jenkins-bot:

[mediawiki/extensions/ExternalData@master] Fix new ExternalData installations by dropping post_vars

https://gerrit.wikimedia.org/r/1077142