Jump to content

Extension:MediaWikiFarm

Issue tracker: #mediawiki-extensions-mediawikifarm
From mediawiki.org
MediaWiki extensions manual
MediaWikiFarm
Release status: beta
Implementation MyWiki
Description Creates single- or multi-versions MediaWiki farms.
Author(s) Seb35talk
Latest version 0.5.0 (2017-04-09)
MediaWiki 1.3+
PHP 5.2+
License GNU General Public License 3.0 or later
Download
Download extension
Git [?]:
Parameters
  • $wgMediaWikiFarmConfigDir
  • $wgMediaWikiFarmSyslog
  • $wgMediaWikiFarmCacheDir
Hooks used
Quarterly downloads 13 (Ranked 116th)
Translate the MediaWikiFarm extension if it is available at translatewiki.net
Issues Open tasks · Report a bug

The MediaWikiFarm extension creates wiki farms (multiple wikis grouped together) with the following main features:

  • Multi-versions: Each wiki can be set in a given version, and all wikis use the same MediaWiki codebase for a given version;
  • Hierarchical configuration: The configuration is written in a hierarchical manner in YAML, PHP, or JSON: default value, default value for the wikis family, value for a single wiki;
  • Performance: Caching, caching, caching; there is one cache per file, caches are written as PHP arrays to benefit from OPcache extension and a final per-wiki LocalSettings.php is issued; with this cache architecture, this extension adds a mean time of 85”s per request compared to a standard LocalSettings.php (on a personal computer; PHP7+OPcache).

Detailed features

[edit]
  • MediaWiki farm: each wiki is identified by its host, with dedicated version, configuration, and data directories;
  • Multi-versions farm: each wiki is one of the installed MediaWiki versions, with extensions and skins selected for this version, and extensions and skins activated or not for this wiki;
  • Hierarchical configuration: each parameter can be specified at different scales: for every wiki in the farm, for a specific family, or for a single wiki;
  • Multi-files configuration: the configuration can be distributed accross files with possibly different permissions, e.g. all passwords in a secret file, other parameters in a public file;
  • Configuration in YAML, JSON, or PHP: these three file formats are supported (YAML needs an external library);
  • Transparent from MediaWiki point of view: the configuration is a classic LocalSettings.php (after compilation from the configuration files) and the initial bootstraping code is hiding itself as much as possible;
  • Per-wiki activation of extensions, even Composer-managed extensions: each extension can be activated per-wiki; Composer-managed extensions must be handled specifically during their installation, but they become then per-wiki activable (together with their dependent librairies and extensions obviously);
  • Caches: all configuration files and final configuration are cached as PHP arrays, interoperable accross PHP versions and cacheable in OPcache;
  • Per-domain and/or per-path URLs: selection of the wiki can be either per-domain (wiki1.org, wiki2.org), per-path (wiki.org/wiki1, wiki.org/wiki2), or a combination of both methods
  • CLI compatible: a command-line script selects the right wiki just like in the Web version;
  • Multi-farms: multiple farms can be created, each one with its own configuration files, possibly shared accross farms to e.g. create pre-production wikis with slightly different configurations;
  • Syntax-error-proof: syntax errors in configuration files do not crash the farm, just silently ignored when cache is activated;
  • Delayed version switch: during version switch, the version does not change until the maintenance/update.php script is run;
  • Custom 404: it can display a customised HTTP 404 page for missing wikis, written in PHP or HTML;
  • Classical LocalSettings.php: although only visible in the cache directory, there is a classical LocalSettings.php;
  • Whole range of MediaWiki versions: every MediaWiki version from 1.1 to 1.28 works in multi-versions farm (small issue for 1.1 and 1.2 still pending);
  • Wide range of PHP versions: every PHP version from 5.2 to 7.1 works (YAML library requires 5.3.2 or 5.5.9, so YAML-to-PHP must be handled separately for old MediaWiki versions);
  • Documented: all functions are documented and a general documentation is available in a subdirectory;
  • Translated: the description on Special:Version is translatable on TranslateWiki although, well, there is only string in the user interface;
  • Unit-tested, well: about 100 tests (~256 assertions) run in 10-15 seconds in the MediaWiki-PHPUnit framework with recent MediaWiki versions with PHP 5.6 to 7.1, totalling almost 100% code coverage with strict coverage activated and global variables under watch; standalone PHPUnit also works (runs in 0.5 seconds);
  • Performance-tested: a basic testing infrastructure measures the performance in order to be aware of the performance impact of various strategies during development (e.g. the per-wiki LocalSettings.php was added because it was 32% (77”s/239”s) quicker than the previous implementation).

Installation

[edit]

There are two installation modes:

  • an almost classical installation in the extensions/ subdirectory, but limited to single-version MediaWiki farms;
  • a more complex installation for multi-versions MediaWiki farms.

Be aware it is non-trivial to set up a MediaWiki farm; this extension only improves the MediaWiki part, after that it remains farm-type configuration of: DNS, webserver, MySQL, HTTPS, domain names, external services (memcached, Parsoid, *oid
), this is a job for a configuration management tool (puppet, Chef, Ansible, Cfengine, etc.). Installation is one thing, but you should read the documentation to understand the goals, the basic concepts, and prepare your naming scheme (the URLs) and directories before you start installing it.


See also: Installation


Development

[edit]

Overview

[edit]
Static call graph of extension code, featuring the three classes.
An alternative view is a visualisation of the interactions between data (object properties) on this page "MediaWikiFarm static analysis"

This extension is the core to run MediaWiki farms, and it is particularly important it runs: 1/ quickly, 2/ on all MediaWiki versions, 3/ on all PHP versions since 5.2. Unfortunately it makes development with more constraints but it can be taken as a challenge :) If a feature cannot meet these requirements, possibly oldest MediaWiki versions can run in a degraded mode (it is already the case because the YAML library does not run on PHP 5.2), but it should be avoided as much as possible. The syntax must be understood by the PHP 5.2 parser (notably no namespaces) and some care must be taken for some details (no short array syntax, no __DIR__). Non-core features (e.g. create statistics) should be created in a separate extension.

Before development, you have to install the farm on your local computer. It was only tested on GNU/Linux Debian with nginx, so please report if you have issues on other systems. The next advices are for the development tools in use; it is better if you can follow them, but it’s fine if you create code review requests without it, if you don’t know these tools. You can browse the documentation in the docs subdirectory and you can run PHPDoc to get a nice HTML summary of the methods (see in Quality tools below how to run it).

You can send code review requests (pull requests in GitHub language) with Gerrit (quick cheat guide: install git-review, create an account and set up your credentials in Gerrit, "git clone" the repository, "git review -s" in the repo, "git review -R" to submit a request, "git commit -a --amend" then "git review -R" to create an amended revision).

Quality tools

[edit]

A PHPUnit test suite can be run with the following command, you have to run it in your farm with a valid MediaWiki version greater than 1.19 (when PHPUnit tests were introduced) (PHPUnit must be installed with Composer’s require-dev in the considered MediaWiki). It should not alter the database, but it cannot be guaranteed. To get coverage results, you have to modify the file [MW]/tests/phpunit/suite.xml and add in the ‎<whitelist> section something like: <directory suffix=".php">/your/farm/directory</directory><exclude><directory suffix=".php">/your/farm/directory</directory>. If possible, please write tests to keep the 100% code coverage. 

 php bin/mwscript.php --wiki=WIKI.YOUR-LOCAL-FARM.net tests/phpunit/phpunit.php --coverage-html coverage-MediaWikiFarm --strict-coverage --group MediaWikiFarm -v

A performance test can be run with a dedicated small script. Before it, you have to change your webserver config so that the main entry point index.php must be /your/farm/directory/tests/perfs/index.php and check MediaWiki runs without errors when you navigate with this special entry point (if there are errors like 500, the performance script will be quicker but will not issue errors). You can set the sample size on the command line (5000 recommended before issuing a code review request). 

 php tests/perfs/perfs.php --wiki=WIKI.YOUR-LOCAL-FARM.net https://WIKI.YOUR-LOCAL-FARM.net 5000

It is better to run the code syntax tool PHP_CodeSniffer (you have to run the Composer’s require-dev section in farm directory). 

 ./vendor/bin/phpcs

A JSON schema is available in the subdirectory docs to describe the main config file farms.yml; it should be updated when needed.

A developer documentation is created with PHPDoc (you have to run the Composer’s require-dev section in farm directory). Please document the new methods/parameters/etc. 

 composer phpdoc

Testing

[edit]

The extension has been tested with a multi-versions installation with MediaWiki ranging from 1.1 to 1.28alpha, with PHP 5.2 (MediaWiki 1.1-1.12), PHP 5.6 (MediaWiki 1.13-1.21), PHP 7.0 (MediaWiki >= 1.22), and PHP 7.1RC1 (MediaWiki 1.28alpha), with nginx 1.6.2 with fastcgi module. One minor issue is still pending to run MW 1.1 and 1.2.

See also User:Seb35/MediaWiki Archaeology.

There is a unit test suite with about 100 tests (250 assertions). This can be run with the customised MediaWiki PHPUnit (see tests/phpunit/phpunit.php in MW directory). This specific test suite can be run separately with --group MediaWikiFarm. It should run without error with strict coverage (--strict-coverage), with a total code coverage of 100%. Only raw scripts and some untestable methods are code-coverage-ignored (4 methods out of 38). Note there is a global function that PHPUnit doesn’t code-coverage, see [1].

MediaWiki deactivated backup of global variables, but it can be re-enabled in this test suite by modifying the object property in constructor of MediaWikiFarmTestCase (slightly slower); in this case, expectedly-modified global variables are declared at the beginning of the test and any other global variable modified reports the test as risky. If a test is really long, possibly PHPUnit computes the differences of values of global variables between the beginning and the end of the test (count 10 minutes for a heavily modified global state).

A performance test is implemented in subdirectory tests/perfs. It is recommended to use it only in a development environment. To use it, you must have the farm installed and point, in the webserver config, the index.php to tests/perfs/index.php, then launch in command line the script php tests/perfs/perfs.php http://WIKI.YOUR-LOCAL-FARM.net [sampleSize]. You must either delete the cache directory just before or give permissions to the script to do it. The first execution creates the cached LocalSettings.php, so it’s normal to see a "high" time in 'config farm', but it then divided by the sample size. With PHP 7.0 and MediaWiki 1.28alpha with the farm cache activated, I have figures (sample size=5000): 157”s for bootstraping farm, 219”s for executing the LS.php of farm (most (~177”s) of it is initialisation of ExtensionRegistry), 221”s for executing the classical LS.php (the two LS.php are strictly the same file, so it is not significant), and 9.984ms for the compilation of configuration (when cache is invalidated). With the same sample size, after the introduction of the cache for server names, the bootstraping time dropped to 85”s.

Future

[edit]

See phabricator:tag/mediawiki-extensions-mediawikifarm for a list of tasks.

Here are other tasks to be transferred in Phabricator:

  1. logging is now implemented, next step is a better handling of errors/warnings
  2. create integration tests (mock some MW parts and check for instance an extension is really activated, or test scenarios like updating a config file or transitionning from no-cache to cache to no-cache)
  3. issue a version 1.0
    1. define the list of features above of what should be in version 1.0
    2. proofread the documentation
    3. last check on the architecture and method names to avoid a major change in the near-future (see also phabricator:T162686)
  4. Check when a skin is default skin and a Composer-installed one, but is not activated (example Chameleon)
  5. Check existence of a wiki in the config files, similarly to this issue
  6. handle constants like CACHE_NONE, or more generally PHP syntax to use e.g. references; possibly this would be another config file with list of PHP-enabled parameters (or even a restricted list of PHP syntaxes authorised)

See also

[edit]
[edit]
Retrieved from "https://www.mediawiki.org/w/index.php?title=Extension:MediaWikiFarm&oldid=6108987"