Page MenuHomePhabricator
Create Task
Maniphest T183015

Create Serbian Elasticsearch Plugin/Analysis Chain Using Serbian Morphological Libraries
Closed, ResolvedPublic
Actions

Description

The consensus is that SCStemmers "Stemmer #4" works well for Serbian Wikipedia and Wiktionary text after a Cyrillic-to-Latin mapping upgrade. It doesn't lose any tokens, and it handles non-Serbian text well.

So, the next step is to create an Elasticsearch plugin based on the stemmer and test that out as part of an analysis chain. (Other elements of the analysis chain include folding accents used primarily to mark pitch accent, which are used in dictionary and encyclopedia entries, but not normal text, and probably additional general ICU folding.)

If it's a success for Serbian, the follow up would be to test it for Croatian, Serbo-Croatian, and Bosnian wikis.

Details

SubjectRepoBranchLines +/-
Initial commit of extra-analysis pluginsearch/extra-analysismaster+1 K -0
Enable Serbian-Specific Analysis Chainmediawiki/extensions/CirrusSearchmaster+290 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

TJones triaged this task as Medium priority.Dec 15 2017, 5:48 PM
TJones created this task.
TJones mentioned this in T178926: Review Serbian Morphological Libraries.Dec 15 2017, 5:55 PM
TJones moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Jan 16 2018, 3:06 PM
gerritbot subscribed.Mar 2 2018, 9:35 PM

Change 415788 had a related patch set uploaded (by Tjones; owner: Tjones):
[search/extra-analysis@master] [WIP] Initial commit of extra-analysis plugin

https://gerrit.wikimedia.org/r/415788

gerritbot added a project: Patch-For-Review.Mar 2 2018, 9:35 PM
TJones moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Mar 6 2018, 2:35 PM
TJones moved this task from Needs review to not in use - please delete on the Discovery-Search (Current work) board.Mar 6 2018, 6:33 PM
gerritbot added a comment.Mar 8 2018, 4:10 PM

Change 417299 had a related patch set uploaded (by Tjones; owner: Tjones):
[mediawiki/extensions/CirrusSearch@master] Enable Serbian-Specific Analysis Chain

https://gerrit.wikimedia.org/r/417299

TJones added a comment.Mar 8 2018, 4:18 PM
  • The Serbian stemmer plugin (in the new search/extra-analysis plugin) is just about ready, in Change 415788 above.
  • The Analysis config to use it, with additional analysis chain config, including diacritic folding, is in Change 417299 above.
  • My write up of my analysis chain analysis is available on MediaWiki.
    • Summary: enabled ICU folding with Serbian exceptions, and it works well with the stemmer, nothing unexpected.
TJones moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Mar 8 2018, 4:18 PM
gerritbot added a comment.Mar 8 2018, 7:09 PM

Change 415788 merged by jenkins-bot:
[search/extra-analysis@master] Initial commit of extra-analysis plugin

https://gerrit.wikimedia.org/r/415788

TJones mentioned this in T189239: Deploy initial version of the extra-analysis plugin.Mar 8 2018, 7:36 PM
gerritbot added a comment.Mar 8 2018, 10:30 PM

Change 417299 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Enable Serbian-Specific Analysis Chain

https://gerrit.wikimedia.org/r/417299

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)).Mar 8 2018, 11:01 PM
TJones moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Mar 8 2018, 11:06 PM
TJones mentioned this in T189265: Re-index Serbian Wikis.Mar 8 2018, 11:09 PM
TJones removed a subtask: T189265: Re-index Serbian Wikis.
Liuxinyu970226 unsubscribed.Mar 9 2018, 10:13 AM
Amire80 moved this task from Untriaged to Search on the I18n board.Mar 12 2018, 1:26 PM
TJones mentioned this in T138857: Serbian language search differentiates between Cyrillic and Latin alphabets.Mar 13 2018, 1:42 PM

When this is done and Serbian wikis are re-indexed (T189265), then T138857: Serbian language search differentiates between Cyrillic and Latin alphabets will be done, too.

debt closed subtask T189239: Deploy initial version of the extra-analysis plugin as Resolved.Apr 27 2018, 8:15 PM
debt closed this task as Resolved.Apr 27 2018, 8:18 PM
debt closed subtask T192395: Create Croatian, Serbo-Croatian, and Bosnian Analysis Chains Using Serbian Morphological Libraries as Resolved.Jun 14 2018, 8:08 PM
Restricted Application added a subscriber: • Petar.petkovic. · View Herald TranscriptJun 14 2018, 8:08 PM
• Petar.petkovic removed a project: Patch-For-Review.Jun 14 2018, 9:22 PM
TJones mentioned this in T200099: Create Esperanto Elasticsearch Plugin Using Esperanto Morphological Libraries.Jul 20 2018, 4:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits