Extra Queries and Filters

The plan is for this to include any extra queries, filters, native scripts, score functions, and anything else we think we end up creating to make search nice for Wikimedia. It contains 2 diffferent plugins:

extra

The extra plugin contains utilities that are generally useful.

Queries:

• source_regex - An nGram accelerated regular expression filter that is generally much much faster than sequentially checking all documents.
• token_count_router - Simple query wrapper that evaluates some conditions based on the number of tokens of the input query.
• simswitcher - Simple query wrapper that allows to override similarity settings at query time (expert: use with caution). * term_freq - Simple term query with filtering based on term frequency.

Native Scripts:

• super_detect_noop - Like detect_noop but supports configurable sloppiness. New in 1.5.0, 1.4.1, and 1.3.1.

Analysis:

• preserve_original - A token filter that wraps a filter chain to keep and emit the original term at the same position. New in 2.3.4.
• term_freq - A token filter to populate the term frequency from the input string. New in 5.5.2.6.

extra-analysis-slovak

This plugin contains a Slovak stemmer.

Analysis:

• slovak_stemmer - A token filter that provides stemming for the Slovak language. New in 5.5.2.4.

extra-analysis-surrogates

This plugin contains a filter to merge UTF-32 high and low surrogates that have been split, particularly by pre–ES 6.4 smartcn.

Analysis:

• surrogate_merger - New in 5.5.2.8.

Installation

Install it like so for Elasticsearch x.y.z:

<= 2.4.1

./bin/plugin --install org.wikimedia.search/extra/x.y.z

>= 5.1.2

./bin/elasticsearch-plugin install org.wikimedia.search:extra:x.y.z
./bin/elasticsearch-plugin install org.wikimedia.search:extra-analysis-slovak:x.y.z
./bin/elasticsearch-plugin install org.wikimedia.search:extra-analysis-ssurrogates:x.y.z

Build

Spotbugs is run during the verify phase of the build to find common issues. The build will break if any issue is found. The issues will be reported on the console.

To run just the check, use mvn spotbugs:check on a project that was already compiled (mvn compile). mvn spotbugs:gui will provide a graphical UI that might be easier to read.

Like all tools, spotbugs is much dumber than you. If you find a false positive, you can ignore it with the @SuppressFBWarnings annotation. You can provide a justification to make document why this rule should be ignored in this specific case. Some rules don't make sense for this project and they can be ignored via src/dev-tools/spotbugs-excludes.xml.