For what it is worth, external search providers such as Google find the article just fine.

I was not able to duplicate this problem on dewiki

We need to update the documentation. But at least:

• "near realtime" means usually a few minutes, but up to 30 minutes is still considered normal conditions. This applies to the full text search. The title suggest index (used for the auto complete in the search box) is updated daily.
• We have a process that reindexes all pages over a period of 8 weeks. So if an update is lost, it will eventually be re-applied.

It is unlikely that we'll be able to investigate the indexing failures. We are working on a new version of our update pipeline (T317045) that should give us better visibility into search failures.

I tried to clarify the section of docs about updates to include the distinction between full-text and title completion search indexes: https://www.mediawiki.org/w/index.php?title=Help:CirrusSearch&diff=prev&oldid=5897819

To check that titles are making it into the primary search index i ran a quick python script (P47281) and ran it for the last 7 days worth of new pages according to the recent changes api. It found 12572 pages that were created and should exist in the enwiki search index. Of these 12 were not found in the search index. A manual check shows them all to be redirects to redirects which we don't index. This looks to be generally working, although there could certainly be edge cases that are not handled.

Poking at the logs for the script that builds the daily autocomplete indices, we may be missing errors that happen there. The logs show that the enwiki completion index failed its daily build from dec 9 2022 through jan 20 2023. This was not identified by any of our monitoring, we should correct that so these errors bubble up sooner and get fixed immediately.

We can use an elasticsearch query to find the oldest dated completion indices. This query will give us the 5 titlesuggest indices with the oldest batch_id (~= indexing timestamp) when issued against the :9243 cluster:

POST /omega:*_titlesuggest,psi:*_titlesuggest,*_titlesuggest/_search
{
    "size": 0,
    "aggs": {
        "by_index": {
            "terms": {
                "field": "_index",
                "order": {"max_batch_id": "asc"}
            },
            "aggs": {
                "max_batch_id": {"max":{"field": "batch_id"}}
            }
        }
    }
}

Not sure yet where it goes. We could integrate this into prometheus-wmf-elasticsearch-exporter and then add an alertmanager rule, i suppose that would be the most straightforward. We could record the age (now - batch_id) of the oldest batch per-cluster and alert if the age gets more than a couple days. It feels a little overkill to run this query every minute though.

Change 911940 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/puppet@production] search: Report age of titlesuggest indices to prometheus

https://gerrit.wikimedia.org/r/911940

Change 911945 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/alerts@master] search: Add alert based on age of titlesuggest indices

https://gerrit.wikimedia.org/r/911945

Change 911940 merged by Ryan Kemper:

[operations/puppet@production] search: Report age of titlesuggest indices to prom

https://gerrit.wikimedia.org/r/911940

Change 913959 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/puppet@production] search: Fix collection of *_titlesuggest metric on small clusters

https://gerrit.wikimedia.org/r/913959

Change 913959 merged by Ryan Kemper:

[operations/puppet@production] search: Fix collection of *_titlesuggest metric on small clusters

https://gerrit.wikimedia.org/r/913959

Change 911945 merged by jenkins-bot:

[operations/alerts@master] search: Add alert based on age of titlesuggest indices

https://gerrit.wikimedia.org/r/911945