Search inside an array object in Elastic Search

Question

Is it possible to search without using nested fields, achieving exact matches within the same object in an array in ElasticSearch?

I can't change the structure of the object.

For example:

{
  "cardaccdetails": [
    {
      "mkt_code": "",
      "cid": ""
    },
    {
      "mkt_code": "",
      "cid": ""
    },
    {
      "mkt_code": "",
      "cid": ""
    },
    {
      "mkt_code": "",
      "cid": ""
    }
  ]
}

How can filter the data such that mkt_code and cid are from same array object?

Answer 1

You can use runtime mapping with painless script for archiving your logic. But this is not recommended as it might impact the performance if you have large dataset.

Lets consider, you have data like below in Elasticsearch index:

"hits": [
      {
        "_index": "test1",
        "_id": "1",
        "_score": 1,
        "_source": {
          "id": "1",
          "cardaccdetails": [
            {
              "mkt_code": "101",
              "cid": "A"
            },
            {
              "mkt_code": "201",
              "cid": "B"
            }
          ]
        }
      },
      {
        "_index": "test1",
        "_id": "2",
        "_score": 1,
        "_source": {
          "id": "1",
          "cardaccdetails": [
            {
              "mkt_code": "101",
              "cid": "B"
            },
            {
              "mkt_code": "201",
              "cid": "A"
            }
          ]
        }
      }
    ]

You can use below query:

POST test1/_search
{
  "runtime_mappings": {
    "isMatch": {
      "type": "boolean",
      "script": {
        "source": """
          for (item in params._source['cardaccdetails']){
            if(item['mkt_code']== '101' && item['cid']== 'A')
                emit(true);
          }
          emit(false);
          """
      }
    }
  }, 
  "query": {
    "match": {
      "isMatch": "true"
    }
  }
}

This will return only one document which is having 101 and A into same object inside array.

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test1",
        "_id": "1",
        "_score": 1,
        "_source": {
          "id": "1",
          "cardaccdetails": [
            {
              "mkt_code": "101",
              "cid": "A"
            },
            {
              "mkt_code": "201",
              "cid": "B"
            }
          ]
        }
      }
    ]
  }
}

You can put this query as Search template and pass value as parameter.

Answer 2

Is it possible to search without using nested fields, achieving exact matches within the same object in an array in ElasticSearch?

Short answer: No

Explanation: In Elasticsearch, non-nested fields within an array are “flattened” at indexing time, so Elasticsearch doesn’t inherently recognize that individual field values in an array of objects are associated within the same object. Without the nested field type, fields in an array of objects are treated as if they were part of a single object, which means searches can’t distinguish between values belonging to separate objects.

Official explanation: The nested type is a specialised version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other. https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

Workaround: If you’re unable to use nested fields, another option is to restructure your data to avoid arrays of objects.