7.3.1 Tag Drilldown and Faceting

Tags are a common and powerful tool for modeling data. Their flexibility makes them pervasive. While basic use cases for tags have simple implementations, providing richer query options can be quite tricky.

Tags generally have one of two search use cases: they are either used to filter, which is a straightforward task, or they are analyzed and matched against searches using the query, which is more complicated. We’ll cover both types of searches in this tutorial.

To understand our modeling for tags in elasticsearch we’ll quickly recap multi-valued fields. Any field in an elasticsearch document can store either one, or many values, depending on whether the data is set to be either an array, or a single value. Lucene does not actually have a notion of arrays, elasticsearch adds this on top, but it does allow any field to hold multiple values. This is an ideal fit for elasticsearch, where we may have a document with multiple tags. We can have one “tags” field in our document, typed as a string, and then provide multiple values for it.

Multi-valued fields work quite simply when the tag content is mapped as not_analyzed static tokens. In this case, a single match query will search the tokens with ease. Working with analyzed tags, however, is much harder. Let’s start by covering the simple case, indexing single token not_analyzed tags.

We’ll be able to re-use the schema in Figure 7.2. We’ll start by building a drilldown based on the department_name field, then move on building a free-text search that can work with the department_name_analyzed field.

7.3.2 Working With Non-Analyzed Tags

Let’s start by looking at the filtering using the department_name field, which has the not_analyzed option set. We’ll be loading a new dataset, products_multi_tagged.eloader, which is very similar to the products.eloader dataset used previously in this chapter. The main difference in this dataset is the addition of multiple tags per product. Let’s take a look at the new data by issuing the query in figure 7.7.

// Load Dataset: products_multi_tagged.eloader
POST products_multi_tagged/_search?pretty=true
{
  "query": {"match_all": {}},
  "facets":{
    "department_name": {"terms": {"field": "department_name"}}}
}

The responses in the facet section for the query in figure 7.7 should look something like that in figure 7.8. This is a single item from the array of results in the facets for the query. The rest has been omitted for brevity’s sake. Notice how multiple values have been set for the department_* fields.

{
  "department_name": ["Housewares", "Kitchen", "Breakast"],
  "department_name_analyzed": ["Housewares", "Kitchen", "Breakast"],
  "department_id": 2,
  "name": "TCPIntl.CoffeeMaker",
  "description": "Deluxe Coffee Maker, made by the trusted folks at TCP International.",
   "price": 4999
}

Searching our multi-valued data is an easy task for a match query. We can simply provide a boolean filter comprised of multiple match queries as must clauses. This will ensure that we match all documents that have exactly those tags. There’s a strong reason to use filters instead of queries in these cases: tags should not have any effect on the final score because we’re simply using them to restrict the dataset. Since similarity is based off calculations hinging on tf*idf scores, this can lead to nonsensically scored results depending on the frequency of filtered terms.

To search for items in both “Housewares” and “Kitchen”, we would issue a query such as that in figure 7.9. This will correctly return the one item we have in both of these departments. Notice that the filter terms match the department names exactly. Alternatively, we could have used the department_id field which is also available, but we’ll stick to names to make these examples more clear.

// Load Dataset: products_multi_tagged.eloader
POST products_multi_tagged/_search?pretty=true
{
  "query": {
    "filtered": {
      "filter": {
        "bool": {
       "should": [
         {"term": {"department_name": "Housewares"}},
         {"term": {"department_name": "Kitchen"}}
       ] }}}},
  "facets": {
    "department_name": {
      "terms": {"field": "department_name"}}}}

7.3.3 Working With Analyzed Tags

We’ve now seen how it’s possible to perform multi-tag facet drill-downs in the simple case where an exact term for each tag is known. For situations where we want tags considered as part of free-text search, things get trickier. In our previous examples we wanted to use our department_name field as a pure filter only. However, a search for “Kitchen TCP”, should turn up the coffee maker by TCP Industries first, not books on networking. This is the primary use-case we’ll analyze going forward in this section.

We’re going to solve this problem by using the position_offset_gap mapping option. If you examine the mapping for the products_multi_tagged index, you’ll notice that the mapping for department_name_analyzed has the setting "position_offset_gap": 1000. This option adjusts the document position data stored alongside terms in Lucene indexes. A position_offset_gap setting of n artificially inflates the distance of values in multi valued fields by a given amount. It is equivalent to concatenating all the values into a single value padded by n stop words. Altering our match with the phrase option, and setting the phrase slop to a number < n + the max length of a value, enables us to isolate matches to individual values within our multi-valued field. Since calculating that value can be tricky, it is recommended to use absurdly large values.

Let’s break down the query in figure 7.10. In this example we’ve changed quite a bit from figure 7.9. We’ve switched away from a filtered query, and have replaced it with a bool query.

// Load Dataset: products_multi_tagged.eloader
POST products_multi_tagged/_search?pretty=true
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "department_name_analyzed": {
              "type": "phrase",
              "slop": 10, 
              "query": "kitchen tcp"}}},
        {"match": {"name": "kitchen tcp"}},
        {"match": {"description": "kitchen tcp"}} ]}},
  "facets": {
    "department_name": {
      "terms": {"field": "department_name"}}}}

The intent behind this query is to handle free text input originating from a user facing search input. The user’s query is seen duplicated across all 3 clauses of the boolean query, a task normally done by application code dynamically building the query. A boolean query is used since we need to search across multiple fields with different options. Since each query in the should clause is a match query, and since multiple terms within a match query are matched with a logical ‘or’, we only need at least one term of the search phrase to match. In this case, at least one field must have the term “kitchen” or “tcp”. Documents that have both fields will score higher. Documents with both terms present more frequently and/or closer together will score higher still.

It is for these reasons that the top result is the “TCP Intl. Coffee Maker”. Its high score is due to its being tagged as “Kitchen” and containing the term term “TCP” in its title. Note that the word “Kitchen” is analyzed, as the lowercase “kitchen” still matches. Other documents do match, such as the kitchen spatula, and some TCP related networking products, but they do not rank as highly. This is the bare framework for implementing this kind of search, a full version would probably also include synonyms for “Kitchen” in the department_name_analyzed field, such as “Cooking”, “Food”, and “Culinary”.