Almost every "why does my search return nothing?" is the same bug. The analyzer that ran when the document was indexed and the analyzer that ran on the query produced different terms, so nothing matched — and because both sides look correct, you can stare at the query for an hour.
This lesson is about seeing the terms instead of guessing at them.
What an analyzer is
An analyzer turns a string into a list of terms. It has exactly three stages, always in this order:
"<p>Sunlit <b>Loft</b></p>"
|
v character filters (0 or more) — rewrite the raw string
"Sunlit Loft"
|
v tokenizer (exactly 1) — split it into tokens
["Sunlit", "Loft"]
|
v token filters (0 or more) — change, add or drop tokens
["sunlit", "loft"]Character filters operate on the whole string before anything is split. The
common one is html_strip, which removes markup:
curl -s localhost:9200/_analyze -H 'Content-Type: application/json' -d '{
"tokenizer": "standard",
"char_filter": ["html_strip"],
"text": "<p>Sunlit <b>Loft</b></p>"
}'
# ["Sunlit", "Loft"]The tokenizer decides where the boundaries are. There is exactly one, and it is the decision that matters most, because no later filter can put back a distinction the tokenizer threw away.
Token filters then reshape the tokens: lowercase them, strip accents, remove stop words, stem them, add synonyms.
The _analyze API
This is the tool. It runs an analyzer and shows you the tokens, without indexing anything.
curl -s localhost:9200/_analyze -H 'Content-Type: application/json' \
-d '{"analyzer": "standard", "text": "wi-fi Loft #2"}'Compare a few tokenizers on the same input and the point of choosing one becomes obvious:
standard ['wi', 'fi', 'loft', '2']
whitespace ['wi-fi', 'enabled', 'loft', '#2']
keyword ['wi-fi enabled loft #2']
simple ['wi', 'fi', 'loft']standard split the hyphen and kept the digit. whitespace kept the
hyphen and the hash but did not lowercase — it is a tokenizer with no filters.
keyword produced one token, which is the "analyzer" a keyword field
effectively uses. simple dropped the digit entirely, because it splits on anything
that is not a letter.
If your search for "wi-fi" fails, one of those four behaviours is why, and thirty seconds with
_analyze tells you which.
Analysing by field, which is the one you want
Naming an analyzer tests an analyzer. Naming a field tests what your index is actually doing:
curl -s localhost:9200/stayhub-properties/_analyze -H 'Content-Type: application/json' \
-d '{"field": "city", "text": "San Francisco"}'
# ['san', 'francisco']
curl -s localhost:9200/stayhub-properties/_analyze -H 'Content-Type: application/json' \
-d '{"field": "city.raw", "text": "San Francisco"}'
# ['San Francisco']Same input, same document, two fields, two completely different indexes behind them. That is the
multi-field from lesson 3 made visible, and it is why aggregations use city.raw and
matching uses city.
explain: true, when the tokens are not enough
Add "explain": true and you get the output of every stage rather than just the end:
curl -s localhost:9200/stayhub-properties/_analyze -H 'Content-Type: application/json' \
-d '{"analyzer": "stayhub_text", "text": "Málaga LOFTS", "explain": true}'tokenizer: standard ['Málaga', 'LOFTS']
filter: lowercase ['málaga', 'lofts']
filter: asciifolding ['malaga', 'lofts']You can see the accent survive the tokenizer, survive lowercasing, and disappear at
asciifolding. When a custom analyzer does something you did not expect, this shows you
which filter did it.
The built-in analyzers
standard is the default and a good one: Unicode-aware word splitting, lowercasing,
no stop words. Use it until you have a reason not to.
english (and its equivalents for other languages) adds stop words and stemming, and
the difference is dramatic:
standard "The Cabins are Running" -> ['the', 'cabins', 'are', 'running']
english "The Cabins are Running" -> ['cabin', 'run']Four tokens become two. "the" and "are" are dropped as stop words; "cabins" is stemmed to
cabin and "running" to run. Now a search for "cabin" finds documents
containing "cabins", which is usually what a user means.
Usually. Stemming is lossy and occasionally embarrassing — it is an algorithm, not a dictionary, so it produces stems that are not words and sometimes collapses words that should stay apart. And stop words mean a search for the band "The The" finds nothing at all.
Whether to use a language analyzer depends on the field. Descriptions: yes. Titles, where the exact words matter more: often not. Names, product codes, cities: no.
Building your own
A custom analyzer is a named combination, declared in the index settings:
"analysis": {
"analyzer": {
"stayhub_text": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "asciifolding"],
}
}
},Then a field asks for it by name:
"title": {"type": "text", "analyzer": "stayhub_text"},Two filters, and the second one is doing real work. asciifolding strips diacritics,
so "Málaga" is indexed as malaga — and a guest who does not type the
accent, which is most of them, still finds the listing. Without it they get nothing and conclude
the city has no listings.
The order matters and is not arbitrary: filters run in sequence, so
["lowercase", "asciifolding"] lowercases first. With most Latin text either order
works; with Turkish dotted and dotless I it does not, which is the general warning about assuming
filter order does not matter.
The tokenizers worth knowing by name
There are a dozen or so. Five cover nearly everything.
standard splits on word boundaries using the Unicode text segmentation rules. It
handles most languages sensibly and is the right default.
keyword emits the input unchanged as a single token. Useless on its own, and the
building block for a normalizer — keyword tokenizer plus lowercase gives you
case-insensitive exact matching.
whitespace splits only on whitespace, keeping punctuation attached. Right for
identifiers and code-like values where ORD-1234 must stay one token.
pattern splits on a regular expression, which is the escape hatch for a format
nothing else understands.
edge_ngram emits every prefix, which is the autocomplete building block shown
below. Its sibling ngram emits every substring, which supports
"contains" matching and produces an enormous index — a 20-character field with
min_gram: 2, max_gram: 20 is hundreds of terms per value. Reach for it knowingly.
Worth adding path_hierarchy to the list, because it solves a problem people
usually solve badly: it turns /us/ca/san-francisco into /us,
/us/ca, /us/ca/san-francisco, so a filter on any level of a hierarchy is
one term query.
Positions, and why phrase search depends on them
An analyzer emits more than the token text. Each token carries a position and character offsets, and both matter.
Positions are what make match_phrase possible. Searching for the phrase
"mountain views" is really "the term mountain at some position n, and
views at n+1". If a token filter removes or reorders tokens, the positions
shift and phrases break in ways that look arbitrary.
The classic instance is stop words. Remove "the" from "the mission" at index time and the two remaining terms sit next to each other; a phrase search for "in the mission" then depends on whether the query side removed it too. This is the real reason modern advice is to leave stop words in and let BM25 handle common terms — it does that well, and it never breaks a phrase.
Offsets are what highlighting uses to find the original substring to wrap in
<mark>. A character filter that changes the string length — such as
html_strip — adjusts offsets so highlighting still lands on the right characters
in the original text.
The bug, in full
Analysis happens twice: once when a document is indexed, once when a query runs. Elasticsearch uses the field's analyzer for both, which is why this mostly works without thinking about it.
You can override the query side, and that is where it goes wrong. Here is an index whose field uses one analyzer to index and a different one to search:
{
"settings": { "analysis": { "analyzer": {
"idx": { "tokenizer": "standard", "filter": ["lowercase"] },
"qry": { "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] }
} } },
"mappings": { "properties": {
"t": { "type": "text", "analyzer": "idx", "search_analyzer": "qry" }
} }
}Index "Málaga" and search for it. Both ways:
query 'malaga' -> 0 hits
query 'málaga' -> 0 hitsZero for both — including the query where the user spelled it exactly as it is
stored. Work it through: the index holds málaga, because idx does not
fold. Every query goes through qry, which folds, so both queries become
malaga. That term is not in the index. Nothing can ever match this field.
No error, no warning, no clue in the query. The only thing that reveals it is running
_analyze on both sides and noticing they disagree.
When a different search analyzer is right
The feature exists for a reason: autocomplete. Index with an edge_ngram filter so
"cabin" is stored as every prefix —
edge_ngram(2,6) "cabin" -> ['ca', 'cab', 'cabi', 'cabin']— and search with a plain analyzer, so a user typing "cab" produces the single term
cab and matches. If you n-grammed the query too, "cab" would become
['ca', 'cab'] and start matching things it should not.
That is the rule: a different search analyzer is correct when it is deliberately narrower than the index one. It is a bug whenever the two merely drifted apart.
Two traps that follow
keyword fields are not analysed at all
No analyzer runs, so lowercase does not apply and neither does
asciifolding. A keyword field holding "Cabin" will never
match "cabin".
The fix for keyword fields is a normalizer, which is an analyzer with no tokenizer — character filters and token filters only, producing exactly one token:
{ "settings": { "analysis": { "normalizer": {
"lowercase_exact": { "type": "custom", "filter": ["lowercase", "asciifolding"] }
} } } }Use it on email addresses, usernames and slugs, where you want exactness and case-insensitivity at the same time.
Changing an analyzer does nothing to existing documents
This is lesson 3's constraint wearing a different hat. Analysis happens at write time, so the terms in the index were produced by whatever the analyzer was then. Change it and the old documents keep their old terms.
Worse, the change is not even allowed on an open index — adding an analyzer requires closing the index, and pointing an existing field at a different one is refused outright, because it would make the field's stored terms unreachable.
So the sequence for an analysis change is always: new index, new mapping, reindex, swap. Which is lesson 15, and is the third time in five lessons that the answer has been the same. That repetition is the argument for reading lesson 15 before you need it.
A worked example: autocomplete
Putting the pieces together on a real requirement — a city box that suggests as the user types.
{
"settings": { "analysis": {
"filter": {
"city_prefix": { "type": "edge_ngram", "min_gram": 2, "max_gram": 15 }
},
"analyzer": {
"city_index": { "tokenizer": "standard",
"filter": ["lowercase", "asciifolding", "city_prefix"] },
"city_search": { "tokenizer": "standard",
"filter": ["lowercase", "asciifolding"] }
}
} },
"mappings": { "properties": {
"city_suggest": { "type": "text",
"analyzer": "city_index",
"search_analyzer": "city_search" }
} }
}"San Francisco" is indexed as sa, san, fr, fra, fran, franc, ... — every
prefix of every token. A user typing fran produces the single term fran,
which matches. Typing f produces nothing, because min_gram is 2; that is
deliberate, since one-letter prefixes match nearly everything and are not worth the index.
Three things this gets right that the naive version does not. The
search_analyzer does not n-gram, for the reason above. Accents are folded on
both sides, so typing "malaga" suggests "Málaga". And because this is an ordinary
text field, the suggestions can be combined with the user's other filters in the same
bool query — which the dedicated completion type cannot easily do.
The cost is index size. Every value becomes a dozen or more terms. On a field of city names that is nothing; do the same to descriptions and the index will surprise you.
More than one language
If your content is in several languages, one analyzer cannot be right for all of it — an English stemmer applied to German produces nonsense. Two approaches work.
A field per language, populated by whichever one applies:
description_en, description_es, each with its own analyzer, queried with
a multi_match across all of them. Simple, and the mapping grows with the languages you
support.
An index per language, with the same field names, searched together through an alias or a comma-separated index list. Better when the languages have very different volumes, since each index can be sized independently.
What does not work is guessing the language at query time from the query itself. Two words are not enough to identify a language, and a wrong guess applies the wrong stemmer to everything.
Synonyms, briefly
A synonym filter maps terms to other terms:
{ "filter": { "stay_synonyms": {
"type": "synonym_graph",
"synonyms": ["flat, apartment", "cabin, cottage, chalet"]
} } }Two decisions come with it. Index-time or query-time? Query-time is almost
always right — a synonym list changes often, and at query time changing it takes effect
immediately instead of requiring a reindex. Use synonym_graph in the
search_analyzer, which is one of the legitimate uses of that setting.
Where does the list live? Inline in the mapping is fine for ten entries and unmanageable at a thousand. The alternative is a file on every node, or the synonyms API, which stores the set in the cluster and can be reloaded.
Be sparing. Every synonym broadens matching, and broad matching makes relevance worse in ways that are hard to attribute later.
Keeping analysis honest as the index changes
Analysis config is the part of a mapping most likely to be edited by someone who is not thinking about the documents already in the index. Two habits keep that from becoming an incident.
Test the analyzer, not the search. A test that indexes a document and asserts a
search returns it passes for many wrong reasons — a different clause matched, a fuzzy
expansion covered the gap. Asserting on _analyze output pins the actual behaviour:
def analyze(text: str, *, analyzer: str | None = None, field: str | None = None) -> list[str]:
body: dict = {"text": text}
if analyzer:
body["analyzer"] = analyzer
if field:
body["field"] = field
result = es.indices.analyze(index=ALIAS, body=body)
return [t["token"] for t in result["tokens"]]With that, assert analyze("Málaga", analyzer="stayhub_text") == ["malaga"] is
a one-line regression test for the entire accent-folding behaviour, and it fails the moment somebody
removes the filter.
Treat an analysis change as a schema migration, because it is one. It needs a new index and a reindex, the same as a type change, and it deserves the same review. The dangerous version is the one that looks small — adding a stemmer, adding a stop-word list — because the change is accepted on the new documents and nobody notices that half the index now behaves differently from the other half.
A routine for when search returns nothing
In order, and it takes about two minutes.
One. Confirm the document is there at all: GET /index/_doc/<id>,
or match_all. If it is not, this is an indexing problem, not an analysis one.
Two. Run _analyze with "field" set to the field you are
querying, and the document's text. Those are the terms in the index.
Three. Run it again with the query string. Those are the terms being looked for.
Four. Compare the two lists. If they share nothing, you have found it. If they
do overlap, the analysis is fine and the problem is the query — most often a
term where a match belongs, or an operator: and requiring
every term when the document only has some.
That routine turns the vaguest failure in Elasticsearch into a two-list comparison, and it works
whether the cause is a keyword field, a stemmer, a missing asciifolding, or an
analyzer somebody changed six months ago and never reindexed.