Configuring Text Analysis Schemes for Amazon CloudSearch

Amazon CloudSearch enables you to configure a language-specific analysis scheme for each text and text-array field. An analysis scheme controls how the contents of the field are processed during indexing. Although the defaults for each language work well in many cases, fine-tuning the analysis options enables you to optimize the search results based on your knowledge of the data you are searching. For a list of supported languages, see Supported Languages.

An analysis scheme specifies the language of the text to be processed and the following analysis options:

During text processing, field values and search terms are converted to lowercase (case-folded), so stopwords, stems, and synonyms are not case-sensitive. For more information about how Amazon CloudSearch processes text during indexing and when handling search requests, see Text Processing in Amazon CloudSearch.

You must specify a language for each analysis scheme and configure an analysis scheme for each text and text-array field. When you configure fields through the Amazon CloudSearch console, the analysis scheme defaults to the _en_default_ analysis scheme. If you do not specify analysis options for an analysis scheme, Amazon CloudSearch uses the default options for the specified language. For information about the defaults for each language, see Language Specific Settings.

The easiest way to define analysis schemes is through the Analysis Schemes page in the Amazon CloudSearch console. You must apply an analysis scheme to a field for it to take effect. You can apply an analysis scheme to a field from the Indexing Options page. You can also define analysis schemes and configure an analysis scheme for each field through the command line tools and AWS SDKs.

When you apply a new analysis scheme to an index field or modify an analysis scheme that's in use, you must explicitly rebuild the index for the changes to be reflected in search results.

Stemming in Amazon CloudSearch

Stemming is the process of mapping related words to a common stem. A stem is typically the root or base word from which variants are derived. For example, run is the stem of running and ran. Stemming is performed during indexing as well as at query time. Stemming reduces the number of terms that are included in the index, and facilitates matches when the search term is a variant of a term that occurs in the content being searched. For example, if you map the term running to the stem run and then search for running, the request matches documents that contain run as well as running.

Amazon CloudSearch supports both algorithmic stemming and explicit stemming dictionaries. You configure algorithmic stemming by specifying the level of stemming that you want to use. The available levels of algorithmic stemming vary depending on the language:

In addition to controlling the degree of algorithmic stemming that's performed, you can specify a stemming dictionary that maps specific related words to a common stem. You specify the dictionary as a JSON object that contains a collection of string:value pairs that map a term to its stem, for example, {"term1": "stem1", "term2": "stem2", "term3": "stem3"}. The stemming dictionary is applied in addition to any algorithmic stemming. This enables you to override the results of the algorithmic stemming to correct specific cases of overstemming or understemming. The maximum size of a stemming dictionary is 500 KB. Stemming dictionary entries must be lowercase.

You use the StemmingDictionary key to define a custom stemming dictionary in an analysis scheme. Because you pass the dictionary to Amazon CloudSearch as a string, you must escape all double quotes within the string. For example, the following analysis scheme defines stems for running and jumping:

{
    "AnalysisSchemeName": "myscheme",
    "AnalysisSchemeLanguage": "en",
    "AnalysisOptions": {
        "AlgorithmicStemming": "light",
        "StemmingDictionary": "{\"running\": \"run\",\"jumping\": \"jump\"}"
    }
} 

If you do not specify the level of algorithmic stemming or a stemming dictionary in your analysis scheme, Amazon CloudSearch uses the default algorithmic stemming level for the specified language. While stemming can help users find relevant documents that might otherwise be excluded from the search results, overstemming can result in too many matches with questionable relevance. The default level of algorithmic stemming configured for each language works well for most use cases. In general, it's best to start with the default and then make adjustments to optimize the relevance of the search results for your use case. For information about the defaults for each language, see Language Specific Settings.

Stopwords in Amazon CloudSearch

Stopwords are words that should typically be ignored both during indexing and at search time because they are either insignificant or so common that including them would result in a massive number of matches.

During indexing, Amazon CloudSearch uses the stopword dictionary when it processes text and text-array fields. In most cases, stopwords are not included in the index. The stopword dictionary is also used to filter search requests.

A stopwords dictionary is a JSON array of terms, for example, ["a", "an", "the", "of"]. The stopwords dictionary must explicitly list each word that you want to ignore. Wildcards and regular expressions are not supported.

You use the Stopwords key to define a custom stopwords dictionary in an analysis scheme. Because you pass the dictionary to Amazon CloudSearch as a string, you must escape all double quotes within the string. For example, the following analysis scheme configures the stopwords a, an, and the:

{
    "AnalysisSchemeName": "myscheme",
    "AnalysisSchemeLanguage": "en",
    "AnalysisOptions": {
        "Stopwords": "[\"a\",\"an\",\"the\"]"
    }
} 

If you do not specify a stopwords dictionary in your analysis scheme, Amazon CloudSearch uses the default stopword dictionary for the specified language. The default stopwords configured for each language work well for most use cases. In general, it's best to start with the default and then make adjustments to optimize the relevance of the search results for your use case. For information about the defaults for each language, see Language Specific Settings.

Synonyms in Amazon CloudSearch

You can configure synonyms for terms that appear in the data that you are searching. That way, if a user searches for the synonym rather than the indexed term, the results will include documents that contain the indexed term. For example, you might define custom synonyms to do the following:

When you define a synonym, the synonym is added to the index everywhere the base token occurs. For example, if you define fish as a synonym of barracuda, the term fish is added to every document that contains the term barracuda. Adding a large number of synonyms can increase the size of the index as well as query latency—synonyms increase the number of matches and the more matches, the longer it takes to process the results.

The synonym dictionary is used during indexing to configure mappings for terms that occur in text fields. No synonym processing is done on search requests. By default, Amazon CloudSearch does not define any synonyms.

You can specify synonyms in two ways:

A synonym dictionary is specified as a JSON object that defines the synonym groups and aliases. The groups value is an array of arrays, where each sub-array is a conflation group. The aliases value is an object that contains a collection of string:value pairs where the string specifies a term and the array of values specifies each of the synonyms for that term. The following example includes both conflation groups and aliases:

{
    "groups": [["1st", "first", "one"], ["2nd", "second", "two"]],
    "aliases": { "youth": ["child", "kid", "boy", "girl"], 
                 "adult": ["men", "women"] }
}

Both groups and aliases support multiword synonyms. In the following example, multiword synonyms are used in a conflation group as well as an alias:

{
    "groups": [["tool box", "toolbox"], ["band saw", "bandsaw"]],
    "aliases": { "workbench": ["work bench"]}
}

You use the Synonyms key to define a custom synonym dictionary in an analysis scheme. Because you pass the dictionary to Amazon CloudSearch as a string, you must escape all double quotes within the string. For example, the following analysis scheme configures aliases for the term youth:

{
    "AnalysisSchemeName": "myscheme",
    "AnalysisSchemeLanguage": "en",
    "AnalysisOptions": {
        "Synonyms": "{\"aliases\": {\"youth\": [\"child\",\"kid\"]}}"
    }
}

Configuring Analysis Schemes Using the Amazon CloudSearch Console

You can define analysis schemes from the Analysis Schemes pane in the Amazon CloudSearch console.

Configuring Analysis Schemes Using the AWS CLI

You use the aws cloudsearch define-analysis-scheme command to define language-specific text processing options, including stemming options, stopwords, and synonyms. For information about installing and setting up the AWS CLI, see the AWS Command Line Interface User Guide.

You specify an analysis scheme as part of the configuration of each text or text-array field. For more information, see configure indexing options.

Configuring Analysis Schemes Using the AWS SDKs

The AWS SDKs (except the Android and iOS SDKs) support all of the Amazon CloudSearch actions defined in the Amazon CloudSearch Configuration API, including DefineAnalysisScheme. For more information about installing and using the AWS SDKs, see AWS Software Development Kits.

Indexing Bigrams for Chinese, Japanese, and Korean in Amazon CloudSearch

Chinese, Japanese, and Korean do not have explicit word boundaries. Simply indexing individual characters (unigrams) can result in matches that aren't very relevant to a search query. One solution is to index bigrams. A bigram is every sequence of two adjacent characters in a string. For example, the following example shows bigrams for the string 我的氣墊船裝滿了鱔魚 :

我的  的氣  氣墊  墊船  船裝  裝滿  滿了  了鱔  鱔魚

While indexing bigrams can improve search result quality, keep in mind that it can significantly increase the size of your index.

When you assign an analysis scheme that sets a field's language to mul, Amazon CloudSearch automatically generates bigrams for all Chinese, Japanese, and Korean text within the field.

For more information about creating and using analysis schemes, see Configuring Analysis Schemes.

If you are indexing Japanese content, you might also be interested in using a custom tokenization dictionary with the standard Japanese language processor. For more information, see Customizing Japanese Tokenization.

Customizing Japanese Tokenization in Amazon CloudSearch

If you need more control over how Amazon CloudSearch tokenizes Japanese, you can add a custom Japanese tokenization dictionary to your analysis scheme. Configuring a custom tokenization dictionary enables you to override how specific entries are tokenized by the standard Japanese language processor. This can improve search result accuracy in some cases, particularly when you need to index and retrieve domain-specific phrases.

A tokenization dictionary is a collection of entries where each entry specifies a set of characters, how the characters should be tokenized, how each token should be pronounced (readings), and a part-of-speech tag. You specify the dictionary as an array, and each dictionary entry is an array of strings. The entries are of the following form:

["<text>","<token 1> ... <token n>","<reading 1> ... <reading n>","<part-of-speech tag>"]

You must specify a reading for each token and the part-of-speech tag for the entry. See Japanese Part-of-Speech Tags for the part of speech tags that are treated as stopwords.

You use the JapaneseTokenizationDictionary key to define a custom tokenization dictionary in an analysis scheme. Because you pass the tokenization dictionary to Amazon CloudSearch as a string, you must escape all double quotes within the string. For example, the dictionary in the following analysis scheme specifies segmentation overrides for Kanji and Katakana compounds, and a custom reading for a proper name:

{       
    "AnalysisSchemeName": "jascheme",
    "AnalysisSchemeLanguage": "ja",
    "AnalysisOptions": {
        "Stopwords": "[\"a\", \"the\"]",
        "AlgorithmicStemming": "full",
        "JapaneseTokenizationDictionary": "[ [\"日本経済新聞\",\"日本 経済 新聞\",\"ニホン ケイザイ シンブン\",\"カスタム名詞\"],[\"トートバッグ\",\"トート バッグ\",\"トート バッグ\",\"かずカナ名詞\"],[\"朝青龍\",\"朝青龍\",\"アサショウリュウ\",\"カスタム人名\"] ]"
    }
}

When configuring an analysis scheme with the AWS CLI, you can store the analysis scheme in a text file and specify that file as the --analysis-scheme value. This enables you to format the scheme so that it's easier to read. For example, if you store the jascheme analysis scheme in a file called jascheme.txt, you can pass that file in when you call aws cloudsearch define-analysis-scheme:

aws cloudsearch define-analysis-scheme --region us-east-1 --domain-name
mydomain --analysis-scheme file://jascheme.txt

For more information about creating and using analysis schemes, see Configuring Analysis Schemes.

Japanese Part-of-Speech Tags in Amazon CloudSearch

When you use a custom tokenization dictionary for Japanese, you specify a part-of-speech tag for each entry. If the part-of-speech tag matches one of the tags configured as a stop tag, the entry is treated as a stopword.

The following table shows the part of speech tags configured as stop tags in Amazon CloudSearch.