Package com.norconex.committer.azuresearch

Class AzureSearchCommitter

java.lang.Object
com.norconex.committer.core3.AbstractCommitter
com.norconex.committer.core3.batch.AbstractBatchCommitter
com.norconex.committer.azuresearch.AzureSearchCommitter
All Implemented Interfaces:
IBatchConsumer, ICommitter, IXMLConfigurable, AutoCloseable
public class AzureSearchCommitter extends AbstractBatchCommitter

Commits documents to Microsoft Azure Search.

Document reference encoding

By default the document reference (Azure Search Document Key) is encoded using URL-safe Base64 encoding. This is Azure Search recommended approach when a document unique id can contain special characters (e.g. a URL). If you know your document references to be safe (e.g. a sequence number), you can set AzureSearchCommitterConfig.setDisableDocKeyEncoding(boolean) to true. To otherwise store a reference value un-encoded, you can additionally store it in a field other than your reference ("id") field.

Single vs multiple values

Fields with single value will be sent as such, while multi-value fields are sent as array. If you have a field defined as an array in Azure Search, sending a single value may cause an error.

It is possible for values to always be sent as arrays for specific fields. This is done using AzureSearchCommitterConfig.setArrayFields(String). It expects comma-separated-value list or a regular expression, depending of the value you set for AzureSearchCommitterConfig.setArrayFieldsRegex(boolean).

Field names and errors

Azure Search will produce an error if any of the documents in a submitted batch contains one or more fields with invalid characters. To prevent sending those in vain, the committer will validate your fields and throw an exception upon encountering an invalid one. To prevent exceptions from being thrown, you can set AzureSearchCommitterConfig.setIgnoreValidationErrors(boolean) to true to log those errors instead.

An exception will also be thrown for errors returned by Azure Search (e.g. a field is not defined in your Azure Search schema). To also log those errors instead of throwing an exception, you can set AzureSearchCommitterConfig.setIgnoreResponseErrors(boolean) to true.

Field naming rules

Those are the field naming rules mandated for Azure Search (in force for Azure Search version 2016-09-01): Search version

  • Document reference (ID): Letters, numbers, dashes ("-"), underscores ("_"), and equal signs ("="). First character cannot be an underscore.
  • Document field name: Letters, numbers, underscores ("_"). First character must be a letter. Cannot start with "azureSearch". Maximum length is 128 characters.

XML configuration usage:


<committer
    class="com.norconex.committer.azuresearch.AzureSearchCommitter">
  <endpoint>(Azure Search endpoint)</endpoint>
  <apiVersion>(Optional Azure Search API version to use)</apiVersion>
  <apiKey>(Azure Search API admin key)</apiKey>
  <indexName>(Name of the index to use)</indexName>
  <disableDocKeyEncoding>[false|true]</disableDocKeyEncoding>
  <ignoreValidationErrors>[false|true]</ignoreValidationErrors>
  <ignoreResponseErrors>[false|true]</ignoreResponseErrors>
  <useWindowsAuth>[false|true]</useWindowsAuth>
  <arrayFields
      regex="[false|true]">
    (Optional fields to be forcefully sent as array, even if single
    value. Unless "regex" is true, expects a CSV list of field names.)
  </arrayFields>
  <proxySettings/>
  <sourceKeyField>
    (Optional document field name containing the value that will be stored
    in Azure Search target document key field. Default is the document
    reference.)
  </sourceKeyField>
  <targetKeyField>
    (Optional name of Azure Search document field where to store a
    document unique key identifier (sourceKeydField).
    Default is "id".)
  </targetKeyField>
  <targetContentField>
    (Optional Azure Search document field name to store document
    content/body. Default is "content".)
  </targetContentField>
</committer>

XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML usage example:


<committer
    class="com.norconex.committer.azuresearch.AzureSearchCommitter">
  <endpoint>https://example.search.windows.net</endpoint>
  <apiKey>1234567890ABCDEF1234567890ABCDEF</apiKey>
  <indexName>sample-index</indexName>
</committer>

The above example uses the minimum required settings.

Author:
Pascal Essiembre