Package com.norconex.committer.cloudsearch

Class CloudSearchCommitter

java.lang.Object
com.norconex.committer.core3.AbstractCommitter
com.norconex.committer.core3.batch.AbstractBatchCommitter
com.norconex.committer.cloudsearch.CloudSearchCommitter
All Implemented Interfaces:
IBatchConsumer, ICommitter, IXMLConfigurable, AutoCloseable
public class CloudSearchCommitter extends AbstractBatchCommitter

Commits documents to Amazon CloudSearch.

Authentication:

An access key and security key are required to connect to and interact with CloudSearch. For enhanced security, it is best to use one of the methods described in DefaultAWSCredentialsProviderChain for setting them (environment variables, system properties, profile file, etc). Do not explicitly set "accessKey" and "secretKey" on this class if you want to rely on safer methods.

CloudSearch ID limitations:

As of this writing, CloudSearch has a 128 characters length limitation on its "id" field. In addition, certain characters are not allowed. By default, an error will result from trying to submit documents with an invalid ID. You can get around this by setting setFixBadIds(boolean) to true. It will truncate references that are too long and append a hash code to it to keep uniqueness. It will also convert invalid characters to underscore. This approach is not 100% collision-free (uniqueness), but it should safely cover the vast majority of cases.

XML configuration usage:


<committer
    class="com.norconex.committer.cloudsearch.CloudSearchCommitter">
  <!-- Mandatory: -->
  <serviceEndpoint>(CloudSearch service endpoint)</serviceEndpoint>
  <!-- Mandatory if not configured elsewhere: -->
  <accessKey>
    (Optional CloudSearch access key. Will be taken from environment
     when blank.)
  </accessKey>
  <secretKey>
    (Optional CloudSearch secret key. Will be taken from environment
     when blank.)
  </secretKey>
  <!-- Optional settings: -->
  <fixBadIds>
    [false|true](Forces references to fit into a CloudSearch id field.)
  </fixBadIds>
  <signingRegion>(CloudSearch signing region)</signingRegion>
  <proxySettings/>
  <sourceIdField>
    (Optional document field name containing the value that will be stored
    in CloudSearch "id" field. Default is the document reference.)
  </sourceIdField>
  <targetContentField>
    (Optional CloudSearch field name to store the document
    content/body. Default is "content".)
  </targetContentField>
</committer>

XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML usage example:


<committer
    class="com.norconex.committer.cloudsearch.CloudSearchCommitter">
  <serviceEndpoint>
    search-example-xyz.some-region.cloudsearch.amazonaws.com
  </serviceEndpoint>
</committer>

The above example uses the minimum required settings (relying on environment variables for AWS keys).

Author:
Pascal Essiembre

  • Field Details

    • FIELD_PATTERN

      public static final Pattern FIELD_PATTERN
      CouldSearch mandatory field pattern. Characters not matching the pattern will be replaced by an underscore.

    • COULDSEARCH_ID_FIELD

      public static final String COULDSEARCH_ID_FIELD
      CloudSearch mandatory ID field
      See Also:

    • DEFAULT_COULDSEARCH_CONTENT_FIELD

      public static final String DEFAULT_COULDSEARCH_CONTENT_FIELD
      Default CloudSearch content field
      See Also:

  • Constructor Details

    • CloudSearchCommitter

      public CloudSearchCommitter()
      Creates a new CloudSearch committer with no configuration.

    • CloudSearchCommitter

      public CloudSearchCommitter(String serviceEndpoint)
      Creates a new CloudSearch committer with the given service endpoint.
      Parameters:
      serviceEndpoint - CloudSearch service endpoint

    • CloudSearchCommitter

      public CloudSearchCommitter(String serviceEndpoint, String signingRegion)
      Creates a new CloudSearch committer with the given service endpoint and signing region.
      Parameters:
      serviceEndpoint - CloudSearch service endpoint
      signingRegion - AWS signing region

  • Method Details

    • getServiceEndpoint

      public String getServiceEndpoint()
      Gets AWS service endpoint.
      Returns:
      AWS service endpoint

    • setServiceEndpoint

      public void setServiceEndpoint(String serviceEndpoint)
      Sets AWS service endpoint.
      Parameters:
      serviceEndpoint - AWS service endpoint

    • getSigningRegion

      public String getSigningRegion()
      Gets the AWS signing region.
      Returns:
      the AWS signing region

    • setSigningRegion

      public void setSigningRegion(String signingRegion)
      Gets the AWS signing region.
      Parameters:
      signingRegion - the AWS signing region

    • getAccessKey

      public String getAccessKey()
      Gets the CloudSearch access key. If null, the access key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
      Returns:
      the access key

    • setAccessKey

      public void setAccessKey(String accessKey)
      Sets the CloudSearch access key. If null, the access key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
      Parameters:
      accessKey - the access key

    • getSecretKey

      public String getSecretKey()
      Gets the CloudSearch secret key. If null, the secret key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
      Returns:
      the secret key

    • setSecretKey

      public void setSecretKey(String secretKey)
      Sets the CloudSearch secret key. If null, the secret key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
      Parameters:
      secretKey - the secret key

    • getTargetContentField

      public String getTargetContentField()
      Gets the name of the CloudSearch field where content will be stored. Default is "content".
      Returns:
      field name

    • setTargetContentField

      public void setTargetContentField(String targetContentField)
      Sets the name of the CloudSearch field where content will be stored. Specifying a null value will disable storing the content.
      Parameters:
      targetContentField - field name

    • getSourceIdField

      public String getSourceIdField()
      Gets the document field name containing the value to be stored in CloudSearch "id" field. Default is not a field, but rather the document reference.
      Returns:
      name of field containing id value

    • setSourceIdField

      public void setSourceIdField(String sourceIdField)
      Sets the document field name containing the value to be stored in CloudSearch "id" field. Set null to use the document reference instead of a field (default).
      Parameters:
      sourceIdField - name of field containing id value, or null

    • isFixBadIds

      public boolean isFixBadIds()
      Gets whether to fix IDs that are too long for CloudSearch ID limitation (128 characters max). If true, long IDs will be truncated and a hash code representing the truncated part will be appended.
      Returns:
      true to fix IDs that are too long

    • setFixBadIds

      public void setFixBadIds(boolean fixBadIds)
      Sets whether to fix IDs that are too long for CloudSearch ID limitation (128 characters max). If true, long IDs will be truncated and a hash code representing the truncated part will be appended.
      Parameters:
      fixBadIds - true to fix IDs that are too long

    • getProxySettings

      public ProxySettings getProxySettings()
      Gets the proxy settings.
      Returns:
      proxy settings

    • setProxySettings

      public void setProxySettings(ProxySettings proxy)
      Sets the proxy settings.
      Parameters:
      proxy - proxy settings

    • initBatchCommitter

      protected void initBatchCommitter() throws CommitterException
      Overrides:
      initBatchCommitter in class AbstractBatchCommitter
      Throws:
      CommitterException

    • commitBatch

      protected void commitBatch(Iterator<ICommitterRequest> it) throws CommitterException
      Specified by:
      commitBatch in class AbstractBatchCommitter
      Throws:
      CommitterException

    • closeBatchCommitter

      protected void closeBatchCommitter() throws CommitterException
      Overrides:
      closeBatchCommitter in class AbstractBatchCommitter
      Throws:
      CommitterException

    • saveBatchCommitterToXML

      protected void saveBatchCommitterToXML(XML xml)
      Specified by:
      saveBatchCommitterToXML in class AbstractBatchCommitter

    • loadBatchCommitterFromXML

      protected void loadBatchCommitterFromXML(XML xml)
      Specified by:
      loadBatchCommitterFromXML in class AbstractBatchCommitter

    • equals

      public boolean equals(Object other)
      Overrides:
      equals in class AbstractBatchCommitter

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class AbstractBatchCommitter

    • toString

      public String toString()
      Overrides:
      toString in class AbstractBatchCommitter