Package com.norconex.committer.solr

Class SolrCommitter

java.lang.Object

com.norconex.committer.core3.AbstractCommitter

com.norconex.committer.core3.batch.AbstractBatchCommitter

com.norconex.committer.solr.SolrCommitter

All Implemented Interfaces:

IBatchConsumer, ICommitter, IXMLConfigurable, AutoCloseable

public class SolrCommitter
extends AbstractBatchCommitter

Commits documents to Apache Solr.

Solr Client:

As of 2.4.0, it is possible to specify which type of Solr Client to use. The expected configuration value of "solrURL" is influenced by the client type chosen. Default client type is HttpSolrClient. The clients are:

HttpSolrClient

For direct access to a single Solr node. Ideal for local development. Needs a Solr URL. Default client.

LBHttpSolrClient

Simple load-balancing as an alternative to an external load balancer. Needs two or more Solr node URLs (comma-separated).

ConcurrentUpdateSolrClient

Optimized for mass upload on a single node. Not best for queries. Needs a Solr URL.

CloudSolrClient

For use with a SolrCloud cluster. Needs a comma-separated list of Zookeeper hosts.

Http2SolrClient

Same as HttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

LBHttp2SolrClient

Same as LBHttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

ConcurrentUpdateHttp2SolrClient

Same as LBHttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

Authentication

Basic authentication is supported for password-protected Solr installations.

XML configuration usage:

<committer
    class="com.norconex.committer.solr.SolrCommitter">
  <solrClientType>
    (See class documentation for options. Default: HttpSolrClient.)
  </solrClientType>
  <solrURL>(URL to Solr)</solrURL>
  <solrUpdateURLParams>
    <param
        name="(parameter name)">
      (parameter value)
    </param>
    <!-- multiple param tags allowed -->
  </solrUpdateURLParams>
  <solrCommitDisabled>[false|true]</solrCommitDisabled>
  <!-- Use the following if authentication is required. -->
  <credentials/>
  <sourceIdField>
    (Optional document field name containing the value that will be stored
    in Solr target ID field. Default is the document reference.)
  </sourceIdField>
  <targetIdField>
    (Optional name of Solr field where to store a document unique
    identifier (sourceIdField).  If not specified, default is "id".)
  </targetIdField>
  <targetContentField>
    (Optional Solr field name to store document content/body.
    Default is "content".)
  </targetContentField>
</committer>

XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEFAULT_SOLR_CONTENT_FIELD	Default Solr content field
static final String	DEFAULT_SOLR_ID_FIELD	Default Solr ID field

Constructor Summary

Constructors

Constructor	Description
SolrCommitter()	Constructor.

Method Summary

Modifier and Type	Method	Description
protected void	addSolrDeleteRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest, DeleteRequest committerRequest)	Adds a delete operation to the given Solr batch request.
protected void	addSolrUpsertRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest, UpsertRequest committerRequest)	Adds an upsert operation to the given Solr batch request.
protected org.apache.solr.common.SolrInputDocument	buildSolrDocument(Properties fields)	Builds a Solr input document from the given field properties.
protected void	closeBatchCommitter()
protected void	commitBatch(Iterator<ICommitterRequest> it)
boolean	equals(Object other)
Credentials	getCredentials()	Gets Solr authentication credentials.
SolrClientType	getSolrClientType()	Gets the Solr client type.
String	getSolrURL()	Gets the Solr URL.
String	getSourceIdField()	Gets the document field name containing the value to be stored in Solr ID field.
String	getTargetContentField()	Gets the name of the Solr field where content will be stored.
String	getTargetIdField()	Gets the name of the Solr field where to store a document unique identifier (sourceIdField).
String	getUpdateUrlParam(String name)	Gets a URL parameter value by its parameter name.
Set<String>	getUpdateUrlParamNames()	Gets the update URL parameter names.
int	hashCode()
protected void	initBatchCommitter()
boolean	isSolrCommitDisabled()	Gets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.
protected void	loadBatchCommitterFromXML(XML xml)
protected void	pushSolrRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest)	Pushes the given Solr update request, optionally committing afterward.
protected void	saveBatchCommitterToXML(XML xml)
void	setCredentials(Credentials credentials)	Sets Solr authentication credentials.
void	setSolrClientType(SolrClientType solrClientType)	Sets the Solr client type.
void	setSolrCommitDisabled(boolean solrCommitDisabled)	Sets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.
void	setSolrURL(String solrURL)	Sets the Solr URL.
void	setSourceIdField(String sourceIdField)	Sets the document field name containing the value to be stored in Solr ID field.
void	setTargetContentField(String targetContentField)	Sets the name of the Solr field where content will be stored.
void	setTargetIdField(String targetIdField)	Sets the name of the Solr field where to store a document unique identifier (sourceIdField).
void	setUpdateUrlParam(String name, String value)	Sets URL parameters to be added on Solr HTTP calls.
String	toString()

Methods inherited from class com.norconex.committer.core3.batch.AbstractBatchCommitter

consume, doClean, doClose, doDelete, doInit, doUpsert, getCommitterQueue, loadCommitterFromXML, saveCommitterToXML, setCommitterQueue

Methods inherited from class com.norconex.committer.core3.AbstractCommitter

accept, addRestriction, addRestrictions, applyFieldMappings, clean, clearFieldMappings, clearRestrictions, close, delete, fireDebug, fireDebug, fireError, fireError, fireInfo, fireInfo, getCommitterContext, getFieldMappings, getRestrictions, init, loadFromXML, removeFieldMapping, removeRestriction, removeRestriction, saveToXML, setFieldMapping, setFieldMappings, upsert

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.norconex.commons.lang.xml.IXMLConfigurable

loadFromXML, saveToXML

Field Details

DEFAULT_SOLR_ID_FIELD

public static final String DEFAULT_SOLR_ID_FIELD

Default Solr ID field

See Also:

• Constant Field Values

DEFAULT_SOLR_CONTENT_FIELD

public static final String DEFAULT_SOLR_CONTENT_FIELD

Default Solr content field

See Also:

• Constant Field Values

Constructor Details

SolrCommitter

public SolrCommitter()

Constructor.

Method Details

getSolrClientType

public SolrClientType getSolrClientType()

Gets the Solr client type.

Returns:

solr client type

setSolrClientType

public void setSolrClientType(SolrClientType solrClientType)

Sets the Solr client type.

Parameters:

solrClientType - solr client type

getSolrURL

public String getSolrURL()

Gets the Solr URL.

Returns:

Solr URL

setSolrURL

public void setSolrURL(String solrURL)

Sets the Solr URL.

Parameters:

solrURL - solrURL

setUpdateUrlParam

public void setUpdateUrlParam(String name,
 String value)

Sets URL parameters to be added on Solr HTTP calls.

Parameters:

name - parameter name

value - parameter value

getUpdateUrlParam

public String getUpdateUrlParam(String name)

Gets a URL parameter value by its parameter name.

Parameters:

name - parameter name

Returns:

parameter value

getUpdateUrlParamNames

public Set<String> getUpdateUrlParamNames()

Gets the update URL parameter names.

Returns:

parameter names

setSolrCommitDisabled

public void setSolrCommitDisabled(boolean solrCommitDisabled)

Sets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.

Parameters:

solrCommitDisabled - true if sending Solr commit is disabled

isSolrCommitDisabled

public boolean isSolrCommitDisabled()

Gets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.

Returns:

true if sending Solr commit is disabled.

getCredentials

public Credentials getCredentials()

Gets Solr authentication credentials.

Returns:

credentials

setCredentials

public void setCredentials(Credentials credentials)

Sets Solr authentication credentials.

Parameters:

credentials - the credentials

getTargetContentField

public String getTargetContentField()

Gets the name of the Solr field where content will be stored. Default is "content".

Returns:

field name

setTargetContentField

public void setTargetContentField(String targetContentField)

Sets the name of the Solr field where content will be stored. Specifying a null value will disable storing the content.

Parameters:

targetContentField - field name

getSourceIdField

public String getSourceIdField()

Gets the document field name containing the value to be stored in Solr ID field. Default is not a field, but rather the document reference.

Returns:

name of field containing id value

setSourceIdField

public void setSourceIdField(String sourceIdField)

Sets the document field name containing the value to be stored in Solr ID field. Set null to use the document reference instead of a field (default).

Parameters:

sourceIdField - name of field containing id value, or null

getTargetIdField

public String getTargetIdField()

Gets the name of the Solr field where to store a document unique identifier (sourceIdField). Default is "id".

Returns:

name of Solr ID field

setTargetIdField

public void setTargetIdField(String targetIdField)

Sets the name of the Solr field where to store a document unique identifier (sourceIdField). If not specified, default is "id".

Parameters:

targetIdField - name of Solr ID field

initBatchCommitter

protected void initBatchCommitter()
                           throws CommitterException

Overrides:

initBatchCommitter in class AbstractBatchCommitter

Throws:

CommitterException

commitBatch

protected void commitBatch(Iterator<ICommitterRequest> it)
                    throws CommitterException

Specified by:

commitBatch in class AbstractBatchCommitter

Throws:

CommitterException

closeBatchCommitter

protected void closeBatchCommitter()
                            throws CommitterException

Overrides:

closeBatchCommitter in class AbstractBatchCommitter

Throws:

CommitterException

pushSolrRequest

protected void pushSolrRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest)
                        throws org.apache.solr.client.solrj.SolrServerException,
IOException,
CommitterException

Pushes the given Solr update request, optionally committing afterward.

Parameters:

solrBatchRequest - the Solr update request to push

Throws:

org.apache.solr.client.solrj.SolrServerException - if Solr returns an error

IOException - if an I/O error occurs communicating with Solr

CommitterException - if the response status is invalid

addSolrUpsertRequest

protected void addSolrUpsertRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest,
 UpsertRequest committerRequest)
                             throws CommitterException

Adds an upsert operation to the given Solr batch request.

Parameters:

solrBatchRequest - the Solr update request to add to

committerRequest - the committer upsert request

Throws:

CommitterException - if the upsert cannot be prepared

addSolrDeleteRequest

protected void addSolrDeleteRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest,
 DeleteRequest committerRequest)

Adds a delete operation to the given Solr batch request.

Parameters:

solrBatchRequest - the Solr update request to add to

committerRequest - the committer delete request

buildSolrDocument

protected org.apache.solr.common.SolrInputDocument buildSolrDocument(Properties fields)

Builds a Solr input document from the given field properties.

Parameters:

fields - document fields and their values

Returns:

Solr input document

loadBatchCommitterFromXML

protected void loadBatchCommitterFromXML(XML xml)

Specified by:

loadBatchCommitterFromXML in class AbstractBatchCommitter

saveBatchCommitterToXML

protected void saveBatchCommitterToXML(XML xml)

Specified by:

saveBatchCommitterToXML in class AbstractBatchCommitter

equals

public boolean equals(Object other)

Overrides:

equals in class AbstractBatchCommitter

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractBatchCommitter

toString

public String toString()

Overrides:

toString in class AbstractBatchCommitter