Frequently Asked Questions - SearchStax

The SearchStax® Frequently Asked Questions page includes the following approved question and answer about our Apache Solr Cloud services.

Some of our replicas are missing!

The easiest mistake to make when creating a new collection is to specify the wrong number of replicas. The number of replicas should equal the number of servers in the cluster. (See What is a collection/core/shard/replica? for help in sorting out these terms.)

The Problem

We showed you how to create a new collection in the Quick Start lesson:

curl 'https://ss394245-us-west-1-aws.searchstax.com/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=test1&numShards=1&replicationFactor=1&maxShardsPerNode=1'

This example works perfectly for creating a collection on a single-node system. However, when clients scale up to a three-node cluster, they sometimes forget to change replicationFactor=1 to replicationFactor=3.

This mistake creates a collection that resides on one node only. That node does all the work while the other two nodes remain idle.

The Symptoms

A replicationFactor error comes to our attention in one of several ways:

  • A client complains about high query latency. One server gets overloaded handing a high volume of queries.
  • A client notices that some servers don't have the right number of cores (replicas).
  • A client complains that a backup/restore operation missed an index.
  • Query service is interrupted when one (out of three) servers stops running. A high-availability cluster should continue to serve queries when a single server is down.
  • A client adds nodes to a cluster. The replica count must be manually changed.

We confirm a replicationFactor error by looking in the deployment's Solr Dashboard.

  1. From the SearchStax Cloud Manager, click the name of the deployment.
  2. On the Deployment Servers page, click the Solr HTTP Endpoint.
  3. On the Solr Dashboard, click Cloud. The graph shows a <collection> → <shard> → <list of replicas>. For a three-node cluster, you should see three replicas.
SearchStax Solr Cloud Graph

If you see only one replica where you expected three, you'll have to add two replicas to the collection. Fortunately, this is easy to do.

The Solution

The solution is to manually add replicas to the nodes that lack them.

The Solr API lets us add a new replica directly to a collection by sending a message to the Solr HTTP Endpoint. The syntax of the message is:

curl '<HTTP Solr Endpoint>/solr/admin/collections?action=ADDREPLICA&collection=<collection-name>&shard=shard1&node=<node-name>'

You already know the values of the HTTP Solr Endpoint and the collection-name. The node-name is not as obvious.

Make a note of the IP address of the single replica that appeared in the graph above. That's the IP of the server that already has a replica. Now we need the names of the two servers that need to have a replica added.

Click Tree under Cloud. Open the list of live_nodes.

SearchStax Solr Cloud Tree

Those are the node-names. Identify the two nodes that need replicas, and plug in all the required values in the ADDREPLICA message.

This is a practical example that we used with Solr 7.3.1:

curl 'https://ss750305-us-west-1-aws.searchstax.com/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1&node=10.0.1.164:8983_solr'

The successful creation of a new replica returns a message similar to this one:

{
  "responseHeader":{
    "status":0,
    "QTime":582},
  "success":{
    "10.0.1.164:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":435},
      "core":"testcollection_shard1_replica_n3"}}}

Repeat this step for the remaining node.

When you return to the Cloud graph display, you will see that your collection now has three replicas.

SearchStax Solr Cloud Graph

We love to answer questions!

Please contact the SearchStax Support Desk immediately if you have any question about Solr Cloud deployments.

Return to Frequently Asked Questions.