SearchStax named one of the “Top 20 Open Source Software Solutions for 2017”. Read more here.
Top 10 Solr Best Practices

It is quite straightforward to install Solr on your local machine and load a collection. This is great when you are either testing out the waters or are starting the development phase of your project. When you start getting ready for production, however, you discover how daunting it can actually be. There are many things which Solr does not print on their “instruction label” that are really important when getting your system production ready.

Here is our list of the top 10 hurdles that stand between the Solr developer and the production system:

  1. Zookeeper Ensemble
  2. High Availability
  3. Security at Transport and at Rest
  4. Access Restrictions
  5. Load Balancer
  6. Logging & Log Rotation
  7. Monitoring and Alerting
  8. Backups
  9. Expandable Storage
  10. Vulnerability Patching

Let’s count up from the bottom all the way up to the most important one.

10. Vulnerability Patching

A typical Solr deployment lives on one or more servers that are accessed via IP address/URLs and ports to those systems. Network settings can sometimes create vulnerabilities in your architecture. It is important to keep an eye on what can be accessed via the internet, and what potential exploits you might be inviting. For instance, just changing the header size in jetty.xml might lead to a potential DDoS exploit.

When you deploy Solr in production, make sure that you only update the settings that you absolutely must change. It’s a good practice to do a regular vulnerability scan on your servers to see if there are any specific vulnerabilities that need to be addressed. In addition, it’s best to keep yourself updated with latest discovered vulnerabilities (https://www.cvedetails.com/vulnerability-list/vendor_id-442/CVS.html) and monitor the Apache Solr mailing lists. Once you’ve discovered a vulnerability for the version of Solr you are using, it’s best to apply the patch in a timely manner.

9. Expandable Storage

It is difficult to predict how large your Solr collection will be when you are first starting out. It is always advisable to change the data directory of your Solr deployment to a dedicated storage device that can be expanded as your data grows. All the major cloud providers allow you to attach an extra storage device to your instance. It can be increased in size later if need be.

  • Add a separate data disk to your instance.
  • Change the data directory of Solr from default to the added disk.
  • Change the size of the disk as required.

8. Backups

It is crucial to protect your Solr data. Make regular backups of your data so you can restore your system quickly and efficiently in case of a failure.

It is important that you:

  • Schedule a backup to happen at least once per day.
  • Store backups in a storage account outside of your deployment.
  • Set a retention policy to clean up out-of-date backups.
  • Do periodic backup verifications.

Also, backing up a multi-node system can be a complex task. You have to mount a shared drive across all the nodes and then trigger backups using the BACKUP API.

Here are some links for further reading:

https://lucene.apache.org/solr/guide/7_7/making-and-restoring-backups.html
https://aws.amazon.com/s3/
https://azure.microsoft.com/en-us/services/storage/blobs/

7. Monitoring & Alerting

Top 10 Solr Best Practices

It is very important to monitor all aspects of your Solr deployment regularly and consistently. Having a clear view of the system-health metrics, such as Memory, System Load Average, JVM Heap, helps you avoid CPU overload and out-of-memory errors. The collection-level metrics, such as indexing performance, search latency, and errors occurring during searching and indexing, are beneficial in tuning the dynamic behavior of your system.

You can do so by:

  • Use metrics provided by your cloud provider.
  • Have a set of APIs that can read and report system statistics.

Solution providers, such as SearchStax, enable these features out of the box. They are essential for keeping the deployment running smoothly.

Here is a link to the SearchStax monitoring documentation:
https://www.searchstax.com/docs/monitoring/

6. Logging & Log Rotation

Top 10 Solr Best Practices

Logging keeps you informed about conditions inside Solr. Solr provides a very granular control over logging levels for each and every component. It is very important that you:

  • Go through this complete list to figure out which information is useful to you.
  • Set the level of logging that you want for each component.
  • Make sure that the log rotation policies are set in accordance with your needs:
    • Rotate logs based on time
    • Rotate logs based on size

For more about configuring logging, follow this link:
https://lucene.apache.org/solr/guide/6_6/configuring-logging.html

5. Load Balancer

A load balancer distributes incoming queries across Solr nodes. Putting a load balancer on top of your Solr cluster helps you achieve a truly distributed and highly-available architecture.

Some clients rely on a combination of SolrCloud and Zookeeper to access Solr in a distributed sense, where the client library gets IP addresses of Solr nodes from zookeeper and then queries them randomly. This method is not recommended, because it is not universally supported and it relies on accessing the Solr nodes directly. A load balancer, on the other hand, adds a layer of security on top of your Solr cluster, and also is a universally-supported way of accessing your Solr cluster in a distributed fashion.

Here are some load balancers that you can deploy:

  • Load balancer provided by your cloud provider.
  • Nginx.

Here are some useful links:

https://www.nginx.com/resources/glossary/load-balancing/
https://aws.amazon.com/elasticloadbalancing/
https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview

4. Security – Access Restrictions

After encrypting your connection to Solr, it is crucial to restrict any unwanted access to it. Solr lets you enable basic authentication for your system, and it lets you add multiple users to it. User can have different levels of access to Solr.

Another great way to secure your Solr is by applying IP filtering to your instance. It can help you restrict access to all ports except the ones being used by Solr and zookeeper. It can also help you control the incoming traffic by allowing access from only specified IP/CIDR ranges. So:

  • Enable basic authentication for Solr.
  • Add IP/CIDR filtering in your security group.

Learn more here:

https://lucene.apache.org/solr/guide/6_6/basic-authentication-plugin.html
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html

3. Security at Transport and at Rest

When setting up your production Solr, it is very important to follow all the attainable security protocols. The first one is TLS. It provides you an end-to-end encryption between Solr and your application. You can either:

  • Update your Solr configurations and enable TLS, or
  • Add a layer of load balancer and enable TLS there.

Also, you should enable disk encryption on your data disk so that all the data stored by Solr is encrypted.

Here are some useful links:

https://lucene.apache.org/solr/guide/6_6/enabling-ssl.html
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html

2. High Availability

Top 10 Solr Best Practices

You do not want your production system to go down, ever, which is why it is so important that your Solr setup supports high availability. It can be done by:

  • Deploying Solr across multiple nodes
  • Putting a replica of each collection on at least 2 nodes.

This will ensure that even if one of the nodes go down, you will still have access to all of your data.

Here are some links to help you set up Solr with high availability:

https://lucene.apache.org/solr/guide/7_7/getting-started-with-solrcloud.html#getting-started-with-solrcloud
https://lucene.apache.org/solr/guide/7_7/collections-api.html#addreplica

1. Zookeeper

Zookeeper is a configuration-management application that comes prepackaged with Solr. It works well in the local environment, but as you increase the traffic or number of nodes, the prepackaged Zookeeper falls short of delivering proper support to your Solr cluster.

When taking your Solr to production:

  • Zookeeper should be installed separately.
  • Change the Solr configurations so that it uses this separate zookeeper.
  • You should have one zookeeper per Solr node, unless you have two Solr nodes. In that case you should have three zookeeper nodes.

Doing all this will insure that all the internal workings of Solr can keep on going smoothly even under the heaviest of traffic.

Some useful links:

https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html
https://stackoverflow.com/questions/24033682/zookeeper-running-on-two-nodes

Conclusion

These are the top 10 things that every person taking Solr to production should keep in mind to ensure a smooth transition when going live. It can be very challenging to execute all of these tasks properly. Solr-as-a-service providers, such as Searchstax, take care of all the hassles of making your deployment production-ready for you. This lets you put your effort on more important things, such as spending time on your own application.