SearchStax named one of the “Top 20 Open Source Software Solutions for 2017”. Read more here.
Accelerating Solr App Development

Overview

We recently began hosting the Solr search infrastructure for a Sitecore client who was looking for increased search performance and scalability. Naturally, they selected a Solr solution. Our client, a large Australian government entity, would see frequent high bursts of traffic and search activity. Their Sitecore deployment is hosted on Microsoft’s Azure Platform as a Service (PaaS).  They needed a highly available and resilient Solr setup on Azure to deal with their traffic spikes.

In addition to the bump in performance and scalability, they were also looking to accelerate their Sitecore search deployment, testing, and release cycle times. Our client wanted to accelerate Solr app development by integrating their automated testing routines to ensure quality releases and minimized cycle times. They wanted to utilize APIs to programmatically create Solr search clusters, utilize them for their automated testing runs, and then tear them down. This has numerous benefits over creating these deployments manually each time OR creating the deployments and letting them sit idle in between releases. In this article, we’ll talk about what we did to help our client meet their goals.

Search Automation

Setting up a search cluster using open-source software such as Solr from scratch can be painful.  Some of the things that an engineer would have to do to set up a cluster include:

  • Allocating Hardware
  • Choosing and installing an Operating System
  • Configuring Disk layouts including swap space, system volumes, and data volumes
  • Setting up a zookeeper cluster
  • Configuring Solr parameters
  • Setting up and configuring a load balancer
  • Setting up monitoring and alerting
  • Many more fiddly details

 

Doing the initial setup can easily take an entire day or more, even for experienced engineers.  And once the system is set up, there is another matter of maintaining the cluster, attending to issues like OS patch upgrades, disk failures, node crashes, etc.

 

Fortunately, software such as SearchStax can ease the pain of setting up a cluster by leveraging modern automation and cloud technology.  With SearchStax, a cluster can be setup in minutes by filling out some simple forms and pushing a button.

 

Point-and-click cluster creation is an amazing feat of modern computation.  However, it turns out that automation techniques can be leveraged even further.  By providing a provisioning API, a cluster can be set up by an automation job, requiring zero human intervention.

 

Why would one want to set up a cluster via an API?  The primary use case for programmatic cluster creation is for integration of testing automation.  While production search environments are long-lived, testing environments tend to be of an ephemeral nature: a cluster is created, tests are run, and the cluster is torn down.  Of course, semi-permanent dedicated clusters can be provisioned for test environments.  This has several downsides:

  • There is a cost to having hardware up and running while it is not being utilized
  • Long-running clusters suffer from bit rot – they can become outdated, and accumulate crusty data and configuration
  • Once a cluster is set up, it is non-trivial to reconfigure it.  For example adding additional nodes or changing storage capacity require specialized knowledge and careful attention to detail.

 

SearchStax provides a REST API that makes it easy to integrate cluster provisioning with an automated testing pipeline.   In the next section, we walk through a typical sequence of API calls that such a job might perform.

API

Obtain Credentials

The first step in the process is to obtain credentials from SearchStax.  Given a username and password, a token is retrieved

Request

curl -X POST \

 https://app.searchstax.com/api/rest/v1/obtain-auth-token/ \

 -d 'username=admin&password=s3cr3t'

Response

{

   "token": "b09c"

}

List Deployments

One might want to view the deployments that are currently up and running.  This is accomplished via the list-deployments API call.  Note that we pass in the token in the Authorization header.

Request

curl -X GET \

 https://app.searchstax.com/api/rest/v2/account/MyAccount/deployment/ \

 -H 'Authorization: Token b09c'

Response

{

   "count": 1,

   "next": null,

   "previous": null,

   "results": [

       {

           "name": "mydeployment1",

           "uid": "ss123456",

           "application": "Solr",

           "application_version": "6.6.2",

           "tier": "Silver",

           "http_endpoint": "https://ss123456-ap-southeast-2-aws.searchstax.com/solr/",

           "status": "Running",

           "provision_state": "Done",

           "termination_lock": true,

           "plan_type": "DedicatedDeployment",

           "plan": "DN1",

           "region_id": "ap-southeast-2",

           "cloud_provider": "Amazon Web Services",

           "cloud_provider_id": "aws",

           "num_additional_app_nodes": 0,

           "deployment_type": "Dedicated Node",

           "num_nodes_default": 1,

           "num_additional_zookeeper_nodes": 0,

           "servers": [

               "ss123456-1"

           ]

       }

   ]

}

Create a Deployment

Now for the main event.  We request a new cluster, passing parameters specifying the cluster name, the Solr version, the cloud provider, the region, whether a termination lock is enabled, and some other parameters.

Request

curl -X POST \

 https://app.searchstax.com/api/rest/v2/account/MyAccount/deployment/ \

 -H 'Authorization: Token b09c' \

 -d 'name=MyDeployment2&application_version=7.1.0&plan_type=DedicatedDeployment&cloud_provider_id=aws&plan=DN1&region_id=us-west-1&termination_lock=true&application=Solr'

 

Response

For a response, we receive an HTTP 201 CREATED status code

Toggle Termination Lock

Notice that in our creation request, we set termination_lock to true.   This protects against accidental deletion of the cluster.  To enable deletion, we need to set the termination_lock to false.

Request

curl -X POST \

 https://app.searchstax.com/api/rest/v2/account/VicRoads2/deployment/ss677906/termination-lock/ \

 -H 'Authorization: Token b09c' \

 -d 'name=MyDeployment2&termination_lock=false'

Response

For the response, we receive an HTTP 200 OK status code.

Delete the Deployment

Once we are through using the cluster, we will want to tear it down using the Delete API.  This can be done as follows:

Request

curl -X DELETE \

 https://app.searchstax.com/api/rest/v2/account/VicRoads2/deployment/ss677906/ \

 -H 'Authorization: Token b09c'

Response

For the response, we receive an HTTP 200 OK status code.

Summary

APIs for provisioning search clusters provide an invaluable tool for automating search related tests.  In this article, we described how cloud and automation technology provide a powerful engine for creating clusters, while provisioning APIs can further accelerate high-quality releases of search applications.

 

Get The Latest Content First!

* = required field