Meltdown (CVE-2017-5754) and Spectre (CVE-2017-5753 and CVE-2017-5715) are two of the most widespread security issues in the design of modern CPUs and can affect your Apache Solr deployments. These vulnerabilities were disclosed on January 4, 2018. They affect almost every computer made in the last 10 years. These vulnerabilities take advantage of certain processor optimizations. They make it possible for hackers to reveal data stored in memory. In this article, we’ll discuss: how to know if your Solr servers are affected, how to protect your Solr instances, and how to apply the patches to your Solr cluster.
How to know if your servers running Apache Solr are affected?
Spectre and Meltdown affect most major processors. Unless your servers that are running Apache Solr or Zookeeper are specifically patched, these servers are most likely affected. Most Operating System providers have released patches that protect against Meltdown. However, updates to protect against Spectre are still being released as it requires more extensive remediation.
If you are running in the cloud and / or in a virtualized environments, the cloud providers need to update the underlying infrastructure. For major cloud providers, below is the current status:
|Cloud Provider||Relevant Link||Cloud Provider Advisory|
|Amazon Web Services (AWS)||AWS Security Bulletin||Issues have been addressed for all Hypervisors. Recommendation for customers to patch their instance operating systems.|
|Microsoft Azure||Azure Guidance||The infrastructure that runs Azure and isolates customer workloads from each other is protected. This means that other customers running on Azure cannot attack your application using these vulnerabilities. Install updates from your operating system provider when available.|
|Google Cloud Platform||Product Status||Infrastructure patched against known attacks. Customers must patch/update guest environment.|
You, as the user still need to protect your servers against these vulnerabilities. And patch the guest operating system that is running on your servers.
How to protect your Apache Solr instances running in the cloud?
A complete protection against these vulnerabilities would likely require CPU design changes. However, software patches can provide mitigation against these exploits. Unfortunately, these patches are also known to decrease the performance of your servers.
As of January 9th, the following Ubuntu Distributions have released updates to provide some patches to these vulnerabilities. Ensure that your OS kernel is updated to at-least the below patch versions to be certain of some mitigation.
|Ubuntu 17.10||kernel 4.13.0-25-generic|
|Ubuntu 16.04||kernel 4.4.0-109-generic|
|Ubuntu 14.04||kernel 3.13.0-139-generic|
To check which version of the kernel you can execute a “hostnamectl status” command that would provide you the details on the Architecture, Operating System and Kernel. Below is what a sample output on a Microsoft Azure VM looks like:
ubuntu@ss756535-1:~$ hostnamectl status Static hostname: ss123456-1 Icon name: computer-vm Chassis: vm Machine ID: fb4644c304c4442c803a1398983580a8 Boot ID: 596c9f251b7444d8b6e8bf0a698bf358 Virtualization: microsoft Operating System: Ubuntu 16.04.3 LTS Kernel: Linux 4.11.0-1015-azure Architecture: x86-64
As you can see above, this server has Ubuntu 16.04 with Kernel 4.11.0-1015. This is higher than the above mentioned, this show that it has been patched.
How can the patches be applied to your Apache Solr Cluster?
Below are the steps you can follow to update your Apache Solr Cluster. These steps can be performed on cloud providers that support VMs (e.g. AWS, Microsoft Azure, Google Cloud Platform). For mission critical sites, our recommendation would be to migrate your search traffic to a different cluster or different data center. However, you could perform these steps in a rolling fashion. If you are setup on SolrCloud with two or more replicas, a single replica can handle your application workload.
We would also like to recommend that you should ensure you have about 200-500mb available storage on your servers to ensure appropriate packages can be downloaded.
- Login to one of your Solr servers
- Stop Solr Service
sudo service solr stop
- Grab the latest of all packages on the server
sudo apt update
- Upgrade the Kernel and Packages
sudo apt dist-upgrade
- Cleanup all old kernel and OS related packages
sudo apt autoremove
- Reboot your machine
- Once the server is back online, ssh in and check the active kernel against the list above to ensure that your kernel has been upgraded.
- If Solr service is not started yet, start the solr service
sudo service solr start
- Go to your Apache Solr admin and ensure the replica is synced up with the leader and is in active state
Our recommendation is that you should check for new updates frequently to ensure that you receive future patches as they become available and apply them.