Jan. 18, 2018
Sameer Maggon
|
Meltdown (CVE-2017-5754) and Spectre (CVE-2017-5753 and CVE-2017-5715) are two of the most widespread security issues in the design of modern CPUs and can affect your Apache Solr deployments. These vulnerabilities were disclosed on January 4, 2018. They affect almost every computer made in the last 10 years. These vulnerabilities take advantage of certain processor optimizations. They make it possible for hackers to reveal data stored in memory. In this article, we’ll discuss: how to know if your Solr servers are affected, how to protect your Solr instances, and how to apply the patches to your Solr cluster.
Spectre and Meltdown affect most major processors. Unless your servers that are running Apache Solr or Zookeeper are specifically patched, these servers are most likely affected. Most Operating System providers have released patches that protect against Meltdown. However, updates to protect against Spectre are still being released as it requires more extensive remediation.
If you are running in the cloud and / or in a virtualized environments, the cloud providers need to update the underlying infrastructure. For major cloud providers, below is the current status:
| Cloud Provider | Relevant Link | Cloud Provider Advisory |
|---|---|---|
| Amazon Web Services (AWS) | AWS Security Bulletin | Issues have been addressed for all Hypervisors. Recommendation for customers to patch their instance operating systems. |
| Microsoft Azure | Azure Guidance | The infrastructure that runs Azure and isolates customer workloads from each other is protected. This means that other customers running on Azure cannot attack your application using these vulnerabilities. Install updates from your operating system provider when available. |
| Google Cloud Platform | Product Status | Infrastructure patched against known attacks. Customers must patch/update guest environment. |
You, as the user still need to protect your servers against these vulnerabilities. And patch the guest operating system that is running on your servers.
A complete protection against these vulnerabilities would likely require CPU design changes. However, software patches can provide mitigation against these exploits. Unfortunately, these patches are also known to decrease the performance of your servers.
As of January 9th, the following Ubuntu Distributions have released updates to provide some patches to these vulnerabilities. Ensure that your OS kernel is updated to at-least the below patch versions to be certain of some mitigation.
| Ubuntu 17.10 | kernel 4.13.0-25-generic |
|---|---|
| Ubuntu 16.04 | kernel 4.4.0-109-generic |
| Microsoft Azure | Azure Guidance |
| Ubuntu 14.04 | kernel 3.13.0-139-generic |
ubuntu@ss756535-1:~$ hostnamectl status
Static hostname: ss123456-1
Icon name: computer-vm
Chassis: vm
Machine ID: fb4644c304c4442c803a1398983580a8
Boot ID: 596c9f251b7444d8b6e8bf0a698bf358
Virtualization: microsoft
Operating System: Ubuntu 16.04.3 LTS
Kernel: Linux 4.11.0-1015-azure
Architecture: x86-64
As you can see above, this server has Ubuntu 16.04 with Kernel 4.11.0-1015. This is higher than the above mentioned, this show that it has been patched.
Below are the steps you can follow to update your Apache Solr Cluster. These steps can be performed on cloud providers that support VMs (e.g. AWS, Microsoft Azure, Google Cloud Platform). For mission critical sites, our recommendation would be to migrate your search traffic to a different cluster or different data center. However, you could perform these steps in a rolling fashion. If you are setup on SolrCloud with two or more replicas, a single replica can handle your application workload.
We would also like to recommend that you should ensure you have about 200-500mb available storage on your servers to ensure appropriate packages can be downloaded.
Login to one of your Solr servers
Stop Solr Service
sudo service solr stop
Cleanup all old kernel and OS related packages
sudo apt update
Upgrade the Kernel and Packages
sudo apt dist-upgrade
Cleanup all old kernel and OS related packages
sudo apt autoremove
Reboot your machine
sudo reboot
Once the server is back online, ssh in and check the active kernel against the list above to ensure that your kernel has been upgraded.
hostnamecrt status
If Solr service is not started yet, start the solr service
sudo service solr start
Go to your Apache Solr admin and ensure the replica is synced up with the leader and is in active state
Logout
exit
Copyrights © SearchStax Inc.2014-2024. All Rights Reserved.
SearchStax Site Search solution is engineered to give marketers the agility they need to optimize site search outcomes. Get full visibility into search analytics and make real-time changes with one click.
close
SearchStax Managed Search service automates, manages and scales hosted Solr infrastructure in public or private clouds. Free up developers for value-added tasks and reduce costs with fewer incidents.
close
close