Solr Service Alerting - SearchStax


Overview

SearchStax® provides two kinds of real-time email alerts:

Either type of alert may optionally invoke a webhook to notify an external bug-tracking system or alerting system.

Both types of alerts create an "incident" report that you can inspect in the SearchStax dashboard.

Premium Alerting

For SearchStax customers with Premium Support Level Agreements (SLAs), we have an internal monitoring system that notifies our on-call support team of any issues.

Contents of this page:

Heartbeat Alerts

Both Zookeeper and Solr send reports of system metrics to SearchStax once per minute. You can set up a "heartbeat" alert to notify you if these reports are interrupted. The system also notifies you when the updates resume.

Best Practice: Avoid false alarms

There are innocent events that can make Pulse miss a beat, such as transient network delays. Alerts have a configurable delay to help avoid false alarms.

Set up a Heartbeat Alert

To set up a heartbeat alert, open the SearchStax dashboard and navigate to a specific deployment.

  1. Expose the details page for that deployment.
  2. Click the Pulse label in the left-side navigation bar.
  3. Open the Alerting menu in the top menu bar.
  4. Select Heartbeat.

SearchStax Heartbeat Alerting

Control Description
Server The Server control offers a list of the servers in this deployment. Select one of them to monitor.
Name Give the alert a name that you will recognize when you see it in email.
Notify if data is missing for more than... When heartbeat data stops flowing, wait this long before triggering the alert.
Max Notifications Alert emails are reissued every two minutes. How many of them do you want to send?
Send alerts to Choose from a list of registered SearchStax users.
Send trigger alert to webhook Invoke this webhook when this alert is triggered.
Send resolve alert to webhook Invoke this webhook when the alert is resolved.

Heartbeat Email

A heartbeat email notification resembles this one:

Dear SearchStax Customer,

The alert ss380502-5 heartbeat alert for your deployment Films (ss380502) has been triggered.

The following host is unreachable.

Host: ss380502-5

To View Metrics in Dashboard: https://app.searchstax.com/admin/deployment/pulse/deployment/ss380502/alert/incident/update/65737

To Edit this Alert: https://app.searchstax.com/admin/deployment/pulse/deployment/ss380502/alert/heartbeat/update/841/

This alert was triggered at 2020-01-15 20:12:27 UTC.

This alert was raised for account SilverSupportAccount.

You will receive a similar "UP" notification when the heartbeat is again detected.

Threshold Alerts

A "threshold" alert watches a specific system metric and sends you email when the metric meets or exceeds some value.

SearchStax allows you to monitor the following metrics:

Set up a Threshold Alert

To set up a threshold alert, open the SearchStax dashboard and navigate to a specific deployment.

  1. Expose the details page for that deployment.
  2. Click the Pulse label in the left-side navigation bar.
  3. Open the Alerting menu in the top menu bar.
  4. Select Threshold.
  5. Click the Create New Alert button.

Threshold Alert SearchStax

Control Description
Host Machine The Host Machine control offers a list of the servers in this deployment. Select one of them to monitor.
Metric Name Choose one of many internal metrics monitored by SearchStax.
Collection Some metrics are collection-specific. Others apply to "all collections."
Alert Name Give the alert a name that you will recognize when you see it in email.
Delay of at least Metric must exceed threshold for this long before triggering the alert.
Max Alerts Alert emails are reissued every two minutes. How many of them do you want to send?
Repeat Every Time to wait between sending repeat email messages.
Send alerts to Choose from a list of registered SearchStax users.
Send trigger alert to webhook Invoke this webhook when this alert is triggered.
Send resolve alert to webhook Invoke this webhook when the alert is resolved.

Receive a Threshold Alert

A threshold email notification resembles this one:

Dear SearchStax Customer,

The alert "Server 5 below 10% CPU" for your deployment Films (ss380502) has been triggered.

Host:           ss380502-5
Metric:         CPU Usage
Name:           "Server 5 below 10% CPU"
Threshold:      < 10.0%
Current Value:  0.01 %

To View Metrics in Dashboard: https://app.searchstax.com/admin/deployment/pulse/deployment/ss380502/system/

To Edit this Alert: https://app.searchstax.com/admin/deployment/pulse/deployment/ss380502/alert/incident/update/6012

This alert was triggered at 2019-12-20 17:51:42 UTC.

This alert was raised for account SilverSupportAccount.

Incidents

To view a list of your heartbeat or threshold incidents, open the SearchStax dashboard and navigate to the deployment in question. (Alternately, there is an incident link in the email message you received. The link takes you directly to the details of the incident.)

  1. Expose the details page for that deployment.
  2. Click the Pulse label in the left-side navigation bar.
  3. Open the Alerting menu in the top menu bar.
  4. Select Incidents.

SearchStax Alerting Incidents

Click the incident to view its details. You'll see a brief description of the incident followed by a timeline of events. Read the timeline from the bottom up.

Questions?

Do not hesitate to contact the SearchStax Support Desk.