Solr and Load Balancer Log Files in SearchStax Cloud


SearchStax Cloud log files help you tune Solr performance and isolate potential bugs in your application. The SearchStax Help Center references many typical errors and other log entries very specifically.

The Solr log files may be viewed in a limited way from the Solr dashboard. Recent solr.log files may be downloaded from SearchStax Cloud. There is a special log in SearchStax Cloud that records your team’s actions in the SearchStax Cloud Dashboard. Some other logs are available through the agency of SearchStax support engineers.

Time Stamps in Solr.log Files

The date/time stamps in solr.log files are in UTC (Coordinated Universal Time).

Contents:

  1. Solr Server Logs
  2. Load Balancer Logs
  3. Nginx Logs
  4. SearchStax Cloud Activity Log

Solr Server Logs

Each Solr instance maintains a solr.log file, managed by Apache Log4j.

This file accumulates log messages at the INFO, WARN, and ERROR levels until the size of the file reaches 33.55 MB or until Solr is restarted. At that point, Solr closes the file, renames it solr.log.N, and opens a new solr.log file. You can see the files in the SearchStax Cloud dashboard, Dedicated Infrastructure > Deployment > Deployment Details > Logs screen.

Solr.log is the live file, containing the most-recent log entries. The file extensions (X in solr.log.X) do not always reflect the age of the file contents. Sort the files by date to put them in chronological sequence.

The files in this list are a snapshot of the Solr log files on the server. This snapshot is updated at the top of every hour. This means that when you experiment with your system, you must wait until the top of the hour to see changes in the logs. (If you are in India, the logs become available at the bottom of every hour.)

Log Retention

The topic of “log retention” using Log4j is a matter of file size rather than age. Please see Configuring Logging for more information.

As expressed in SearchStax Cloud, a new solr.log file is initiated each time a Solr instance is started. That file continues to record log messages until either Solr stops, or until the size of solr.log exceeds 33.55 MB. The solr.log file of a small, stable system can accumulate log messages for weeks or months without rolling over.

SearchStax Cloud retains a maximum of ten rollover files. Therefore, when solr.log rolls over for the eleventh time, the oldest file will be deleted.

    Note that you must examine the logs from each Solr node in your cluster. Solr problems sometimes occur on only one node.

    Logging Levels

    The remarks in this section assume that you want to modify the default logging behavior of Log4j in SearchStax Cloud.

    You can adjust Solr logging in the Solr Dashboard:

    Searchstax Solr Log Files

    Solr Dashboard log changes affect only one server

    SearchStax Cloud connects to the Solr Dashboard of one Solr node chosen randomly by a load balancer. Changes to logging levels affect only one server of the cluster. When you return to the Solr Dashboard to undo your log settings later, you might not be connected to the same node. This can make it difficult to back out your changes.

    To cleanly reset all servers to default logging, perform a rolling restart of the cluster. Each server returns to default logging as it restarts.

    You can monitor significant Solr events on an adjacent screen in the Solr dashboard:

    Searchstax Solr Log Files

    See Configuring Logging in the Apache Solr documentation.

    Load Balancer Logs

    All traffic to a SearchStax cluster passes through a Load Balancer that distributes requests across the available servers.

    Due to technical considerations, query logs are most naturally obtained from the Solr cluster’s load balancer. SearchStax Cloud makes the load balancer logs available through the deployment dashboard.

    These are access logs that record the incoming queries, like this one:

    {"httpRequest":{"remoteIp":"76.9.66.238","requestMethod":"GET","requestSize":"118","requestUrl":"https://ss123456-us-west2-gcp.searchstax.com/solr/filmcollection/select?q=director_name:%22Steven%20Spielberg%22&wt=json&indent=true&fl=movie_title","responseSize":"956","serverIp":"10.76.0.2","status":200,"userAgent":"curl/7.58.0"},"insertId":"1sxeg25ff8ohxo","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry","enforcedSecurityPolicy":{"configuredAction":"ACCEPT","name":"ss123456-armor","outcome":"ACCEPT","priority":2147483647},"statusDetails":"response_sent_by_backend"},"logName":"projects/searchstax-production/logs/requests","receiveTimestamp":"2019-01-09T15:00:29.930210771Z","resource":{"labels":{"backend_service_name":"","forwarding_rule_name":"ss123456-https-forwarding","project_id":"searchstax-production","target_proxy_name":"ss123456-https-proxy","url_map_name":"ss123456-loadbalancer","zone":"global"},"type":"http_load_balancer"},"severity":"INFO","spanId":"e549479c9a69ea78","timestamp":"2019-01-09T15:00:29.513737511Z","trace":"projects/searchstax-production/traces/b8e7795aa135c44964cd1a8343a2f34c"}

    SearchStax updates these logs once per hour.

    Nginx Logs

    SearchStax Cloud single-node deployments don’t have Load Balancers, but do use Nginx to handle incoming requests.

    SearchStax premium clients may ask our Support Desk for help in obtaining the Nginx logs when necessary.

    SearchStax Cloud Activity Log

    SearchStax Cloud automatically records all significant user actions that occur in the SearchStax Cloud Dashboard. To view them, click Activity Log in the left-side navigation panel.

    Activity Log entries are retained indefinitely.

    Help with Log Files

    Premium customers may ask SearchStax support for help tracking down memory and performance issues. This often involves enabling logging on servers that the SearchStax end-user cannot reach.

    For this type of help please reach out to the SearchStax Support Desk.

    Questions?

    Do not hesitate to contact the SearchStax Support Desk.