We sometimes get a complaint that a collection has stopped responding during batch ingestion of new documents. Frequently, the root problem is too-frequent attempts to “commit” the new documents to the index.
Solr supports rapid indexing by temporarily accumulating the new index records in memory. This is very fast, but the new records are not searchable and have not been “persisted” to permanent storage.
A commit (sometimes called a “hard commit”) flushes the new records to index files on disk. The commit also initializes a new Searcher to make the new items available to queries. This sounds very attractive to new Solr users, who sometimes demand a commit after every add (commit=true) or within a second of every add (commitWithin=1000).
Unfortunately, the commit process is expensive. Ingestion waits while Solr writes records to disk. Then Zookeeper distributes the updated files to the appropriate replicas across the cluster.
If commit requests are too frequent, the system bogs down in file operations and replication. The CPU gets overloaded. Zookeeper gets far enough behind to trigger recovery mode for some the replicas. Everything grinds to a halt.
For instance, the system shown above is running 250 commits per minute, which is excessive.
The solution is simple. Don’t demand frequent commits when uploading large batches of data. Immediate commits are for systems that add infrequent but time-critical records. Experience (and common sense) tell us that very few Solr systems need to provide second-by-second index updates during bulk uploads of thousands of records. By default, Solr autoCommits every 15 seconds. (The setting is in solrconfig.xml.) This is usually sufficient to persist the new records, distribute them among replicas, and keep search results fresh.