Model Simulation - SearchStax


SearchStax® Analytics let you record user events and play them back as Model Simulations. This feature lets you directly compare two Solr configurations using exactly the same input.

Professional Services Engagement

Some of the features described on this page require the participation of SearchStax Professional Services.

Contents of this page:

Model Simulation

A simulation runs recorded user actions from a production system against a development system. It then reports how well the development system performed compared to the production system. This lets you make a series of development tests that are exactly comparable to one another and to the production system.

Javascript Events Library

SearchStax Analytics features, including Model Simulations, depend on collecting user events from your running applications.

See Recording Events to learn how to capture user events on your web page. Event messages are sent to a SearchStax Analytics App for collation and display.

The events are recorded in the SearchStax Analytics App's database, and can be retrieved for playback by the simulation.

Setting Up a Simulation

A simulation is profiled as part of a specific SearchStax Analytics App.

Open SearchStax and navigate to the list of Analytics Apps. Click on an app to display its dashboard. Click on Model Simulation to view the list of simulations (if any). SearchStax A/B Simulation To profile a new simulation, click the New Simulation button in the upper right. This opens a form for simulation properties. Search A/B Simulation

Parameter Description
Simulation Name The human-readable name of the simulation.
API Endpoint The base URL of the search endpoint.
Query String Params Additional parameters concatenated to the URL specified by API endpoint. For example, if the API endpoint is http://foo.com and the Query String params is key=foo, then the resulting URL would be http://foo.com?key=foo.
Range Start Timestamp marking the beginning of the events to replay. "2017-10-10 17:00:00".
Range End Timestamp marking the end of the events to replay. "2017-10-19 17:00:00".
Page Key The name of the pagination parameter. For example, if the API endpoint is http://foo.com and the page key is page, then the first URL generated will be http://foo.com?page=1 (assuming that Start Page = 1). The next URL generated would be http://foo.com?page=2, etc.
Start Page The number of the starting page. Can be 0 or 1.
Page Size Key The name of the parameter to use for the page size (# of results returned per page). For example, if the Page Size Key is size, and Page Size is 20, then the URL might look like http://foo.com?page=2&size=20.
Page Size The value of the parameter to use for page size (20 in the previous example).
Max Pages The maximum number of pages to crawl. For example, if Start Page = 1, and Max Pages = 10, then the last URL generated in the sequence would be something like http://foo.com?page=10.
Path to objects list After a URL is generated and sent to a search server, the server generates a document that is presumed to be in JSON format. This expression tells the simulator where to look for items in that document. Paths are separated by the "|" delimiter. For example, the _embedded,events path means look in the _embedded section of the document, then look in the events section. The simulator expects an array of JSON dictionaries in that part of the document.
Request Type The HTTP method to use when accessing the search server, either GET or POST. (Typically GET, since we're fetching data and not uploading data.)
Top Percentile Percentage of total requests, ranked by frequency, to use in the simulation. 10% would include the most-popular queries, which in some situations might all be the same query. Higher percentages include a greater mix of less-frequent queries. The run time goes up exponentially with the percentage.
Delay Between Requests Number of milliseconds the crawler waits between HTTP requests.
Required Mappings These required settings map fields in the response items (the JSON dictionaries) to event data. They are used by the simulator to compute simulation stats based the output of crawled data.

query: Required field mapping from the analytics query field to your API field.

cDocId: Required field mapping the analytics cDocId field to your API field.

Regex to extract id from cDocId? A regular expression that can be used to extract the cDocId from the cDocId field. Leave empty if you have an exact match.

Is cDocId exact? If checked, the value in the cDocId field will be matched exactly against event data (used for MRR and ACP calculations). If unchecked, the regex is used instead of an exact match.
Optional Mappings These optional settings map fields in the response items (the JSON dictionaries) to event data. They are used by the simulator to compute simulation stats based the output of crawled data.

Analytics key: The name of a field in an event. Used only if the API key is not an expression containing %(foo)s expressions. Example: event = {query:foo, bar=baz}, Analytics Key=bar, API Key=yow. This will generate a URL such as http://foo.com?yow=baz.

API Key: Can be a simple string (for example, "yow" in the previous example), or an expression such as param=%(field1)s,%(field2)s. In the latter case, the Analytics key is ignored and URL parameters are generated based by reading values from fields denoted in the parenthesis and generating an expression substituting the %(field)s patterns with the values. For example if the API key is latlong=%(lat)s,%(lon)s, and event = {query:foo, lat=2, lon=4}, the following URL might be generated: http://foo.com?latlong=2,4.

Note that a new simulation begins to run as soon as you save it.

Simulations can run for hours...

Simulations can run for great lengths of time. Range Start, Range End, and Top Percentile all have an impact on run time. (Start small.)

Running a Simulation

The architecture of running a test involves both SearchStax and a simulator application that feeds input to the treatment deployment.
SearchStax A/B Simulation

The simulator collects the result events from the treatment and relays them to SearchStax for display.

SearchStax Professional Services will set up the simulator for you.

Viewing the Results

When the test is complete, return to the list of simulations. Click the "+" to the left of the simulation name to unfold the table of results. (The plus changes to a minus in the expanded view.) There you will find a display comparing the metrics of the simulation with those of the production system. SearchStax A/B Simulation

The Download Data button displays the URLs that were generated by the simulator from the recorded data.

Questions?

Do not hesitate to contact the SearchStax Support Desk.