Optimizely
If you’re considering SearchStax Site Search or have recently purchased a site search solution, you’re probably thinking through data ingestion. Data ingestion is often a major consideration in getting site search up and running, and developing a sound approach to data import is critical as it directly impacts the quality and relevance of search results.
In the context of site search, data ingestion is the process of importing and loading data from one or more data sources and making it available in a structured format that can be indexed and searched by a search engine. The data may include website content, program details, research, documents and more. Data ingestion also involves repeatedly pulling in data on a real-time or regular batched basis.
The goal of data ingestion for site search is to ensure that the search engine can quickly and accurately retrieve relevant information for a user’s search query and improve the overall user experience.
Getting Data into SearchStax Site Search
With SearchStax Site Search, data ingestion means getting data into a search index so it can be accessed via a search request on a website or in a custom application. This post looks at the various ways to load data into Site Search, identifies the sources and types of data we support and provides recommendations for best practices.
There are three main ways to load data into SearchStax Site Search:
- SearchStax Crawler
- CMS Connectors for Sitecore and Drupal
- Ingest APIs
SearchStax Crawler for Site Search
SearchStax Site Search Crawler (Crawler) makes it easier and faster to get started with Site Search. Crawler works with any website content management system CMS and digital experience platform (DXP).
Crawler can automatically discover and index content across your entire website without any modification to your code. You can get your site search up and running in minutes with customized field extraction, search configuration and relevancy tuning – all easily accessible through the Site Search dashboard.
How Does Crawler Work?
Crawler starts with an entry URL or sitemap URL. From there it will automatically discover additional pages on your site by looking for navigation, links and sitemap entries for each page. Once every page has been discovered and crawled, each page gets indexed and is ready for search discovery.
SearchStax Site Search Crawler performs a daily crawl to keep your search results current as your content evolves over time.
CMS Connectors for Sitecore and Drupal
If you use Sitecore or Drupal for your content management system CMS, SearchStax has integration modules that automate the data indexing process and offer the added advantage of real-time updates to your search index with every content addition, change or deletion.
Sitecore Integration
The SearchStax Site Search Connector for Sitecore is available for Sitecore versions starting with version 9 through version 10.3. The Connector integrates with the Sitecore Indexing Manager and automatically indexes all Sitecore content items out-of-the-box. Additional information can be found in the Sitecore Connector product documentation.
Drupal Integration
The SearchStax Site Search Connector for Drupal automatically tracks all search results known to the Drupal Search API. Once the Drupal Connector is installed and configured, it automatically indexes any new or updated content in the Drupal environment. The module adds search functionality while requiring virtually no changes to the Drupal website. The Drupal integration was developed by Thomas Seidl (drunken monkey), the creator and maintainer of the Drupal Search API, and follows all Drupal open source code guidelines. Additional information can be found in the Drupal Connector product documentation or from the Drupal Connector module page at Drupal.org.
SearchStax Ingest APIs
The SearchStax Data Ingest API is a service that allows you to index and search structured data in your SearchStax search service. The API enables you to send data to your search service in real-time, making it immediately searchable by users. Customers can also use the SearchStax Ingest API to load documents into their Site Search application.
The Ingest APIs simplify the data ingestion process by enabling a customer or an implementation partner to create a small piece of code to get data from any source and push it into SearchStax Site Search. You can index individual JSON documents, multiple JSON documents or a JSON file with an array of JSON objects. You can also index XML documents by sending one or multiple tags. Additional information on using the Ingest APIs can be found in the Site Search product documentation.
Sources and Types of Data for SearchStax Site Search
The primary use cases for SearchStax Site Search involve adding search capabilities to any content management systems such as Sitecore, Drupal, Acquia, Adobe AEM, WordPress, Hubspot, Optimizely, Coremedia, Hannon Hill, Magnolia, Salesforce DXP, any HTML website and custom apps.
If you are working with Sitecore or Drupal, most customers will use the SearchStax Connectors for these CMS solutions. For other content sources, you will either use our Crawler or have developers use the ingest APIs to index your content.
As far as content, the following types of data can be managed by SearchStax Site Search: HTML web pages, PDFs, Word documents, Excel spreadsheets, Powerpoint files, text files, rich text format (RTF) and Visio drawing files (VSD).
SearchStax Site Search enables marketers and developers to deliver powerful site search at scale. Talk to a Search Expert with our search experts to see how search can improve the visitor experience and gain actionable insights to quickly optimize the search experience.
Data Ingest for Site Search FAQs
Get Our Newsletter
The Stack is delivered bi-monthly with industry trends, insights, products and more
