Defining document indexes

You can use two types of search indexes for the pages of websites (i.e. documents in the content tree):

To define which documents an index covers, specify allowed or excluded content:

1. Open the Smart search application.
2. Edit () the index.
3. Select the Indexed content tab.
4. Click Add allowed content or Add excluded content.
5. Open the Sites tab and assign the websites where you wish to use the index.
6. Switch to the Cultures tab and select which language versions of the website’s documents are indexed.
   • You must assign at least one culture in order for the index to be functional.
   • If you have a multi-site index, you can select the cultures separately for each site.

Adding allowed content

Allowed content defines which of the website’s documents are included in the index. Specify documents using a combination of the following options:

• Path - path expression identifying the documents that should be indexed.
• Document types - allows you to limit which document types are included in the index.

The following properties define types of additional content that you can include in Documents search indexes. The settings are not available for Documents crawler indexes:

• Include ad-hoc forums - includes the content of ad-hoc forums placed on the specified documents (if there are any).
• Include blog comments - includes blog comments posted for blog post documents.
• Include message boards - includes message boards placed on the specified documents.
• Include attachment content - if checked, the index includes the text content of files attached to the specified documents. See Searching attachment files for more information.
• Include categories - if checked, the index stores the display names of Categories assigned to the specified documents. This allows users to find documents that belong to categories whose name matches the search expression.

Examples

Adding excluded content

Excluded content allows you to remove documents or entire website sections from the allowed content. For example, if you allow /% and exclude /Special‑pages/% at the same time, the index will include all documents on the site except for the ones found under the /Special-pages node.

You can specify the following options:

• Path - path expression identifying the documents that should be excluded.
• Document types - allows you to limit which document types are excluded from the index.

Examples

Configuring search settings for document fields

Documents are often complex data structures with many different fields. Not all fields may be relevant to the search that you are implementing. Document types allow you to adjust how the system indexes specific fields. We recommend indexing only necessary fields to keep your indexes as small (and fast) as possible.

To edit the field search settings for document types:

1. Open the Document types application.
2. Edit () a document type.
3. Open the Search fields tab.

In the top part of the tab, configure how the system displays documents of the given type in search results:

• Title field - select the document field whose value is used for the title of search results.
• Content field - the field whose value is used for the content extract of search results.
• Image field - the field that contains the image displayed next to search results.
• Date field - the field whose value is used for the date and time displayed in search results.

The table in the bottom section of the tab determines how the smart search indexes the document type’s fields (as defined on the Fields tab). You can set the following search options for individual fields:

When running searches using document indexes, the system returns results according to the field search settings of individual document types. The document type search settings are shared by all document indexes in the system.

SKU (product) and general document fields

To configure the field search settings for E-commerce SKUs (products):

1. Open the Modules application.
2. Edit () the E-commerce module.
3. Open the Classes tab.
4. Edit the SKU class.
5. Select the Search tab.
6. Click Customize.

You can configure the search settings for fields just like for document types. The SKU fields are joined together with general document fields, such as fields that store the content of editable regions on pages (DocumentContent) or the content of text widgets (DocumentWebParts).

Configuring document crawler indexes

Document crawler search indexes read the content of pages while logged in under a user account. You can configure the following properties for every document crawler index (on the General tab of the index editing interface):

By default, document crawlers also index documents that use redirection from the site’s main domain name to a domain alias. To only allow indexing for pages that use the website’s main domain, set the CMSCrawlerAllowSiteAliasRedirect key to false in your application’s web.config file:

<add key="CMSCrawlerAllowSiteAliasRedirect" value="false" />

The key applies to all document crawler indexes in the system.

Customizing how crawlers process page content (API)

By default, the system converts the HTML output of documents to plain text before saving it to document crawler indexes:

• Strips all HTML tags
• Removes the Head tag, Style tags and all JavaScript
• Converts all whitespace formatting to simple spaces

If you wish to index the content of any tags or exclude parts of the page output, you can customize how the crawlers process the HTML. You need to implement your custom functionality in a handler of the OnHtmlToPlainText event of the CMS.Search.SearchCrawler class. This event occurs whenever a document search crawler processes the HTML output of a page.

To assign a method as the handler for the OnHTMLToPlainText event, add a new class to the App_Code folder of your web project (or CMSApp_AppCode -> Old_App_Code on web application installations). For example, you can define the content of the class as shown below:

using System;
using System.Web;

using CMS.Base;
using CMS.Search;
using CMS.Helpers;

[DocumentCrawlerContentLoader]
public partial class CMSModuleLoader
{
    /// <summary>
    /// Attribute class for assigning event handlers.
    /// </summary>
    private class DocumentCrawlerContentLoaderAttribute : CMSLoaderAttribute
    {
        /// <summary>
        /// Called automatically when the application starts.
        /// </summary>
        public override void Init()
        {
            // Assigns a handler for the OnHtmlToPlainText event
            SearchCrawler.OnHtmlToPlainText += new SearchCrawler.HtmlToPlainTextHandler(SearchHelper_OnHtmlToPlainText);
        }

        // Add your custom HTML processing actions and return the result as a string
        static string SearchHelper_OnHtmlToPlainText(string plainText, string originalHtml)
        {
            string outputResult = originalHtml;

            // Removes new line entities
            outputResult = outputResult.Replace("\n", " ");

            // Removes tab spaces
            outputResult = outputResult.Replace("\t", " ");

            // Removes JavaScript
            outputResult = HTMLHelper.RegexHtmlToTextScript.Replace(outputResult, " ");

            // Removes tags
            outputResult = HTMLHelper.RegexHtmlToTextTags.Replace(outputResult, " ");

            // Decodes HTML entities
            outputResult = HttpUtility.HtmlDecode(outputResult);

            return outputResult;
        }

    }
}

The OnHTMLToPlainText event provides the following string parameters to the handler:

• plainText - the page output already stripped of all tags and converted to plain text
• originalHTML - the raw page HTML code without any modifications