The thesaurus lookup is performed at query time, and the dictionary can be set up with support for various languages. The thesaurus must be deployed as a csv file using powershell, it has columns for key, synonym, and an optional language. The “key” column can be a phrase, and so can the synonym. ie. “Go fishing” can be a synonym for “hunt for fish”. To provide several synonyms for a word, simply add it several times. To make the synonym go both ways, add a second entry with the word and synonym switching place. (See Microsoft for further details )

The thesaurus is simple, but it works. Or, does it really? Testing synonyms fetched from the internet on a SharePoint search index populated with US Court records crawled from theinfo.org.  AKA is a common legal term synonymous with “also known as”. So let’s try that out.

Searching for “aka” yields some hundred results, searching for “also known as” yields a different set of results of roughly the same size. By adding synonyms we would expect the search results to combine the hits for both queries into one, so to speak.
So I create a thesaurus containing:

Key,Synonym,Language
aka,also known as

And upload it with the powershell command:

$searchApp = Get-SPEnterpriseSearchServiceApplication
Import-SPEnterpriseSearchThesaurus -SearchApplication $searchApp -Filename \\spbox\temp\thesaurus.csv

I wait for a few seconds, and search for “aka”.
Now, I would expect to find hits containing “also known as”. But where is it? I have to scroll and page down to the bottom of page 3 before I find it:

Why? Could the ULS logs provide any clues? Turning on verbose logging on the Search Query Processing, and search again. Now, in the ULS logs there are entries with “After thesaurus tree modification” (the indentation is mine, trying to make it look a little clearer, also abbreviated it some).

Microsoft.Office.Server.Search.Query.Pipeline.Executors.LinguisticQueryProcessingExecutor : After thesaurus tree modification:
	'AndNode(FirstChild=StringNode
(FirstChild=WordsNode(FirstChild=TokenNode(FirstChild=null,NextSibling=OnearNode(FirstChild=TokenNode(FirstChild=null,NextSibling=TokenNode
		(FirstChild=null,NextSibling=TokenNode
			(FirstChild=null,NextSibling=null,Length=1,Linguistics=True,Token=<strong>as</strong>,Weight=1),
			Length=1,Linguistics=True,Token=<strong>known</strong>,Weight=1),
			Length=1,Linguistics=True,Token=<strong>also</strong>,Weight=1),NextSibling=null
,ExtraTermsAllowed=0,<strong>Weight=0.2</strong>),
Length=1,Linguistics=True,Token=<strong>aka</strong>,Weight=1),NextSibling=null),

The synonyms are given a weight of 0.2. The original term has a weight of 1, theoretically that means the synonym has 20% percent weight of the original term. Perhaps we could make the synonyms show up by increasing the weight.  So, how can we do that? Apparently, there is no way.  Not unless we open up the magic box of the Ceres shell.

We begin by connecting to the Interactionengine:

Add-PsSnapin Microsoft.SharePoint.Powershell
& "C:\Program Files\Microsoft Office Servers\15.0\Search\Scripts\ceresshell.ps1"
Connect-System -Uri  (Get-SPEnterpriseSearchServiceApplication).SystemManagerLocations[0] -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity
Connect-Engine -NodeTypes InterActionEngine

Now, let’s try to rip out the configurations of the SharePointSearchProvider flow:

$flowname = Microsoft.SharePointSearchProviderFlow
Get-Flow $flowname  > $flowname.txt

Reading the flow configurations for the SharePointSearchProviderFlow, we find an option named synonymWeight.

<Operator name="Linguistics" type="LinguisticQueryProcessing">
  lt;Property name="querySpellingCorrectionTokenLimit" value="10" />
...
<Property name="stemWeight" value="0.2" />

&lt;Property name=<strong>"synonymWeight" value="0.2" </strong>/&gt;

&lt;/Properties&gt;

&lt;/Operator&gt;

Strangely, it has the weight 0.2. Ring any bells, anyone? It is the same weight we saw in the ULS. Now, let us try to see what happens if we increase the weight here.
So, we set the synonymWeight to 1, and upload the file.

Remove-Flow $flowname
Get-Content $flowname.txt | Out-String | Add-Flow $flowname
Stop-Flow –FlowName $flowname –ForceAll

Now, when searching for “aka”, we get hits containing “also know as” on the first search results page.
Voila!

When Microsoft bought FAST, they were in the process of developing improvements to their search solution, code named Mars.  Among the improvements were graphical interfaces for the flow engines for Query (IMS) and Content Processing (CTS). These were packaged and sold As FSIS, Fast Search for Internet Sites. SharePoint 2013 has integrated the pipeline workflow from CTS and IMS, and is using it internally. There is, however, no (apparent) option for configuring the flows by the end user. Of course, there is no nice graphical wizards for drawing up the pipeline steps as in FSIS.

Buried deep down in the folders of SharePoint, there is a file called ceresshell.ps1.  Ceres is a dwarf planet circulating between Mars and Jupiter. Incidentally, we can use the Ceres shell to gain access to the configurations of the internal SharePoint flows. Once Mars was reached, the team behind SharePoint search kept on pushing into space, until they reached solid ground on the next planet, Ceres.

So what is the Ceres shell, and what can it do? Looking at the contents of the file, it is a powershell script that loads some snap ins.

Add-PSSnapin hostcontrollerpssnapin
Add-pssnapin junopssnapin
Add-pssnapin searchcorepssnapin
Add-pssnapin enginepssnapin
Add-pssnapin analysisenginepssnapin

So, we start up a SharePoint powershell session as Farm administrator, and load in the Ceres shell.

PS> & "C:\Program Files\Microsoft Office Servers\15.0\Search\Scripts\ceresshell.ps1"

Now the Ceres cmdlets have been loaded up and are ready to use.

Before we can have any fun with this, we need to connect to the “system”.

Connect-System  -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity

Once connected to the system, we need to connect to the “engine”

Connect-Engine

Now we are ready. To get a list of all flows available for inspection, type

Get-Flow

To show the flow configuration for a single flow:

Get-Flow Microsoft.CrawlerIndexingSubFlow

Here is a part of that file

<OperatorGraph name="Microsoft.CrawlerIndexingSubFlow" xmlns="http://schemas.microsoft.com/ceres/studio/2009/10/flow">

<Parameters />

<Operators>

<Operator name="MarsWriter" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.MarsWriter">

<Properties>

<Property name="callbackType" value=""Completed"" />

<Property name="callbackWarningField" value=""ParsingErrors"" />

<Property name="commitInterval" value="1" />

<Property name="crawledPropertyBuckets" value="["content"]" />

<Property name="defaultMaxIndexSize" value="524288" />

<Property name="defaultMaxResultSize" value="16384" />

<Property name="idField" value=""externalId"" />

<Property name="managedPropertiesListName" value=""ManagedProperties"" />

<Property name="managedPropertyBuckets" value="["ManagedPropertiesBucket"]" />

<Property name="marsCallbackInfoProperty" value=""marsLinkDBSynchronization"" />

<Property name="provideCallbacks" value="True" />

<Property name="siteCollectionIdField" value=""sitecollectionid"" />

<Property name="tenantIdField" value=""tenantId"" />

<Property name="truncatedFlagField" value=""IsPartiallyProcessed"" />

</Properties>

</Operator>

<Operator name="SubFlowInput" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.SubFlow.SubFlowInput">

<Targets>

<Target>

<operatorMoniker name="/Microsoft.CrawlerIndexingSubFlow/MarsWriter" />

Interesting, isn’t it? There seems to be some configurable values for stuff like “defaultMaxResultSize”.

Now you see how you can open the Ceres shell, and look at the flow configurations.

Next post will show you how you can modify values in the existing flows.