The thesaurus lookup is performed at query time, and the dictionary can be set up with support for various languages. The thesaurus must be deployed as a csv file using powershell, it has columns for key, synonym, and an optional language. The “key” column can be a phrase, and so can the synonym. ie. “Go fishing” can be a synonym for “hunt for fish”. To provide several synonyms for a word, simply add it several times. To make the synonym go both ways, add a second entry with the word and synonym switching place. (See Microsoft for further details )

The thesaurus is simple, but it works. Or, does it really? Testing synonyms fetched from the internet on a SharePoint search index populated with US Court records crawled from theinfo.org.  AKA is a common legal term synonymous with “also known as”. So let’s try that out.

Searching for “aka” yields some hundred results, searching for “also known as” yields a different set of results of roughly the same size. By adding synonyms we would expect the search results to combine the hits for both queries into one, so to speak.
So I create a thesaurus containing:

Key,Synonym,Language
aka,also known as

And upload it with the powershell command:

$searchApp = Get-SPEnterpriseSearchServiceApplication
Import-SPEnterpriseSearchThesaurus -SearchApplication $searchApp -Filename \\spbox\temp\thesaurus.csv

I wait for a few seconds, and search for “aka”.
Now, I would expect to find hits containing “also known as”. But where is it? I have to scroll and page down to the bottom of page 3 before I find it:

Why? Could the ULS logs provide any clues? Turning on verbose logging on the Search Query Processing, and search again. Now, in the ULS logs there are entries with “After thesaurus tree modification” (the indentation is mine, trying to make it look a little clearer, also abbreviated it some).

Microsoft.Office.Server.Search.Query.Pipeline.Executors.LinguisticQueryProcessingExecutor : After thesaurus tree modification:
	'AndNode(FirstChild=StringNode
(FirstChild=WordsNode(FirstChild=TokenNode(FirstChild=null,NextSibling=OnearNode(FirstChild=TokenNode(FirstChild=null,NextSibling=TokenNode
		(FirstChild=null,NextSibling=TokenNode
			(FirstChild=null,NextSibling=null,Length=1,Linguistics=True,Token=<strong>as</strong>,Weight=1),
			Length=1,Linguistics=True,Token=<strong>known</strong>,Weight=1),
			Length=1,Linguistics=True,Token=<strong>also</strong>,Weight=1),NextSibling=null
,ExtraTermsAllowed=0,<strong>Weight=0.2</strong>),
Length=1,Linguistics=True,Token=<strong>aka</strong>,Weight=1),NextSibling=null),

The synonyms are given a weight of 0.2. The original term has a weight of 1, theoretically that means the synonym has 20% percent weight of the original term. Perhaps we could make the synonyms show up by increasing the weight.  So, how can we do that? Apparently, there is no way.  Not unless we open up the magic box of the Ceres shell.

We begin by connecting to the Interactionengine:

Add-PsSnapin Microsoft.SharePoint.Powershell
& "C:\Program Files\Microsoft Office Servers\15.0\Search\Scripts\ceresshell.ps1"
Connect-System -Uri  (Get-SPEnterpriseSearchServiceApplication).SystemManagerLocations[0] -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity
Connect-Engine -NodeTypes InterActionEngine

Now, let’s try to rip out the configurations of the SharePointSearchProvider flow:

$flowname = Microsoft.SharePointSearchProviderFlow
Get-Flow $flowname  > $flowname.txt

Reading the flow configurations for the SharePointSearchProviderFlow, we find an option named synonymWeight.

<Operator name="Linguistics" type="LinguisticQueryProcessing">
  lt;Property name="querySpellingCorrectionTokenLimit" value="10" />
...
<Property name="stemWeight" value="0.2" />

&lt;Property name=<strong>"synonymWeight" value="0.2" </strong>/&gt;

&lt;/Properties&gt;

&lt;/Operator&gt;

Strangely, it has the weight 0.2. Ring any bells, anyone? It is the same weight we saw in the ULS. Now, let us try to see what happens if we increase the weight here.
So, we set the synonymWeight to 1, and upload the file.

Remove-Flow $flowname
Get-Content $flowname.txt | Out-String | Add-Flow $flowname
Stop-Flow –FlowName $flowname –ForceAll

Now, when searching for “aka”, we get hits containing “also know as” on the first search results page.
Voila!

When Microsoft bought FAST, they were in the process of developing improvements to their search solution, code named Mars.  Among the improvements were graphical interfaces for the flow engines for Query (IMS) and Content Processing (CTS). These were packaged and sold As FSIS, Fast Search for Internet Sites. SharePoint 2013 has integrated the pipeline workflow from CTS and IMS, and is using it internally. There is, however, no (apparent) option for configuring the flows by the end user. Of course, there is no nice graphical wizards for drawing up the pipeline steps as in FSIS.

Buried deep down in the folders of SharePoint, there is a file called ceresshell.ps1.  Ceres is a dwarf planet circulating between Mars and Jupiter. Incidentally, we can use the Ceres shell to gain access to the configurations of the internal SharePoint flows. Once Mars was reached, the team behind SharePoint search kept on pushing into space, until they reached solid ground on the next planet, Ceres.

So what is the Ceres shell, and what can it do? Looking at the contents of the file, it is a powershell script that loads some snap ins.

Add-PSSnapin hostcontrollerpssnapin
Add-pssnapin junopssnapin
Add-pssnapin searchcorepssnapin
Add-pssnapin enginepssnapin
Add-pssnapin analysisenginepssnapin

So, we start up a SharePoint powershell session as Farm administrator, and load in the Ceres shell.

PS> & "C:\Program Files\Microsoft Office Servers\15.0\Search\Scripts\ceresshell.ps1"

Now the Ceres cmdlets have been loaded up and are ready to use.

Before we can have any fun with this, we need to connect to the “system”.

Connect-System  -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity

Once connected to the system, we need to connect to the “engine”

Connect-Engine

Now we are ready. To get a list of all flows available for inspection, type

Get-Flow

To show the flow configuration for a single flow:

Get-Flow Microsoft.CrawlerIndexingSubFlow

Here is a part of that file

<OperatorGraph name="Microsoft.CrawlerIndexingSubFlow" xmlns="http://schemas.microsoft.com/ceres/studio/2009/10/flow">

<Parameters />

<Operators>

<Operator name="MarsWriter" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.MarsWriter">

<Properties>

<Property name="callbackType" value=""Completed"" />

<Property name="callbackWarningField" value=""ParsingErrors"" />

<Property name="commitInterval" value="1" />

<Property name="crawledPropertyBuckets" value="["content"]" />

<Property name="defaultMaxIndexSize" value="524288" />

<Property name="defaultMaxResultSize" value="16384" />

<Property name="idField" value=""externalId"" />

<Property name="managedPropertiesListName" value=""ManagedProperties"" />

<Property name="managedPropertyBuckets" value="["ManagedPropertiesBucket"]" />

<Property name="marsCallbackInfoProperty" value=""marsLinkDBSynchronization"" />

<Property name="provideCallbacks" value="True" />

<Property name="siteCollectionIdField" value=""sitecollectionid"" />

<Property name="tenantIdField" value=""tenantId"" />

<Property name="truncatedFlagField" value=""IsPartiallyProcessed"" />

</Properties>

</Operator>

<Operator name="SubFlowInput" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.SubFlow.SubFlowInput">

<Targets>

<Target>

<operatorMoniker name="/Microsoft.CrawlerIndexingSubFlow/MarsWriter" />

Interesting, isn’t it? There seems to be some configurable values for stuff like “defaultMaxResultSize”.

Now you see how you can open the Ceres shell, and look at the flow configurations.

Next post will show you how you can modify values in the existing flows.

Still though, it would be cool to have a PowerShell script that runs a query through a flow, and displays the results. Turns out it’s not that hard to build! Though I have to admit it gets a little complex, and is not exactly FSIS standard procedure. But I found it interesting to put together and useful to have around – perhaps you will too.

There’s a link to the source code at the end of this post, and I will discuss the most important parts of the script below. So let’s dive right in! The script takes two parameters: a flow name and a query that will target the ESP default composite field. In this experimental version of the script I haven’t polished the output, and actually only prints the first result document. Feel free to modify it to allow more advanced (arbitrary?) queries and stylish output – just make sure to report back when you’re done!

Here’s a screenshot of what the script produces today:

The IMS nodes expose two external interfaces that clients can use to send query requests: a WCF interface and a HTTP interface. At first look, the HTTP interface probably sounds like the easy way out. But it turns out the FSIS Shell is already prepared to run WCF queries.

How is that you wonder? When you run the built-in FSIS shell, you’ll notice that it preloads the shell with a bunch of important FSIS snap-ins. It also loads a special application configuration, more precisely %FSIS_HOME%\Powershell\Powershell.exe.config. This file defines two important WCF endpoints: NetTcpBinding_IProcessingEngineManager and NetTcpBinding_IProcessingEngine. It also defines which certificate to use in order to secure the communication. This is exactly what we need.

There’s one caveat with this approach: you’ll have to run the PowerShell console with your FSIS service user. This is because the certificate is fetched from its personal certificate store. Let me know if you find an elegant way around this.

How do we actually run the flow? Well, there is an Execute-Flow cmdlet. No, there are actually two. One from the snap-in EnginePSSnapIn for running flows in general, and one from InteractionEnginePSSnapIn that targets IMS flows. When you start the FSIS shell, they’re loaded in this order:

Add-PSSnapin HostControllerPSSnapIn
Add-PSSnapin JunoPSSnapin
Add-PSSnapin EnginePSSnapIn
Add-PSSnapin InteractionEnginePSSnapIn

Notice the problem? Yup, the one from InteractionEnginePSSnapIn will actually overwrite the one from EnginePSSnapIn. This is most likely a bug in the current FSIS release. But for the purpose of this blog post, it really doesn’t matter. We want the one from InteractionEnginePSSnapIn anyway.

Execute-Flow takes a Context as parameter. This context object is populated with a query and an IMS flow, and is then passed from task to task through the specified flow, until it comes out on the other end – hopefully with some search results attached to it.

Now is probably the time to say that this script won’t work for all types of flows. It depends on what type of tasks you put in your flow, and how you create the actual query that you’ll put on the context object. In this example, I’m using the simplest IMS flow possible: just a flow input, an ESP lookup, and the flow output:

So let’s create the context and make sure we target the IMS flow that was supplied when the script was called. The particular class we need is already loaded into the FSIS Shell environment, so it’s pretty easy:

$context = New-Object Microsoft.Ceres.InteractionEngine.Services.Context
$context.EnsureRouting().FlowName = $flowName

Building the query is harder. To do that, we have to go a little outside the scope of the FSIS shell. We have to use the same assembly that FSIS uses under the hood. Most users would probably never need to do this, but rather use IMS UI Toolkit to build queries and render results.

The DLL we’re interested in is called Microsoft.Ceres.InteractionEngine.ContextModifiers.dll. We load it into PowerShell with the one-liner below. Notice that $env:FSIS_HOME is PowerShell’s way of referencing an environment variable, in this case the installation directory of FSIS.

[Reflection.Assembly]::LoadFile($env:FSIS_HOME + "AdminUI\site\bin\Microsoft.Ceres.InteractionEngine.ContextModifiers.dll")

After loading this assembly, we can reference classes in the ContextModifiers.Ims namespace. It contains lots of interesting classes for building advanced queries and parsing results. For now, we settle on using the ImsTermsOperator class, which translates into a standard text query. The ImsTermsOperator has several properties that control linguistics, if multiple terms within it should be ANDed or ORed, etc. For now, we just specify the “Terms” property to whatever is passed in when the script is called. Since we’re not specifying the $searchOp.Scope property, the query will target the default composite field.

$searchOp = New-Object Microsoft.Ceres.InteractionEngine.ContextModifiers.Ims.ImsTermsOperator
$searchOp.Terms = $query

Next step is to add the $searchOp to the context, and for that we need to use a wrapper class called ImsSearch. Take a look in the source code below for the gory details about this. For now, let’s just skip right to the grand finale: we connect to IMS using Connect-ProcessingEngine, and run the flow with the Execute-Flow cmdlet; passing in the context object and getting back an updated version bundled with the search results:

Connect-ProcessingEngine $imsHost $imsPort
$context = Execute-Flow $context

As the $context variable is serialized during transfer back and forth to IMS, we also need to explicitly parse out the search results, and cast it into an object of type Ims.ImsSearchResults. It looks a little messy, but is actually very straight-forward.

Take a look in the code for yourself: Query-IMS-Flow.ps1.

All of this could of course also be written in your favorite .NET language. Then you could also get around the certificate issue quite easily. I guess I’m just a PowerShell fanboy.

The default installation optionally creates and configures a self-signed CA, and issues all necessary certificates. This is handy for development and test installations, when it might not be possible to use a PKI infrastructure. Or simply just when you don’t want to flesh out the money for a trusted root CA.

However, you don’t want to have your production environment running on self-signed certificates. You want to use a proper, trusted, CA.

If you opt to use a trusted CA right from the get-go, when you first install FSIS, the process for doing so is fairly well-documented. But if you want to migrate an existing installation, going from using the self-signed certificates to relying on a trusted CA, there’s at least one caveat you should know about. In the documentation about renewing certificates, and changing the CA, there’s a small note saying:

The procedure assumes that you do not change the name of the CA. SecurityConfig -useNewCerts offers no way to update the CA subject name in the configuration. The name of the CA is used for authorization throughout the system. If it changes, all configuration files referencing this name on all hosts must be updated and all the hosts must be restarted.

This is crucial information, because if you update the CA, it’s very likely that the subject name of the new certificate does not match the old from the self-signed version. Other than that, the process of moving to a trusted CA is straight-forward. Put the new certificates in place, run SecurityConfig with the -useNewCerts options, and restart the FSIS service.

This is just what I did yesterday. Unfortunately for me, after deploying the new certificates, my client application got an Access Denied response, or at best a “The credentials supplied to the package were not recognized”. On my FSIS node, I set up a trace of the communication, but that only told me that the connection was dropped. After lots of debugging, I luckily stumbled over the note in the documentation.

So I needed to update those “configuration files”. As the author wasn’t kind enough to specify which these were, I ran a string search over the whole FSIS install directory. The files that I needed to update were the Node.config files in FSIS’s AdminNode and ImsNode services:

C:\Users\[My FSIS user]\AppData\Local\FSIS\Nodes\Fsis\AdminNode1\Node.config
C:\Users\[My FSIS user]\AppData\Local\FSIS\Nodes\Fsis\InteractionEngineNode1\Node.config

Depending on your setup, you might have several such files, spread over several machines. In any case, those files contain the following XML node:

<add key="IssuerAuthorizationManager.AuthorizedIssuer" value="CN=fsisIssuer@MY_HOSTNAME" />

After updating the CN name, and restarting the FSIS service, my client application finally started to work as expected.

UPDATE

The procedure above; updating the InteractionEngineNodeX\Node.config, will of course only affect the settings for that particular node. After some more digging around, I’ve found out how to make the changes kick in also for new nodes added later on.

The template file for the IMS Node.config files lives here (yes, no file suffix):

%FSIS_HOME%\Resources\Templates\InteractionEngineNode

This file contains the following line:

<add key="IssuerAuthorizationManager.AuthorizedIssuer" value="${Managed.CertificateIssuerName}" />

You can replace ${Managed.CertificateIssuerName} with something like CN=fsisIssuer@MY_HOSTNAME, but even better is to replace the actual variable being expanded. It is defined under the <appsettings> here:

%FSIS_HOME%\Host Controller\HostControllerService.exe.config

But many handled exceptions can be equally frustrating. I recently wrote a small demo application, showing a client how to integrate against FSIS from within their particular server environment. Everything looked okay (doesn’t it always :)), but whenever I ran my queries, FSIS just replied with “Socket Connection was aborted”.

A good place to start when debugging WCF problems is to enable tracing. I added the following XML snippet to my app.config, and opened the generated file with Microsoft Service Trace Viewer. Unfortunately for me, there were no additional error messages or hints on what the problem might be.

<system.diagnostics>
  <sources>
    <source name="System.ServiceModel"
            switchValue="Information, ActivityTracing"
            propagateActivity="true">
      <listeners>
        <add name="traceListener"
             type="System.Diagnostics.XmlWriterTraceListener"
             initializeData="C:\Client-WcfTrace.svclog" />
      </listeners>
    </source>
  </sources>
</system.diagnostics>

Digging further, I enabled tracing in the FSIS endpoint as well. The application configuration for the IMS node is by default sitting in the FSIS user’s home directory, more precisely:

C:\Users\[FSIS service user]\AppData\Local\FSIS\Nodes\Fsis\InteractionEngineNode1\Node.config

Under the System.Diagnostics XML node I added a source tag for System.ServiceModel, so that the whole segment looked like:

<system.diagnostics>
  <switches>
    <add name="NodeRunner" value="Info" />
    <add name="Bootstrap" value="Info" />
    <add name="Debug" value="0" />
  </switches>
  <trace autoflush="false" indentsize="4">
    <listeners>
      <remove name="Default" />
    </listeners>
  </trace>
  <sources>
    <source name="System.ServiceModel"
            switchValue="Information, ActivityTracing"
            propagateActivity="true">
      <listeners>
        <add name="wcfTraceListener"
             type="System.Diagnostics.XmlWriterTraceListener"
             initializeData="C:\Server-WcfTrace.svclog" />
      </listeners>
    </source>
  </sources>
</system.diagnostics>

Sure enough! When inspecting the resulting log file in Trace Viewer, I found a red line indicating exactly what the problem was. In my case, I had messed up my certificates: “the remote SSL client failed to provide a required certificate”.

Read more about WCF tracing on MSDN, and if you haven’t already, make sure to also read my brief introduction to FAST Search for Internet Sites.

For the first time since Microsoft’s acquisition of FAST, the broad audience can now reap the benefits of a high-end platform for enterprise search. This post is a first look at FSIS; a brief overview of the new components and what has kept Microsoft busy throughout the time since the acquisition. The post is simply intended to serve as an introduction to the curious, and makes no attempt to be comprehensive.

Previous incarnations of FAST technology have been equipped with powerful and flexible support for processing documents and indexing them accordingly. Out of a UX perspective, FAST technology has not been equally strong. With FSIS, Microsoft has recognized this imbalance by targeting what we at Comperio refer to as “search logic”. This area, which is both broad and complex, has given birth to several products and frameworks – one of them being Comperio Front. The future will have to decide if this first version of FSIS is strong enough to manage on its own.

The FSIS architecture

Both FSIS, and its sister-product FAST Search Server 2010 for Internal Applications (FSIA), are built around the proven technology in FAST ESP. More specifically, the very core of both these products is FAST ESP 5.3 with service pack 3 or higher.

In FSIS, ESP is fairly well hidden inside of the Microsoft packaging. New modules which encapsulate the existing ESP interfaces provide a high level of abstraction. Most prominent of the new modules are inarguably Content Transformation Services (CTS) and Interaction Management Services (IMS). These handle management and processing of indexing and query flows. There is also FAST Search Designer, a Visual Studio 2008 add-on, which makes it easy to create, debug, and orchestrate these flows.  A fourth module, Search Business Manager targets business-users who need to adjust the overall search experience, and perhaps want to try out new features using the integrated A/B testing functionality.

There are also additional supporting modules, e.g. the IMS UI toolkit which provides a foundation for new web application projects. Another example is the Content Distributor Emulator, which provides backwards-compatibility for legacy ESP connectors. But on a high level, with only the most important modules listed, the structure of FSIS corresponds to the figure below.

Anyone adventurous enough to seek complete and fine-grained control of their search solution will have to go deeper than the outer FSIS layers. ESP is still the go-to place for all advanced customizations, and is still available in its completeness.

Content and interaction workflows

The right part of the figure is typically described as the “content side” or the “source side” of the search solution. This is where CTS is located. Data to be indexed is sent to CTS, where it is manipulated and prepared for indexing according to business-rules and requirements. For example, it might be necessary to enrich certain documents with synonyms or additional keywords. All such logic is graphically mapped up in workflows, using either preconfigured or custom data manipulation “tasks”.

Those of us with experience of development from ESP will feel right at home in the workflow task library in CTS. Many features and functions are direct derivatives of those found in the traditional ESP Document Processing framework, but CTS provides a much-needed face-lift. Most developers will surely appreciate the improved development and debugging possibilities. It should be noted that the traditional Document Processing pipelines are still available; they are in fact still running behind the scenes.

IMS is located on the left side of the figure. Queries are both going in, and coming back out through the workflows in IMS. Just like documents flow through CTS, queries and result sets flow through IMS. The big difference is of course that instead of manipulating content, IMS brings support for tailoring the search experience: e.g. branching up a query into several federated searches, or conditionally enriching or expanding queries. As mentioned at the top of this post, FAST has traditionally been lagging behind in this area. In that regard, IMS is extremely welcome news for developers and business-users alike.

In future blog entries we will dig deeper and explore more of the capabilities in the new FAST-based offerings from Microsoft. We will surely revisit the concept of “search logic”, and how FSIS attempt to solve such problems. Until then, feel free to shoot us any comments or questions!