For quite some time, we didn’t know for sure which functionality was available for the different deployments and licenses. In January or so, an article surfaced on TechNet describing which features are available for the various SharePoint Online plans. But the on-premise perspective was still missing. Until a few weeks ago, when the very same Technet page was updated to also include information for on-premise solutions. Confusingly, the page is still called SharePoint Online Service Description.

In any case, what should you look out for from a Search perspective?

How about the cloud?

If you have small requirements on search, e.g. you only need to index whatever is stored in SharePoint and you’re happy with how most things are out of the box, then by all means investigate the various cloud plans. However, as they all lack the connector framework, you can’t index anything living outside of SharePoint. In addition, all cloud plans are weak on the content processing side. None of them can do custom entity extraction, i.e. using dictionaries to pull metadata out of unstructured text, and adding your own logic to content processing is off the table as well.

In other words, you’ll have trouble getting your content into the index, and there’s no possibility to process that content in order to enable a richer search experience. Still, if your search requirements are low, and all your important stuff already sits in SharePoint, this may not be a problem for you.

Additionally – and this is a huge bummer for many smaller organizations who’s planning to go with SharePoint Online – none of the cloud plans include the possibility to use the Content Search Web Part. This one you’ll really miss if you’re planning to build search-driven applications around your portal.

In other words, if you’re looking for more advanced search technology than the very basics, an on-premise solution is the way to go. No surprises here.

 On-premise solutions

The only combination of deployment type and license model that gives you a 100% complete feature set is the on-premise Enterprise license. And if you’re looking to implement Enterprise Search in your organization, the choice is very simple. You need the Enterprise CAL.

Foundation is completely off the table for more advanced solutions, due to the missing connector framework. The Standard license is somewhat better, but just like the cloud offerings it’s very weak on the content processing side and there’s no tunable relevancy either. This may be alright for small shops with small requirements on search, but also keep in mind that you’re missing out on the Content Search Web Part and the more advanced feature sets of the Query Rule engine. If those two had been included, the Standard license would have been a lot more pleasing for many small organizations.

Here’s the link to the spreadsheet again http://sdrv.ms/13k5u9f, and it’s also available for your viewing pleasure right here on the blog. Feel free to hook me up on Twitter or via the comment field below if you want to discuss the various deployments and licenses for SharePoint search!

In my previous blog post, Examining the new search core in SharePoint 2013, I attached a presentation on Slideshare. The first few slides summarized FAST’s product history, both before and after it was acquired by Microsoft. I also boldly stated that we’re seeing the “End of an era, the birth of a new age”. As cheesy as it may sound, at least from a technology perspective, I find this to be very true.

FAST Search and Transfer, the Norwegian company that sprung from the university in Trondheim, was founded as early as 1997. That’s a staggering 16 years ago. Even 4 years before the initial release of SharePoint.

Although it’s debatable when FAST initially entered the Enterprise Search market, the first release of FAST Data Search (FDS) was certainly a milestone. Much later, Microsoft acquired FAST and snapped up its hugely successful successor, the Enterprise Search Platform (ESP). It was later turned into the products FSIA/FSIS and also served as the backbone for FAST Search Server 2010 for SharePoint (FS4SP).

One thing that strikes me when reminiscing about FAST, is how the technology stack can be traced throughout its history. Many of the internal FAST processes we saw in FS4SP were actually already introduced back in FDS (of course in varying shape and form). But like I said in my last blog post, the idea that “FAST is now fully integrated” in SharePoint 2013 is just… well, wrong. The “technology trace” is now gone.

The figure below outlines FAST’s product history. As you can see, FS4SP was a merge between the FAST and the SharePoint worlds. And it showed. A Microsoft technician, who shall remain unnamed, once referred to it as the “bastard child of the two paradigms”. Although a little harsh – FS4SP was arguably a great improvement to the search-deprived SharePoint world – there was certainly some truth to it.

Eventually, with the arrival of SharePoint 2013, Microsoft released a well-tailored and cohesive search experience. But there was no longer an obvious technical inheritance from the earlier FAST product line, which is why SharePoint 2013 isn’t about having FAST “fully integrated”. It’s, simply put, a whole new platform. Of course with a few exceptions, e.g. some functionality from FSIS carried over to the new release. But generally speaking, we’re looking at a completely new platform, rewritten from the ground up.

In other words, the pre-MS technology is no longer around and the “technology trace” that we saw in earlier products is now cut off. Additionally, the “FAST” trademark has now slowly started the process of fading into obscurity.

All said and done, it’s only appropriate that FAST – the company, its technology and the trademark – is finally given a proper farewell. Without further ado, I give you its obituary: FAST, may you rest in peace, among friends in Enterprise Software heaven.

(Sorry, just couldn’t help but to shamelessly plug Comperio’s consulting services )

While researching, talking about, and listening on what people say about search in SharePoint 2013, I’ve noticed two major themes.

The idea that “FAST is now fully integrated”

…and while this is certainly true, it’s also not even close to the full story. Search in SharePoint 2013 is a completely new technology, unparalleled in the Microsoft world. The actual index is fundamentally different, boosting better performance by using in-memory technology and even being atomic!

The excitement about the User Experience

A few years back, FAST technology was sold and marketed by technical things like linear scaling, content processing power, and low search times. That’s light years away from what excites people about search in SharePoint 2013. Quite the contrary, the web is full of people buzzing about the actual search experience. How the default search center looks cleaner. How well document previews are integrated. That the addition of automatic search analytics makes relevance calculation… wait for it… self-learning! And so on.

I can’t help but think — actually I’m hoping — that this marks a shift in how people think about search: that it’s not defined by the internal technicalities, but the user experiences we can build on top of it.

And while UX is arguably one of the most important factors of a search solution, I’ll always have a weak spot for what’s actually going on under the hood. So if you’re like me, check out the presentation below!

The release of SharePoint 2013 is touted by Microsoft as a bridge from the past to the future; a pivotal shift from server- and wave-based releases to a rapid release cycle, combining the muscles from the cloud, fresh design principles and Enterprise Social technology. They’re proud, and they should be.

In the keynote, Microsoft also revealed some of the rationale behind the recent acquisition of Yammer, seeing how it complements SharePoint’s strong document-management skills with Yammer’s deep expertise in making the enterprise more social.

Jeff Teper, another high-profile SharePoint executive, told the crowds that over the last three years, Microsoft’s engineering team has centralized their effort around three important pillars: Experiences, Innovation and Ecosystem. This shows off in a dramatically improved user experience, better development tools and a completely revamped application model paving the way for us partner companies to more rapidly and with more flexibility deploy innovative applications in the cloud.

Another consistent theme throughout the keynote, was how Search now has a bigger role across SharePoint and throughout the Microsoft stack. FAST is now fully integrated, and the new search core drives both content and recommendations throughout the suite.

We now have a much better baseline when implementing Search technology in the enterprise. Hopefully this is finally the time when custom implementations of Enterprise Search will stop ending up as just a search box, and instead unleash the potential of Your organization’s accumulated knowledge and tackle today’s overwhelming growth of information.

They say that everyone dies twice. First when you actually die, and then finally when your name is mentioned for the last time.

Back before online search revolutionized ranking algorithms, people sorted their search results by date, alphabetically or by any other static metadata. The concept of a relevance-based result set, where items are evaluated against your search terms at query-time, was rare. Now, we’ve gotten used to sorting things by relevance. A concept as easy to understand as it is hard to implement.

Rank models, i.e. the math behind calculating a metric for relevance, sparked an interest in me the moment I first heard of them. And ever since, every time someone wanted to sort a result set by the “old-fashioned way”, e.g. by date, I always cringed. But I swallowed my search geek-pride and moved on.

How search engines calculate relevance and rank is way out of the scope of this post, but because of my long-running loathing of non-relevance based sorting, I want to highlight a small but glorious thing in the new SharePoint 2013 search GUI, that may have bypassed even early adopters in the findability scene. My heart almost skipped a beat when I first noticed it at one of last week’s sessions at the Microsoft SharePoint conference in Las Vegas.

Alternate sorting in SharePoint 2010

In the search center in SharePoint 2010, both the one for Enterprise Search and FAST, the default was to sort by relevance. Great, but there was a drop down that allowed you to change over to sorting by date or, in the case of FAST, any other managed property which had been enabled for sorting. Even so, I’ve experienced time after time that the drop down was not used very often. Why? Probably because dates and many other metadata is several times better navigated using drill down facets, and this is something that comes natural to many end-users over time.

Of course there are exceptions and occasions when an alternate static sorting over a single metadata property makes sense, but for a good portion of search centers, I’m very convinced that the drop down adds little more than just some clutter to the GUI.

In the SharePoint 2013 search center, you can still sort by any property, of course. But the drop down in the default GUI is not there anymore. You’re encouraged to sort by relevance, and that’s it. There are two things to this:

• Firstly, the functionality to sort on a single static metadata property is still there behind the scenes, and very easy to expose to the user if it’s necessary in your solution.
• Secondly, the engine driving the relevance calculation is much improved and it’s now easier than ever (in the SharePoint world) to tune your relevance model; allowing it to span over several metadata and dynamically evaluate search words against the index corpus.

But not having that “sort by” drop down in the default GUI is a huge step.

The search box in SharePoint 2013, no static sorting to be seen

This is not only about getting a cleaner GUI, but more importantly also about getting to a point where users have started to assume that sorting by relevance just works. Silently, but surely, search engines are maturing, and we’ve – equally silently – started to assume they work better and better.

My inner search geek has started to wonder that perhaps static sorting first died when the rules of online search changed with PageRank, and people got accustomed to sorting by relevance. And with having that alternate sorting drop down removed from the default SharePoint 2013 GUI, perhaps this is the time when static sorting fades into obscurity – and dies the second time.

Dear static sorting, R.I.P.

Let’s take it from the beginning. Using Best Bets, i.e. hard-wiring a certain result item to a particular search term, is a great way for Search Administrators to make sure end-users find what they’re looking for. Interestingly enough, there’s sometimes a slight contempt towards Best Bets in the technical community. Some people seem to think that the search solution’s relevance model should be able to stand on its own, and make sure the best results are always returned at the top of the results anyway.

But as a Search Administrator, if you could increase the perceived end-user experience a lot by a just a little manual (or semi-automatic) labor, wouldn’t you do it? Sure you would. One way of doing that is simply to look into the top X search queries in your query logs, and spend a few minutes on each search query and hard-wire a certain result item against it.

Anyway – we’re on the Technology section, so let’s get dirty.

In FS4SP, Best Bets are managed through the FAST Search Keywords page under Site Collection Administration. Adding a new one is easy: you start by creating a keyword, and then attach a Best Bet to it. Simple enough.

Let’s say you added the keyword “Comperio”, with the definition “Microsoft Search Partner of the Year 2011”. You then attached a Best Bet to it, linking to Comperio’s web site. With the minimal configuration, you’d end up with something like this:

Cool! You got yourself a Best Bet, and this is how they’re shown on the majority of FS4SP deployments out there. However, the Best Bet is not the only thing showing. The first star is showing the keyword itself, a long with its definition.

Let’s say you want to save some pixel real-estate, and decides that showing just the link is enough. At the Edit Keyword page, there’s no switch saying if to show the definition or not, but you can remove the text altogether from the text area shown below. That ought to do it!

Unfortunately, the result is not much better. We still get two stars, only now the keyword definition text is removed. The keyword itself is floating around by its own above the Best Bet – and not a single pixel was saved from vertical misery.

There’s no other setting on the FAST Search Keywords page to fix this – but there’s still a nice work-around. Enter PowerShell!

In an FS4SP shell, get hold of a Search Setting Group object corresponding to your Site Collection. If you only have one site collection, you can do:

$ssg = Get-FASTSearchSearchSettingGroup

If you have more than one, and haven’t named the group manually, pass in the id of your site collection to the the cmdlet’s –Name parameter. Running the two lines below in a SharePoint Management shell will print the id:

$site = Get-SPSite -Identity <a href="http://&lt;site collection root&gt;">http://&lt;site collection root&gt;</a>
$site.ID

When you’ve got hold of the search setting group, you can use it to view all internal information about the keyword:

PS C:\FASTSearch\bin> $ssg.Keywords.GetKeyword("comperio")
Synonyms        : {}
BestBets        : {Comperio}
FeaturedContent : {}
Promotions      : {}
Group           : Microsoft.SharePoint.Search.Extended.Administration.WCFClient.SearchSettingGroupImplWCF
Definition      : <DIV> </DIV>
Term            : comperio
LastChanged     : 1/27/2012 12:36:57 AM
Id              : 2

Looking at the Definition property, you’ll notice that’s it not actually empty at all – even though you’ve clearly removed all text from the definition text area. It contains the HTML entity for a non-breaking space wrapped in a <div> tag! Turns out that if you use the GUI to clear the definition, this is literally as empty you can make it. Even on a brand new keyword, without even specifying a definition. Bug or feature? I don’t know. But now knowing what the problem is, you can easily fix it for yourself by running:

PS C:\FASTSearch\bin> $keyword = $ssg.Keywords.GetKeyword("comperio")
PS C:\FASTSearch\bin> $keyword.Definition = $null

For the most part, Microsoft has done a great job integrating FAST into the SharePoint world, but there are certainly some areas where you notice there are actually two paradigms behind the scene: how things work in SharePoint, and how things (used to) work in FAST.

The primary key of the FS4SP index is one of those areas. In this post, I’ll try to make this a little more understandable.

First of all, the place where most people will notice that a primary key actually exists is in the crawl log on the FAST Content SSA.

Each indexed item is assigned an auto-incremented integer referred to as Item ID. All right, so let’s dig a little deeper.

If you’ve been reading up on FS4SP, you probably already know that there is an internal FAST-process called “qrserver”. It is responsible for receiving queries from the FAST Query SSA and internally forwarding the query to the actual index. You might also know that this process exposes a small web interface. For security reasons, it’s only available from within the server it runs on. More specifically: http://localhost:13280.

Searching for something here will return results in an internal XML format. The actual result items are listed under the <RESULTSET> tag a bit down in the XML. There are lots of things to talk about here, e.g. that the naming convention used internally in FAST is quite different to what is used when the results come back through the FAST Query SSA. A managed property for example, is referred to as a “field” internally. But let’s not fret over that now. Instead, let’s look at the first few properties of the first result:

After a few quick comparisons of the crawl log on the FAST Content SSA and the search results from the qrserver, it’s clear that the Item ID is stored in FS4SP’s index inside the property contentid. When comparing with the Item IDs listed on the FAST Content SSA, we also notice that SharePoint is prefixing the Item ID with “ssic://” when it’s stored in the index. In other words, the true primary key as used internally, is based on the pattern “ssic://[auto-incremented integer]”.

But as anyone who’s been using FAST pre-Microsoft can tell you, the contentid is actually not the primary key of the index. The real primary key is what’s stored inside the property internalid. The value of this property is the MD5 digest of the contentid, concatenated with the name of the content collection it is stored in. Let’s double-check, using our example. We had these two:

<FIELD NAME="internalid">8a832873c701c00135ce827d6c64c09c_sp</FIELD>
<FIELD NAME="contentid">ssic://33</FIELD>

Calculating the MD5 digest of the contentid correctly yields the internalid (sans the collection suffix):

md5(“ssic://33”) == “8a832873c701c00135ce827d6c64c09c”

Since the internalid is suffixed with the name of the collection, we can actually put several items with the same contentid into the index. The requirement being that they’re stored in separate collections, so that the concatenated internalid value becomes unique. In FS4SP however, we often use only the default “sp” collection. Luckily, SharePoint makes sure to assign the Item IDs so that they’re unique across all collections, hence creating unique internalid:s even though the items are in the same collection.

However, there are ways in FS4SP to index data without going through the FAST Content SSA, i.e. you can index data without letting SharePoint know about it. This happens when you’re using any of the FAST Search specific connectors or the command-line tool docpush. These tools talk directly with the index, bypassing SharePoint completely. Thus, the content id item won’t be assigned an Item ID using the “ssic://” pattern.

So, what happens instead? Let’s try it out. Using the docpush tool, we can send an arbitrary web page into the index:

Using the qrserver web interface, we inspect what was indexed:

In this case, the contentid property is the URL of the web page we specified. This makes sense as the URL is unique for the whole web, and as such it is also a good candidate for being a primary key in the index. A URL is just a special case of a URI, which is what many of the FS4SP command-line tools use when referring to the primary key of the index. Examples being the docpush tool (when deleting a item from the index with the –d switch) and the waadmin tool (used for retrieving link cardinality data for an indexed item).

To sum up with some key points:

• The primary key of the index is stored in a property called contentid, though in the SharePoint GUIs it is referred to as an Item ID and look slightly different. They relate to one another as: [contentid] = ssic://[item id]
• Items that are indexed using the connectors of the FAST Content SSA are assigned a contentid on the form “ssic://”…
• Items that are indexed with the FAST Search specific connectors or the docpush tool do not follow the same pattern, but are typically a proper URL or a value from a database.
• If a command-line tool calls for a “URI” to an indexed item, use whatever is stored in the item’s contentid property.

The Microsoft SharePoint Conference is consistently the number one event on SharePoint. This time, even though it wasn’t even a release year, more than 7000 participants attended the conference, beating both the Microsoft Partner and the Build conference.

Microsoft SharePoint Conference 2011 venue center

We’re sincerely impressed by the professionalism and the amount of planning that must have gone into this event. Microsoft really did an amazing job; all the way from when the first invitations were sent off to when the last booth had been disassembled. As presenters on the conference, we also had the pleasure of working with Microsoft behind the scenes, and we can assure you that the success of SPC 2011 was in no way a coincidence.

Early Monday morning, Jared Spataro, Jeff Teper and Kurt DelBene kick-started the conference with an inspiring keynotes session. Besides giving the many thousands in the crowd an outline of the week, they also shared a glimpse of their opinions and the outlook of SharePoint technology going forward. If you didn’t get a chance of attending the conference, make sure to watch the keynote online.

As a company specializing in Enterprise Search and FAST Search for SharePoint, we’re of course extra thrilled that Microsoft – in the actual keynote -  gave us a live demonstration of a SharePoint and FAST Search installation containing more than 100 million documents. Of course, they didn’t just show it to us; an assistant was brought onto the scene and pulled the power cord to the server rack, thus triggering a full farm failure. In just a few seconds, the solution had recovered and was back online, fully operational. The crowd was sheering!

As for the actual sessions, SPC 2011 really had something for everyone. The amount of sessions, hands-on labs and activities going on was simply staggering. The Enterprise Search area alone, whether having a business-oriented or technical focus, was covered by more than 30 sessions! Including our own day-long pre-conference training session Learning to develop a high-end search solution with FAST Search for SharePoint 2010 and our break-out session on User-Centric Design for Deploying FAST Search Server 2010 for SharePoint.

Comperio hands-on lab on FAST Search for SharePoint at SPC 2011

But besides all interesting sessions, attending SPC 2011 was of course about more than just learning new material. We also had an absolute blast at Disneyland, rented just for this occasion, and at the many other venues set up specifically for us attending SPC 2011. Most of the time, the partying started right when we stepped out from the elevator, as the lobby in our hotel turned out to be one of the hot spots for grabbing a “SharePint” after the sessions.

We’ve met so many interesting and inspiring people from all over the world; whether that someone was a SharePoint developer from down-under in trouble with their FAST Search installation or people from Fortune 100 companies around the globe looking to finally to tackle their Knowledge Management challenges.

Everyone at the conference must have heard the key message from Microsoft as loud and clear as us at Comperio: Enterprise Search is the solution to a wide range of problems in your organization, and FAST Search technology will help you working towards solving those problems.

This is an exciting time to be in the search business!

But even if you do, you may have noticed that the FAST Search Web crawler is still turned on behind the scenes (on a default single-node installation). The Node Controller, i.e. the “nctrl” tool that handles FS4SP internal processes, is set to automatically start the Web crawler and its associated processes by default. In this blog post, we’ll circumvent this in order to gain some insight into the inner workings of the Node Controller.

Run a “nctrl status”, and you will see all running FS4SP-processes:

The processes in red are all related to the crawler:

crawler
The main component of the FAST Search Web crawler.

browserengine
A component which emulates a real web browser, and makes sure the crawler “see” what a regular user see while browsing the web.

webanalyzer, fdmworker, walinkstorereceiver, walookupdb0
Components of the Web Analyzer, which calculates a link graph of the crawled content. This data can be used for ranking purposes (compare to Google’s PageRank algorithm). Please note that the Web Analyzer is also used for relevance calculations in SharePoint crawls – and not only for crawled external web content.

If you don’t plan to use the FAST Search Web Crawler, and don’t care for the output from the Web Analyzer module, you can shave off something like 50-200 MB of RAM usage by stopping these processes. Although it’s an easy operation, don’t do this unless you understand the steps involved, the consequences, how to roll back the changes, and most importantly: that you now tread into unsupported and undocumented waters. Also, even though I and a few other people have tested this without running into problems, it’s impossible to guarantee that your particular configuration is unaffected. But at the very least, please read on to gain some insight into how the Node Controller works.

If you still want to disable the FAST Search Web Crawler, stop all crawler-related processes by running the following command in a FS4SP shell:

nctrl stop browserengine crawler fdmworker walinkstorerreceiver walookupdb0 webanalyzer

Running “nctrl status” will now show you this:

As the crawler processes are now in the “User suspended” mode, they will remain suspended even after a full restart of FS4SP.

If you like, you can go even further and clean up the output of “nctrl status” by removing the now disabled processes from the list altogether. To do this, open up the Node Controller’s configuration file, i.e. %FASTSEARCH%\etc\NodeConf.xml. At the top, you will see something like:

<startorder>
  <proc>nameservice</proc>
  <proc>configserver</proc>
  <proc>contentdistributor</proc>
  <proc>indexer</proc>
  <proc>search-1</proc>
  <proc>topfdispatch</proc>
  <proc>samworker</proc>
  <proc>samadmin</proc>
  <proc>qrserver</proc>
  <proc>qrproxy</proc>
  <proc>indexingdispatcher</proc>
  <proc>browserengine</proc>
  <proc>fdmworker</proc>
  <proc>walookupdb0</proc>
  <proc>walinkstorerreceiver</proc>
  <proc>webanalyzer</proc>      
  <proc>sprel</proc>
  <proc>spelltuner</proc>
</startorder>

Comment out all crawler-related processes, like this:

<startorder>
  <proc>nameservice</proc>
  <proc>configserver</proc>
  <proc>contentdistributor</proc>
  <proc>indexer</proc>
  <proc>search-1</proc>
  <proc>topfdispatch</proc>
  <proc>samworker</proc>
  <proc>samadmin</proc>
  <proc>qrserver</proc>
  <proc>qrproxy</proc>
  <proc>indexingdispatcher</proc>
  <!--
  <proc>browserengine</proc>
  <proc>fdmworker</proc>
  <proc>walookupdb0</proc>
  <proc>walinkstorerreceiver</proc>
  <proc>webanalyzer</proc> 
  -->    
  <proc>sprel</proc>
  <proc>spelltuner</proc>
</startorder>

Save the file, and then issue the command “nctrl reloadcfg” to make the Node Controller pick up the changes. All processes you commented out are not stripped from the “nctrl status” output. But the “crawler” process remains, indeed it was not listed under the <startorder> tag that you just edited. This is because the crawler is administered through the “nctrl” tool itself, and not through its configuration file. Run “nctrl remove crawler” to get rid of it as well.

Your status output should now look like the below. Nice and clean!

Of course, turning off functionality like this is an advanced operation – not recommended to most users. However, even though you don’t want to do this yourself, at least you’ve now gained some insight into how FS4SP’s Node Controller works. With a little imagination and extrapolating you can now even add your own custom applications to the Node Controller, and make sure they are started alongside FS4SP. Let me know if you have any questions, or run into problems.

Update:

If you’re serious about getting rid of the FAST Search Web crawler, Mikael from Tech and Me kindly pointed out an alternative strategy for disabling it: change the deployment.xml accordingly, and run the Set-FASTSearchConfiguration cmdlet. This will however require you to shut down FS4SP (nctrl stop), which is not necessary using the above procedure. But it’s arguably a cleaner approach for getting rid of the crawler. TechNet has more information about deployment reconfigurations in general. Please note that this will overwrite your %FASTSEARCH%\etc\NodeConf.xml – editing it manually is unsupported after all.

Still though, it would be cool to have a PowerShell script that runs a query through a flow, and displays the results. Turns out it’s not that hard to build! Though I have to admit it gets a little complex, and is not exactly FSIS standard procedure. But I found it interesting to put together and useful to have around – perhaps you will too.

There’s a link to the source code at the end of this post, and I will discuss the most important parts of the script below. So let’s dive right in! The script takes two parameters: a flow name and a query that will target the ESP default composite field. In this experimental version of the script I haven’t polished the output, and actually only prints the first result document. Feel free to modify it to allow more advanced (arbitrary?) queries and stylish output – just make sure to report back when you’re done!

Here’s a screenshot of what the script produces today:

The IMS nodes expose two external interfaces that clients can use to send query requests: a WCF interface and a HTTP interface. At first look, the HTTP interface probably sounds like the easy way out. But it turns out the FSIS Shell is already prepared to run WCF queries.

How is that you wonder? When you run the built-in FSIS shell, you’ll notice that it preloads the shell with a bunch of important FSIS snap-ins. It also loads a special application configuration, more precisely %FSIS_HOME%\Powershell\Powershell.exe.config. This file defines two important WCF endpoints: NetTcpBinding_IProcessingEngineManager and NetTcpBinding_IProcessingEngine. It also defines which certificate to use in order to secure the communication. This is exactly what we need.

There’s one caveat with this approach: you’ll have to run the PowerShell console with your FSIS service user. This is because the certificate is fetched from its personal certificate store. Let me know if you find an elegant way around this.

How do we actually run the flow? Well, there is an Execute-Flow cmdlet. No, there are actually two. One from the snap-in EnginePSSnapIn for running flows in general, and one from InteractionEnginePSSnapIn that targets IMS flows. When you start the FSIS shell, they’re loaded in this order:

Add-PSSnapin HostControllerPSSnapIn
Add-PSSnapin JunoPSSnapin
Add-PSSnapin EnginePSSnapIn
Add-PSSnapin InteractionEnginePSSnapIn

Notice the problem? Yup, the one from InteractionEnginePSSnapIn will actually overwrite the one from EnginePSSnapIn. This is most likely a bug in the current FSIS release. But for the purpose of this blog post, it really doesn’t matter. We want the one from InteractionEnginePSSnapIn anyway.

Execute-Flow takes a Context as parameter. This context object is populated with a query and an IMS flow, and is then passed from task to task through the specified flow, until it comes out on the other end – hopefully with some search results attached to it.

Now is probably the time to say that this script won’t work for all types of flows. It depends on what type of tasks you put in your flow, and how you create the actual query that you’ll put on the context object. In this example, I’m using the simplest IMS flow possible: just a flow input, an ESP lookup, and the flow output:

So let’s create the context and make sure we target the IMS flow that was supplied when the script was called. The particular class we need is already loaded into the FSIS Shell environment, so it’s pretty easy:

$context = New-Object Microsoft.Ceres.InteractionEngine.Services.Context
$context.EnsureRouting().FlowName = $flowName

Building the query is harder. To do that, we have to go a little outside the scope of the FSIS shell. We have to use the same assembly that FSIS uses under the hood. Most users would probably never need to do this, but rather use IMS UI Toolkit to build queries and render results.

The DLL we’re interested in is called Microsoft.Ceres.InteractionEngine.ContextModifiers.dll. We load it into PowerShell with the one-liner below. Notice that $env:FSIS_HOME is PowerShell’s way of referencing an environment variable, in this case the installation directory of FSIS.

[Reflection.Assembly]::LoadFile($env:FSIS_HOME + "AdminUI\site\bin\Microsoft.Ceres.InteractionEngine.ContextModifiers.dll")

After loading this assembly, we can reference classes in the ContextModifiers.Ims namespace. It contains lots of interesting classes for building advanced queries and parsing results. For now, we settle on using the ImsTermsOperator class, which translates into a standard text query. The ImsTermsOperator has several properties that control linguistics, if multiple terms within it should be ANDed or ORed, etc. For now, we just specify the “Terms” property to whatever is passed in when the script is called. Since we’re not specifying the $searchOp.Scope property, the query will target the default composite field.

$searchOp = New-Object Microsoft.Ceres.InteractionEngine.ContextModifiers.Ims.ImsTermsOperator
$searchOp.Terms = $query

Next step is to add the $searchOp to the context, and for that we need to use a wrapper class called ImsSearch. Take a look in the source code below for the gory details about this. For now, let’s just skip right to the grand finale: we connect to IMS using Connect-ProcessingEngine, and run the flow with the Execute-Flow cmdlet; passing in the context object and getting back an updated version bundled with the search results:

Connect-ProcessingEngine $imsHost $imsPort
$context = Execute-Flow $context

As the $context variable is serialized during transfer back and forth to IMS, we also need to explicitly parse out the search results, and cast it into an object of type Ims.ImsSearchResults. It looks a little messy, but is actually very straight-forward.

Take a look in the code for yourself: Query-IMS-Flow.ps1.

All of this could of course also be written in your favorite .NET language. Then you could also get around the certificate issue quite easily. I guess I’m just a PowerShell fanboy.