A few years ago Comperio launched a nifty service for Posten Norge, Norway’s postal service. Through the service, retail companies can upload their catalogues and seasonal flyers to make the products listed within searchable. Although the catalogue handling and processing is also very interesting, we’re going to focus on the search side of things in this post. As Comperio has a long relationship and a great deal of experience with FAST ESP, this first iteration of Posten’s Tilbudssok used it as the search backend. It also incorporated Comperio Front, our search middleware product, which recently had a big release. .

Newer is better

Unfortunately, FAST ESP is getting on a bit and as a result Tilbudssok has been limited by what we can coax out of it. To ensure we provide the best possible search solution we decided it was time to upgrade and chose Elasticsearch as the best candidate. If you are unfamiliar with Elasticsearch, take a moment to browse our other blog posts on the subject. The resulting project had three main requirements:

• Replace Fast ESP with Elasticsearch while otherwise maintaining as much of the existing architecture as possible
• Add geodata to products such that a user could find the nearest store where they were available
• Setup sexy log analysis with Logstash and Kibana

Data Sources, Ingestion and Processing

The data source for the search system is a MySQL database populated with catalogue and product data. A separate Comperio system generates this data when Posten’s customers upload PDFs of their brochures i.e. we also fully own the entire data generation process.

The FAST ESP based solution made use of FAST’s JDBC connector to feed data directly to the search index. Inspired by Christoffers blog post, we made use of the JDBC plugin for Elasticsearch. This allowed us to use the same SQL statements to feed Elasticsearch. It took us no more than a couple of hours, including some time wrestling with field mappings, to populate our Elasticsearch index with the same data as the FAST one.

We then needed to add store geodata to the index. As mentioned earlier, we completely own the data flow. We simply extended our existing catalogue/product uploader system to include a store uploader service. Google’s geocoder handled converted addresses to coordinates for use with Elasticsearch’s geo distance sorting. We now had store data in our database. An extra JDBC river and another round of mapping wrestling got that same data into the Elasticsearch index.

Our approach

Before the conversion to Elasticsearch, the Posten system architecture was typical of most Comperio projects. Users interact with a Java based frontend web application. This in turn sends queries to Comperio’s search abstraction layer, Comperio Front. This formats requests such that the system’s search engine, in our case FAST ESP, can understand them. Upon receiving a response from the search engine, Front then formats it into a frontend friendly format i.e. JSON or XML depending on developer preference.

Unfortunately, when we started the project, Front’s Elasticsearch adapter was still a bit immature. It also felt a bit over kill to include it when Elasticsearch has such a robust Java API already. I saw an opportunity to reduce the system’s complexity and learn more about interacting with Elasticsearch’s Java API and took it. With what I learnt, we could later beef up Front’s Elasticsearch adapter for future projects.

As a side note, we briefly flirted with the idea of replacing the entire frontend with a hipstery Javascript/Node.js ecosystem. It was trivial to throw together a working system very quickly but in the interest of maintaining existing architecture and trying to keep project run time down we opted to stick with the existing Java based MVC framework.

After a few rounds of Googling, struggling with documentation and finally simply diving into the code, I was able to piece together the bits of the Elasticsearch Java API puzzle. It is a joy to work with! There are builder classes for pretty much everything. All of our queries start with a basic SearchRequestBuilder. Depending on the scenario, we can then modify this SRB with various flavours of QueryBuilders, FilterBuilders, SortBuilders and AggregationBuilders to handle every potential use case. Here is a greatly simplified example of a filtered search with aggregates:

Logstash and Kibana

With our Elasticsearch based system up ready to roll, the next step was to fulfil our sexy query logging project requirement. This raised an interesting question. Where are the query logs? As it turns out, (please contact us if we’re wrong), the only query logging available is something called slow logging. It is a shard level log where you can set thresholds for the query or fetch phase of the execution. We found this log severely lacking in basic details such as hit count and actual query parameters. It seemed like we could only track query time and the query string.

Rather than fight with this slow log, we implemented our own custom logger in our web app to log salient parts of the search request and response. To make our lives easier everything is logged as JSON. This makes hooking up with Logstash trivial, as our logstash config reveals:

Kibana 4, the latest version of Elastic’s log visualisation suite, was released in February, around the same time as we were wrapping up our logging logic. We had been planning on using Kibana 3, but this was a perfect opportunity to learn how to use version 4 and create some awesome dashboards for our customer:

Kibana 4 is wonderful to work with and will generate so much extra value for Posten and their customers.

Conclusion

• Although the Elasticsearch Java API itself is well rounded and complete, its documentation can be a bit frustrating. But this is why we write blog posts to share our experiences!
• Once we got past the initial learning curve, we were able to create an awesome Elasticsearch Java API toolbox
• We were severely disappointed with the built in query logging. I hope to extract our custom logger and make it more generic so everyone else can use it too.
• The Google Maps API is fun and super easy to work with

Rivers as a data ingestion tool have long been marked for deprecation. When we next want to upgrade our Elasticsearch version we will need to replace them entirely with some other tool. Although Logstash is touted as Elasticsearch’s main equivalent of a connector framework, it currently lacks classic Enterprise search data source connectors. Apache Manifold is a mature open source connector framework that would cover our needs. The latest release has not been tested with the latest version of Elasticsearch, but it supports versions 1.1-3.

Once the solution goes live, during April, Kibana will really come into its own as we get more and more data.

Getting the correct result at the top of your search results isn’t easy. Anyone working within search quickly realizes this. Tuning the underlying ranking model is a job that just doesn’t end. There is an entire profession about search engine optimization, making sure your site gets as high as possible on Google (and Bing, I guess). If it is not the top result on Google, it is somehow your fault and not Google’s.

In my previous blog post, Examining the new search core in SharePoint 2013, I attached a presentation on Slideshare. The first few slides summarized FAST’s product history, both before and after it was acquired by Microsoft. I also boldly stated that we’re seeing the “End of an era, the birth of a new age”. As cheesy as it may sound, at least from a technology perspective, I find this to be very true.

FAST Search and Transfer, the Norwegian company that sprung from the university in Trondheim, was founded as early as 1997. That’s a staggering 16 years ago. Even 4 years before the initial release of SharePoint.

Although it’s debatable when FAST initially entered the Enterprise Search market, the first release of FAST Data Search (FDS) was certainly a milestone. Much later, Microsoft acquired FAST and snapped up its hugely successful successor, the Enterprise Search Platform (ESP). It was later turned into the products FSIA/FSIS and also served as the backbone for FAST Search Server 2010 for SharePoint (FS4SP).

One thing that strikes me when reminiscing about FAST, is how the technology stack can be traced throughout its history. Many of the internal FAST processes we saw in FS4SP were actually already introduced back in FDS (of course in varying shape and form). But like I said in my last blog post, the idea that “FAST is now fully integrated” in SharePoint 2013 is just… well, wrong. The “technology trace” is now gone.

The figure below outlines FAST’s product history. As you can see, FS4SP was a merge between the FAST and the SharePoint worlds. And it showed. A Microsoft technician, who shall remain unnamed, once referred to it as the “bastard child of the two paradigms”. Although a little harsh – FS4SP was arguably a great improvement to the search-deprived SharePoint world – there was certainly some truth to it.

Eventually, with the arrival of SharePoint 2013, Microsoft released a well-tailored and cohesive search experience. But there was no longer an obvious technical inheritance from the earlier FAST product line, which is why SharePoint 2013 isn’t about having FAST “fully integrated”. It’s, simply put, a whole new platform. Of course with a few exceptions, e.g. some functionality from FSIS carried over to the new release. But generally speaking, we’re looking at a completely new platform, rewritten from the ground up.

In other words, the pre-MS technology is no longer around and the “technology trace” that we saw in earlier products is now cut off. Additionally, the “FAST” trademark has now slowly started the process of fading into obscurity.

All said and done, it’s only appropriate that FAST – the company, its technology and the trademark – is finally given a proper farewell. Without further ado, I give you its obituary: FAST, may you rest in peace, among friends in Enterprise Software heaven.

(Sorry, just couldn’t help but to shamelessly plug Comperio’s consulting services )

In this example I have simply crawled the c: drive on my virtual machine with SharePoint to generate some entries in the crawl log. 269 errors aren’t that many; you could click the link and inspect the log manually in a reasonable amount of time. Viewing the logs will let you inspect 50 documents at a time.

This gets boring really fast. Therefore we started to use the API for extracting crawl logs from SharePoint [3] programmatically. Because of the API’s deprecated status the documentation is a bit on the thin side. Luckily there are some examples in the blogosphere [1,2] that helped us along and we could soon dump the crawl logs to file and ctrl-f for our missing document. Of course, the script could find the log entry and only show you the relevant one. However with a large crawl log it can take hours to extract all the log entries. It is better to dump them all to file should you need to look for a second or third file later.

Using grep, findstr or ctrl-f for searching feels wrong when you work as a search consultant. Instead the natural instinct when something large should be searched through is to index it. One option was to index it back into SharePoint through BDC (and get a possible nasty loop with reading and creating logs) or to use the old Fast API directly, bypassing SharePoint. Both felt a little heavy duty for this one-off job, and that’s when we put ElasticSearch to play. ElasticSearch is a fairly new search engine built on Lucene, and sometimes referred to as “The new kid on the block” in the Open Source Community where there’s some friendly rivalry between the Solr and ElasticSearch camps. It is a 20MB download and is ready to index documents mere seconds after you unzip the package. To index a document you simply POST some JSON to a REST endpoint and that is it. With so many libraries offering JSON-serialization and REST calls, building the world’s simplest connector doesn’t take long. We ported the old PowerShell script to C# and added 30 lines of code. With the removal of file creation code we actually reduced the lines of code overall. The indexing specific part of the script looks like this:

All we do is populate a little DTO, serialize it and push it to ElasticSearch. The indexing takes very little time, in fact fetching the logs from SharePoint is the slowest part of the connector. Notice that the call to index the document isn’t even done asynchronously. After indexing some data and indexing the logs ElasticSearch-head[4] is the easiest way to have a look. The tab “Structured Query” lets you create queries without any prior knowledge of the query syntax. As an example, here I have queried for all failing txt files in the log. (Some parts of the screen shot was dropped to make the image fit the page).

If you have a lot of failures in your SharePoint crawl log it makes sense to start with your most common cause of error. In the API each log entry has a property called ErrorId, although StatusCode would be a more fitting name (In PowerShell it seems to be called ErrorId and in C# MessageId). So in order to get the most common cause for errors one should look at the distribution of ErrorIds. In Fast Search for SharePoint this could be done by placing the ErrorIds in a managed property and configure it with the refiner property, effortlessly producing the distribution of ErrorIds. In ElasticSearch you specify refiners, called facets, at query time. In my limited test case the distribution looks like this:

It is clear that ErrorId 0 is the most common followed by 5 and 748. What the specific ErrorIds mean is unfortunately not documented. The easiest is to query for log entries with an ErrorId and look at the error description. As mentioned ErrorId 0 means OK and 1 means deleted. All failures that happen in the Fast pipeline will be wrapped with ErrorId 11. One of the ErrorIds is simply a statement from SharePoint saying that a crawl rule was honored.
Using this as a dashboard and thinking beyond a sample crawl, you could easily envision this being a tool for checking:

• How often and when a document was crawled
• Does the crawler experience issues at certain times of the day
• What crawl rates are the crawler able to sustain in the long run
• What file suffixes are most prone to errors
• What top level folders are the most troublesome to index

The code linked below should be seen as proof of concept. You are free to download and it and it will work. Some parameters on the SharePoint API-calls can surely be set better. ElasticSearch provides a bulk-update API that should be used when indexing large datasets. If you are running ElasticSearch on a non-standard port you need to change the code. Also note the complete lack of error handling.

What about that missing file? Turns out it was there all along. The customer was searching for the filename and didn’t recognize the second hit that was displaying his document’s title, not filename. Oh well.

1: http://blogs.msdn.com/b/spses/archive/2011/06/22/exporting-sharepoint-2010-search-crawl-logs.aspx
2: http://blogs.msdn.com/b/russmax/archive/2012/01/28/sharepoint-powershell-script-series-part-5-exporting-the-crawl-log-to-a-csv-file.aspx
3: http://msdn.microsoft.com/en-us/library/ms514229.aspx
4: http://mobz.github.com/elasticsearch-head
5: http://www.elasticsearch.org

The script can be found here: Program.cs

Comperio consultant, Marcus Johansson, co-authors ‘Working with Microsoft FAST Search Server 2010 for SharePoint’.

As most search techies will testify, the life of a consultant working on complex enterprise search projects with large customers, can be quite a demanding one. To commit to investing personal time, outside of all this project work is, to put pen to paper and document a series of learnings and tips in a structured and user-friendly format, is no mean feat. So on this note, we salute Comperio consultant Marcus Johansson and congratulate him on the release of his first book earlier this month - Working with Microsoft FAST Search Server 2010 for SharePoint.

Marcus co-authored the book with Mikael Svenson and Robert Piddocke, who are also highly regarded contributors to the search community. Here, in Marcus’ own words, is some background on the book:

The book is split into two parts: the first one explains what you’ll need to know to deploy and administrate a solution. The second one is targeting developers who want to build their own search solutions on top of FAST Search for SharePoint. Although we expect most readers to have a SharePoint background, people who’s worked with FAST technology in previous incarnations will hopefully feel at home as well.

Recognizing  FAST Search for SharePoint’s popularity, and seeing that the product is many people’s first exposure of Enterprise Search technology, the first two chapters introduce the reader to important concepts and terminology. We felt it was critical to get this backdrop right as there are vastly different data and user experience opportunities (and challenges) in Search technology, than in e.g. a database-driven solution.

The next few chapters  target the IT Pro audience and deal with such things as architecture, scale-out, deployment and security. Scaling, in particular, is covered in depth seeing how FAST Search for SharePoint is often used for top-tier solutions both in  query load and content volume. Additionally, day-to-day operations and the various methods of maintenance is covered in detail. We show you how to interact with and follow-up the system through the SharePoint GUI, but also how to work with the solutions through PowerShell and, for those adventurous people who wants to truly integrate the platform in their application environments, how you can integrate against FAST Search for SharePoint’s native APIs even for administrational and operational tasks.

The second part of the book is a good fit for application developers who want to create their own search solutions on top of FAST Search for SharePoint, whether that means extending a default FAST Search Center or completely build their own search-driven application truly taking advantage of the powerful platform. Doing so, developers will realize an Enterprise Search platform, such as FAST Search for SharePoint, relies not only on a competent and flexible index, but also on a framework for advanced content processing and the possibilities to query into the data using a rich query language.

Finally, the book is wrapped up with a tutorial-like section on how to resolve common problems and how to attack frequent development scenarios.

At Comperio, we see how more and more of our clients realize the potential in not only implementing a global search experience in their enterprise, but to deploy targeted search-driven applications across their organizations. One user group might need a mash-up of data aggregated from several internal content sources, and perhaps another group needs to trigger a certain function that he or she perceives as a single operation, but in reality hits 2-3 different source systems in the backend. A search-driven interface is the very sweet spot for building such applications, and  I’m hopeful this book will give you the tools of the trade to implement them in reality.

Happy reading!

In recognition of Marcus’ achievement, the team here in Comperio have given him temporary clearance to use lines like ‘As I say in my book, Working with Microsoft FAST Search Server 2010 for SharePoint, …’ in internal meetings and possibly even at customer presentations.

The book is now available for purchase on Amazon - or you can win one of three free copies being offered by Comperio, by being one of the first to tweet this post mentioning @comperiosearch.

Dr. Olstad speaks on how Norway is a hotbed for search technology development and this search ecosystem has it’s source at NTNU in Trondheim, which have been a feeder university to search giants like Yahoo, Google and Microsoft.

With the tidal growth of data (particularly unstructured) in the past decade, it is no surprise that enterprise search has seen impressive increases in levels of demand. Hence, says Dr. Olstad, the competition for bright young knowledge management or search graduates has intensified. He also points out that several search start-ups have risen out of Norway, benefitting from the knowledge and experience in the local search ecosystem.

One example of these Norwegian search start-ups is Comperio, which has matured to being 50 employees strong with offices in Oslo, Stockholm, London and Boston. Comperio is a Systems Integrator (SI) focused on implementing FAST Search projects for enterprise customers. These search solutions, more recently developed on the FAST Search for SharePoint platform, offer an alternative to traditional integration between IT systems. ‘Today, a lot of IT budgets devoted to integrating old and new solutions. It costs time and money. We rather use search technology’, says Comperio founder and CEO Jørn Ellefsen.

Comperio has developed search solutions in Norway for such recognisable names as, Sintef, Posten, DSS, DNV and Innovation Norway. Outside of Norway, Comperio has delivered large projects for the likes of UBS and Shell. ‘The search-based technology we are developing is largely generic, and it allows us to reuse the solutions from previous projects with new customers. This means that it is both cheaper and better for everyone. This market is starting to wake up in earnest and will grow quickly when you discover how effective the technology is. I think the market for our services will increase tenfold over the next three years’, added Ellefsen.

Comperio are currently hiring for their Oslo and London offices. See the open positions @ http://www.comperiosearch.com/about-comperio/work-for-us/.

Original article from http://www.tu.no/it/article292738.ece.

Microsoft offers three different flavors of search for SharePoint 2010: Foundation, Standard and Enterprise. For each level upwards you get more feature and better search capabilities. One of the visual features included with FAST for SharePoint is Thumbnails and Previews for the search results, as listed on the comparison table below (Compare SharePoint Editions – Search).

Wouldn’t it be nice to have this feature on all versions of SharePoint? I will tell you how, as I demonstrated in a proof of concept talk at Arctic SharePoint Challenge 2011.

The Thumbnail and Preview feature is depending on Office Web Apps, which is an additional module you can install on all versions of SharePoint 2010 in order to get web based editing capabilities of Office documents (Word, Excel, PowerPoint and OneNote). Office Web Apps itself is free, but requires Volume Licensing for Microsoft Office 2010.

The API’s used for generating the thumbnails and the PowerPoint previews are the Office Mobile PowerPoint Web Handler Protocol and the Office Mobile Word Web Handler Protocol, both included with Office Web Apps. These API’s include functions to retrieve generated images of the documents as well as functions to retrieve the textual content within the documents. The API works for doc, docx, ppt and pptx files (Office 97-2010 formats).

The Word API is limited to generating a thumbnail for the first page in your document, while the PowerPoint API can generate an image of any slide as seen below.

If you have installed the Enterprise version of SharePoint you have the possibility to create a new Search Center based on the “FAST Search Center” template, even though FAST is not installed.

The FAST Search Center template features XSLT in the Core Results Web Part which renders javascript calls to functions in a file called search.js located in the _layouts folder. Search.js further implements the communication with the mobile API’s to generate the images.

There are quite a few lines of javascript to implement this feature, but if you are a bit experienced with javascript and ajax programming you would be able to program this feature yourself. This will allow you to get thumbnails and PowerPoint previews with any version of SharePoint, from the free Foundation version to the Enterprise version.

If you are less handy with javascript you can copy out the XSLT used in the Core Results WebPart in the FAST Search Center site template, modify it some, and use it with a search site based on the “Enterprise Search Center” template instead. This way you will get additional visual improvements to the standard search page without having to install FAST.

The quick way is to replace some of the xsl:params set by the web part with pre-set variables instead. If you have FAST installed, then the parameters will be set via the Core Results Webpart. As we are using the oob search we have to set these parameters as variables instead.

<xsl:variable name=“OpenPreviewLink“ select=“‘Preview’“ />

<xsl:variable name=“ClosePreviewLink“ select=“‘Close Preview’“ />

<xsl:variable name=“ThumbnailTooltipLoadingFailed“ select=“‘Loading failed!’“ />

<xsl:variable name=“ThumbnailTooltipLoading“ select=“‘Click to toggle preview’“ />

<xsl:variable name=“AAMZone“ select=“0“ />

<xsl:variable name=“PreviewWindowSize“ select=“160“ />

<xsl:variable name=“EnableDocumentPreviewPowerPoint“ select=“‘true’“ />

<xsl:variable name=“EnableDocumentPreviewWord“ select=“‘true’“ />

<xsl:variable name=“ConcurrentDocumentPreview“ select=“10“ />

<xsl:variable name=“TotalDocumentPreview“ select=“10“ />

I also had to modify the xslt two more places. Around line 665 and line 725 locate the FST_CheckForPreview call and change the function parameter

‘<xsl:value-of select=“sitename“/>’

with

‘http://rootsite’

where rootsite is the name of your SharePoint server url.

This was to ensure the calls to the Mobile API went to the top level site, as they failed when called on sub-site url’s.

That being said, there are several other good reasons for deploying FAST for SharePoint in your organization besides these visual enhancements. Capabilities to enrich and modify your content prior to indexing, thus tailoring search experience towards the real needs of your users, as well as the powerful query capabilities offered via FAST Query Language are just a couple of arguments for choosing FAST.

But then again, it never hurts to have moderate amounts of eye-candy on your search page, as it will increase the perceived quality of the results.

flickr.com/theredproject

Technology has no inherent business value. Technology doesn’t provide improved findabiliy, decision-making or knowledge management by itself. Technology just provides features that help users perform tasks, thereby reaching their goals. Only when user goals and business goals align, can you hope to create business value. Without people, there is no purpose – and without purpose, there is no value.

Enterprise search technology is no different from the rest in this respect. In fact, I know only 3 good reasons for investing in search technology.

Spending time and money on an enterprise search platform like FAST Search For SharePoint (FS4SP) may be a good idea if you believe the success of your business depends on:

1. Consolidation of information from different data sources and formats.
2. Content refinement and entity extraction.
3. Linguistic processing and advanced querying capabilities.

There is of course more to search than this, but these features are in my opinion the hallmarks of FS4SP and the old FAST ESP – what enterprise search does better, faster and with higher tractability than any other available technology. Search is by no means the only way to access information, and similar technologies (like data warehouse, ETL and topic maps) will let you do similar things, each having their own particular strengths and weaknesses.

The true value of enterprise search technology is not bound to fair promises of improved efficiency and profitability, but to the marvels it can do in the hands of brilliant engineers – without requiring any prior knowledge of the context of use. Business value is only attainable when technological capabilities form a higher unity with user needs and business goals through cross-diciplinary user-centered design.

The success of your business may depend on improved findability, decision-making and knowledge management, and that is the true value of designing the user experience of search.

Note: this post is cross-posted at Things On Top.

Instead of having a deployment script creating your properties, you could have an event receiver which creates your mappings once a site or list is provisioned. Or when some other action is going on.

So how do you go about doing this?

Well, first off Crawled and Managed Properties for FAST are a part of the index schema and is accessible via the following two namespaces:

Microsoft.SharePoint.Search.Extended.Administration Microsoft.SharePoint.Search.Extended.Administration.Schema

which are the namespaces used for working with the FAST components.

Below is a quick example showing how this can be done in code, with comments along the way to guide the process. It’s all pretty straight forward.

const string propertyName = “productname”;

// Get a reference to the Index Schema via the SchemaContext class

SchemaContext schemaContext = new SchemaContext();

Schema schema = schemaContext.Schema;

// Assign a guid to the custom Crawled Propery Category

Guid myGuid = new Guid(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1);

Category catGroup;

// Create if it does not exist

if (!schema.AllCategories.Contains(myGuid))

{

catGroup = schema.AllCategories.Create(“My Crawled Properties”, myGuid);

}

else

{

catGroup = schema.AllCategories[myGuid];

}

// Create a new crawled property if it doesn’t exist

CrawledProperty cp = catGroup.GetAllCrawledProperties().Where(

c => c.Name == propertyName && c.VariantType == 31 && c.Propset == myGuid).FirstOrDefault();

if (cp == null) cp = catGroup.CreateCrawledProperty(propertyName, myGuid, 31);

// Create a new managed property if it doesn’t exist

ManagedProperty mp;

if (!schema.AllManagedProperties.Contains(propertyName))

{

mp = schema.AllManagedProperties.Create(propertyName, “”, ManagedType.Text);

}

else

{

mp = schema.AllManagedProperties[propertyName];

}

// Create a new crawled to managed property mapping if it doesn’t exit

CrawledPropertyMapping mapping = mp.GetCrawledPropertyMappings();

if (!mapping.Contains(cp))

{

mapping.Add(cp);

mp.SetCrawledPropertyMappings(mapping);

}

// Get a reference to the default full text index (first one)

FullTextIndex fti = schema.AllFullTextIndecies.First();

// Get the full text index mapping from the Managed Property

FullTextIndexMappingCollection fullTextIndexMappingCollection = mp.GetFullTextIndexMappings();

FullTextIndexMapping indexMapping =

fullTextIndexMappingCollection.Where(ftpMap => ftpMap.FullTextIndex.Name == fti.Name).FirstOrDefault();

// Set a mapping to importance level 6, if it’s not already set to something else

if (indexMapping == null || !fullTextIndexMappingCollection.Contains(indexMapping))

{

fullTextIndexMappingCollection.Create(fti, 6);

}

With FS4SP,  these query suggestions are automatically created over a period of time based on how often the users search for a specific query within a certain timeframe. Sometimes, it can however make sense to modify the query suggestions manually – and to help us do that, we can always lean on PowerShell.

The script I have implemented, has the following parameters:

.\QuerySuggestions.ps1 -file [fileName] -action [add|delete] -queryssa [querySSAName]

where file is the file with the query suggestion phrases (one per line), action decides if the query suggestions should be added or deleted, queryssa is the name of the Query SSA. And now for the actual code.

#Get input
param
(
    [string]$file,
    [string]$action,
    [string]$queryssa
)

function LoadQueryCompletion()
{
    # Read input file with stuff that you should add to the Query suggestion list
    if (test-path $file)
    {
        log VERBOSE "Reading file: $file"
	$input = Get-Content $file
    }
    else
    {
        log ERROR "File noe found. Exiting script."
        exit 1
    }
    # Get SSA search application
    $searchapp = Get-SPEnterpriseSearchServiceApplication -Identity $queryssa

    # Populating a dictionary with the query suggestions that have already been added before
    $query_suggestions = Get-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication $searchapp -Language En-Us -Type QuerySuggestionAlwaysSuggest
    $already_dict = @{}
    foreach ($term in $query_suggestions)
    {
        if (!$term){ continue }
        $already_dict.Add($term.Phrase,$term.Phrase)
    }

    if ($action -eq "add")
	{
	    foreach ($entry in $input) {
	        if ($already_dict.ContainsKey($entry))
	        {
	            log WARNING "Key already added: $entry"
	        }
	        else
	        {

			try {
				log VERBOSE "Adding entry ($entry) with language En-US"
			    	$nooutput = new-spenterprisesearchlanguageresourcephrase -SearchApplication $searchapp -Language En-Us -Type QuerySuggestionAlwaysSuggest -Name $entry
				$already_dict.Add($entry,$entry)
			}catch {
				log ERROR "Could not add entry ($entry). Exiting script."
				exit 1
			}
	        }
	    }
	}
	elseif ($action -eq "delete")
	{
		foreach ($entry in $input) {
			try {
				if ($already_dict.ContainsKey($entry)) {
					log VERBOSE "Removing entry ($entry) with language En-US"
			    	        $nooutput = remove-spenterprisesearchlanguageresourcephrase -SearchApplication $searchapp -Language En-Us -Type QuerySuggestionAlwaysSuggest -Identity $entry -Confirm:$false
					$already_dict.Remove($entry)
				}else{
					log WARNING "Entry ($entry) has not been added as a query suggestion"
				}
			}catch {
				log ERROR "Could not remove entry ($entry)."
			}
	    }
	}
	else
	{
		log ERROR "-action parameter contains an invalid value. Exiting script."
	}

    #Run timerjob manually to upload entries from DB
    Start-SPTimerJob -Identity "prepare query suggestions"

    log VERBOSE "Finished loading query completion phrases."
}

#Main
$spInstalled = Get-PSSnapin | Select-String Sharepoint
if (!$spInstalled)
{
    Add-PSSnapin Microsoft.Sharepoint.PowerShell
}
LoadQueryCompletion

If you want to include the logging output , add this to the top of the PowerShell script.

# Log level to color mapping
$lc = @{
	VERBOSE="green";
	WARNING="yellow";
	ERROR="red";
	MESSAGE="blue";
}
function log([string] $level, [string] $message) {
	$date = get-date -uformat "%G-%m-%d %T"
	write-host "[$date] " -nonewline
	write-host -foregroundcolor $lc[$level] $level.padright(7) -nonewline
	write-host " $message"
}

If you’re really picky (like me), you should also consider having some kind of validation on the input parameters. For instance verifying that the file exists, that the action parameter contains a valid value, and that the Query SSA exists. You can check the last-mentioned by adding the following lines of code to the script.

$ssaExists = Get-SPEnterpriseSearchServiceApplication | Select-String $queryssa
if ($ssaExists -eq $null)
{
	log ERROR "-queryssa is invalid. Search Service Application does not exist. Exiting script."
	$queryssas = Get-SPEnterpriseSearchServiceApplication | Where-Object {$_.DefaultSearchProvider -eq "FASTSearch"}
	foreach ($queryssa in $queryssas)
        {
		if ($queryssa)
    	    	{
    			$name = $queryssa.Name
    			log VERBOSE "Found the following Query SSA: $name"
    	    	}
        }
}

After doing that, you should be ready to run the script on the SharePoint Admin server.

Afterwards, go to your FAST Search Center, and start typing one of the terms you have added. The result can look something like in the screenshot below.