Comperio gjør det enkelt å finne riktig dokument selv om du ikke er godt kjent med Public 360. Ved å gjøre innholdet i Public 360 tilgjengelig i en søkeindeks kan man få søketreff på tvers av kilder. SharePoint 2013 tilbyr en slik søkeindeks. Denne vil kunne gi presise treff fordi spørringen henter treff basert på data både fra dokumentet og metadata til dokumentet. På den måten kan man utforske hva som finnes av innhold og historikk knyttet til et tema eller en sak.
For å få tilgang på dokumentene i Public 360 må man koble kildesystemet sammen med søkemotoren med en konnektor.

Comperio laget i første omgang en konnektor for å kunne indeksere innhold fra Public 360 basert på Fast Search Server for SharePoint 2010. Siden har vi laget en ny versjon for Microsoft SharePoint 2013.
Når vi laget konnektoren så bruke vi Microsofts rammeverk for å lage tilpassede BCS (Business Connectivity Services) konnektorer til SharePoint 2013.

Så, hvordan fungerer konnektoren for indeksering av innhold?

FileFinder web servicen er endepunktet i Public 360, og det brukes til å hente Public 360 dokumenter. Fra denne web servicen får vi all informasjon om hvert dokument, som egenskaper, metadata og rettighetsinformasjon. Videre blir hvert dokument og dets egenskaper plukket opp av SharePoint 2013, hvor dokumentet crawles, prosesseres og lagres i søkeindeksen.

Og hvordan fungerer konnektoren ved søk utført av en bruker?

Når en bruker gjør et søk i søkeboksen, så legges det til hva slags tilganger brukeren har på spørringen. Disse tilgangene hentes fra Public 360 ved hjelp av en SharePoint security trimmer som settes opp for Public 360. Dette fører til at kun dokumenter som brukeren har rettigheter til blir returnert som en del av søkeresultatet.

Ta kontakt for å høre mer om Public 360 og SharePoint.

Download the presentations:

• Leveraging Search Based Applications on new Intranet
  (David Wright, Lead SharePoint Architect, Coutts)
• Searching Internally Using Enterprise Social Networks
  (Zane Freame, Technical Sales Professional, Microsoft UK)
• Enterprise Search Strategy by Comperio Search
  (Job Maelane, John Thompson, Comperio Search)

Some pictures from the event…

Comperio Search welcomes its guests...

Zane Freame, Technical Sales Professional, Microsoft UK

David Wright, Lead SharePoint Architect, Coutts

Trond Renshuslokken, Comperio CEO

The thesaurus lookup is performed at query time, and the dictionary can be set up with support for various languages. The thesaurus must be deployed as a csv file using powershell, it has columns for key, synonym, and an optional language. The “key” column can be a phrase, and so can the synonym. ie. “Go fishing” can be a synonym for “hunt for fish”. To provide several synonyms for a word, simply add it several times. To make the synonym go both ways, add a second entry with the word and synonym switching place. (See Microsoft for further details )

The thesaurus is simple, but it works. Or, does it really? Testing synonyms fetched from the internet on a SharePoint search index populated with US Court records crawled from theinfo.org.  AKA is a common legal term synonymous with “also known as”. So let’s try that out.

Searching for “aka” yields some hundred results, searching for “also known as” yields a different set of results of roughly the same size. By adding synonyms we would expect the search results to combine the hits for both queries into one, so to speak.
So I create a thesaurus containing:

Key,Synonym,Language
aka,also known as

And upload it with the powershell command:

$searchApp = Get-SPEnterpriseSearchServiceApplication
Import-SPEnterpriseSearchThesaurus -SearchApplication $searchApp -Filename \\spbox\temp\thesaurus.csv

I wait for a few seconds, and search for “aka”.
Now, I would expect to find hits containing “also known as”. But where is it? I have to scroll and page down to the bottom of page 3 before I find it:

Why? Could the ULS logs provide any clues? Turning on verbose logging on the Search Query Processing, and search again. Now, in the ULS logs there are entries with “After thesaurus tree modification” (the indentation is mine, trying to make it look a little clearer, also abbreviated it some).

Microsoft.Office.Server.Search.Query.Pipeline.Executors.LinguisticQueryProcessingExecutor : After thesaurus tree modification:
	'AndNode(FirstChild=StringNode
(FirstChild=WordsNode(FirstChild=TokenNode(FirstChild=null,NextSibling=OnearNode(FirstChild=TokenNode(FirstChild=null,NextSibling=TokenNode
		(FirstChild=null,NextSibling=TokenNode
			(FirstChild=null,NextSibling=null,Length=1,Linguistics=True,Token=<strong>as</strong>,Weight=1),
			Length=1,Linguistics=True,Token=<strong>known</strong>,Weight=1),
			Length=1,Linguistics=True,Token=<strong>also</strong>,Weight=1),NextSibling=null
,ExtraTermsAllowed=0,<strong>Weight=0.2</strong>),
Length=1,Linguistics=True,Token=<strong>aka</strong>,Weight=1),NextSibling=null),

The synonyms are given a weight of 0.2. The original term has a weight of 1, theoretically that means the synonym has 20% percent weight of the original term. Perhaps we could make the synonyms show up by increasing the weight.  So, how can we do that? Apparently, there is no way.  Not unless we open up the magic box of the Ceres shell.

We begin by connecting to the Interactionengine:

Add-PsSnapin Microsoft.SharePoint.Powershell
& "C:\Program Files\Microsoft Office Servers\15.0\Search\Scripts\ceresshell.ps1"
Connect-System -Uri  (Get-SPEnterpriseSearchServiceApplication).SystemManagerLocations[0] -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity
Connect-Engine -NodeTypes InterActionEngine

Now, let’s try to rip out the configurations of the SharePointSearchProvider flow:

$flowname = Microsoft.SharePointSearchProviderFlow
Get-Flow $flowname  > $flowname.txt

Reading the flow configurations for the SharePointSearchProviderFlow, we find an option named synonymWeight.

<Operator name="Linguistics" type="LinguisticQueryProcessing">
  lt;Property name="querySpellingCorrectionTokenLimit" value="10" />
...
<Property name="stemWeight" value="0.2" />

&lt;Property name=<strong>"synonymWeight" value="0.2" </strong>/&gt;

&lt;/Properties&gt;

&lt;/Operator&gt;

Strangely, it has the weight 0.2. Ring any bells, anyone? It is the same weight we saw in the ULS. Now, let us try to see what happens if we increase the weight here.
So, we set the synonymWeight to 1, and upload the file.

Remove-Flow $flowname
Get-Content $flowname.txt | Out-String | Add-Flow $flowname
Stop-Flow –FlowName $flowname –ForceAll

Now, when searching for “aka”, we get hits containing “also know as” on the first search results page.
Voila!

When Microsoft bought FAST, they were in the process of developing improvements to their search solution, code named Mars.  Among the improvements were graphical interfaces for the flow engines for Query (IMS) and Content Processing (CTS). These were packaged and sold As FSIS, Fast Search for Internet Sites. SharePoint 2013 has integrated the pipeline workflow from CTS and IMS, and is using it internally. There is, however, no (apparent) option for configuring the flows by the end user. Of course, there is no nice graphical wizards for drawing up the pipeline steps as in FSIS.

Buried deep down in the folders of SharePoint, there is a file called ceresshell.ps1.  Ceres is a dwarf planet circulating between Mars and Jupiter. Incidentally, we can use the Ceres shell to gain access to the configurations of the internal SharePoint flows. Once Mars was reached, the team behind SharePoint search kept on pushing into space, until they reached solid ground on the next planet, Ceres.

So what is the Ceres shell, and what can it do? Looking at the contents of the file, it is a powershell script that loads some snap ins.

Add-PSSnapin hostcontrollerpssnapin
Add-pssnapin junopssnapin
Add-pssnapin searchcorepssnapin
Add-pssnapin enginepssnapin
Add-pssnapin analysisenginepssnapin

So, we start up a SharePoint powershell session as Farm administrator, and load in the Ceres shell.

PS> & "C:\Program Files\Microsoft Office Servers\15.0\Search\Scripts\ceresshell.ps1"

Now the Ceres cmdlets have been loaded up and are ready to use.

Before we can have any fun with this, we need to connect to the “system”.

Connect-System  -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity

Once connected to the system, we need to connect to the “engine”

Connect-Engine

Now we are ready. To get a list of all flows available for inspection, type

Get-Flow

To show the flow configuration for a single flow:

Get-Flow Microsoft.CrawlerIndexingSubFlow

Here is a part of that file

<OperatorGraph name="Microsoft.CrawlerIndexingSubFlow" xmlns="http://schemas.microsoft.com/ceres/studio/2009/10/flow">

<Parameters />

<Operators>

<Operator name="MarsWriter" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.MarsWriter">

<Properties>

<Property name="callbackType" value=""Completed"" />

<Property name="callbackWarningField" value=""ParsingErrors"" />

<Property name="commitInterval" value="1" />

<Property name="crawledPropertyBuckets" value="["content"]" />

<Property name="defaultMaxIndexSize" value="524288" />

<Property name="defaultMaxResultSize" value="16384" />

<Property name="idField" value=""externalId"" />

<Property name="managedPropertiesListName" value=""ManagedProperties"" />

<Property name="managedPropertyBuckets" value="["ManagedPropertiesBucket"]" />

<Property name="marsCallbackInfoProperty" value=""marsLinkDBSynchronization"" />

<Property name="provideCallbacks" value="True" />

<Property name="siteCollectionIdField" value=""sitecollectionid"" />

<Property name="tenantIdField" value=""tenantId"" />

<Property name="truncatedFlagField" value=""IsPartiallyProcessed"" />

</Properties>

</Operator>

<Operator name="SubFlowInput" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.SubFlow.SubFlowInput">

<Targets>

<Target>

<operatorMoniker name="/Microsoft.CrawlerIndexingSubFlow/MarsWriter" />

Interesting, isn’t it? There seems to be some configurable values for stuff like “defaultMaxResultSize”.

Now you see how you can open the Ceres shell, and look at the flow configurations.

Next post will show you how you can modify values in the existing flows.

For quite some time, we didn’t know for sure which functionality was available for the different deployments and licenses. In January or so, an article surfaced on TechNet describing which features are available for the various SharePoint Online plans. But the on-premise perspective was still missing. Until a few weeks ago, when the very same Technet page was updated to also include information for on-premise solutions. Confusingly, the page is still called SharePoint Online Service Description.

In any case, what should you look out for from a Search perspective?

How about the cloud?

If you have small requirements on search, e.g. you only need to index whatever is stored in SharePoint and you’re happy with how most things are out of the box, then by all means investigate the various cloud plans. However, as they all lack the connector framework, you can’t index anything living outside of SharePoint. In addition, all cloud plans are weak on the content processing side. None of them can do custom entity extraction, i.e. using dictionaries to pull metadata out of unstructured text, and adding your own logic to content processing is off the table as well.

In other words, you’ll have trouble getting your content into the index, and there’s no possibility to process that content in order to enable a richer search experience. Still, if your search requirements are low, and all your important stuff already sits in SharePoint, this may not be a problem for you.

Additionally – and this is a huge bummer for many smaller organizations who’s planning to go with SharePoint Online – none of the cloud plans include the possibility to use the Content Search Web Part. This one you’ll really miss if you’re planning to build search-driven applications around your portal.

In other words, if you’re looking for more advanced search technology than the very basics, an on-premise solution is the way to go. No surprises here.

 On-premise solutions

The only combination of deployment type and license model that gives you a 100% complete feature set is the on-premise Enterprise license. And if you’re looking to implement Enterprise Search in your organization, the choice is very simple. You need the Enterprise CAL.

Foundation is completely off the table for more advanced solutions, due to the missing connector framework. The Standard license is somewhat better, but just like the cloud offerings it’s very weak on the content processing side and there’s no tunable relevancy either. This may be alright for small shops with small requirements on search, but also keep in mind that you’re missing out on the Content Search Web Part and the more advanced feature sets of the Query Rule engine. If those two had been included, the Standard license would have been a lot more pleasing for many small organizations.

Here’s the link to the spreadsheet again http://sdrv.ms/13k5u9f, and it’s also available for your viewing pleasure right here on the blog. Feel free to hook me up on Twitter or via the comment field below if you want to discuss the various deployments and licenses for SharePoint search!

While researching, talking about, and listening on what people say about search in SharePoint 2013, I’ve noticed two major themes.

The idea that “FAST is now fully integrated”

…and while this is certainly true, it’s also not even close to the full story. Search in SharePoint 2013 is a completely new technology, unparalleled in the Microsoft world. The actual index is fundamentally different, boosting better performance by using in-memory technology and even being atomic!

The excitement about the User Experience

A few years back, FAST technology was sold and marketed by technical things like linear scaling, content processing power, and low search times. That’s light years away from what excites people about search in SharePoint 2013. Quite the contrary, the web is full of people buzzing about the actual search experience. How the default search center looks cleaner. How well document previews are integrated. That the addition of automatic search analytics makes relevance calculation… wait for it… self-learning! And so on.

I can’t help but think — actually I’m hoping — that this marks a shift in how people think about search: that it’s not defined by the internal technicalities, but the user experiences we can build on top of it.

And while UX is arguably one of the most important factors of a search solution, I’ll always have a weak spot for what’s actually going on under the hood. So if you’re like me, check out the presentation below!

They say that everyone dies twice. First when you actually die, and then finally when your name is mentioned for the last time.

Back before online search revolutionized ranking algorithms, people sorted their search results by date, alphabetically or by any other static metadata. The concept of a relevance-based result set, where items are evaluated against your search terms at query-time, was rare. Now, we’ve gotten used to sorting things by relevance. A concept as easy to understand as it is hard to implement.

Rank models, i.e. the math behind calculating a metric for relevance, sparked an interest in me the moment I first heard of them. And ever since, every time someone wanted to sort a result set by the “old-fashioned way”, e.g. by date, I always cringed. But I swallowed my search geek-pride and moved on.

How search engines calculate relevance and rank is way out of the scope of this post, but because of my long-running loathing of non-relevance based sorting, I want to highlight a small but glorious thing in the new SharePoint 2013 search GUI, that may have bypassed even early adopters in the findability scene. My heart almost skipped a beat when I first noticed it at one of last week’s sessions at the Microsoft SharePoint conference in Las Vegas.

Alternate sorting in SharePoint 2010

In the search center in SharePoint 2010, both the one for Enterprise Search and FAST, the default was to sort by relevance. Great, but there was a drop down that allowed you to change over to sorting by date or, in the case of FAST, any other managed property which had been enabled for sorting. Even so, I’ve experienced time after time that the drop down was not used very often. Why? Probably because dates and many other metadata is several times better navigated using drill down facets, and this is something that comes natural to many end-users over time.

Of course there are exceptions and occasions when an alternate static sorting over a single metadata property makes sense, but for a good portion of search centers, I’m very convinced that the drop down adds little more than just some clutter to the GUI.

In the SharePoint 2013 search center, you can still sort by any property, of course. But the drop down in the default GUI is not there anymore. You’re encouraged to sort by relevance, and that’s it. There are two things to this:

• Firstly, the functionality to sort on a single static metadata property is still there behind the scenes, and very easy to expose to the user if it’s necessary in your solution.
• Secondly, the engine driving the relevance calculation is much improved and it’s now easier than ever (in the SharePoint world) to tune your relevance model; allowing it to span over several metadata and dynamically evaluate search words against the index corpus.

But not having that “sort by” drop down in the default GUI is a huge step.

The search box in SharePoint 2013, no static sorting to be seen

This is not only about getting a cleaner GUI, but more importantly also about getting to a point where users have started to assume that sorting by relevance just works. Silently, but surely, search engines are maturing, and we’ve – equally silently – started to assume they work better and better.

My inner search geek has started to wonder that perhaps static sorting first died when the rules of online search changed with PageRank, and people got accustomed to sorting by relevance. And with having that alternate sorting drop down removed from the default SharePoint 2013 GUI, perhaps this is the time when static sorting fades into obscurity – and dies the second time.

Dear static sorting, R.I.P.