FAST ESP was, and still is, an incredibly flexible and scalable enterprise search platform and is used by customers with sophisticated search needs and a significant requirement for updating the frequency, relevance adjustments, scalability and stability. The platform has very rich semantic and linguistic characteristics, and a wide range of connectors to the underlying systems, and connection points which can search and present relevant content adapted to the users context and device. These features generate revenue and/ or reduce costs for more than 2,600 customers worldwide that use or have used the FAST ESP platform.

FAST Search and Transfer was acquired by Microsoft in 2007. Following the acquisition, it became clear that FAST ESP will not be continuing as an independent cross-platform search engine. FAST ESP will however, exist in mainstream support until 2013 (2015 for customers with a License Grant), and extended support on all OS platforms until 2018 (2020 for License Grant customers).

During this period it is possible and safe to use and even develop new services on the FAST ESP search engine. Our team of Application Management Services (AMS) consultants have extensive experience in the FAST and FAST ESP platform – many of them formerly worked in consultancy, support or services in Fast Search and Transfer / Microsoft.

While Comperio’s AMS services can customise and sustain stable operation of customers on the FAST ESP platform, we also actively advise on search strategy, design, solution architecture and new search technology, as needs and technology evolve.

Our long-term plan with Comperio Front has been making customers less dependent on the underlying search engine, and the situation with FAST ESP shows that this has been a good strategy. For our customers with Comperio Front, we are now able to provide effective transitions to new technology platforms, as well as new corporate business models, and the search engine’s position as an information hub are preserved in Comperio Front.

When it comes to transitions to new technologies post the FAST ESP era, more choices and opportunities are opening up. For many traditional FAST ESP customers in e-commerce, catalog, classified ads and media, FAST for SharePoint is not necessarily a natural choice straight away, especially if they use anything other than Microsoft technology. Comperio has experience and dialogue with several customers on the transition to open source platforms and cloud services such as Solr, Elasticsearch, Amazon Cloud Search and also Google Custom Search – when called for.

That said, Microsoft plans to further use the advanced capabilities of FAST in their products, both in Office/SharePoint/Exchange/Office365, as well as in its search engine on the internet – Bing. Microsoft’s release of FAST Search for SharePoint 2010 is an example of this and the next generation of Microsoft products will show this more clearly. For many FAST ESP customers in the business market, it is natural to follow this path, as long as it is possible to build the solutions they want. And this is also where Comperio Front comes in, as a building block to make it possible to customize the search based solutions in SharePoint in a flexible manner.

Microsoft’s development team for search, or FAST R&D department, is based in Torggata, Oslo, and works closely with both the Bing and the Microsoft Office Division development teams.
With Comperio’s close affiliation from a geographical, historical and technical perspective, we continue to follow the FAST / Microsoft team in their development (and vice-versa) with technical and strategic dialogue constantly held.

But above anything else, we want to help our clients to great search experiences!

You can learn more about FAST ESP at http://www.comperiosearch.com/products/fast-esp/ and go to www.comperiosearch.com to see other products and services offered by Comperio.

FAST ESP var og er fremdeles en utrolig fleksibel og skalerbar enterprise search platform og blir brukt av kunder med avanserte søkebehov og ekstreme behov for oppdateringshyppighet, relevansjusteringer, skalerbarhet og stabilitet. Plattformen har meget rike semantiske og lingvistiske egenskaper, og et stort utvalg av connectorer til underliggende systemer, samt tilkoblingspunkter for å kunne søke og presentere relevant innhold tilpasset brukerens kontekst og device. Disse egenskapene er inntektskilde og kostnadsreduserende faktor for mer enn 2600 kunder verden over som benytter eller har benyttet FAST ESP plattformen.

FAST Search and Transfer ble kjøpt av Microsoft I 2007. Etter oppkjøpet er det etter hvert blitt klart at FAST ESP ikke blir videreført som selvstendig crossplattform søkemotor. FAST ESP vil imidlertid eksistere i mainstream support fram til 2013 (2015 for kunder med en Licence Grant), og extended support på alle OS plattformer fram til 2018 (2020 for License Grant kunder).

I denne perioden er det fullt mulig og trygt å benytte og til og med utvikle nye tjenester på FAST ESP søkemotor. Vårt team av Application Management Services  (AMS) konsulenter har lang fartstid fra FAST og med FAST ESP plattformen., og har blant annet jobbet med konsulenttjenster, AMS og support internt hos Fast Search and Transfer/Microsoft. Samtidig som vi med våre AMS tjenester kan oppretteholde stabil drift av din FAST ESP plattform, vil vi være aktive rådgivere og løsningsarktekter over på en ny søketeknologi ettersom behovene melder seg.

Vår langsiktige plan med Comperio Front har vært å gjøre våre kunder mindre avhengige av underliggende søkemotor, og situasjonen med FAST ESP viser at dette har vært en god anbefaling. For våre kunder med Comperio Front er vi nå i stand til å tilby effektive  overganger til nye teknologiplattformer, samtidig som bedriftens forretningslogikk og søkemotorens posisjon som  informasjonsnav er ivaretatt i Comperio Front.

Når det gjelder overgangene til nye teknologier i en etter FAST ESP- era er det flere valg og muligheter som åpner seg. For mange tradisjonelle FAST ESP kunder innen ehandel, katalog, rubrikk og media er ikke FAST for Sharepoint nødvendigvis er et naturlig valg for øyeblikket, spesielt ikke dersom man ellers benytter noe annet enn Microsoft teknologi. Comperio har erfaring og dialog med flere kunder om overgang til opensource plattformer og Cloudtjenester som Solr, Elasticsearch og Amazon Cloudsearch, og også Google Custom Search, alt etter behov.

Når dette er sagt er Microsoft’s plan framover å bruke de avanserte egenskapene til FAST i sine produkter videre, både for kontorstøtte i Office/Sharepoint/Exchange/Office365, så vel som i sin søkemotor på internett – Bing. Microsofts release av FAST Search for Sharepoint 2010 er et eksempel på dette, og neste generasjons Microsoft produkter vil vise dette enda tydeligere. For mange FAST ESP kunder i bedriftsmarkedet er det naturlig å følge dette sporet, så lenge det er mulig å bygge de løsningene man ønsker. Og også her er kommer Comperio Front inn som en byggestein for å gjøre det mulig å tilpasse søkebaserte løsninger i Sharepoint på en fleksibel måte.

Microsofts utviklingsteam for søk er lokalisert med basis i FAST R&D avdeling i Torggata Oslo, og samarbeider tett med så vel Bing som Microsoft Office sine utviklingsteam. Med vår nære tilknytning, geografisk, historisk og teknisk fortsetter Comperio å følge FAST/Microsoft i deres videre utvikling, og vårt Microsoft team er hele tiden i Front med det nye som leveres fra Microsoft.

Men først av alt ønsker vi å hjelpe våre kunder til gode søkeopplevelser!

Composite field context weight sums
Weights are not relative to its containing element. I.e. the field-weights within the context part of a composite-rank do not sum up to a 100% which most people find intuitive. If you have two fields weighted 100 and 100 and get a hit in both fields, then hit in the composite becomes 200, instead of 100 as one could expect from a “100% hit”.

The single-field-composite warning
A search on non-composite fields do not generate dynamic rank. For this you will need to wrap it in a composite field, which is perfectly ok. However, when you bliss (upload) an index profile, ESP will spew out warnings. These can be ignored. (But remember to add the composite field reference to the rank profile, as usual, or it will have no effect.)

The context/occurence oddity
An often encountered problem is a composite with fields that have values contain the same tokens (words). A search will then get context hits in multiple fields and get rank contributions from each. However, in most cases you would only like to have a rank contribution from the hit in the highest weighted field. I have tried to turn off whatever I could find in config files, but not been able to solve this problem. There are however a few ways around this.

One involves ripping out duplicate words from the lower weighted fields during document processing, but that makes those fields useless for other purposes.

Another solution involves splitting the fields into singular composite fields and rewriting the query to contain a lot of parts searching each field with the same words, and joining them with the ANY operator.

A third solution is join the fields in question in a field-ref-group in the composite field. This will count multiple field hits as a hit the one group and assign a rank contribution according to the field-ref-group’s weight. But you will no longer be able to assign individual weights to each field.

The default oddity
One composite field must be tagged with

default="yes"

If you have no default composite field, ESP can start to act funny, such as swapping int32 and string types in the result(!)

The quality oddity
This refers to the static boost contribution to the result defined by the “quality” element of the rank-profile.

If the quality element is not present, the default weight is 50, and the default quality field is “hwboost”. This is a magic field that is hard coded and not defined in the index profile. However, try to specify

<quality weight="50" field-ref="hwboost"/>

and you will get an error that the field is not defined. This can seemingly safely be explicitly defined in the index profile as

<field name="hwboost" type="uint32" index="yes" sort="yes"/>

The default start value of hwboost is 10000, and this can be added to or subtracted from during document processing.

The quality weight is limited to steps of 50 (0, 50, 100, 150, …). These values are actually transformed to multipliers 0, 1, 2, … So a weight value of “50″ does not mean half or 50%. With a default hwboost, a weight of “50″ transforms to multiplier 1, i.e. 1*10000=10000.

You can specify your own quality field. It must be of type uint32 (not int32!), index and sorting set to “yes”.

After changing the values, run

bliss-core -C index-profile.xml
view-admin -m refresh

and then wait a minute or so for the views to refresh.

For the first time since Microsoft’s acquisition of FAST, the broad audience can now reap the benefits of a high-end platform for enterprise search. This post is a first look at FSIS; a brief overview of the new components and what has kept Microsoft busy throughout the time since the acquisition. The post is simply intended to serve as an introduction to the curious, and makes no attempt to be comprehensive.

Previous incarnations of FAST technology have been equipped with powerful and flexible support for processing documents and indexing them accordingly. Out of a UX perspective, FAST technology has not been equally strong. With FSIS, Microsoft has recognized this imbalance by targeting what we at Comperio refer to as “search logic”. This area, which is both broad and complex, has given birth to several products and frameworks – one of them being Comperio Front. The future will have to decide if this first version of FSIS is strong enough to manage on its own.

The FSIS architecture

Both FSIS, and its sister-product FAST Search Server 2010 for Internal Applications (FSIA), are built around the proven technology in FAST ESP. More specifically, the very core of both these products is FAST ESP 5.3 with service pack 3 or higher.

In FSIS, ESP is fairly well hidden inside of the Microsoft packaging. New modules which encapsulate the existing ESP interfaces provide a high level of abstraction. Most prominent of the new modules are inarguably Content Transformation Services (CTS) and Interaction Management Services (IMS). These handle management and processing of indexing and query flows. There is also FAST Search Designer, a Visual Studio 2008 add-on, which makes it easy to create, debug, and orchestrate these flows.  A fourth module, Search Business Manager targets business-users who need to adjust the overall search experience, and perhaps want to try out new features using the integrated A/B testing functionality.

There are also additional supporting modules, e.g. the IMS UI toolkit which provides a foundation for new web application projects. Another example is the Content Distributor Emulator, which provides backwards-compatibility for legacy ESP connectors. But on a high level, with only the most important modules listed, the structure of FSIS corresponds to the figure below.

Anyone adventurous enough to seek complete and fine-grained control of their search solution will have to go deeper than the outer FSIS layers. ESP is still the go-to place for all advanced customizations, and is still available in its completeness.

Content and interaction workflows

The right part of the figure is typically described as the “content side” or the “source side” of the search solution. This is where CTS is located. Data to be indexed is sent to CTS, where it is manipulated and prepared for indexing according to business-rules and requirements. For example, it might be necessary to enrich certain documents with synonyms or additional keywords. All such logic is graphically mapped up in workflows, using either preconfigured or custom data manipulation “tasks”.

Those of us with experience of development from ESP will feel right at home in the workflow task library in CTS. Many features and functions are direct derivatives of those found in the traditional ESP Document Processing framework, but CTS provides a much-needed face-lift. Most developers will surely appreciate the improved development and debugging possibilities. It should be noted that the traditional Document Processing pipelines are still available; they are in fact still running behind the scenes.

IMS is located on the left side of the figure. Queries are both going in, and coming back out through the workflows in IMS. Just like documents flow through CTS, queries and result sets flow through IMS. The big difference is of course that instead of manipulating content, IMS brings support for tailoring the search experience: e.g. branching up a query into several federated searches, or conditionally enriching or expanding queries. As mentioned at the top of this post, FAST has traditionally been lagging behind in this area. In that regard, IMS is extremely welcome news for developers and business-users alike.

In future blog entries we will dig deeper and explore more of the capabilities in the new FAST-based offerings from Microsoft. We will surely revisit the concept of “search logic”, and how FSIS attempt to solve such problems. Until then, feel free to shoot us any comments or questions!