How to “spy” the data in a custom pipeline extensibility stage with FS4SP

In the old FAST a much used stage during development is the “Spy” stage. What this stage does is dump out a log file of all current attributes and the values assigned to them at that point in the content processing pipeline.

Fortunately for us, this stage still exists in FS4SP, and it might help you when testing and debugging your crawling.

In order to enable the spy stage, first stop the FAST configserver

nctrl stop configserver

Second, open up %FASTSEARCH%\etc\pipelineconfig.xml

Typically you want to add your spy stage before or after the custom extensibility. In the example below I have added it before.


After the edit, save your file, and start the configserver up again.

nctrl start configserver

If you watch the %FASTSEARCH\var\log folder during indexing you will see a file named spy.txt appear which contains all current fields available to you.


The file is overwritten by each processed file and will contain information from the latest document only. If you index using only one document processor it’s still a valuable tool during development to check that you are receiving the data you expect for your custom stage.

Article written by

Computer literate and search enthusiast with an interest for sharepoint, coding, and life in general :)

2 response to: «How to “spy” the data in a custom pipeline extensibility stage with FS4SP»

  1. February 23, 2011 at 08:47 | Permalink

    This is certainly a pipeline stage that FAST engineers have used more than once when creating search solutions. Can you follow the same approach for custom stages?

  2. February 23, 2011 at 09:28 | Permalink

    You could in theory create custom python stages, but there is no documentation on the new interfaces for properties. So you would have to do a lot of trial an error I guess.

Leave a response

XHTML: These tags are allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

301 Moved Permanently

Moved Permanently

The document has moved here.


Comperio AS
Øvre Slottsgate 27
NO-0157 Oslo,
+47 22 33 71 00
View map


Search Provider Sverige AB
Gamla Brogatan 34
SE-11 120 Stockholm
+46 8-21 49 00
View map