Search your site for a phrase you know is written inside a document and you get nothing back. Search the document's title and it comes up fine. Search the text inside it, zero results.
So here's what's going on. When Sitefinity indexes a file for search it doesn't read the file the way you do, it hands the file off to a small piece of code called a text extractor whose only job is to open that file type and hand back the words as plain text. Those words go into the search index. If no extractor exists for a file type, or the extractor blows up, the file still gets indexed but with an empty body. Title searchable, contents invisible.
And nothing tells you. No error in the admin UI, no failed upload, no document that looks broken. Search just quietly stops finding things.
What Sitefinity actually ships
Extractors for exactly five file types: PDF, DOCX, HTML, plain text and RTF. That's the whole list.
So PowerPoint files, Excel files and macro-enabled Word files (.docm) have NEVER had their contents indexed on any Sitefinity site, out of the box. And a chunk of your PDFs are probably failing too, for a completely separate reason, even though PDF is on the supported list.
Both are fixable because Sitefinity lets you register your own extractors through a config file. Four of them below plus the config that turns them on. If you just want the code, skip ahead.
Signs this is your problem
Search the Sitefinity error log for these. Any of them means documents are landing in the index with no text:
MIME type is not supportedrepeated once per document, per indexing pass. That's a file type with no extractor.InvalidStructureTreeException, or a long Telerik stack trace mentioningRichMedia, on a PDF. That's the PDF problem below.- A reindex that runs forever and produces a searchable index where every document only matches on its filename.
I found this the way most people probably do. A user insisted a document existed, search disagreed, and the document was sitting right there in the library.
What actually goes wrong
Two separate problems, and it took me a while to see they were separate.
First one, whole formats have no extractor at all. PowerPoint is the big one. There is no PPTX extractor in Sitefinity, period, so in a library heavy with lecture slides a large fraction of the corpus is indexed title-only. Excel, same story. So is macro-enabled Word (.docm), which is more irritating because it's the same format as DOCX underneath. The stock extractor is registered against the DOCX MIME type, .docm announces a different one, so it just falls through. In the log that's your "MIME type is not supported" warning, once per document, per indexing pass.
The second one is subtler because PDF does have a stock extractor and it still fails. Decompiling it explains why:
// Sitefinity's DefaultPdfTextExtractor, paraphrased
RadFixedDocument document = provider.Import(doc, timeout);
document.DocumentUnhandledException += (s, e) => { e.Handled = true; };
Look at the order. The tolerant exception handler gets attached to the document after Import() has already returned. That handler catches problems during export, but a malformed structure tree or a RichMedia annotation throws during import, before there's a document to attach a handler to. So the handler does nothing.
Symptom is an InvalidStructureTreeException stack trace in the log and a PDF in the index with an empty body. In a decade-old library of scanned handouts and PDFs with embedded video, that happens a LOT.
ITextExtractor
Sitefinity's extraction is properly pluggable, which is the only reason any of this is fixable. You write a class, name it in a config file, and Sitefinity starts calling it for that file type. ITextExtractor is three members:
public interface ITextExtractor
{
string MimeType { get; }
void Initialize(string mimeType, NameValueCollection config);
void GetText(Stream doc, Stream text);
}
GetText reads the file from doc and writes plain text to text. That's the whole contract.
Registration lives in App_Data/Sitefinity/Configuration/DocumentServiceConfig.config, and the factory instantiates your type by name through Activator.CreateInstance:
<?xml version="1.0" encoding="utf-8"?>
<documentServiceConfig>
<extractorSettings>
<add mimeType="application/pdf"
extractorType="Your.Namespace.PdfTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.openxmlformats-officedocument.presentationml.presentation"
extractorType="Your.Namespace.PptxTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.ms-powerpoint.presentation.macroEnabled.12"
extractorType="Your.Namespace.PptxTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.openxmlformats-officedocument.presentationml.slideshow"
extractorType="Your.Namespace.PptxTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
extractorType="Your.Namespace.XlsxTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.ms-excel.sheet.macroEnabled.12"
extractorType="Your.Namespace.XlsxTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.template"
extractorType="Your.Namespace.XlsxTextExtractor, Your.Assembly" />
<add mimeType="application/vnd.ms-word.document.macroEnabled.12"
extractorType="Your.Namespace.WordTextExtractor, Your.Assembly" />
</extractorSettings>
</documentServiceConfig>
Worth knowing before you go hunting for it: this merges, it does not replace. Adding entries here leaves the stock DOCX, HTML, plain and RTF registrations intact. A MIME type you name explicitly gets overridden by yours, everything else stays as shipped. I checked that against a live instance rather than trusting the docs, and it behaves.
You need one registration per MIME type even when a single class handles several, which is why PPTX, PPTM and PPSX all point at the same extractor. Initialize is where each instance learns which MIME type it got created for.
The thing that catches people out: registering an extractor does NOTHING to documents already in the index. Sitefinity extracts text at index time, not at search time, so existing documents keep whatever body text they had (usually none) until you rebuild the index from the search settings screen. Deploy the DLL, add the config, then reindex.
A shared helper for writing the text back out
Every extractor ends the same way, turning a StringBuilder into UTF-8 bytes on the output stream. There's a trap in doing that the obvious way.
internal static class TextExtractorOutput
{
internal static void WriteUtf8(StringBuilder builder, Stream text)
{
// Raw byte write instead of StreamWriter: disposing a writer would close the
// caller's output stream before DocumentService reads it back
var bytes = Encoding.UTF8.GetBytes(builder.ToString());
text.Write(bytes, 0, bytes.Length);
}
}
Wrap the output in using (var writer = new StreamWriter(text)) and you close the stream you were handed. Sitefinity then reads back from a closed stream, you get an empty body, and there's nothing obvious pointing at the cause. Write the bytes directly and leave the stream's lifetime to whoever created it.
Fixing PDFs that index with no text
The whole point here is attaching the tolerant handler to ImportSettings so it's live during import, then skipping the structure tree entirely. Text extraction never needs the structure tree (it's an accessibility and reading-order artifact) so IgnoreMarkedContent costs you nothing and removes the most common failure surface in one line.
using System;
using System.Collections.Specialized;
using System.IO;
using System.Text;
using Telerik.Sitefinity.Services.Documents;
using Telerik.Windows.Documents.Fixed.FormatProviders;
using Telerik.Windows.Documents.Fixed.FormatProviders.Pdf;
using Telerik.Windows.Documents.Fixed.FormatProviders.Text;
using Telerik.Windows.Documents.Fixed.Model;
public class PdfTextExtractor : ITextExtractor
{
public string MimeType { get; private set; }
public void Initialize(string mimeType, NameValueCollection config)
{
this.MimeType = mimeType;
}
public void GetText(Stream doc, Stream text)
{
var provider = new PdfFormatProvider();
provider.ImportSettings.ReadingMode = ReadingMode.OnDemand;
// Text extraction never reads the structure tree, and a malformed one is the
// single most common import failure in legacy PDFs
provider.ImportSettings.IgnoreMarkedContent = true;
// THE fix: subscribed on ImportSettings, so it is live during Import().
// The stock extractor subscribes on the document afterwards, too late for
// anything thrown while parsing.
provider.ImportSettings.DocumentUnhandledException += (sender, e) =>
{
e.Handled = true;
};
var timeout = TimeSpan.FromMinutes(5);
RadFixedDocument document = provider.Import(doc, timeout);
if (document == null)
{
return;
}
document.DocumentUnhandledException += (sender, e) =>
{
e.Handled = true;
};
var exporter = new TextFormatProvider();
var settings = new TextFormatProviderSettings("\r\n", string.Empty);
var extracted = exporter.Export(document, settings, timeout);
TextExtractorOutput.WriteUtf8(new StringBuilder(extracted), text);
}
}
Couple of notes if you're copying this. ReadingMode lives in Telerik.Windows.Documents.Fixed.FormatProviders, NOT the .Pdf namespace you'd expect, which is an easy twenty minutes lost to CS0103. And ReadingMode.OnDemand keeps page content lazy, which matters when the indexer is chewing through hundreds of files and you'd rather not hold every page of every PDF in memory at once.
This does not turn every broken PDF into clean text. What it fixes is the population of structurally damaged but genuinely textual PDFs that the stock extractor throws away wholesale. A PDF that's purely scanned images is a different case, and the next bit covers where that would hook in.
Where OCR would go, if you have a service for it
Some PDFs have no text layer at all. Somebody scanned a paper handout, or printed to PDF from an image, and every page is a picture of words. There's nothing for TextFormatProvider to export, so extraction "succeeds" and returns an empty string. In the index that's indistinguishable from a file that failed.
You can detect it cheaply though. If a PDF has pages but almost no extracted characters, it's an image-only document. That's where an OCR service would slot in.
private void Extract(Stream doc, Stream text)
{
// ... import and export exactly as above ...
var extracted = exporter.Export(document, settings, timeout);
// A PDF with pages but effectively no characters is a scan. Roughly 20 chars
// per page is well under any real document and well over an empty string.
var looksLikeAScan = document.Pages.Count > 0
&& extracted.Trim().Length < document.Pages.Count * 20;
if (looksLikeAScan)
{
// extracted = OcrPages(document);
}
TextExtractorOutput.WriteUtf8(new StringBuilder(extracted), text);
}
// Sketch only. Fill in with whichever OCR service you already pay for:
// Azure AI Document Intelligence, AWS Textract, Google Document AI, or a
// self-hosted Tesseract sidecar. They all take an image and return text.
//
// private string OcrPages(RadFixedDocument document)
// {
// var builder = new StringBuilder();
//
// foreach (var page in document.Pages)
// {
// // Telerik can rasterize a page for you. The exact provider type moved
// // between versions (it is Skia-based in current releases), so check the
// // Document Processing docs for the one your assemblies actually ship.
// byte[] pageImage;
// using (var buffer = new MemoryStream())
// {
// // imageProvider.Export(page, buffer);
// pageImage = buffer.ToArray();
// }
//
// // Your service call. Keep the per-page timeout short: this runs on the
// // indexing thread, and a stalled HTTP call holds up the whole reindex.
// // builder.AppendLine(ocrClient.Recognize(pageImage));
// }
//
// return builder.ToString();
// }
Before you wire that up though, none of these are code problems.
It costs money per page, and a reindex processes every document again. A library with a few thousand scanned pages can turn one "rebuild the index" click into a real invoice. If you do this, cache the OCR result somewhere keyed by the document so a second reindex reads the cache instead of re-billing you.
It's slow. OCR is seconds per page against milliseconds for normal extraction, and it runs inline on the indexing thread. A few hundred scanned documents can stretch a reindex from minutes into hours. Doing the OCR in a separate background job and writing the result into a field the indexer reads is the saner architecture, it's just a bigger build.
And it's fallible in a way normal extraction isn't. OCR on a bad scan produces plausible-looking wrong words, which then land in your search index as real terms. Usually still better than an empty document, but "searchable" and "accurate" have come apart at that point.
I left this as a hook rather than an implementation. The image-only PDFs in the library were a small enough slice that the cost and complexity weren't worth it yet, and the detection line above at least tells you how big that slice actually is if you log it.
Indexing PowerPoint slides, including speaker notes
No Telerik dependency needed here. The OpenXML SDK (DocumentFormat.OpenXml) already ships with Sitefinity so this is free. PPTX, PPTM and PPSX are all ZIP packages of XML, and PresentationDocument.Open reads all three.
The useful trick is that all visible slide text lives in a:t elements no matter how deeply shapes, groups, tables and text boxes are nested. Walking Descendants<Drawing.Text>() gets you everything without modeling the shape tree at all.
using System.Collections.Specialized;
using System.IO;
using System.Text;
using DocumentFormat.OpenXml.Packaging;
using Telerik.Sitefinity.Services.Documents;
public class PptxTextExtractor : ITextExtractor
{
public string MimeType { get; private set; }
public void Initialize(string mimeType, NameValueCollection config)
{
// One registration per mime; PresentationDocument also opens pptm and ppsx
this.MimeType = mimeType;
}
public void GetText(Stream doc, Stream text)
{
var builder = new StringBuilder();
using (var presentation = PresentationDocument.Open(doc, false))
{
var presentationPart = presentation.PresentationPart;
if (presentationPart == null)
{
return;
}
foreach (var slidePart in presentationPart.SlideParts)
{
// a:t elements hold all visible slide text regardless of shape nesting
foreach (var textNode in slidePart.Slide.Descendants<DocumentFormat.OpenXml.Drawing.Text>())
{
builder.AppendLine(textNode.Text);
}
// Speaker notes often carry the searchable substance of a lecture slide
var notes = slidePart.NotesSlidePart;
if (notes != null)
{
foreach (var noteText in notes.NotesSlide.Descendants<DocumentFormat.OpenXml.Drawing.Text>())
{
builder.AppendLine(noteText.Text);
}
}
}
}
TextExtractorOutput.WriteUtf8(builder, text);
}
}
Speaker notes deserve their own mention. A slide will often read "Management" over a diagram while the notes pane holds the actual prose, the drug names and the caveats and the sentence somebody is going to search for six months later. Indexing slides but skipping notes throws away the most searchable text in the file, and it's four extra lines to include it.
Indexing Excel spreadsheets
Spreadsheets keep their strings in a shared-string table rather than inline, so a naive walk over cells hands you a pile of integer indices. You have to dereference them.
using System.Collections.Generic;
using System.Collections.Specialized;
using System.IO;
using System.Linq;
using System.Text;
using DocumentFormat.OpenXml.Packaging;
using Telerik.Sitefinity.Services.Documents;
public class XlsxTextExtractor : ITextExtractor
{
public string MimeType { get; private set; }
public void Initialize(string mimeType, NameValueCollection config)
{
// One registration per mime; SpreadsheetDocument also opens xlsm and xltx
this.MimeType = mimeType;
}
public void GetText(Stream doc, Stream text)
{
var builder = new StringBuilder();
using (var spreadsheet = SpreadsheetDocument.Open(doc, false))
{
var workbookPart = spreadsheet.WorkbookPart;
if (workbookPart == null)
{
return;
}
// Cell text mostly lives in the shared string table; inline strings are rare
var sharedStrings = new List<string>();
var sharedStringPart = workbookPart.SharedStringTablePart;
if (sharedStringPart != null)
{
sharedStrings = sharedStringPart.SharedStringTable
.Elements<DocumentFormat.OpenXml.Spreadsheet.SharedStringItem>()
.Select(item => item.InnerText)
.ToList();
}
foreach (var sheetPart in workbookPart.WorksheetParts)
{
foreach (var cell in sheetPart.Worksheet.Descendants<DocumentFormat.OpenXml.Spreadsheet.Cell>())
{
if (cell.CellValue == null)
{
continue;
}
var value = cell.CellValue.InnerText;
if (cell.DataType != null && cell.DataType.Value == DocumentFormat.OpenXml.Spreadsheet.CellValues.SharedString)
{
int index;
if (int.TryParse(value, out index) && index >= 0 && index < sharedStrings.Count)
{
builder.AppendLine(sharedStrings[index]);
}
}
else
{
builder.AppendLine(value);
}
}
}
}
TextExtractorOutput.WriteUtf8(builder, text);
}
}
That bounds check on the index isn't ceremony. A corrupt or hand-generated workbook can carry a shared-string index past the end of the table, and an unguarded lookup throws in the middle of indexing over a value nobody would have searched for anyway. Skip the cell, keep the document.
Indexing macro-enabled Word files (.docm)
Smallest of the four and the most purely bureaucratic. A .docm file is DOCX with macros, the content is identical. It only misses out because of the MIME type mismatch from earlier.
using System.Collections.Specialized;
using System.IO;
using System.Text;
using DocumentFormat.OpenXml.Packaging;
using Telerik.Sitefinity.Services.Documents;
public class WordTextExtractor : ITextExtractor
{
public string MimeType { get; private set; }
public void Initialize(string mimeType, NameValueCollection config)
{
this.MimeType = mimeType;
}
public void GetText(Stream doc, Stream text)
{
var builder = new StringBuilder();
using (var word = WordprocessingDocument.Open(doc, false))
{
var body = word.MainDocumentPart?.Document?.Body;
if (body == null)
{
return;
}
foreach (var paragraph in body.Descendants<DocumentFormat.OpenXml.Wordprocessing.Paragraph>())
{
builder.AppendLine(paragraph.InnerText);
}
}
TextExtractorOutput.WriteUtf8(builder, text);
}
}
Leave plain .docx with the built-in extractor. No reason to take ownership of a format Sitefinity already handles.
Failure behavior matters more than the parsing
Sitefinity runs inbound publishing pipes through a helper that amounts to try { action(item); } catch { Log.Error(...); } with no rethrow. Which keeps one bad file from killing an entire reindex, fair enough. The cost is that an exception in the indexing path makes the document silently disappear from the index. No error page, no failed job, just a document that is no longer findable and a line in a log file nobody reads.
So the rule for extractor code is that a failure should cost you one document's body text. Never the document itself, and never the run.
Don't let an exception escape GetText if you can avoid it. Catch it, report it somewhere you actually look, and return with an empty output stream so the document still gets indexed by title and metadata.
Report to your error tracker rather than only the log file. I route extractor failures to Sentry with the MIME type and stream length attached. One caveat worth designing around: GetText only receives a stream, so the document ID lives in the caller and isn't available to you. If you need to identify the specific file, match on timestamp against the log.
And cap the reporting. A systemic problem during a full reindex fails once per document, and twenty thousand documents means twenty thousand identical events, which is exactly how a useful signal ends up muted as noise. I cap at ten per extractor per app domain and carry a counter in the payload so the real scale stays visible.
That last one turned out to be the biggest quality-of-life win of the whole exercise. Before, a reindex dumped a sixty-plus-line Telerik stack trace into the error log for every failing PDF, hundreds of times over, burying anything else happening on the site. Same situation now produces ten events with a count attached, and a log you can still read.
After deploying it
Formats that had no extractor at all now index their contents: slides plus speaker notes, spreadsheet cells, macro-enabled documents. The "MIME type is not supported" warnings stopped, because those MIME types are now supported.
On the PDF side, the structurally broken legacy files that the stock extractor gave up on during import come back with text. Not all of them, image-only scans still need OCR, but the ones whose only sin was a malformed structure tree or a RichMedia annotation are fine now.
It's maybe 300 lines total and most of that is boilerplate around four small parsers. Almost none of the time went into the parsing.