Enabling more robust hyphenated word indexing in Sphider

The problem

Lately I've been hacking around in Sphider. I've been using that as the search engine behind my website for a couple of months now, it's quite encompassing, however there are quite a few annoyances for me personally.

So slowly I've been fixing these items as time permits. Many of the hacks are rather specific to my site, but this one could be of help to anyone who wants better indexing of hyphenated words.

Essentially, Sphider currently indexes hyphenated words as a single word, so "light-weight" is simply indexed as "light-weight". I have an article containing this word within the title, so I would have hoped that searching for "light weight" would bring up that article. Unfortunately, it did not. So I rooted around and found where the words were prepared and inserted into the array.

The solution

What this little modification does is take a hyphenated word, such as "light-weight", it will then split that into four separate words, "light, weight, light-weight, and lightweight", and add them into an array. Then we use this new array to index them all against the URL, so now if you remove the hyphen from the word and simply use the component words your search results should still be accurate. Not perfect, but it works well for me.

The code

Making this change is easy, simply back-up your spiderfuncs.php file and work off a copy. Within the function unique_array (around line 320), find the following code;

if ($stem_words == 1) {
    $newarr = Array();
    foreach ($arr as $val) {
        $newarr[] = stem($val);
    }
    $arr = $newarr;
}
sort($arr);
reset($arr);
$newarr = array ();

Now you want to add the new code just before the array is sorted and reset;

if ($stem_words == 1) {
    $newarr = Array();
    foreach ($arr as $val) {
        $newarr[] = stem($val);
    }
    $arr = $newarr;
}
// BEGIN HYPHENATED WORD INDEXING FIX
// This code splits and also joins hypenated words for indexing, while maintaining the whole hypenated word
// example; light-weight will be split into four separate words, "light, weight, light-weight, and lightweight"
$hyphenarray = array();  // create placeholder array
foreach ($arr as $val) {
    if (strpos($val,"-") == true) {  // word contains a hyphen
        $bits = explode("-",$val);  // let's break it up
        foreach ($bits as $bitsval) {
            $hyphenarray[] = $bitsval;  // add each individual word to array
        }
        $joined = str_replace("-","",$val);  // remove hyphens to join word
        $hyphenarray[] = $joined;  // add the joined word to array
        $hyphenarray[] = $val;  // add the hypenated word to array
    } else {
        $hyphenarray[] = $val;  // word does not contain hyphen, add to array & carry on
    }
}
$arr = $hyphenarray;
// END HYPHENATED WORD INDEXING FIX
sort($arr);
reset($arr);
$newarr = array ();

Finish

And that's it, it should index hyphenated words as their component words now too.