Creating better search suggestions with Sphider

The problem

If you use the PHP web Sphider, you may or may not have noticed that the "did you mean" function isn't very accurate much of the time. Why is that? Simple, here is what it does with the search terms:

1. Use MySQL's SOUNDEX function to find the close matches
2. Use PHP's levenshtein function to find the first closest levenshtein distance.

Well, this is OK, but often gives poor results, especially as it keeps the first closest result & stops, even if better matches exist later in the result array.

The solution

So what I have done is to add two additional levels of matching, after getting the levenshtein distance, we then try and match the metaphone keys, and finally check the keywords using PHPs similar_text function to see if there is a better match than the current one. So this is what we end up with:

1. Use MySQL's SOUNDEX function to find the closest matches
2. Use PHP's levenshtein function to find the closest levenshtein distance.
3. Use PHP's metaphone function to match keys, if they match, then we perform the next step (#4).
4. Use PHP's similar_text function to see if this result is better than the last one.

The code

OK, how to implement? Easy, open the include/searchfuncs.php file, around line 333, find the following;

$near_word ="";
while ($row=mysql_fetch_row($result)) {
    
    $distance = levenshtein($row[0], $word);
    if ($distance < $max_distance && $distance <4) {
	$max_distance = $distance;
	$near_word = $row[0];
    }
}

Now REPLACE that with the following code (indent as appropriate);

// BEGIN BETTER SEARCH SUGGESTION FIX
$near_word ="";
$max_similar = 0;
while ($row=mysql_fetch_row($result)) {
    $distance = levenshtein($row[0], $word);
    if ($distance = $distance) {
            if (metaphone($row[0]) == metaphone($word)) {
                $similar = similar_text($row[0],$word);
                if ($similar >= $max_similar) {
                    $max_distance = $distance;
                    $max_similar = $similar;
                    $near_word = $row[0];
                }
            }
        } else {
            $max_distance = $distance;
            $near_word = $row[0];
    }
}
// END BETTER SEARCH SUGGESTION FIX

Finish

Now save and upload, and now your "did you mean" search suggestions should be much much more accurate.