Creating better search suggestions with Sphider

Test Sphider

Sun, 27 Nov 2011 13:58:49 +0000

Hi again,

I've been searching and I guess the solution lies in "preg_match" in spiderfunc.php and "$entities = array" part in comonfuncs.php. But I could not do it..

Test Sphider

Fri, 25 Nov 2011 23:23:21 +0000

It sounded a good idea. I tested it. But this time, it didn't highlighted the word found and also it didn't found with capital letters :)
In previous version when I typed çanakkale it asked "did you mean Çanakkale" and clicking on Çanakkale brought Çanakkale results not ÇANAKKALE. In this version, when I type Çanakkale it asked "did you mean çanakkale" which is weird because I only have Çanakkale and ÇANAKKALE in page. It finds the page but no highlighting. :))

I also removed that special characters from remove accents part in commonfuncs.php file but no help.

It is really interesting and drives me crazy. I could not find any solution to this. It clearly recognizes the words but we need to tell ğ=Ğ, ş=Ş or Ç=ç so that it matches all and highlights them.

If you find any ideas or any solutions to this, I'll keep an eye on this page..

Thank you for everything.

Matt

Fri, 25 Nov 2011 21:47:49 +0000

Ahhh, you're quite right, haha, this one got butchered, I better check my other articles. Thanks for the trouble shooting!

Regarding the case-sensitive searching, I believe that Sphider itself converts all the strings to lowercase using PHP's strtolower function before storing them in the database so that the md5 hashes match... i.e. searching on "GaMe" would match "game" or "GAME". It's possible that it's mucking up the latin 5 characters when it's doing that, so the md5 hashes aren't matching when searching.

For a quick fix, I would try replacing all the strtolower function calls in spiderfuncs.php, searchfuncs.php, etc with the mb_strtolower function then re-indexing & checking again, but that's just a wild guess at this point.

Test Sphider

Fri, 25 Nov 2011 21:32:36 +0000

Thank you again. I appreciate your help. "Did you mean" function is now working but one "}" character should be deleted.

Fow the case problem.. I read the url you've supplied. But I dont want the query to be converted into lower case or upper case. What I want is :

When you search a query "book", it matches all word regardless of case right? (highlights book and Book and BOOK). But in case of latin 5 characters it doesn't work in the same way. It become case sensitive. Do you have any idea to tell the script to search everything case intensive?

Matt

Fri, 25 Nov 2011 20:31:27 +0000

Hi again,

Sorry about that, I just took a look at the code. It seems when I moved to a new content management system recently a part of the code block to be replaced was stripped out of the original article. Try replacing the code block now listed above.

For finding the lowercase, you can try using PHP's multibyte function;
http://php.net/manual/en/function.mb-strtolower.php

Something like this maybe?
[php]
$query = mb_strtolower(str_replace('"','',$query), 'iso-8859-9');
[/php]

Test Sphider

Fri, 25 Nov 2011 09:21:12 +0000

Regarding my other question..

My html page charsets are windows-1254. I changed my database collation to latin5_turkish_ci.
I added this code to database.php:
mysql_query("SET NAMES 'latin5'");
mysql_query("SET CHARACTER SET latin5");
mysql_query("SET COLLATION_CONNECTION = 'latin5_turkish_ci'");
No problems displaying the characters. I have "AĞRI" in one of my page title. When I search "ağrı",it only search for "ağrı" not "AĞRI" and as a result highlights "ağrı" only.

Thanks again for your help in advance.

Test Sphider

Fri, 25 Nov 2011 09:17:43 +0000

I meant the code actually is not consisting of 4 lines in 1.3.5 version:
The 4th line and the remaining part in my case is:

[code]
if ($distance < $max_distance && $distance <4) {
$max_distance = $distance;
$near_word = $row[0];
}
}
if ($near_word != "" && $word != $near_word) {
$near_words[$word] = $near_word;
}
}
[/code]

When I change this whole code with the code you provided, "did you mean function" is not working.

Matt

Thu, 24 Nov 2011 23:14:32 +0000

Hi there,

Yes, as I noted, you need to REPLACE those 4 lines with the code I provided.

I'm not certain about case sensitivity with non Latin languages, I will have to investigate that & get back to you.

Test Sphider

Thu, 24 Nov 2011 22:52:33 +0000

hi,
did you mean replacing these whole codes with the code you provided?:
starting from
$near_word ="";

ends with

$near_words[$word] = $near_word;
}

}

second question:

sphider does not do case intensive searches for non english characters. How can we correct this?

Matt

Thu, 19 May 2011 18:40:41 +0000

Hi Willy,

Yes, do that on both instances... to make it easier, we could simply put this at the top of the function below the list of global variables, and that would cover it throughout;

[code]$query=strtolower($query);[/code]

but either way works :)