2009-02-18

Search: Better, Faster, Stronger... Geekier

I've taken the plunge - the search feature (and back end of the Search API) is now powered by a real honest-to-Turing database engine. Yes... I've entered the 1970s!

If you were playing with the site for the last hour or so, you may have noticed that search went wonky. As usual, code that worked fine in my staging environment (my laptop) ran into scaling/performance issues when in the production environment - 130k separate INSERT statements took longer than ASP.NET likes to wait. A little fiddling with DataTables and SqlBulkCopy and now a rebuild of the database takes under ten seconds. Hooray!

What you'll find:
  • Search is better. Instead of exact word matches it will now do stem matches. So "Sol" will find "Solomani Rim (sector)" and "Sol (subsector)"
  • Search is faster. Previously, the search was done by creating an in-memory hashtable-based index that could be disposed of at ASP.NET's whim. So the first search on a cold index would take >10 seconds; subsequent searches would be faster... until the server decided to reclaim memory. I don't believe my hosting provider executes the site across multiple hosts, but that's now feasible at least.
  • Search is stronger. The database can be rebuilt in a few seconds from the raw data files (and ginormous metadata map) but that will only be necessary on data updates.
Now, all that said...

  • It's not going to give any additional result data back just yet, nor allow searching on other fields. The outstanding request - that should now be easy to service - is to allow searching by UWP. I have the data ready, I just need to write the glue, but I'm out of time for tonight.
  • The results are fairly arbitrary - matching sectors, matching subsectors, then matching worlds - and a max of 20 results total. Previously that wasn't a problem, but with the looser matching it could be less than idea. Feedback?
What else do you want? What problems have you noticed? Ω

1 comment:

BeRKA said...

Nice! :-)

It now seems a lot faster than last time i searched.