This isn’t going to be an overly long post, but I wanted to express my gratitude for the LINQ extension methods which make a lot of repetitive code no longer necessary. These methods have been available for use since .NET 3, and I have been using them for almost as long. I know I’m pretty late to the party here in praising them, but I needed to get this out anyway!

The real love comes from the fact I no longer have to write wrappers around containers, or even templated (generic) methods of my own to do searching within containers. How many times in the past have I had to write a loop to iterate over a collection of objects to pick out a subset of that collection based on a defined criteria? Too many to count. It’s one of those frustrations that you couldn’t solve very easily without resorting custom filtering code and ugly callbacks, since you might never know what criteria you need to search on.

Enter lambdas (closures) and the IEnumerable extension methods to the rescue! We can now write something as simple as:

subset = set.Where( x => x.SomeProperty == criterion );

And huzzah! I have a subset of the items I wanted. The other lovely fact here, is that the LINQ is lazy… it doesn’t actually evaluate this set until I make a reference to it. While I probably shouldn’t be calling this code if I’m not going to be referencing it, I am a fan of lazy evaluation (lazy loading, lazy anything really).

As with anything in the development world, this sort of method can be dangerous if you don’t use it right. For example, using code like this to filter (or sort) data instead of letting your database do it is pretty silly. On the other hand, if you need to select a set of data from the database and need both the full set, and a subset of the same data… this method is absolutely wonderful!

Most of the time, that’s not a big deal. There are some days though that I want to see real time stats about what is happening to my sites and Google Analytics just doesn’t cut it. So with a quick search, I was able to find Piwik, an open source and real time analytics package. I’m only in the testing phase at the moment, but in general the setup was fairly easy… I guess we’ll see how it goes!

I was actually very close to reinstalling Ubuntu 10.10 this evening. There’s always an issue when installing/upgrading and this time, as it always seems to be, there was a problem with the sound. At first I didn’t even think it was the sound though! Upon loading Firefox 4 (so that I could download Google Chrome), I saw a youtube video on my homepage. I figured I should check it out quickly since flash is also a sore spot with Ubuntu every once in a while. It played, which was a good sign, but it seemed to play is super fast forward no matter what I did.

So I sharpened up my google-fu, and started trying to figure out why this was happening to no avail, including several re-installs of the flash plugin. With that hitting a sore spot, I decided I should try some music. It happened to do the same thing! Finally, after what seemed like forever, I was able to find a vague post on launchpad discussing hardware selections. A-HA, I said to myself. I remember this from the days when I had a Creative Soundblaster Live! card, and the motherboard also had onboard sound.

I fiddled with settings for a few minutes and managed to get things going, but this dilemma really drives me crazy. I don’t understand why these types of issues are so darn hard to figure out. Ubuntu seemed to think I wanted sound going out from my video card over DVI… but when that didn’t work, it didn’t ever consider another option for sound at all. Nope, it just decided to not work. I have the patience, albeit just enough of it, to try and figure these issues out, but I would never, not in a million years, expect an every day user to try to figure an issue like this out.

So… Mark Shuttleworth and Ubuntu Developer alike. I ask you, why can’t things like this be EASY?! I installed Windows 7 last night on the same PC, there wasn’t a single issue (other than wiping out the MBR… but that’s a gripe for another day). I won’t say that Linux hasn’t come a long way in the last 5 or 6 years, but I still think there’s a long, long way to go.

Also… why the heck do you have to change the default applications every damn release. It’s driving me insane!

And… I broke the pi script (well, technically all my “projects” pages) with a little rewrite rule I was testing the other day. I forgot to remove it and it made a nasty redirect loop for things that actually existed. Oh well, it’s fixed for pi day!

If you want your content to work with “ajax” and “web 2.0″ do the mass of your HTML rendering first. Don’t break your site with ridiculous things just because “it’s cool!”, that doesn’t fly.

Cheers

Michael

When I sit down to design a data model, as I have done many, many, MANY times in the past I always try to think about 3rd normal form as a base. I know that full normalization isn’t always the most efficient when you actually need to run an application, but my theory is always to start with a nicely normalized set of schemas/entities and if necessary I denormalize to improve performance at a later point.

Anyway, I’m starting to get off on a tangent. One of the things we have to consider when designing databases is how our data will be accessed. How often we will search based on certain criteria and create indexes accordingly to speed up these requests for data. This is usually about as complex as databases get for me, minus writing complex queries. I don’t usually get down to the details like partitioning and replication and all the grimy stuff.

On the most recent project I’ve been hacking away on for work (I’ve discussed this a couple times already regarding the MVP pattern), I ran into an issue I had never had to tackle before. We have fields in our database which are large blocks of text. These blocks may be searched by our users. OK, this is easy so far… I can just use a “LIKE” search to find the text I need (ignore the fact this could never make use of an index unless I did “searchterm%”). But if we think about this, we want people’s searches to be a little smarter than a simple like query since these searches are the real starting point of our application.

So we have an issue at this point, how to we support smart searches? For example, if I have some text in my database like “I made a database”, and my user enters a search for “databases” is a like query going to match this? Obviously the answer is no, the database isn’t that smart and neither is my code. It would make sense for this to work, how would we do it? Should we teach the search code to know the English language? Probably not, but the answer is probably obvious for anyone who has worked with large pieces of text before ( on-line forum software anyone??). A full text index can do this for us!

Note: The examples below are for SQL Server… I do not know if they work in MySQL or other databases. I assume the approach is similar though.

I’ve never really worked on a project before where I’ve needed to deal with such large blocks of text, so using a full text index was definitely not the first thing which came to my mind. The first thing I managed to get working was just the searching itself. Rather than using a LIKE query I was able to do a CONTAINS search, which means rather than searching like this: “LIKE ‘%this%’”, I was able to search like this: “CONTAINS( TextField, ‘this’ )”. And to tell you the truth I was really amazed at the speed improvement over the like search I was doing before. Running the query through the analyzer gave me half the time the original search was taking.

So the question now is, how do we do the expanded search we want? Well that’s also pretty easy! SQL Server has this wonderful operator for fulltext searches called ‘FORMSOF’. Now there are two ways we can make FORMSOF work that I know of, there may be more (Remember what I said before, not a DB Expert! ): INFLECTIONAL and THESAURUS. In our case we want to use the INFLECTIONAL search, which means it will search based on the root of the word. Searching for databases will use database as the root and turn up matches for both words. I probably picked a bad example there since I can’t think of another way to form the word database. As another example let’s use a verb! If we tried the word search we would end up with matches for “search”, “searching”, “searches”, and “searched”. Do you see what I’m talking about now?

For clarity’s sake, I will actually show you a clause using the above. So our search would be like this: CONTAINS( searchfield, ‘FORMSOF(INFLECTIONAL, “word”)’) . Take note that the FORMSOF operator is inside the quotes. To the contains function it just looks like a string. Also note the word we are searching for is in double quotes inside the formsof operator!

And really that should be it. You should now be able to use a full text index on your database and know how to make use of it! Happy searching!

Anyway, we’ll see how this goes. As of right now theme integration has failed horribly. Maybe I need to do some more research to see if there’s a better option for forums which integrate nicely with a Zend Framework site without me coding it from scratch.