Besides the beauty or the complexity of a CMS, I think that the plagin is what makes or breaks a CMS.
I installed a video plugin in Joomla called "HDVideoShare" and an image gallery plugin called "Phoca Gallery" (this one is actually free). I looked for such equivalent plugins in WordPress and I couldn't find any. That was enough for me to stick with Joomla.
What makes you chose a CMS over another?