Everyday use

I'd recommend the E-M5, as the X100 has VERY poor autofocus performance.  Worse, the X100 will indicate it HAS sharp autofocus when it really doesn't.

Also, I've been on a number of shoots where liquids have dropped onto the camera (tipsy wedding revelers), and lens elements with embedded liquids is something you don't need.  The weather-sealed Olympus wins here.

Finally, m43 format is on a tear in Japan, so I expect that many more lenses will be available for the E-M5 from 3rd party manufacturers. 

going with olympus will need extra lens costs and you already have a 5d and probably don't need small camera with interchangeable lenses.

I would go for the x100. Its much nicer looking and I think it has a better censor. Also prime lens is not a disadvantage. Use your legs to arrange the distance for better composition.