I've seen a plenty of websites that use modal style windows particularly for photo galleries. They can be used for a variety of other content as well, giving a clean looking UI. A little bit of polish, that give a fair amount of bang visually. Unfortunately, the javascript behind it has always been a bit arcane. I stumbled across a few snippets of code that claimed to handle it, but I was worried about continuing cross platform compatibility. Luckily, a smattering of jQuery based projects have been developed to solve that problem.
One you might look at is FancyBox (http://fancybox.net/). I haven't been able to take a chance to use it, but if the landing page for the project is any indication you can make some smooth UI with it. Definitely something I will use in my next personal project. Almost makes me want to write something sooner rather than latter just for the cool factor.