I have had this problem for a while now.
On my server, I run X11 whenever I work with it. Otherwise, I quit X11 since it otherwises uses a lot of resources. This gives my users a much better experience (i.e. better browsing and a lot of it happen at night for me because I live in California and I have loads of hits from France, Algeria, Tunisia, Marocco, etc.)
So... my wife and I both have a Unix account and we both start our own version of X11.
Hers rarely get stuck, mine seems to get stuck once a week or once a month.
Often, it is because after she quits I have to wait for