Check your logfiles, keep your eyes on server performance by something like htop, maybe there's not enough RAM and the server starts hanging itself.
What are your server specs? CPU's, RAM, swap...
Install htop and monitor the performance of your server. You say you can't SSH to your server, so...