Yeah it is taking some time to open.
If you`ve got a dedicated server or VPS, check whats running. You may have a cron job thats running more often then it needs to (once every second instead of hour) or you may have an endless loop going on somewhere.
Next, check your logs to make sure its not just really heavy normal traffic, or if someone`s attempting a brute force breach of your server. (Generally noted by constant connection to ports other than 80, 25, or 110, that fail at authentication)
If all else fails, try a gracefull reboot (you may have to contact the hosting provider if you don`t have access to power management).