I've recently moved from a managed hosting company to pure, raw Amazon instance and I have to say it's fun. I've set up all the LAMP stack, email server, ssl/certs and a few other services and it feels good. Of course within the first maybe 2 months my machine got attacked by a DDOS targeting WordPress installations and essentially went down.
It took me a while, one evening and one morning of stressful rebooting and fiddling to finally figure out what is going on. In the meanwhile I've mastered every aspect of running an AWS instance like detaching/attaching volumes, changing volume type, changing instance type, reassigning elastic ip's you name it. The thing manifested itself in a substantial (like 99%) IOWAIT time on the machine CPU (top -> wa) and resulting sluggishness of response. So if you are apparently running out of IO on your server, it likely is not without some shady reason - amazon limits for IO are well within what should be enough for a small server with a few websites.
Finally I looked up my apache logs (which I should have done in the first place) to see that I'm being bombarded by large post requests somewhere from Russia. Every such request invoked full php stack and hogged the IO at 30MB/s constant request. I guess it's been a while since I regularly did sysadmin work. Time to refresh the old habits.
The fix is dirty, rewrite rule on each virtual host to send such requests directly to hell and updating all of my stuff to the latest version. I'm already looking forward to the next thing. What will that be? Email server exploit? Another WordPress hack? Should I move to Cloudflare and let them have all the fun or should I continue doing it myself?
I'll stick to doing it myself for now, since then I may have something to write about in this category . Unless I fail and this blog is taken down together with the rest of my host. Wish me luck.