I would attempt to rank what I feel is important for a security setup to protect it's keys.
- PROPER AND SECURE CODING PRACTICE WITH PROPER CODE REVIEWS!
- Use programming languages that will help lessen confusion and mistakes like these.
- Ensure that cryptographic libraries should be doing what they are suppose to do ! Cryptography is a hard subject and not everyone is good at this subject but everyone wants to use Cryptography to protect their digital assets. An application programmer who uses a particular cryptographic library may not properly understand encryption so it's hard to proof if the library is stable and "sane".
- Basic education on Cryptography to CS/IT/BIS students should be made mandatory due to the wide spread use of Cryptography in modern computing.
- Key handling to be performed somewhere else. Segregate the system into layers and sandbox. If one part is compromised, you have other layers to support and maintain a certain level of security.
- Integrate a FIPS-140-2 Level 3 or higher installation of Hardware Security Module (HSM) to solely handle the keys. Keys should always be handled in a HSM with FIPS-140-2 Level 3 or higher environment whereby the keys will never leave the secure environment and the HSM should be in working condition clean of no recent tamper events. Not everyone can afford a FIPS-140-2 Level 3 or higher HSM so it's pretty much a luxury item for those who can afford them. For those who can afford these HSMs, they should be using them properly.
If you noticed, proper education and secure coding has been ranked on top of everything else. The reason is due to the fact that, no matter how good the libraries and hardware/software are, proper understanding and usage is the key to security. Most of the standard algorithms out there that are well known are usually secure. It is the mismanagement on the human aspect that is causing so much security problems.