Infrastructure
This chapter contains guidelines for infrastructure, IaC and wider Dev-Ops.
Tooling
What these guidelines want to address
- Empowering developers to deploy and change infrastructure
- Smoother on-boarding and issue resolution
- Replication, code quality, and documentation
General tips
Use Infrastructure as Code
These guidelines have examples for our most used stacks - use them whenever possible. If you find a use-case not covered by the examples, try to contribute the solution back to these guidelines.
Document changes (besides the ones done with IaC)
Having a log that is always used while working with the infrastructure is a great way to track issues later on. It is also a great way to make sure development and production are in sync.
Temporary changes tend to become permanent. For example, if a developer requires access to the production database, do not open it up to the internet. Define a bastion server in IaC that needs to be turned on/off when needed and requires a RSA key to access.
Run IaC manually
It might be tempting to run terraform in CI but that kind of action usually requires a lot of access privileges and thus a lot of things can go wrong and with grave consequences.
IaC tooling will display a change-set - always check that the additions or removals will not affect the things you were not trying to edit.
While we protect our selves by making the databases and data storage non-destroyable it is not 100% safe and would at minimum result in hours of work and downtime fixing it.
Pipelines should only be used to deploy application code.
Avoid monoliths
The modules should have clear separation - changing an SPA setting should not have any effect on the backend. Adding a new endpoint should not affect the database. Syncing up different environments should be done with small configuration changes.
A good guide of how to separate modules is to think about these scenarios:
- what if we need to add another SPA for administration?
- what if we need to add another API with public access?
- what if we need to scale up the database production?
- what if we need to move one environment to another AWS account?
Cleanly separate environments
CI/CD tooling should not have access to environments the pipeline is not actively working on. One option (if the client is up to it) is to separate completely by provider accounts.
Reuse modules
Following the tips above, you would get modules that are re-usable. Try to contribute them to these guidelines!
Keep an eye on billing
Setting a billing alert, while also helpful with not burning up the clients’ budget, will be a good indicator for services unintentionally left running or worse, alert of a security breach.
Scaling down staging and development servers will also save resources.
Deploy the exact same code in multiple stages
Code in staging should match code in production perfectly (aside from environment variables and scaling). To achieve that, we use the same code repository for all stages and build code by git sha.
Example of how this is done with Docker and Node.js
Make simple ENV changes possible
Adding or changing an ENV should generally not involve re-deploying the stack.
An example on how to use AWS SSM is available here.