Towards Better Configuration & Environment Variables

We build configuration tooling here at Prefab, so it was a little embarrassing that our own local development configuration was a mess. We fixed it, we feel a lot better about it and we think you might dig it.

So What Was Wrong?

We used our own dynamic configuration for much of our config and that worked well, but when we needed environment variables everything started to fall apart. The pain points were:

Defaults In Multiple Places

Environment variables sound nice: "I'll just have my app be configurable from the outside". But in practice it can get messy. What are the default values? Do I need to specify defaults for everything? How do I share those defaults. When do I have fallback values and when do I blow up?

We had ended up with defaults in:

Ruby code:

# puma.rb
ENV.fetch('WEB_CONCURRENCY') { 2 }

A .env.example file:

GCS_BUCKET_NAME=app-development-billing-csv-uploads

Other yaml configs like config/application.yml:

STRIPE_SECRET_KEY: sk_test_1234566

production:
  STRIPE_PRODUCT_BUSINESS_MONTHLY_2022_PRICE_ID: price_1234556

And in Terraform in another rep:

resource "kubernetes_config_map" "configs" {
  metadata {
    name = "configs"
  }

  data = {
    "redis.uri" = "${local.redis_base_uri}/1"
  }
}

Per Env Configuration All Over the Place

Beyond defaults, where do I put the environment specific overrides? Are these all in my devops CD pipeline? That's kinda a pain. Where are the production overrides? Could be anywhere! We had them in each of:

1. config/production.rb
2. database.yml production: section
3. config/application.yml production: section

Duplicated Defaults Across Repos

Because we have multiple services, we also had some of the defaults in ruby .env.example also showing up in our Java app in a src/main/resources/application-development.yml.

No Easy Way to Share Secrets / API Keys

As if all of ^ wasn't enough of a mess. Secrets had to have an entirely different flow. We were good about not committing anything to source control, but it was a pain to get the secrets to the right place and easy to forget how to do it.

Summary

We were surviving, but it wasn't fun and the understanding / context fell out of our heads quickly meaning that whenever we needed to change something we had to reboot how things worked into our working memory and it took longer than it needed to. For a longer rant on environment variables, check out 9 Things I Hate About Environment Variables.

What Would Be Better?

So, what would be better? We wanted:

• A single place to look to see all of our of my configuration
• Developers have a single API key to manage, no local env var mysteries
• Defaults that are easy to override for local dev, but weren't footguns leading to Works On My Machine issues
• Easy to share configuration between projects
• Interoperability with our Terraform / IaaS / Kubernetes variables
• A system that supports secrets as well as configuration
• Telemetry on what values are actually being used in production for our IaaS / Terraform provided values

We had a ton of the infrastructure in place to support this from our dynamic configuration work, but when it came to environment variables we were still in the stone age.

Our Dream

Our dream looked like this. With just a single api key and callsite, like:

#.env
# One API Key per developer
PREFAB_API_KEY=123-Development-P456-E2-SDK-c12c561b-22c3-4a52-9c38-a8f24355c102

#database.yaml
default: &default
  database: <%= Prefab.get("postgres.db.name") %>

We wanted to be able to see all of my configuration in one place in:

The UI

The Prefab Config UI for a config using normal strings as well as provided strings.

It's clear what the value is in every environment and I can see which environments are getting the value from a Terraform provided env var.

The CLI

The Prefab CLI running a prefab info call.

code (main)  $ prefab info
? Which item would you like to see? postgres.db.name

- Default: unused
- Development: forcerankit_dev
- Production: POSTGRES_DB_NAME via ENV
- Staging: POSTGRES_DB_NAME via ENV
- Test: forcerankit_test

Evaluations over the last 24 hours:

Production: 4
- 100% - forcerankit
Staging: 1
- 100% - forcerankit

My Editor

And even a hover in my editor using the VSCode extension.

What We Did to Enable This

There were 3 big things we needed to support to make this happen: Environment variables, Datafiles & Secrets.

Provided ENV Vars as a Config Type

First we needed to allow a config value to be "provided by" an environment variable. You can now do that within the Prefab UI or CLI.

If you check the box for "Provide by ENV VAR" you can then specify the ENV VAR name for any environments that it should be provided in.

Datafile Support

Datafile support allows the Prefab client to start up using a single local file instead of reaching out to the Prefab API. This is useful for CI environments where you may want perfect reproducibility and no external network calls. You can generate a datafile for your local environment and then commit it to source control. This allows you to have a single source of truth for your configuration and secrets.

In our goal of having a "Single Source of Truth" for our configuration, the original system of default files like .prefab.default.config.yaml ended up being more of a hindrance than a help. There's a big difference between a UI that is all-knowing and a system that has partial knowledge that could be overridden by other files, re-introducing complexity into the system.

Making the API all-knowing is lovely, but if everything is in the API, what do we do for CI / Testing?

Our solution is to have 2 modes:

1. Live mode.
2. Datafile mode. Offline, load a single datafile.

The datafiles are easy to generate. You simply run prefab download --env test and it will download a datafile for the test environment. You can then commit that datafile to source control and use it in CI.

In CI environments you can then run PREFAB_DATAFILE=prefab.test.123.config.json and it will use that datafile instead of reaching out to the API.

Secrets

The last big piece of this work was supporting secrets. If we were going to clean this all up once and for all, it just didn't work to still be on our own for secrets. I'll cover that in a future blog post, but if you're interested in our Secrets Management Beta please let us know. It's a zero-trust, CLI based solution that we think hits the nail on the head of being dead simple and easy to use.

What's Next?

We're really happy with how this turned out. Everything just feels... right. Configuration is important. Configuration goes in one place. It sounds like that should be easy, but from my experience up until now it's not the world many of us have been living in.

If you've been living in a monolith world deploying to heroku, you've long been enjoying the simple pleasure of heroku config:set GITHUB_USERNAME=joesmith. But if you have more than one application, apps in different languages, or weren't deploying to something simple like heroku, the story has been much worse.

What we've built has been a big improvement for us and we think it will be for you too. We're going to be rolling this out to all of our SDKs over the next few weeks. We'd love to hear what you think.