> If you want secure config storage on Kubernetes, you end up using Secrets, which ends up being key-value like env vars anyway.

You can load the secret file directly into the app, no need to load it as env vars or keep it strictly as key-value pairs.

> If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.

You can use tools like ejson[1] or sops[2] to get encrypted files checked into Git that have key level granularity on diffs.

There is definitely a place for higher level abstractions than Kubernetes. Mostly it gives operators a standard platform to build from when teams outgrow the PaaS sandbox.

[1] https://github.com/Shopify/ejson [2] https://github.com/getsops/sops

> You can load the secret file directly into the app

Now you're getting away from the spirit of 12factor and hard-coupling them again. The intent is for the app to consume the secrets but have no knowledge or care where they came from.

Edit: misread as "load the secrets directly into the app." Yeah, this is just env vars but different.

My trick for this with k8s Secrets, when there's some sort of config file that absolutely must be consumed from disk, is to

1. create a Secret that defines the file as a key

2. mount that Secret into the container (.secret.secretName in the volume definition) under an arbitrary conventional path, e.g. /secrets

3. define an env-var that gives the full path to the file; where the fallback when this env-var isn't defined is to find the file in the working directory with a conventional filename.

If your own code uses the file, it can read the path from this env-var; while if the core of your container is some opaque tool that doesn't know about the env-var but does take the file's path on the command-line, then you can wrap the tool in an entrypoint script that reads the env-var and passes it as a command-line arg to the tool.

IMHO, this approach is flexible but principle-of-least-surprise obeying for both development and production deployment.

That sounds an awful lot like a .env file to me.

yup thats exactly what .env is, and tbh that doesn't change anything in the bigger picture

...no? It's the opposite.

A .env file allows you to go from knowing/assuming where a particular file is, to having environment variables.

What I'm describing is that you can go from being passed environment variables directly, to knowing where certain files are; which therefore allows those files to be mounted into a container-image in arbitrary places, rather than encoding brittle assumptions about config-file locations into the binary within the container-image — assumptions that usually get in the way during development, when you're not running the binary inside a container.

You still didn't understand my point. The thing that I'm talking about is using env-vars to point at config files that do not themselves contain env-vars, but rather are opaque app-specific config files, in formats like YAML or whatever-else.

I don't use kubernates. We have JVM servers running on AWS. We load secrets from Secrets Manager. Configs are mostly HOCON files stored in S3. The knowledge of where to find them is configured via env vars.

We use a custom setup in which the config values are loaded from multiple sources actually, so we could put everything into secrets manager, or load them all from env vars or HOCON files.

If we had to set every config value in ECS / lambda using env vars, it would be a major pain in the ass, and error prone.

> We load secrets from Secrets Manager

And how your apps authenticate to Secrets Manager ? Did you ever call `env` on a pod that has IRSA configured ?

This is just a middle step to do exactly the same thing but instead of using envFromSecret you use envFromSM

Not OP but an IAM role with scoped access to secrets. Better again, using secretsFrom in the task definition which injects the secret as an environment variable for you.

IRSA in pods and containers is translated to env vars. That was the point.

That if you dont want to pull in k8s dependencies to the code, at the end every approach ends up as a mounted file or env vars.

> Now you're getting away from the spirit of 12factor

The entire point here is that 12-factor isn't the be-all, end-all, so this is irrelevant.

> The intent is for the app to consume the secrets but have no knowledge or care where they came from.

This doesn't really make sense, and is an impossible requirement. The app will always have to know where configuration or secrets come from. Environment variables is just one option of "knowing where they come from". Choosing file storage is just as valid.

I'm not really kubernetes expert, but don't they come from /my-secrets and the application doesn't need to care how that got mounted?

> somewhat influenced by how Heroku works

The Twelve-Factor App is marketing content produced by the former Founder & CTO of Heroku. It's a great piece of content no doubt, but it should be viewed with that lens.

If it's marketing content then it's not doing a very good job. I have known about the 12factor app principles for a long time, and only found out today that it has ties to Heroku.

Still, Heroku would look impressively good if you wanted to deploy a 12-factor app, won't it?

Not all marketing is that straightforward.

Likewise.

ExternalSecrets with K8s. Put them into SOPS, Vault, your cloud’s secrets handler etc. - then reference them in the app as though they exist. They get loaded at deployment.

Doesn't break diffs, doesn’t directly couple anything.

I'd say 99% of my configs are not secret, at least not from containers running nearby.

I'm fine reading a few passwords and tokens from a secure key-value store. But the rest of my config can be a bit more expressive.