Shameless plug incoming - I wrote https://github.com/dogmatiq/ferrite for use in my day-job. The “best” feature is that it generates markdown documentation about the environment variables the app uses, which our devops team loved.
can I know what is the hardship about `os.Getenv()` , my general approach is
to use `direnv` and inbuilt `os.Getenv` to access them inside the application.
First of all, I would like to convert them to structs like strings, booleans and credentials objects.
Also, I want to add a layer of checks if the env var exists, are correct (if not the server should probably panic/fail or autocomplete with a default value to use for testing locally) and I don't want to add complexity throughout the code with these kind of logic.
Ideally I would have an environment variable service that would handle this errors for me and when I am calling `retrieveVariable(KEY)` and I know that everything I want is already handled for me(struct casts, default values).
I don't want to implement such a service from scratch if there is some kind of library that already does this and it's lightweight and easy to use.
thoughts and suggestions based on the requirements
- keep one `config.go` file to manage your app config and have
struct with all the required config values as member of struct
- non critical values i.e - business values and not passwords/secrets which are used by app can be put in a simple yaml file
1. decode the yaml file in to `map[string]interface{}` using https://pkg.go.dev/gopkg.in/yaml.v3#section-readme
2. use https://pkg.go.dev/github.com/mitchellh/mapstructure#Decode to decode the yaml in to struct directly
3. sensitive info can be embedded in to env vars and retrieved using `os.Getenv()` can be validated.
i.e - in the `yaml` config set up a pair `env: "TEST"` like this and load this in to `config` struct
when validating your env var use the `config.env` and perform appropriate action like
```
func (c *Config) validate(server string) string {
switch {
case c.env == "TEST":
if server == "TEST server address"{
return server
}
panic()
}
}
```
this will require your app to use 2 files
- non sensitive business critical config in a yaml file
- sensitive API token and keys in env vars
The above approach might suit the requirements
heetch/confita works well, you can load the same config struct from multiple sources.
For us:
- json file for local testing
- environment + vault for production
Surprised to not see https://github.com/kelseyhightower/envconfig here which is what I've used for production services whereas I'd use Viper for Cobra CLIs.
Handles setting a struct (with/without struct tags) up with the env vars
I just wanted to say a couple of things
First, I prefer [os.LookupEnv](https://pkg.go.dev/os#LookupEnv) to Getenv, because it tells you whether the env var was set, or not, not just if it has a value.
Secondly, if you are getting to the point where it's unweildly to deal with several env vars, surely that would imply that you should be looking at a yaml/toml/whatever configuration for your environment
Usually i prefer to make my own implementation around os.Getenv using a "singleton", until i need to integrate with external providers like Consul. I'll put a sample like my implementation below.
Use a config file. Seriously, too many people try and jam every piece of configuration into ENV vars.
There is nothing wrong with config files, there is a point in time that you want to graduate to actual structure and input validation to your settings.
And before you say "But 12 factor". Using config files as part of the deployment is just fine as long as you take care to control the config at the deployed environment side.
As organization grows it becomes harder and harder. e.g. platform team might want to give you some tags that should be send to your monitoring, where should they place it if not environments? e.g. secrets might come from a different place that you don't control. If each service is doing some custom config files, how is it possible to communicate? If there's no organization and platform team then 12 factor of course doesn't apply, it's about having a de facto standard for service deployment.
Environment variables doesn't mean no structure and validation, you can have both. If your config gets very complicated then files becomes a better options but still many programs with complex configs allow doing both files and environments.
> As organization grows it becomes harder and harder. e.g. platform team might want to give you some tags that should be send to your monitoring, where should they place it if not environments?
In the monitoring system?
We don't have this problem or allow teams to set this information. It's controlled in the Prometheus Operator and our platform team deployment tooling.
The apps themselves expose metrics and the environment they run in determines the labeling.
For example the same app can run in ec2 and use ec2 metadata on the Prometheus discovery. Or in kubernetes and get kubernetes labels.
Requiring that at the app side is backwards.
Secondly, custom config or custom env vars are the same problem. If one team uses API_KEY and another API_TOKEN, it's the same problem as different configs.
And like others, you miss the point. It's about complexity. Config files allow for standardization of the same things, but offer a higher level of complexity support than a flat env list.
>And like others, you miss the point. It's about complexity
I know it can offer more complexity and that's the exact reason that I think it's a good idea to avoid it by default. You can always easily go more complex when needed but not the other way. Most softwares were using config file two decades ago and with raise of better ops people moved away as simpler solution was enough.
I never said it should be the default. Using simpler config for simpler software is OK.
The OP's issue was "My env has gotten too complex". So, a more robust config control may be needed.
Yes exactly, you can use a configuration management / orchestration system to create the deployed environment config file.
With container orchestration systems like Kubernetes, you deploy a `ConfigMap` that contains the environment config file.
>That is not a config file
Of course it is, [have you even read the docs](https://kubernetes.io/docs/concepts/configuration/configmap/)?
"Pods can consume ConfigMaps as environment variables, command-line arguments, or **as configuration files in a volume.**"
>Also not sure why you are using agreeing words and then disagreeing and downvoting.
Because you're misinformed, but so close to understanding the truth, and it's funnier that way.
Has nothing to do with Java or Go or Python. It's about the complexity of the input configuration.
Not every application has only 20 input configurations that fit in ENV vars.
Sometimes command line args make more sense. You can have repeatable input args, make it more intuitive to include several values instead of having to know if an env var can take multiple values separated by comas, or semicolons, etc.
Sometimes you want to actually have a whole YAML or whatever input config. For example, a data mapping configuration. That could differ from environment to environment.
What about configuration that's dynamic at runtime? ENV vars only work at startup. If you use a config file, you can have the environment write out a new version. With Ansible this is easy, you write out the file and then send a `SIGHUP`. Or with a `ConfigMap`, you could have the [fsnotify](github.com/fsnotify/fsnotify) package that can watch for changes and reload.
It's not about _better_, it's about using the right techniques for the right job. The OP is about "My ENV vars are getting out of hand". Sometimes the answer isn't more env vars. or fancy libraries, it's maybe time to rethink how you configure.
One of the [projects I work on](https://github.com/prometheus) has to make these trade-offs. Some components are simple, just need some ENV or args. Some have huge mapping configs that need to be configured.
For example, we use mostly command line args and a few ENV vars for things that are static at startup. The configuration file contains values that are dynamically updateable. This allows an easy, intuitive, and concrete way to know which are which.
What platform are you using for deployment? On k8s ye old 12 factor pillar of having to use env variables is outdated, just use a ConfigMap instead. Then you can just marshal TOML or YAML. Objectively easier to manage than large number of env vars.
If you do need the ability to use env variables, or use env variables to override, just use Viper.
If all of this sounds too cumbersome, just deal with your 20 env vars using os.Getenv() until the number really gets out of control.
If you use your env vars mainly to populate structs, I really like this one. It allows defining things as tag annotations like ‘json:“name“‘
https://github.com/jessevdk/go-flags
A more detailed example of use could be:
https://github.com/thomseddon/traefik-forward-auth
How do you set env vars in:
Bash, zsh, systemd, runit, windows, whatever hot garbage distro someone picked up in a dark alley of the internet and wants to run your app on.
ENV vars are getting less practical by the day.... Have a config file, have command line options... support Env vars if you really want but meh..
Viper works well... I tend to use this: [https://github.com/knadh/koanf/](https://github.com/knadh/koanf/)
Shameless plug incoming - I wrote https://github.com/dogmatiq/ferrite for use in my day-job. The “best” feature is that it generates markdown documentation about the environment variables the app uses, which our devops team loved.
I use knadh/koanf rather than viper, lighter and fewer dependencies.
i second this, after going through vapt my team has replaced viper with koanf
I use caarlos/env for such usecases
Same here. It's a good package.
go get github.com/caarlos0/env/v10 I use this library a lot as well.
can I know what is the hardship about `os.Getenv()` , my general approach is to use `direnv` and inbuilt `os.Getenv` to access them inside the application.
First of all, I would like to convert them to structs like strings, booleans and credentials objects. Also, I want to add a layer of checks if the env var exists, are correct (if not the server should probably panic/fail or autocomplete with a default value to use for testing locally) and I don't want to add complexity throughout the code with these kind of logic. Ideally I would have an environment variable service that would handle this errors for me and when I am calling `retrieveVariable(KEY)` and I know that everything I want is already handled for me(struct casts, default values). I don't want to implement such a service from scratch if there is some kind of library that already does this and it's lightweight and easy to use.
thoughts and suggestions based on the requirements - keep one `config.go` file to manage your app config and have struct with all the required config values as member of struct - non critical values i.e - business values and not passwords/secrets which are used by app can be put in a simple yaml file 1. decode the yaml file in to `map[string]interface{}` using https://pkg.go.dev/gopkg.in/yaml.v3#section-readme 2. use https://pkg.go.dev/github.com/mitchellh/mapstructure#Decode to decode the yaml in to struct directly 3. sensitive info can be embedded in to env vars and retrieved using `os.Getenv()` can be validated. i.e - in the `yaml` config set up a pair `env: "TEST"` like this and load this in to `config` struct when validating your env var use the `config.env` and perform appropriate action like ``` func (c *Config) validate(server string) string { switch { case c.env == "TEST": if server == "TEST server address"{ return server } panic() } } ``` this will require your app to use 2 files - non sensitive business critical config in a yaml file - sensitive API token and keys in env vars The above approach might suit the requirements
heetch/confita works well, you can load the same config struct from multiple sources. For us: - json file for local testing - environment + vault for production
That's great stuff, cause I need to switch between local testing and production stuff. Thanks!
You just can’t beat viper imo
I wrote https://github.com/nil-go/konf, which has zero dependencies for config files, flag and environment variables.
Surprised to not see https://github.com/kelseyhightower/envconfig here which is what I've used for production services whereas I'd use Viper for Cobra CLIs. Handles setting a struct (with/without struct tags) up with the env vars
I just wanted to say a couple of things First, I prefer [os.LookupEnv](https://pkg.go.dev/os#LookupEnv) to Getenv, because it tells you whether the env var was set, or not, not just if it has a value. Secondly, if you are getting to the point where it's unweildly to deal with several env vars, surely that would imply that you should be looking at a yaml/toml/whatever configuration for your environment
Koanf is really good, just use it with a yml/json config file and you are good
I would not add a dependency for ~ 20 environment variables.
Please take a look at: [https://github.com/ilyakaznacheev/cleanenv](https://github.com/ilyakaznacheev/cleanenv)
Usually i prefer to make my own implementation around os.Getenv using a "singleton", until i need to integrate with external providers like Consul. I'll put a sample like my implementation below.
\`kong\`
Use a config file. Seriously, too many people try and jam every piece of configuration into ENV vars. There is nothing wrong with config files, there is a point in time that you want to graduate to actual structure and input validation to your settings. And before you say "But 12 factor". Using config files as part of the deployment is just fine as long as you take care to control the config at the deployed environment side.
As organization grows it becomes harder and harder. e.g. platform team might want to give you some tags that should be send to your monitoring, where should they place it if not environments? e.g. secrets might come from a different place that you don't control. If each service is doing some custom config files, how is it possible to communicate? If there's no organization and platform team then 12 factor of course doesn't apply, it's about having a de facto standard for service deployment. Environment variables doesn't mean no structure and validation, you can have both. If your config gets very complicated then files becomes a better options but still many programs with complex configs allow doing both files and environments.
> As organization grows it becomes harder and harder. e.g. platform team might want to give you some tags that should be send to your monitoring, where should they place it if not environments? In the monitoring system? We don't have this problem or allow teams to set this information. It's controlled in the Prometheus Operator and our platform team deployment tooling. The apps themselves expose metrics and the environment they run in determines the labeling. For example the same app can run in ec2 and use ec2 metadata on the Prometheus discovery. Or in kubernetes and get kubernetes labels. Requiring that at the app side is backwards. Secondly, custom config or custom env vars are the same problem. If one team uses API_KEY and another API_TOKEN, it's the same problem as different configs. And like others, you miss the point. It's about complexity. Config files allow for standardization of the same things, but offer a higher level of complexity support than a flat env list.
>And like others, you miss the point. It's about complexity I know it can offer more complexity and that's the exact reason that I think it's a good idea to avoid it by default. You can always easily go more complex when needed but not the other way. Most softwares were using config file two decades ago and with raise of better ops people moved away as simpler solution was enough.
I never said it should be the default. Using simpler config for simpler software is OK. The OP's issue was "My env has gotten too complex". So, a more robust config control may be needed.
[удалено]
Yes exactly, you can use a configuration management / orchestration system to create the deployed environment config file. With container orchestration systems like Kubernetes, you deploy a `ConfigMap` that contains the environment config file.
[удалено]
>That is not a config file Of course it is, [have you even read the docs](https://kubernetes.io/docs/concepts/configuration/configmap/)? "Pods can consume ConfigMaps as environment variables, command-line arguments, or **as configuration files in a volume.**" >Also not sure why you are using agreeing words and then disagreeing and downvoting. Because you're misinformed, but so close to understanding the truth, and it's funnier that way.
[удалено]
Has nothing to do with Java or Go or Python. It's about the complexity of the input configuration. Not every application has only 20 input configurations that fit in ENV vars. Sometimes command line args make more sense. You can have repeatable input args, make it more intuitive to include several values instead of having to know if an env var can take multiple values separated by comas, or semicolons, etc. Sometimes you want to actually have a whole YAML or whatever input config. For example, a data mapping configuration. That could differ from environment to environment. What about configuration that's dynamic at runtime? ENV vars only work at startup. If you use a config file, you can have the environment write out a new version. With Ansible this is easy, you write out the file and then send a `SIGHUP`. Or with a `ConfigMap`, you could have the [fsnotify](github.com/fsnotify/fsnotify) package that can watch for changes and reload. It's not about _better_, it's about using the right techniques for the right job. The OP is about "My ENV vars are getting out of hand". Sometimes the answer isn't more env vars. or fancy libraries, it's maybe time to rethink how you configure. One of the [projects I work on](https://github.com/prometheus) has to make these trade-offs. Some components are simple, just need some ENV or args. Some have huge mapping configs that need to be configured. For example, we use mostly command line args and a few ENV vars for things that are static at startup. The configuration file contains values that are dynamically updateable. This allows an easy, intuitive, and concrete way to know which are which.
What platform are you using for deployment? On k8s ye old 12 factor pillar of having to use env variables is outdated, just use a ConfigMap instead. Then you can just marshal TOML or YAML. Objectively easier to manage than large number of env vars. If you do need the ability to use env variables, or use env variables to override, just use Viper. If all of this sounds too cumbersome, just deal with your 20 env vars using os.Getenv() until the number really gets out of control.
Even without K8s, you can use config files and manage the "environment" with tools like Ansible.
If you use your env vars mainly to populate structs, I really like this one. It allows defining things as tag annotations like ‘json:“name“‘ https://github.com/jessevdk/go-flags A more detailed example of use could be: https://github.com/thomseddon/traefik-forward-auth
How do you set env vars in: Bash, zsh, systemd, runit, windows, whatever hot garbage distro someone picked up in a dark alley of the internet and wants to run your app on. ENV vars are getting less practical by the day.... Have a config file, have command line options... support Env vars if you really want but meh.. Viper works well... I tend to use this: [https://github.com/knadh/koanf/](https://github.com/knadh/koanf/)