* add env var LOG_COLORS=[1|true|0|false] to toggle colored log output (enabled by default) by @iwilltry42 in <https://github.com/rancher/k3d/pull/951>
- `k3s-manifests-custom` -> `/var/lib/rancher/k3s/server/manifests/custom` (not K3s default: this is just some sub-directory inside the auto-deploy manifests directory which will also be parsed)
* Compatibility Tests by @iwilltry42 in <https://github.com/rancher/k3d/pull/956>
- `k3s-containerd` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml` (use with caution, K3s generates this file!)
* Volume Shortcuts and k3d-managed volumes by @iwilltry42 in <https://github.com/rancher/k3d/pull/916>
- `k3s-containerd-tmpl` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl` (used by K3s to generate the real config above)
* Use some destination shortcuts with the `--volume/-v` flag that k3d automatically expands
- `k3s-registry-config` -> `/etc/rancher/k3s/registries.yaml` (or just use `--registry-config`)
- non-existing named volumes starting with a `k3d-` prefix will now be created and managed by `k3d`
* `k3s-manifests-custom` -> `/var/lib/rancher/k3s/server/manifests/custom` (not K3s default: this is just some sub-directory inside the auto-deploy manifests directory which will also be parsed)
* `k3s-containerd` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml` (use with caution, K3s generates this file!)
* `k3s-containerd-tmpl` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl` (used by K3s to generate the real config above)
* `k3s-registry-config` -> `/etc/rancher/k3s/registries.yaml` (or just use `--registry-config`)
* k3d-managed volumes
* non-existing named volumes starting with a `k3d-` prefix will now be created and managed by `k3d`
* JSON schema versions in-repo to link to from schemastore.org by @iwilltry42 in <https://github.com/rancher/k3d/pull/942>
### Changed
* Config file compatible with Kustomize by @erikgb in <https://github.com/rancher/k3d/pull/945>
* chore: update direct dependencies by @iwilltry42 in <https://github.com/rancher/k3d/pull/935>
### Fixed
* serverlb should be created before using and restarted unless stopped by @wymli in <https://github.com/rancher/k3d/pull/948>
* fix typo in node.go by @eltociear in <https://github.com/rancher/k3d/pull/949>
* mutex on ensureToolsNode to avoid duplicate container name causing error by @iwilltry42 in <https://github.com/rancher/k3d/pull/952>
* detect '--disable=coredns' and conditionally disable injection by @iwilltry42 in <https://github.com/rancher/k3d/pull/955>
* invert logic for LOG_LEVEL parsing by @myitcv in <https://github.com/rancher/k3d/pull/958>
### Deprecated
* SimpleConfig API version `k3d.io/v1alpha3` is now deprecated in favor of `k3d.io/v1alpha4`
### Removed
### Removed
- unused volume validation functionality in `cmd/util`, does not affect the CLI (#916)
* unused volume validation functionality in `cmd/util`, does not affect the CLI (#916)
### Compatibility
This release was automatically tested with the following setups:
#### Docker
* 20.10.5
* 20.10.12
**Expected to Fail** with the following versions:
* <= 20.10.4 (due to runc, see <https://github.com/rancher/k3d/issues/807>)
#### K3s
We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel:
* Channel v1.23
* Channel v1.22
**Expected to Fail** with the following versions:
* <= v1.18 (due to not included, but expected CoreDNS in K3s)
## v5.2.2
## v5.2.2
### Fixes
### Fixes
- mitigate issue when importing images from multiple tars (#881, @sbaier1)
* mitigate issue when importing images from multiple tars (#881, @sbaier1)
- fix: cluster delete should not fail if no cluster was found by config file (#886, @kuritka)
* fix: cluster delete should not fail if no cluster was found by config file (#886, @kuritka)
### Misc
### Misc
- docs: new page about k3d concepts, incl. nodefilters (#888)
* docs: new page about k3d concepts, incl. nodefilters (#888)
- cluster network: reserve IP extra IP for k3d-tools container in k3d-managed IPAM to avoid conflicts
* cluster network: reserve IP extra IP for k3d-tools container in k3d-managed IPAM to avoid conflicts
- process the SimpleConfig before validating it to avoid early exit in hostnetwork mode (#860)
* process the SimpleConfig before validating it to avoid early exit in hostnetwork mode (#860)
- error out if `K3D_FIX_DNS=1` is set and user tries to mount a file to `/etc/resolv.conf` (conflict)
* error out if `K3D_FIX_DNS=1` is set and user tries to mount a file to `/etc/resolv.conf` (conflict)
- clusterStart: only run actions which are necessary given the start reason (e.g. `cluster start` vs. `cluster create`)
* clusterStart: only run actions which are necessary given the start reason (e.g. `cluster start` vs. `cluster create`)
- fix injection of `host.k3d.internal` based on resolving `host.docker.internal` (#872)
* fix injection of `host.k3d.internal` based on resolving `host.docker.internal` (#872)
- also now uses `host.docker.internal` in kubeconfig based on certain conditions (see PR)
* also now uses `host.docker.internal` in kubeconfig based on certain conditions (see PR)
### Misc
### Misc
- tests/e2e: parellelize and cleanup tests -> cut execution speed in half (#848 ͑)
* tests/e2e: parellelize and cleanup tests -> cut execution speed in half (#848 ͑)
- also run some make targets in parallel
* also run some make targets in parallel
- new env var `E2E_PARALLEL=<int>` to configure parallelism
* new env var `E2E_PARALLEL=<int>` to configure parallelism
- test output is now redirected to files inside the runner and only the logs of failed tests will later be output
* test output is now redirected to files inside the runner and only the logs of failed tests will later be output
- Update dependencies, including docker, containerd & k8s
* Update dependencies, including docker, containerd & k8s
- docs: clarify usage of local registries with k3d
* docs: clarify usage of local registries with k3d
- docs: fix port numbers in registry usage guide
* docs: fix port numbers in registry usage guide
### Notes
### Notes
- k3d v5.x.x requires at least docker version 20.10.4
* k3d v5.x.x requires at least docker version 20.10.4
## v5.1.0
## v5.1.0
### Features
### Features
- clusterCreate: `--image` option (also in config file) magic words to follow K3s channels (#841)
* clusterCreate: `--image` option (also in config file) magic words to follow K3s channels (#841)
-`latest`/`stable` to follow latest/stable channels of K3s
*`latest`/`stable` to follow latest/stable channels of K3s
-`+<channel>` (prefix `+`) where `<channel>` can as well be `latest` or `stable`, but also e.g. `v1.21`
*`+<channel>` (prefix `+`) where `<channel>` can as well be `latest` or `stable`, but also e.g. `v1.21`
- k3d will then check the K3s channel server to get the latest image for that channel
* k3d will then check the K3s channel server to get the latest image for that channel
### Enhancements
### Enhancements
- nodeHooks: add descriptions and log them for more verbosity (#843)
* nodeHooks: add descriptions and log them for more verbosity (#843)
-`node create`: inject `host.k3d.internal` into `/etc/hosts` similar to the `cluster create` command (#843)
*`node create`: inject `host.k3d.internal` into `/etc/hosts` similar to the `cluster create` command (#843)
### Fix
### Fix
-`--network host`: do not do any network magic (like `host.k3d.internal` injection, etc.) when `host` network is used (#844)
*`--network host`: do not do any network magic (like `host.k3d.internal` injection, etc.) when `host` network is used (#844)
### Misc
### Misc
- CI/Makefile: build with `-mod vendor`
* CI/Makefile: build with `-mod vendor`
- docs: document using some K3s features in k3d, including `servicelb`, `traefik`, `local-storage-provisioner` and `coredns` (#845)
* docs: document using some K3s features in k3d, including `servicelb`, `traefik`, `local-storage-provisioner` and `coredns` (#845)
## v5.0.3
## v5.0.3
### Enhancements & Fixes
### Enhancements & Fixes
- simplified way of getting a Docker API Client that works with Docker Contexts and `DOCKER_*` environment variable configuration (#829, @dragonflylee)
* simplified way of getting a Docker API Client that works with Docker Contexts and `DOCKER_*` environment variable configuration (#829, @dragonflylee)
- fix: didn't honor `DOCKER_TLS` environment variables before
* fix: didn't honor `DOCKER_TLS` environment variables before
## v5.0.2
## v5.0.2
### Enhancements
### Enhancements
- CoreDNS Configmap is now edited in the auto-deploy manifest on disk instead of relying on `kubectl patch` command (#814)
* CoreDNS Configmap is now edited in the auto-deploy manifest on disk instead of relying on `kubectl patch` command (#814)
- refactor: add cmd subcommands in a single function call (#819, @moeryomenko)
* refactor: add cmd subcommands in a single function call (#819, @moeryomenko)
- handle ready-log-messages by type and intent & check them in single log streams instead of checking whole chunks every time (#818)
* handle ready-log-messages by type and intent & check them in single log streams instead of checking whole chunks every time (#818)
### Fixes
### Fixes
- fix: config file check failing with env var expansion because unexpanded input file was checked
* fix: config file check failing with env var expansion because unexpanded input file was checked
### Misc
### Misc
- cleanup: ensure that connections/streams are closed once unused (#818)
* cleanup: ensure that connections/streams are closed once unused (#818)
- cleanup: split type definitions across multiple files to increase readability (#818)
* cleanup: split type definitions across multiple files to increase readability (#818)
- docs: clarify `node create` help text about cluster reference (#808, @losinggeneration)
* docs: clarify `node create` help text about cluster reference (#808, @losinggeneration)
- refactor: move from io/ioutil (deprecated) to io and os packages (#827, @Juneezee)
* refactor: move from io/ioutil (deprecated) to io and os packages (#827, @Juneezee)
## v5.0.1
## v5.0.1
### Enhancement
### Enhancement
- add `HostFromClusterNetwork` field to `LocalRegistryHosting` configmap as per KEP-1755 (#754)
* add `HostFromClusterNetwork` field to `LocalRegistryHosting` configmap as per KEP-1755 (#754)
### Fixes
### Fixes
- fix: nilpointer exception on failed exec process with no returned logreader
* fix: nilpointer exception on failed exec process with no returned logreader
- make post-create cluster preparation (DNS stuff mostly) more resilient (#780)
* make post-create cluster preparation (DNS stuff mostly) more resilient (#780)
- fix v1alpha2 -> v1alpha3 config migration (and other related issues) (#799)
* fix v1alpha2 -> v1alpha3 config migration (and other related issues) (#799)
### Misc
### Misc
- docs: fix typo (#784)
* docs: fix typo (#784)
- docs: fix usage of legacy `--k3s-agent/server-arg` flag
* docs: fix usage of legacy `--k3s-agent/server-arg` flag
## v5.0.0
## v5.0.0
@ -164,269 +211,269 @@ The demo repository has also been updated to work with k3d v5: <https://github.c
### Breaking Changes
### Breaking Changes
- new syntax for nodefilters
* new syntax for nodefilters
- dropped the usage of square brackets `[]` for indexing, as it caused problems with some shells trying to interpret them
* dropped the usage of square brackets `[]` for indexing, as it caused problems with some shells trying to interpret them
- new syntax: `@identifier[:index][:opt]` (see <https://github.com/rancher/k3d/discussions/652>)
* new syntax: `@identifier[:index][:opt]` (see <https://github.com/rancher/k3d/discussions/652>)
- example for a port-mapping: `--port 8080:80@server:0:proxy`
* example for a port-mapping: `--port 8080:80@server:0:proxy`
- identifier = `server`, index = `0`, opt = `proxy`
* identifier = `server`, index = `0`, opt = `proxy`
-`opt` is an extra optional argument used for different purposes depending on the flag
*`opt` is an extra optional argument used for different purposes depending on the flag
- currently, only the `--port` flag has `opt`s, namely `proxy` and `direct` (see other breaking change)
* currently, only the `--port` flag has `opt`s, namely `proxy` and `direct` (see other breaking change)
- port-mapping now go via the loadbalancer (serverlb) by default
* port-mapping now go via the loadbalancer (serverlb) by default
- the `--port` flag has the `proxy` opt (see new nodefilter syntax above) set by default
* the `--port` flag has the `proxy` opt (see new nodefilter syntax above) set by default
- to leverage the old behavior of direct port-mappings, use the `direct` opt on the port flag
* to leverage the old behavior of direct port-mappings, use the `direct` opt on the port flag
- the nodefilter `loadbalancer` will now do the same as `servers:*;agents:*` (proxied via the loadbalancer)
* the nodefilter `loadbalancer` will now do the same as `servers:*;agents:*` (proxied via the loadbalancer)
- flag `--registries-create` transformed from bool flag to string flag: let's you define the name and port-binding of the newly created registry, e.g. `--registry-create myregistry.localhost:5001`
* flag `--registries-create` transformed from bool flag to string flag: let's you define the name and port-binding of the newly created registry, e.g. `--registry-create myregistry.localhost:5001`
### Fixes
### Fixes
- cleaned up and properly sorted the sanitization of existing resources used to create new nodes (#638)
* cleaned up and properly sorted the sanitization of existing resources used to create new nodes (#638)
### Features & Enhancements
### Features & Enhancements
- new command: `k3d node edit` to edit existing nodes (#615)
* new command: `k3d node edit` to edit existing nodes (#615)
- currently only allows `k3d node edit NODE --port-add HOSTPORT:CONTAINERPORT` for the serverlb/loadbalancer to add new ports
* currently only allows `k3d node edit NODE --port-add HOSTPORT:CONTAINERPORT` for the serverlb/loadbalancer to add new ports
- pkg: new `NodeEdit` function
* pkg: new `NodeEdit` function
- new (hidden) command: `k3d debug` with some options for debugging k3d resources (#638)
* new (hidden) command: `k3d debug` with some options for debugging k3d resources (#638)
- e.g. `k3d debug loadbalancer get-config` to get the current loadbalancer configuration
* e.g. `k3d debug loadbalancer get-config` to get the current loadbalancer configuration
- loadbalancer / k3d-proxy (#638)
* loadbalancer / k3d-proxy (#638)
- updated fork of `confd` to make usage of the file backend including a file watcher for auto-reloads
* updated fork of `confd` to make usage of the file backend including a file watcher for auto-reloads
- this also checks the config before applying it, so the lb doesn't crash on a faulty config
* this also checks the config before applying it, so the lb doesn't crash on a faulty config
- updating the loadbalancer writes the new config file and also checks if everything's going fine afterwards
* updating the loadbalancer writes the new config file and also checks if everything's going fine afterwards
- some settings of the loadbalancer can now be configured using `--lb-config-override`, see docs at <https://k3d.io/v5.0.0/design/defaults/#k3d-loadbalancer>
* some settings of the loadbalancer can now be configured using `--lb-config-override`, see docs at <https://k3d.io/v5.0.0/design/defaults/#k3d-loadbalancer>
- helper images can now be set explicitly via environment variables: `K3D_IMAGE_LOADBALANCER`&`K3D_IMAGE_TOOLS` (#638)
* helper images can now be set explicitly via environment variables: `K3D_IMAGE_LOADBALANCER`&`K3D_IMAGE_TOOLS` (#638)
- concurrently add new nodes to an existing cluster (remove some dumb code) (#640)
* concurrently add new nodes to an existing cluster (remove some dumb code) (#640)
-`--wait` is now the default for `k3d node create`
*`--wait` is now the default for `k3d node create`
- normalized flag usage for k3s and runtime (#598, @ejose19)
* normalized flag usage for k3s and runtime (#598, @ejose19)
- rename `k3d cluster create --label` to `k3d cluster create --runtime-label` (as it's labelling the node on runtime level, e.g. docker)
* rename `k3d cluster create --label` to `k3d cluster create --runtime-label` (as it's labelling the node on runtime level, e.g. docker)
- config option moved to `options.runtime.labels`
* config option moved to `options.runtime.labels`
- add `k3d cluster create --k3s-node-label` to add Kubernetes node labels via k3s flag (#584, @developer-guy, @ejose, @dentrax)
* add `k3d cluster create --k3s-node-label` to add Kubernetes node labels via k3s flag (#584, @developer-guy, @ejose, @dentrax)
- new config option `options.k3s.nodeLabels`
* new config option `options.k3s.nodeLabels`
- the same for `k3d node create`
* the same for `k3d node create`
- improved config file handling (#605)
* improved config file handling (#605)
- new version `v1alpha3`
* new version `v1alpha3`
- warning when using outdated version
* warning when using outdated version
- validation dynamically based on provided config apiVersion
* validation dynamically based on provided config apiVersion
- new default for `k3d config init`
* new default for `k3d config init`
- new command `k3d config migrate INPUT [OUTPUT]` to migrate config files between versions
* new command `k3d config migrate INPUT [OUTPUT]` to migrate config files between versions
- currently supported migration `v1alpha2` -> `v1alpha3`
* currently supported migration `v1alpha2` -> `v1alpha3`
- pkg: new `Config` interface type to support new generic `FromViper` config file parsing
* pkg: new `Config` interface type to support new generic `FromViper` config file parsing
- changed flags `--k3s-server-arg`&`--k3s-agent-arg` into `--k3s-arg` with nodefilter support (#605)
* changed flags `--k3s-server-arg`&`--k3s-agent-arg` into `--k3s-arg` with nodefilter support (#605)
- new config path `options.k3s.extraArgs`
* new config path `options.k3s.extraArgs`
- config file: environment variables (`$VAR`, `${VAR}` will be expanded unconditionally) (#643)
* config file: environment variables (`$VAR`, `${VAR}` will be expanded unconditionally) (#643)
- docker context support (#601, @developer-guy ʢ)
* docker context support (#601, @developer-guy ʢ)
- Feature flag using the environment variable `K3D_FIX_DNS` and setting it to a true value (e.g. `export K3D_FIX_DNS=1`) to forward DNS queries to your local machine, e.g. to use your local company DNS
* Feature flag using the environment variable `K3D_FIX_DNS` and setting it to a true value (e.g. `export K3D_FIX_DNS=1`) to forward DNS queries to your local machine, e.g. to use your local company DNS
### Misc
### Misc
- tests/e2e: timeouts everywhere to avoid killing DroneCI (#638)
* tests/e2e: timeouts everywhere to avoid killing DroneCI (#638)
- logs: really final output when creating/deleting nodes (so far, we were not outputting a final success message and the process was still doing stuff) (#640)
* logs: really final output when creating/deleting nodes (so far, we were not outputting a final success message and the process was still doing stuff) (#640)
- tests/e2e: add tests for v1alpha2 to v1alpha3 migration
* tests/e2e: add tests for v1alpha2 to v1alpha3 migration
- docs: use v1alpha3 config version
* docs: use v1alpha3 config version
- docs: update general appearance and cleanup
* docs: update general appearance and cleanup
## v4.4.8
## v4.4.8
## Enhancements
## Enhancements
- Improved DroneCI Pipeline for Multiarch Images and SemVer Tags (#712)
* Improved DroneCI Pipeline for Multiarch Images and SemVer Tags (#712)
- **Important**: New images will not have the `v` prefix in the tag anymore!
* **Important**: New images will not have the `v` prefix in the tag anymore!
- but now real releases will use the "hierarchical" SemVer tags, so you could e.g. subscribe to rancher/k3d-proxy:4 to get v4.x.x images for the proxy container
* but now real releases will use the "hierarchical" SemVer tags, so you could e.g. subscribe to rancher/k3d-proxy:4 to get v4.x.x images for the proxy container
## Fixes
## Fixes
- clusterCreate: do not override hostIP if hostPort is missing (#693, @lukaszo)
* clusterCreate: do not override hostIP if hostPort is missing (#693, @lukaszo)
- imageImport: import all listed images, not only the first one (#701, @mszostok)
* imageImport: import all listed images, not only the first one (#701, @mszostok)
- clusterCreate: when memory constraints are set, only pull the image used for checking the edac folder, if it's not present on the machine
* clusterCreate: when memory constraints are set, only pull the image used for checking the edac folder, if it's not present on the machine
- fix: update k3d-tools dependencies and use API Version Negotiation, so it still works with older versions of the Docker Engine (#679)
* fix: update k3d-tools dependencies and use API Version Negotiation, so it still works with older versions of the Docker Engine (#679)
### Misc
### Misc
- install script: add darwin/arm64 support (#676, @colelawrence)
* install script: add darwin/arm64 support (#676, @colelawrence)
- new flag: `k3d image import --keep-tools` to not delete the tools node container after importing the image(s) (#672)
* new flag: `k3d image import --keep-tools` to not delete the tools node container after importing the image(s) (#672)
- improve image name handling when importing images (#653, @cimnine)
* improve image name handling when importing images (#653, @cimnine)
- normalize image names internally, e.g. strip prefixes that docker adds, but that break the process
* normalize image names internally, e.g. strip prefixes that docker adds, but that break the process
- see <https://k3d.io/usage/commands/k3d_image_import/> for more info
* see <https://k3d.io/usage/commands/k3d_image_import/> for more info
### Fixes
### Fixes
- Use default gateway, when bridge network doesn't have it (#666, @kuritka)
* Use default gateway, when bridge network doesn't have it (#666, @kuritka)
- Start an existing, but not running tools node to re-use it when importing an image (#672)
* Start an existing, but not running tools node to re-use it when importing an image (#672)
### Misc
### Misc
- deps: switching back to upstream viper including the StringArray fix
* deps: switching back to upstream viper including the StringArray fix
- docs: reference to "nolar/setup-k3d-k3s" step for GitHub Actions (#668, @nolar)
* docs: reference to "nolar/setup-k3d-k3s" step for GitHub Actions (#668, @nolar)
- docs: updated and simplified CUDA guide (#662, @vainkop) (#669)
* docs: updated and simplified CUDA guide (#662, @vainkop) (#669)
## v4.4.6
## v4.4.6
### Fixes
### Fixes
- fix an issue where the cluster creation would stall waiting for the `starting worker processes` log message from the loadbalancer/serverlb
* fix an issue where the cluster creation would stall waiting for the `starting worker processes` log message from the loadbalancer/serverlb
- this was likely caused by a rounding issue when asking docker to get the container logs starting at a specific timestamp
* this was likely caused by a rounding issue when asking docker to get the container logs starting at a specific timestamp
- we now drop subsecond precision for this to avoid the rounding issue, which was confirmed to work
* we now drop subsecond precision for this to avoid the rounding issue, which was confirmed to work
- see issues #592ɭ
* see issues #592ɭ
### Misc
### Misc
- to debug the issue mentioned above, we introduced a new environment variable `K3D_LOG_NODE_WAIT_LOGS`, which can be set to a list of node roles (e.g. `K3D_LOG_NODE_WAIT_LOGS=loadbalancer,agent`) to output the container logs that k3d inspects
* to debug the issue mentioned above, we introduced a new environment variable `K3D_LOG_NODE_WAIT_LOGS`, which can be set to a list of node roles (e.g. `K3D_LOG_NODE_WAIT_LOGS=loadbalancer,agent`) to output the container logs that k3d inspects
## v4.4.5
## v4.4.5
### Fixes
### Fixes
- overall: use the getDockerClient helper function everywhere to e.g. support docker via ssh everywhere
* overall: use the getDockerClient helper function everywhere to e.g. support docker via ssh everywhere
- nodeCreate: do not copy meminfo/edac volume mounts from existing nodes, to avoid conflicts with generated mounts
* nodeCreate: do not copy meminfo/edac volume mounts from existing nodes, to avoid conflicts with generated mounts
- kubeconfig: fix file handling on windows (#626 + #628, @dragonflylee)
* kubeconfig: fix file handling on windows (#626 + #628, @dragonflylee)
### Misc
### Misc
- docs: add [FAQ entry](https://k3d.io/faq/faq/#nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied) on nf_conntrack_max: permission denied issue from kube-proxy (#607)
* docs: add [FAQ entry](https://k3d.io/faq/faq/#nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied) on nf_conntrack_max: permission denied issue from kube-proxy (#607)
- docs: cleanup, fix formatting, etc.
* docs: cleanup, fix formatting, etc.
- license: update to include 2021 in time range
* license: update to include 2021 in time range
- docs: link to AutoK3s (#614, @JacieChao)
* docs: link to AutoK3s (#614, @JacieChao)
- tests/e2e: update the list of tested k3s versions
* tests/e2e: update the list of tested k3s versions
## v4.4.4
## v4.4.4
### Enhancements
### Enhancements
- nodes created via `k3d node create` now inherit the registry config from existing nodes (if there is any) (#597)
* nodes created via `k3d node create` now inherit the registry config from existing nodes (if there is any) (#597)
- the cgroupv2 hotfix (custom entrypoint script) is now enabled by default (#603)
* the cgroupv2 hotfix (custom entrypoint script) is now enabled by default (#603)
- disable by setting the environment variable `K3D_FIX_CGROUPV2=false`
* disable by setting the environment variable `K3D_FIX_CGROUPV2=false`
### Fixes
### Fixes
- fix using networks without IPAM config (e.g. `host`)
* fix using networks without IPAM config (e.g. `host`)
### Misc
### Misc
- docs: edit links on k3d.io now point to the correct branch (`main`)
* docs: edit links on k3d.io now point to the correct branch (`main`)
- docs: new FAQ entry on spurious PID entries when using shared mounts (#609, @leelavg)
* docs: new FAQ entry on spurious PID entries when using shared mounts (#609, @leelavg)
## v4.4.3
## v4.4.3
### Highlights
### Highlights
- cgroupv2 support: to properly work on cgroupv2 systems, k3s has to move all the processes from the root cgroup to a new /init cgroup and enable subtree_control
* cgroupv2 support: to properly work on cgroupv2 systems, k3s has to move all the processes from the root cgroup to a new /init cgroup and enable subtree_control
- this is going to be included in the k3s agent code directly (<https://github.com/k3s-io/k3s/pull/3242>)
* this is going to be included in the k3s agent code directly (<https://github.com/k3s-io/k3s/pull/3242>)
- for now we're overriding the container entrypoint with a script that does this (#579, compare <https://github.com/k3s-io/k3s/pull/3237>)
* for now we're overriding the container entrypoint with a script that does this (#579, compare <https://github.com/k3s-io/k3s/pull/3237>)
- thanks a lot for all the input and support @AkihiroSuda
* thanks a lot for all the input and support @AkihiroSuda
- **Usage**: set the environment variable `K3D_FIX_CGROUPV2` to a `true` value before/when creating a cluster with k3d
* **Usage**: set the environment variable `K3D_FIX_CGROUPV2` to a `true` value before/when creating a cluster with k3d
- e.g. `export K3D_FIX_CGROUPV2=1`
* e.g. `export K3D_FIX_CGROUPV2=1`
### Fixes
### Fixes
- fix: docker volume not mountable due to validation failure
* fix: docker volume not mountable due to validation failure
- was not able to mount named volume on windows as we're checking for `:` meant for drive-letters and k3d separators
* was not able to mount named volume on windows as we're checking for `:` meant for drive-letters and k3d separators
- k3d-proxy: rename udp upstreams to avoid collisions/duplicates (#564)
* k3d-proxy: rename udp upstreams to avoid collisions/duplicates (#564)
### Features
### Features
- add *hidden* command `k3d runtime-info` used for debugging (#553)
* add *hidden* command `k3d runtime-info` used for debugging (#553)
- this comes with some additions on package/runtime level
* this comes with some additions on package/runtime level
- add *experimental*`--subnet` flag to get some k3d IPAM to ensure that server nodes keep static IPs across restarts (#560)
* add *experimental*`--subnet` flag to get some k3d IPAM to ensure that server nodes keep static IPs across restarts (#560)
### Misc
### Misc
- docs: fix typo (#556, @gcalmettes)
* docs: fix typo (#556, @gcalmettes)
- docs: fix typo (#561, @alechartung)
* docs: fix typo (#561, @alechartung)
- ci/drone: pre-release on `-dev.X` tags
* ci/drone: pre-release on `-dev.X` tags
- ci/drone: always build no matter the branch name (just not release)
* ci/drone: always build no matter the branch name (just not release)
- docs: add automatic command tree generation via cobra (#562)
* docs: add automatic command tree generation via cobra (#562)
- makefile: use `go env gopath` as install target for tools (as per #445)
* makefile: use `go env gopath` as install target for tools (as per #445)
- JSONSchema: add some examples and defaults (now also available via <https://raw.githubusercontent.com/rancher/k3d/main/pkg/config/v1alpha2/schema.json> in your IDE)
* JSONSchema: add some examples and defaults (now also available via <https://raw.githubusercontent.com/rancher/k3d/main/pkg/config/v1alpha2/schema.json> in your IDE)
## v4.4.1
## v4.4.1
### Fixes
### Fixes
- use viper fork that contains a fix to make cobra's `StringArray` flags work properly
* use viper fork that contains a fix to make cobra's `StringArray` flags work properly
- this fixes the issue, that flag values containing commas got split (because we had to use `StringSlice` type flags)
* this fixes the issue, that flag values containing commas got split (because we had to use `StringSlice` type flags)
- this is to be changed back to upstream viper as soon as <https://github.com/spf13/viper/pull/398> (or a similar fix) got merged
* this is to be changed back to upstream viper as soon as <https://github.com/spf13/viper/pull/398> (or a similar fix) got merged
## v4.4.0
## v4.4.0
### Features / Enhancements
### Features / Enhancements
- Support for Memory Limits using e.g. `--servers-memory 1g` or `--agents-memory 1.5g` (#494, @konradmalik)
* Support for Memory Limits using e.g. `--servers-memory 1g` or `--agents-memory 1.5g` (#494, @konradmalik)
- enabled by providing fake `meminfo` files
* enabled by providing fake `meminfo` files
### Fixes
### Fixes
- fix absolute paths in volume mounts on Windows (#510, @markrexwinkel)
* fix absolute paths in volume mounts on Windows (#510, @markrexwinkel)
### Documentation
### Documentation
- clarify registry names in docs and help text
* clarify registry names in docs and help text
- add usage section about config file (#534)
* add usage section about config file (#534)
- add FAQ entry on certificate error when running behind corporate proxy
* add FAQ entry on certificate error when running behind corporate proxy
- use the new `//go:embed` feature to directly embed the jsonschema in the binary (#529)
* use the new `//go:embed` feature to directly embed the jsonschema in the binary (#529)
- Add a status column to `k3d registry list` output (#496, @ebr)
* Add a status column to `k3d registry list` output (#496, @ebr)
- Allow non-prefixed (i.e. without `k3d-` prefix) user input when fetching resources (e.g. `k3d node get mycluster-server-0` would return successfully)
* Allow non-prefixed (i.e. without `k3d-` prefix) user input when fetching resources (e.g. `k3d node get mycluster-server-0` would return successfully)
### Fixes
### Fixes
- Allow absolute paths for volumes on Windows (#510, @markrexwinkel)
* Allow absolute paths for volumes on Windows (#510, @markrexwinkel)
- fix nil-pointer exception in case of non-existent IPAM network config
* fix nil-pointer exception in case of non-existent IPAM network config
- Properly handle combinations of host/hostIP in kubeAPI settings reflected in the kubeconfig (#500, @fabricev)
* Properly handle combinations of host/hostIP in kubeAPI settings reflected in the kubeconfig (#500, @fabricev)
### Misc
### Misc
- docs: fix typo in stop command help text (#513, @searsaw)
* docs: fix typo in stop command help text (#513, @searsaw)
- ci/ghaction: AUR (pre-)release now on Ubuntu 20.04 and latest archlinux image
* ci/ghaction: AUR (pre-)release now on Ubuntu 20.04 and latest archlinux image
- REMOVE incomplete and unused `containerd` runtime from codebase, as it was causing issues to build for windows and hasn't made any progress in quite some time now
* REMOVE incomplete and unused `containerd` runtime from codebase, as it was causing issues to build for windows and hasn't made any progress in quite some time now
## v4.2.0
## v4.2.0
### Features / Enhancements
### Features / Enhancements
- add processing step for cluster config, to configure it e.g. for hostnetwork mode (#477, @konradmalik)
* add processing step for cluster config, to configure it e.g. for hostnetwork mode (#477, @konradmalik)
- allow proxying UDP ports via the load balancer (#488, @k0da)
* allow proxying UDP ports via the load balancer (#488, @k0da)
### Fixes
### Fixes
- fix usage of `DOCKER_HOST` env var for Kubeconfig server ref (trim port)
* fix usage of `DOCKER_HOST` env var for Kubeconfig server ref (trim port)
- fix error when trying to attach the same node (e.g. registry) to the same network twice (#486, @kuritka)
* fix error when trying to attach the same node (e.g. registry) to the same network twice (#486, @kuritka)
- different log messages per role and start-place (that we wait for to consider a node to be ready)
* different log messages per role and start-place (that we wait for to consider a node to be ready)
- module: `NodeStartOpts` now accept a `ReadyLogMessage` and `NodeState` now takes a `Started` timestamp string
* module: `NodeStartOpts` now accept a `ReadyLogMessage` and `NodeState` now takes a `Started` timestamp string
### Fixes
### Fixes
- do not ignore `--no-hostip` flag and don't inject hostip if `--network=host` (#471, @konradmalik)
* do not ignore `--no-hostip` flag and don't inject hostip if `--network=host` (#471, @konradmalik)
- fix: `--no-lb` ignored
* fix: `--no-lb` ignored
- fix: print error cause when serverlb fails to start
* fix: print error cause when serverlb fails to start
### Misc
### Misc
- tests/e2e: add config override test
* tests/e2e: add config override test
- tests/e2e: add multi server start-stop cycle test
* tests/e2e: add multi server start-stop cycle test
- tests/e2e: improved logs with stage and test details.
* tests/e2e: improved logs with stage and test details.
- builds&tests: use Docker 20.10 and BuildKit everywhere
* builds&tests: use Docker 20.10 and BuildKit everywhere
- :memo: docs: add <https://github.com/AbsaOSS/k3d-action> (GitHub Action) as a related project (#476, @kuritka)
* :memo: docs: add <https://github.com/AbsaOSS/k3d-action> (GitHub Action) as a related project (#476, @kuritka)
### Tested with
### Tested with
- E2E Tests ran with k3s versions
* E2E Tests ran with k3s versions
- v1.17.17-k3s1 (see Known Issues below)
* v1.17.17-k3s1 (see Known Issues below)
- v1.18.15-k3s1 (see Known Issues below)
* v1.18.15-k3s1 (see Known Issues below)
- v1.19.7-k3s1
* v1.19.7-k3s1
- v1.20.2-k3s1
* v1.20.2-k3s1
### Known Issues
### Known Issues
- automatic multi-server cluster restarts tend to fail with k3s versions v1.17.x & v1.18.x and probably earlier versions (using dqlite)
* automatic multi-server cluster restarts tend to fail with k3s versions v1.17.x & v1.18.x and probably earlier versions (using dqlite)
- Using Viper brings us lots of nice features, but also one problem:
* Using Viper brings us lots of nice features, but also one problem:
- We had to switch StringArray flags to StringSlice flags, which
* We had to switch StringArray flags to StringSlice flags, which
- allow to use multiple flag values comma-separated in a single flag, but also
* allow to use multiple flag values comma-separated in a single flag, but also
- split flag values that contain a comma into separate parts (and we cannot handle issues that arise due to this)
* split flag values that contain a comma into separate parts (and we cannot handle issues that arise due to this)
- so if you rely on commas in your flag values (e.g. for `--env X=a,b,c`), please consider filing an issue or supporting <https://github.com/spf13/viper/issues/246> and <https://github.com/spf13/viper/pull/398>
* so if you rely on commas in your flag values (e.g. for `--env X=a,b,c`), please consider filing an issue or supporting <https://github.com/spf13/viper/issues/246> and <https://github.com/spf13/viper/pull/398>
-`--env X=a,b,c` would be treated the same as `--env X=a`, `--env b`, `--env c`
*`--env X=a,b,c` would be treated the same as `--env X=a`, `--env b`, `--env c`
## v4.0.0
## v4.0.0
@ -499,41 +546,41 @@ The demo repository has also been updated to work with k3d v5: <https://github.c
**If you're using k3d as a Go module, please have a look into the code to see all the changes!**
**If you're using k3d as a Go module, please have a look into the code to see all the changes!**
- We're open for chats via Slack or GitHub discussions
* We're open for chats via Slack or GitHub discussions
- Module is now on `github.com/rancher/k3d/v4` due to lots of breaking changes
* Module is now on `github.com/rancher/k3d/v4` due to lots of breaking changes
-`pkg/cluster` is now `pkg/client`
*`pkg/cluster` is now `pkg/client`
-`ClusterCreate` and `NodeCreate` don't start the entities (containers) anymore
*`ClusterCreate` and `NodeCreate` don't start the entities (containers) anymore
-`ClusterRun` and `NodeRun` orchestrate the new Create and Start functionality
*`ClusterRun` and `NodeRun` orchestrate the new Create and Start functionality
-`NodeDelete`/`ClusterDelete` now take an additional `NodeDeleteOpts`/`ClusterDeleteOpts` struct to toggle specific steps
*`NodeDelete`/`ClusterDelete` now take an additional `NodeDeleteOpts`/`ClusterDeleteOpts` struct to toggle specific steps
- NodeSpec now features a list of networks (required for registries)
* NodeSpec now features a list of networks (required for registries)
- Check the documentation, help text and tutorials for more details
* Check the documentation, help text and tutorials for more details
- Communicate managed registry using the LocalRegistryHostingV1 spec from [KEP-1755](https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md)
* Communicate managed registry using the LocalRegistryHostingV1 spec from [KEP-1755](https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md)
- interesting especially for tools that reload images, like Tilt or Skaffold
* interesting especially for tools that reload images, like Tilt or Skaffold
- **Config File Support**
* **Config File Support**
- Put all your CLI-Arguments/Flags into a more readable config file and re-use it everywhere (keep it in your repo)
* Put all your CLI-Arguments/Flags into a more readable config file and re-use it everywhere (keep it in your repo)
- Note: this is not always a 1:1 matching in naming/syntax/semantics
* Note: this is not always a 1:1 matching in naming/syntax/semantics
-`k3d cluster create --config myconfig.yaml`
*`k3d cluster create --config myconfig.yaml`
```yaml
```yaml
apiVersion: k3d.io/v1alpha1
apiVersion: k3d.io/v1alpha1
@ -547,30 +594,30 @@ The demo repository has also been updated to work with k3d v5: <https://github.c
- loadbalancer
- loadbalancer
```
```
- Check out our test cases in [pkg/config/test_assets/](./pkg/config/test_assets/) for more config file examples
* Check out our test cases in [pkg/config/test_assets/](./pkg/config/test_assets/) for more config file examples
- **Note**: The config file format (& feature) might still be a little rough around the edges and it's prone to change quickly until we hit a stable release of the config
* **Note**: The config file format (& feature) might still be a little rough around the edges and it's prone to change quickly until we hit a stable release of the config
- [WIP] Support for Lifecycle Hooks
* [WIP] Support for Lifecycle Hooks
- Run any executable at specific stages during the cluster and node lifecycles
* Run any executable at specific stages during the cluster and node lifecycles
- e.g. we modify the `registries.yaml` in the `preStart` stage of nodes
* e.g. we modify the `registries.yaml` in the `preStart` stage of nodes
- Guides will follow
* Guides will follow
- Print container creation time (#431, @inercia)
* Print container creation time (#431, @inercia)
- add output formats for `cluster ls` and `node ls` (#439, @inercia)
* add output formats for `cluster ls` and `node ls` (#439, @inercia)
#### Fixes
#### Fixes
- import image: avoid nil pointer exception in specific cases
* import image: avoid nil pointer exception in specific cases
- cluster delete: properly handle node and network (#437)
* cluster delete: properly handle node and network (#437)
- --port: fix bnil-pointer exception when exposing port on non-existent loadbalancer
* --port: fix bnil-pointer exception when exposing port on non-existent loadbalancer
- completion/zsh: source completion file
* completion/zsh: source completion file
#### Misc
#### Misc
- Now building with Go 1.15
* Now building with Go 1.15
- same for the k3d-tools code
* same for the k3d-tools code
- updated dependencies (including Docker v20.10)
* updated dependencies (including Docker v20.10)
- tests/e2e: add `E2E_INCLUDE` and rename `E2E_SKIP` to `E2E_EXCLUDE`
* tests/e2e: add `E2E_INCLUDE` and rename `E2E_SKIP` to `E2E_EXCLUDE`
- tests/e2e: allow overriding the Helper Image Tag via `E2E_HELPER_IMAGE_TAG`
* tests/e2e: allow overriding the Helper Image Tag via `E2E_HELPER_IMAGE_TAG`
With each release, we test if k3d works with specific versions of Docker and K3s, to ensure, that at least the most recent versions of Docker and the active releases (i.e. non-EOL release channels, similar to Kubernetes) work properly with it.
The tests happen automatically in GitHub Actions.
Some versions of Docker and K3s are expected to fail with specific versions of k3d due to e.g. incompatible dependencies or missing features.
We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel.
## Releases
### v5.3.0 - 03.02.2022
#### Docker
* 20.10.5
* 20.10.12
**Expected to Fail** with the following versions:
* <= 20.10.4 (due to runc, see <https://github.com/rancher/k3d/issues/807>)
#### K3s
* Channel v1.23
* Channel v1.22
**Expected to Fail** with the following versions:
* <= v1.18 (due to not included, but expected CoreDNS in K3s)
iwilltry42
3 years ago