- docs(podman): add usage for rootless mode on macOS (#1314)
### Changed
- **Potentially Breaking**: For people using k3d as a module: switch from netaddr.af to netipx + netip (changed some code around `host.k3d.internal` and the docker runtime)
- **Potentially Breaking**: K3d config directory may change for you: Adhere to XDG's configuration specification (#1320)
**Note**: This is the **first independent release** of k3d
- k3d moved from rancher/k3d to k3d-io/k3d
- k3d is fully community-owned
- k3d does not depend on any company's toolchain or accounts
**Note 2**: You can now fund the work on k3d using GitHub Sponsors ([@iwilltry42](https://github.com/sponsors/iwilltry42)) or IssueHunt ([k3d-io/k3d](https://issuehunt.io/r/k3d-io/k3d))
### Added
- GitHub Actions Release Workflow (#977 Ѐ)
- Replaces DroneCI
- Now uses `buildx`&`buildx bake` for multiplatform builds (instead of VMs with the according architectures)
- Now pushes to GHCR instead of DockerHub
- docs: added FAQ entry on using Longhorn in k3d
- docs: added config file tip that k3d expands environment variables
- docs: added section about using k3d with Podman (#987)
- docs: add connect section on homepage (#988)
- added `k3d node create --k3s-arg` flag (#1032)
### Changed
- references to rancher/k3d updated to k3d-io/k3d (#976)
- reference to rancher/k3s updated to k3s-io/k3s (#985)
- explicitly set `bridge` mode for k3d-created networks for Podman compatibility (#986)
- use secure defaults for curl in install script (#999)
- chore: update docs requirements and re-run docgen for commands (#1033)
- change: no default image for node creation in local cluster where image should be copied from existing nodes (#1034)
### Fixed
- fixed volume shortcuts not working because clusterconfig was not being processed
- fixed AUR Release pipeline with more relaxed version selection (#966)
- fixed ZSH completion output (#1014)
- Do not defer goroutine to delete tools node, as this leads to errors
- Hotfix: switch default for image import to original tools-node mode, as the new direct mode fails fairly often
- GetGatewayIP for host.k3d.internal should error out if there's no gateway defined (#1027)
- Store hostAliases in label to persist them across cluster stop/start (#1029)
### Deprecated
- DockerHub Images: k3d's images will now be pushed to GHCR under <https://github.com/orgs/k3d-io/packages?repo_name=k3d>
This release was [automatically tested](https://github.com/k3d-io/k3d/actions/runs/2044325827) with the following setups:
#### Docker
- 20.10.5
- 20.10.12
**Expected to Fail** with the following versions:
-<= 20.10.4 (due to runc, see <https://github.com/rancher/k3d/issues/807>)
#### K3s
We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel:
- Channel v1.23
- Channel v1.22
**Expected to Fail** with the following versions:
-<= v1.18 (due to not included, but expected CoreDNS in K3s)
- new config options to configure extra hosts by @iwilltry42 in <https://github.com/rancher/k3d/pull/938>
- host pid mode support for k3s-server and k3s-agent by @hlts2 in <https://github.com/rancher/k3d/pull/929>
- SimpleConfig v1alpha4 by @iwilltry42 in <https://github.com/rancher/k3d/pull/944>
- add env var LOG_COLORS=[1|true|0|false] to toggle colored log output (enabled by default) by @iwilltry42 in <https://github.com/rancher/k3d/pull/951>
- Compatibility Tests by @iwilltry42 in <https://github.com/rancher/k3d/pull/956>
- Volume Shortcuts and k3d-managed volumes by @iwilltry42 in <https://github.com/rancher/k3d/pull/916>
- Use some destination shortcuts with the `--volume/-v` flag that k3d automatically expands
-`k3s-manifests-custom` -> `/var/lib/rancher/k3s/server/manifests/custom` (not K3s default: this is just some sub-directory inside the auto-deploy manifests directory which will also be parsed)
-`k3s-containerd` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml` (use with caution, K3s generates this file!)
-`k3s-containerd-tmpl` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl` (used by K3s to generate the real config above)
-`k3s-registry-config` -> `/etc/rancher/k3s/registries.yaml` (or just use `--registry-config`)
- k3d-managed volumes
- non-existing named volumes starting with a `k3d-` prefix will now be created and managed by `k3d`
- JSON schema versions in-repo to link to from schemastore.org by @iwilltry42 in <https://github.com/rancher/k3d/pull/942>
We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel:
- simplified way of getting a Docker API Client that works with Docker Contexts and `DOCKER_*` environment variable configuration (#829, @dragonflylee)
- fix: didn't honor `DOCKER_TLS` environment variables before
**Info**: <https://k3d.io> is now versioned, so you can checkout different versions of the documentation by using the dropdown menu in the page title bar!
- dropped the usage of square brackets `[]` for indexing, as it caused problems with some shells trying to interpret them
- new syntax: `@identifier[:index][:opt]` (see <https://github.com/rancher/k3d/discussions/652>)
- example for a port-mapping: `--port 8080:80@server:0:proxy`
- identifier = `server`, index = `0`, opt = `proxy`
-`opt` is an extra optional argument used for different purposes depending on the flag
- currently, only the `--port` flag has `opt`s, namely `proxy` and `direct` (see other breaking change)
- port-mapping now go via the loadbalancer (serverlb) by default
- the `--port` flag has the `proxy` opt (see new nodefilter syntax above) set by default
- to leverage the old behavior of direct port-mappings, use the `direct` opt on the port flag
- the nodefilter `loadbalancer` will now do the same as `servers:*;agents:*` (proxied via the loadbalancer)
- flag `--registries-create` transformed from bool flag to string flag: let's you define the name and port-binding of the newly created registry, e.g. `--registry-create myregistry.localhost:5001`
- new command: `k3d node edit` to edit existing nodes (#615)
- currently only allows `k3d node edit NODE --port-add HOSTPORT:CONTAINERPORT` for the serverlb/loadbalancer to add new ports
- pkg: new `NodeEdit` function
- new (hidden) command: `k3d debug` with some options for debugging k3d resources (#638)
- e.g. `k3d debug loadbalancer get-config` to get the current loadbalancer configuration
- loadbalancer / k3d-proxy (#638)
- updated fork of `confd` to make usage of the file backend including a file watcher for auto-reloads
- this also checks the config before applying it, so the lb doesn't crash on a faulty config
- updating the loadbalancer writes the new config file and also checks if everything's going fine afterwards
- some settings of the loadbalancer can now be configured using `--lb-config-override`, see docs at <https://k3d.io/v5.0.0/design/defaults/#k3d-loadbalancer>
- helper images can now be set explicitly via environment variables: `K3D_IMAGE_LOADBALANCER`&`K3D_IMAGE_TOOLS` (#638)
- concurrently add new nodes to an existing cluster (remove some dumb code) (#640)
-`--wait` is now the default for `k3d node create`
- normalized flag usage for k3s and runtime (#598, @ejose19)
- rename `k3d cluster create --label` to `k3d cluster create --runtime-label` (as it's labelling the node on runtime level, e.g. docker)
- config option moved to `options.runtime.labels`
- add `k3d cluster create --k3s-node-label` to add Kubernetes node labels via k3s flag (#584, @developer-guy, @ejose, @dentrax)
- new config option `options.k3s.nodeLabels`
- the same for `k3d node create`
- improved config file handling (#605)
- new version `v1alpha3`
- warning when using outdated version
- validation dynamically based on provided config apiVersion
- new default for `k3d config init`
- new command `k3d config migrate INPUT [OUTPUT]` to migrate config files between versions
- currently supported migration `v1alpha2` -> `v1alpha3`
- pkg: new `Config` interface type to support new generic `FromViper` config file parsing
- changed flags `--k3s-server-arg`&`--k3s-agent-arg` into `--k3s-arg` with nodefilter support (#605)
- new config path `options.k3s.extraArgs`
- config file: environment variables (`$VAR`, `${VAR}` will be expanded unconditionally) (#643)
- docker context support (#601, @developer-guy ʢ)
- Feature flag using the environment variable `K3D_FIX_DNS` and setting it to a true value (e.g. `export K3D_FIX_DNS=1`) to forward DNS queries to your local machine, e.g. to use your local company DNS
- tests/e2e: timeouts everywhere to avoid killing DroneCI (#638)
- logs: really final output when creating/deleting nodes (so far, we were not outputting a final success message and the process was still doing stuff) (#640)
- tests/e2e: add tests for v1alpha2 to v1alpha3 migration
- Improved DroneCI Pipeline for Multiarch Images and SemVer Tags (#712)
- **Important**: New images will not have the `v` prefix in the tag anymore!
- but now real releases will use the "hierarchical" SemVer tags, so you could e.g. subscribe to rancher/k3d-proxy:4 to get v4.x.x images for the proxy container
- to debug the issue mentioned above, we introduced a new environment variable `K3D_LOG_NODE_WAIT_LOGS`, which can be set to a list of node roles (e.g. `K3D_LOG_NODE_WAIT_LOGS=loadbalancer,agent`) to output the container logs that k3d inspects
- cgroupv2 support: to properly work on cgroupv2 systems, k3s has to move all the processes from the root cgroup to a new /init cgroup and enable subtree_control
- this is going to be included in the k3s agent code directly (<https://github.com/k3s-io/k3s/pull/3242>)
- for now we're overriding the container entrypoint with a script that does this (#579, compare <https://github.com/k3s-io/k3s/pull/3237>)
- thanks a lot for all the input and support @AkihiroSuda
- **Usage**: set the environment variable `K3D_FIX_CGROUPV2` to a `true` value before/when creating a cluster with k3d
- ci/drone: always build no matter the branch name (just not release)
- docs: add automatic command tree generation via cobra (#562)
- makefile: use `go env gopath` as install target for tools (as per #445)
- JSONSchema: add some examples and defaults (now also available via <https://raw.githubusercontent.com/rancher/k3d/main/pkg/config/v1alpha2/schema.json> in your IDE)
- use the new `//go:embed` feature to directly embed the jsonschema in the binary (#529)
- Add a status column to `k3d registry list` output (#496, @ebr)
- Allow non-prefixed (i.e. without `k3d-` prefix) user input when fetching resources (e.g. `k3d node get mycluster-server-0` would return successfully)
- docs: fix typo in stop command help text (#513, @searsaw)
- ci/ghaction: AUR (pre-)release now on Ubuntu 20.04 and latest archlinux image
- REMOVE incomplete and unused `containerd` runtime from codebase, as it was causing issues to build for windows and hasn't made any progress in quite some time now
- automatic multi-server cluster restarts tend to fail with k3s versions v1.17.x & v1.18.x and probably earlier versions (using dqlite)
- Using Viper brings us lots of nice features, but also one problem:
- We had to switch StringArray flags to StringSlice flags, which
- allow to use multiple flag values comma-separated in a single flag, but also
- split flag values that contain a comma into separate parts (and we cannot handle issues that arise due to this)
- so if you rely on commas in your flag values (e.g. for `--env X=a,b,c`), please consider filing an issue or supporting <https://github.com/spf13/viper/issues/246> and <https://github.com/spf13/viper/pull/398>
-`--env X=a,b,c` would be treated the same as `--env X=a`, `--env b`, `--env c`
- Check the documentation, help text and tutorials for more details
- Communicate managed registry using the LocalRegistryHostingV1 spec from [KEP-1755](https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md)
- interesting especially for tools that reload images, like Tilt or Skaffold
- **Config File Support**
- Put all your CLI-Arguments/Flags into a more readable config file and re-use it everywhere (keep it in your repo)
- Note: this is not always a 1:1 matching in naming/syntax/semantics
- Check out our test cases in [pkg/config/test_assets/](./pkg/config/test_assets/) for more config file examples
- **Note**: The config file format (& feature) might still be a little rough around the edges and it's prone to change quickly until we hit a stable release of the config