From bb8a2b0bb57ad69680e4e2a0772a0bf31f35a457 Mon Sep 17 00:00:00 2001 From: iwilltry42 Date: Thu, 3 Feb 2022 13:40:42 +0100 Subject: [PATCH] v5.3.0: Changelog + Compatibility Docs --- CHANGELOG.md | 631 ++++++++++++++++++++------------------ docs/faq/compatibility.md | 28 ++ 2 files changed, 367 insertions(+), 292 deletions(-) create mode 100644 docs/faq/compatibility.md diff --git a/CHANGELOG.md b/CHANGELOG.md index e9ff1b6d..bceeaa8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,153 +1,200 @@ # Changelog -## v5.3.0 +## v5.3.0 - 03.02.2022 **Note:** Now trying to follow a standard scheme defined by ### Added -- Volumemount-Shortcuts (#916) - - Use some destination shortcuts with the `--volume/-v` flag that k3d automatically expands - - `k3s-storage` -> `/var/lib/rancher/k3s/storage` - - `k3s-manifests` -> `/var/lib/rancher/k3s/server/manifests` - - `k3s-manifests-custom` -> `/var/lib/rancher/k3s/server/manifests/custom` (not K3s default: this is just some sub-directory inside the auto-deploy manifests directory which will also be parsed) - - `k3s-containerd` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml` (use with caution, K3s generates this file!) - - `k3s-containerd-tmpl` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl` (used by K3s to generate the real config above) - - `k3s-registry-config` -> `/etc/rancher/k3s/registries.yaml` (or just use `--registry-config`) -- k3d-managed volumes (#916) - - non-existing named volumes starting with a `k3d-` prefix will now be created and managed by `k3d` +* new config options to configure extra hosts by @iwilltry42 in +* host pid mode support for k3s-server and k3s-agent by @hlts2 in +* SimpleConfig v1alpha4 by @iwilltry42 in +* add env var LOG_COLORS=[1|true|0|false] to toggle colored log output (enabled by default) by @iwilltry42 in +* Compatibility Tests by @iwilltry42 in +* Volume Shortcuts and k3d-managed volumes by @iwilltry42 in + * Use some destination shortcuts with the `--volume/-v` flag that k3d automatically expands + * `k3s-storage` -> `/var/lib/rancher/k3s/storage` + * `k3s-manifests` -> `/var/lib/rancher/k3s/server/manifests` + * `k3s-manifests-custom` -> `/var/lib/rancher/k3s/server/manifests/custom` (not K3s default: this is just some sub-directory inside the auto-deploy manifests directory which will also be parsed) + * `k3s-containerd` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml` (use with caution, K3s generates this file!) + * `k3s-containerd-tmpl` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl` (used by K3s to generate the real config above) + * `k3s-registry-config` -> `/etc/rancher/k3s/registries.yaml` (or just use `--registry-config`) + * k3d-managed volumes + * non-existing named volumes starting with a `k3d-` prefix will now be created and managed by `k3d` +* JSON schema versions in-repo to link to from schemastore.org by @iwilltry42 in + +### Changed + +* Config file compatible with Kustomize by @erikgb in +* chore: update direct dependencies by @iwilltry42 in + +### Fixed + +* serverlb should be created before using and restarted unless stopped by @wymli in +* fix typo in node.go by @eltociear in +* mutex on ensureToolsNode to avoid duplicate container name causing error by @iwilltry42 in +* detect '--disable=coredns' and conditionally disable injection by @iwilltry42 in +* invert logic for LOG_LEVEL parsing by @myitcv in + +### Deprecated + +* SimpleConfig API version `k3d.io/v1alpha3` is now deprecated in favor of `k3d.io/v1alpha4` ### Removed -- unused volume validation functionality in `cmd/util`, does not affect the CLI (#916) +* unused volume validation functionality in `cmd/util`, does not affect the CLI (#916) + +### Compatibility + +This release was automatically tested with the following setups: + +#### Docker + +* 20.10.5 +* 20.10.12 + +**Expected to Fail** with the following versions: + +* <= 20.10.4 (due to runc, see ) + +#### K3s + +We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel: + +* Channel v1.23 +* Channel v1.22 + +**Expected to Fail** with the following versions: + +* <= v1.18 (due to not included, but expected CoreDNS in K3s) ## v5.2.2 ### Fixes -- mitigate issue when importing images from multiple tars (#881, @sbaier1) -- fix: cluster delete should not fail if no cluster was found by config file (#886, @kuritka) +* mitigate issue when importing images from multiple tars (#881, @sbaier1) +* fix: cluster delete should not fail if no cluster was found by config file (#886, @kuritka) ### Misc -- docs: new page about k3d concepts, incl. nodefilters (#888) - - +* docs: new page about k3d concepts, incl. nodefilters (#888) + * ## v5.2.1 ### Features & Enhancements -- improved Podman compatibility (#868, @serverwentdown) - - last missing piece: release of -- improved error handling and logs when waiting for container logs (ca47fac) +* improved Podman compatibility (#868, @serverwentdown) + * last missing piece: release of +* improved error handling and logs when waiting for container logs (ca47fac) ### Fixes -- fix: only replace default api host with docker host (#879) -- fix: use available hardcoded K3s version in version.go (0bbb5b9) +* fix: only replace default api host with docker host (#879) +* fix: use available hardcoded K3s version in version.go (0bbb5b9) ## v5.2.0 ### Features & Enhancements -- Improve image import performance (#826, @sbaier1) - - **New flag**: `k3d image import --mode [auto | direct | tools]` - - `tools` is the old default, which spawns a `k3d-tools` container for importing - - `auto` is the new default to automatically detect which mode should work best - - `direct` directly streams the images into the node containers without the `k3d-tools` container -- Enhanced usability of nodefilters & error messages for wrong usage (#871) -- **New command**: `k3d version list [k3s | k3d | k3d-proxy | k3d-tools]` to get image tags that can be used with k3d (#870) - - e.g. use `k3d version list k3s --format repo` to get the latest image available for K3s and use it via `k3d cluster create --image ` - - Docs: [docs/usage/commands/k3d_version_list.md](./docs/usage/commands/k3d_version_list.md) +* Improve image import performance (#826, @sbaier1) + * **New flag**: `k3d image import --mode [auto | direct | tools]` + * `tools` is the old default, which spawns a `k3d-tools` container for importing + * `auto` is the new default to automatically detect which mode should work best + * `direct` directly streams the images into the node containers without the `k3d-tools` container +* Enhanced usability of nodefilters & error messages for wrong usage (#871) +* **New command**: `k3d version list [k3s | k3d | k3d-proxy | k3d-tools]` to get image tags that can be used with k3d (#870) + * e.g. use `k3d version list k3s --format repo` to get the latest image available for K3s and use it via `k3d cluster create --image ` + * Docs: [docs/usage/commands/k3d_version_list.md](./docs/usage/commands/k3d_version_list.md) ### Fixes -- cluster network: reserve IP extra IP for k3d-tools container in k3d-managed IPAM to avoid conflicts -- process the SimpleConfig before validating it to avoid early exit in hostnetwork mode (#860) -- error out if `K3D_FIX_DNS=1` is set and user tries to mount a file to `/etc/resolv.conf` (conflict) -- clusterStart: only run actions which are necessary given the start reason (e.g. `cluster start` vs. `cluster create`) -- fix injection of `host.k3d.internal` based on resolving `host.docker.internal` (#872) - - also now uses `host.docker.internal` in kubeconfig based on certain conditions (see PR) +* cluster network: reserve IP extra IP for k3d-tools container in k3d-managed IPAM to avoid conflicts +* process the SimpleConfig before validating it to avoid early exit in hostnetwork mode (#860) +* error out if `K3D_FIX_DNS=1` is set and user tries to mount a file to `/etc/resolv.conf` (conflict) +* clusterStart: only run actions which are necessary given the start reason (e.g. `cluster start` vs. `cluster create`) +* fix injection of `host.k3d.internal` based on resolving `host.docker.internal` (#872) + * also now uses `host.docker.internal` in kubeconfig based on certain conditions (see PR) ### Misc -- tests/e2e: parellelize and cleanup tests -> cut execution speed in half (#848 & #849) - - also run some make targets in parallel - - new env var `E2E_PARALLEL=` to configure parallelism - - test output is now redirected to files inside the runner and only the logs of failed tests will later be output -- Update dependencies, including docker, containerd & k8s -- docs: clarify usage of local registries with k3d -- docs: fix port numbers in registry usage guide +* tests/e2e: parellelize and cleanup tests -> cut execution speed in half (#848 & #849) + * also run some make targets in parallel + * new env var `E2E_PARALLEL=` to configure parallelism + * test output is now redirected to files inside the runner and only the logs of failed tests will later be output +* Update dependencies, including docker, containerd & k8s +* docs: clarify usage of local registries with k3d +* docs: fix port numbers in registry usage guide ### Notes -- k3d v5.x.x requires at least docker version 20.10.4 +* k3d v5.x.x requires at least docker version 20.10.4 ## v5.1.0 ### Features -- clusterCreate: `--image` option (also in config file) magic words to follow K3s channels (#841) - - `latest`/`stable` to follow latest/stable channels of K3s - - `+` (prefix `+`) where `` can as well be `latest` or `stable`, but also e.g. `v1.21` - - k3d will then check the K3s channel server to get the latest image for that channel +* clusterCreate: `--image` option (also in config file) magic words to follow K3s channels (#841) + * `latest`/`stable` to follow latest/stable channels of K3s + * `+` (prefix `+`) where `` can as well be `latest` or `stable`, but also e.g. `v1.21` + * k3d will then check the K3s channel server to get the latest image for that channel ### Enhancements -- nodeHooks: add descriptions and log them for more verbosity (#843) -- `node create`: inject `host.k3d.internal` into `/etc/hosts` similar to the `cluster create` command (#843) +* nodeHooks: add descriptions and log them for more verbosity (#843) +* `node create`: inject `host.k3d.internal` into `/etc/hosts` similar to the `cluster create` command (#843) ### Fix -- `--network host`: do not do any network magic (like `host.k3d.internal` injection, etc.) when `host` network is used (#844) +* `--network host`: do not do any network magic (like `host.k3d.internal` injection, etc.) when `host` network is used (#844) ### Misc -- CI/Makefile: build with `-mod vendor` -- docs: document using some K3s features in k3d, including `servicelb`, `traefik`, `local-storage-provisioner` and `coredns` (#845) +* CI/Makefile: build with `-mod vendor` +* docs: document using some K3s features in k3d, including `servicelb`, `traefik`, `local-storage-provisioner` and `coredns` (#845) ## v5.0.3 ### Enhancements & Fixes -- simplified way of getting a Docker API Client that works with Docker Contexts and `DOCKER_*` environment variable configuration (#829, @dragonflylee) - - fix: didn't honor `DOCKER_TLS` environment variables before +* simplified way of getting a Docker API Client that works with Docker Contexts and `DOCKER_*` environment variable configuration (#829, @dragonflylee) + * fix: didn't honor `DOCKER_TLS` environment variables before ## v5.0.2 ### Enhancements -- CoreDNS Configmap is now edited in the auto-deploy manifest on disk instead of relying on `kubectl patch` command (#814) -- refactor: add cmd subcommands in a single function call (#819, @moeryomenko) -- handle ready-log-messages by type and intent & check them in single log streams instead of checking whole chunks every time (#818) +* CoreDNS Configmap is now edited in the auto-deploy manifest on disk instead of relying on `kubectl patch` command (#814) +* refactor: add cmd subcommands in a single function call (#819, @moeryomenko) +* handle ready-log-messages by type and intent & check them in single log streams instead of checking whole chunks every time (#818) ### Fixes -- fix: config file check failing with env var expansion because unexpanded input file was checked +* fix: config file check failing with env var expansion because unexpanded input file was checked ### Misc -- cleanup: ensure that connections/streams are closed once unused (#818) -- cleanup: split type definitions across multiple files to increase readability (#818) -- docs: clarify `node create` help text about cluster reference (#808, @losinggeneration) -- refactor: move from io/ioutil (deprecated) to io and os packages (#827, @Juneezee) +* cleanup: ensure that connections/streams are closed once unused (#818) +* cleanup: split type definitions across multiple files to increase readability (#818) +* docs: clarify `node create` help text about cluster reference (#808, @losinggeneration) +* refactor: move from io/ioutil (deprecated) to io and os packages (#827, @Juneezee) ## v5.0.1 ### Enhancement -- add `HostFromClusterNetwork` field to `LocalRegistryHosting` configmap as per KEP-1755 (#754) +* add `HostFromClusterNetwork` field to `LocalRegistryHosting` configmap as per KEP-1755 (#754) ### Fixes -- fix: nilpointer exception on failed exec process with no returned logreader -- make post-create cluster preparation (DNS stuff mostly) more resilient (#780) -- fix v1alpha2 -> v1alpha3 config migration (and other related issues) (#799) +* fix: nilpointer exception on failed exec process with no returned logreader +* make post-create cluster preparation (DNS stuff mostly) more resilient (#780) +* fix v1alpha2 -> v1alpha3 config migration (and other related issues) (#799) ### Misc -- docs: fix typo (#784) -- docs: fix usage of legacy `--k3s-agent/server-arg` flag +* docs: fix typo (#784) +* docs: fix usage of legacy `--k3s-agent/server-arg` flag ## v5.0.0 @@ -164,269 +211,269 @@ The demo repository has also been updated to work with k3d v5: ) - - example for a port-mapping: `--port 8080:80@server:0:proxy` - - identifier = `server`, index = `0`, opt = `proxy` - - `opt` is an extra optional argument used for different purposes depending on the flag - - currently, only the `--port` flag has `opt`s, namely `proxy` and `direct` (see other breaking change) -- port-mapping now go via the loadbalancer (serverlb) by default - - the `--port` flag has the `proxy` opt (see new nodefilter syntax above) set by default - - to leverage the old behavior of direct port-mappings, use the `direct` opt on the port flag - - the nodefilter `loadbalancer` will now do the same as `servers:*;agents:*` (proxied via the loadbalancer) -- flag `--registries-create` transformed from bool flag to string flag: let's you define the name and port-binding of the newly created registry, e.g. `--registry-create myregistry.localhost:5001` +* new syntax for nodefilters + * dropped the usage of square brackets `[]` for indexing, as it caused problems with some shells trying to interpret them + * new syntax: `@identifier[:index][:opt]` (see ) + * example for a port-mapping: `--port 8080:80@server:0:proxy` + * identifier = `server`, index = `0`, opt = `proxy` + * `opt` is an extra optional argument used for different purposes depending on the flag + * currently, only the `--port` flag has `opt`s, namely `proxy` and `direct` (see other breaking change) +* port-mapping now go via the loadbalancer (serverlb) by default + * the `--port` flag has the `proxy` opt (see new nodefilter syntax above) set by default + * to leverage the old behavior of direct port-mappings, use the `direct` opt on the port flag + * the nodefilter `loadbalancer` will now do the same as `servers:*;agents:*` (proxied via the loadbalancer) +* flag `--registries-create` transformed from bool flag to string flag: let's you define the name and port-binding of the newly created registry, e.g. `--registry-create myregistry.localhost:5001` ### Fixes -- cleaned up and properly sorted the sanitization of existing resources used to create new nodes (#638) +* cleaned up and properly sorted the sanitization of existing resources used to create new nodes (#638) ### Features & Enhancements -- new command: `k3d node edit` to edit existing nodes (#615) - - currently only allows `k3d node edit NODE --port-add HOSTPORT:CONTAINERPORT` for the serverlb/loadbalancer to add new ports - - pkg: new `NodeEdit` function -- new (hidden) command: `k3d debug` with some options for debugging k3d resources (#638) - - e.g. `k3d debug loadbalancer get-config` to get the current loadbalancer configuration -- loadbalancer / k3d-proxy (#638) - - updated fork of `confd` to make usage of the file backend including a file watcher for auto-reloads - - this also checks the config before applying it, so the lb doesn't crash on a faulty config - - updating the loadbalancer writes the new config file and also checks if everything's going fine afterwards - - some settings of the loadbalancer can now be configured using `--lb-config-override`, see docs at -- helper images can now be set explicitly via environment variables: `K3D_IMAGE_LOADBALANCER` & `K3D_IMAGE_TOOLS` (#638) -- concurrently add new nodes to an existing cluster (remove some dumb code) (#640) - - `--wait` is now the default for `k3d node create` -- normalized flag usage for k3s and runtime (#598, @ejose19) - - rename `k3d cluster create --label` to `k3d cluster create --runtime-label` (as it's labelling the node on runtime level, e.g. docker) - - config option moved to `options.runtime.labels` - - add `k3d cluster create --k3s-node-label` to add Kubernetes node labels via k3s flag (#584, @developer-guy, @ejose, @dentrax) - - new config option `options.k3s.nodeLabels` - - the same for `k3d node create` -- improved config file handling (#605) - - new version `v1alpha3` - - warning when using outdated version - - validation dynamically based on provided config apiVersion - - new default for `k3d config init` - - new command `k3d config migrate INPUT [OUTPUT]` to migrate config files between versions - - currently supported migration `v1alpha2` -> `v1alpha3` - - pkg: new `Config` interface type to support new generic `FromViper` config file parsing -- changed flags `--k3s-server-arg` & `--k3s-agent-arg` into `--k3s-arg` with nodefilter support (#605) - - new config path `options.k3s.extraArgs` -- config file: environment variables (`$VAR`, `${VAR}` will be expanded unconditionally) (#643) -- docker context support (#601, @developer-guy & #674) -- Feature flag using the environment variable `K3D_FIX_DNS` and setting it to a true value (e.g. `export K3D_FIX_DNS=1`) to forward DNS queries to your local machine, e.g. to use your local company DNS +* new command: `k3d node edit` to edit existing nodes (#615) + * currently only allows `k3d node edit NODE --port-add HOSTPORT:CONTAINERPORT` for the serverlb/loadbalancer to add new ports + * pkg: new `NodeEdit` function +* new (hidden) command: `k3d debug` with some options for debugging k3d resources (#638) + * e.g. `k3d debug loadbalancer get-config` to get the current loadbalancer configuration +* loadbalancer / k3d-proxy (#638) + * updated fork of `confd` to make usage of the file backend including a file watcher for auto-reloads + * this also checks the config before applying it, so the lb doesn't crash on a faulty config + * updating the loadbalancer writes the new config file and also checks if everything's going fine afterwards + * some settings of the loadbalancer can now be configured using `--lb-config-override`, see docs at +* helper images can now be set explicitly via environment variables: `K3D_IMAGE_LOADBALANCER` & `K3D_IMAGE_TOOLS` (#638) +* concurrently add new nodes to an existing cluster (remove some dumb code) (#640) + * `--wait` is now the default for `k3d node create` +* normalized flag usage for k3s and runtime (#598, @ejose19) + * rename `k3d cluster create --label` to `k3d cluster create --runtime-label` (as it's labelling the node on runtime level, e.g. docker) + * config option moved to `options.runtime.labels` + * add `k3d cluster create --k3s-node-label` to add Kubernetes node labels via k3s flag (#584, @developer-guy, @ejose, @dentrax) + * new config option `options.k3s.nodeLabels` + * the same for `k3d node create` +* improved config file handling (#605) + * new version `v1alpha3` + * warning when using outdated version + * validation dynamically based on provided config apiVersion + * new default for `k3d config init` + * new command `k3d config migrate INPUT [OUTPUT]` to migrate config files between versions + * currently supported migration `v1alpha2` -> `v1alpha3` + * pkg: new `Config` interface type to support new generic `FromViper` config file parsing +* changed flags `--k3s-server-arg` & `--k3s-agent-arg` into `--k3s-arg` with nodefilter support (#605) + * new config path `options.k3s.extraArgs` +* config file: environment variables (`$VAR`, `${VAR}` will be expanded unconditionally) (#643) +* docker context support (#601, @developer-guy & #674) +* Feature flag using the environment variable `K3D_FIX_DNS` and setting it to a true value (e.g. `export K3D_FIX_DNS=1`) to forward DNS queries to your local machine, e.g. to use your local company DNS ### Misc -- tests/e2e: timeouts everywhere to avoid killing DroneCI (#638) -- logs: really final output when creating/deleting nodes (so far, we were not outputting a final success message and the process was still doing stuff) (#640) -- tests/e2e: add tests for v1alpha2 to v1alpha3 migration -- docs: use v1alpha3 config version -- docs: update general appearance and cleanup +* tests/e2e: timeouts everywhere to avoid killing DroneCI (#638) +* logs: really final output when creating/deleting nodes (so far, we were not outputting a final success message and the process was still doing stuff) (#640) +* tests/e2e: add tests for v1alpha2 to v1alpha3 migration +* docs: use v1alpha3 config version +* docs: update general appearance and cleanup ## v4.4.8 ## Enhancements -- Improved DroneCI Pipeline for Multiarch Images and SemVer Tags (#712) - - **Important**: New images will not have the `v` prefix in the tag anymore! - - but now real releases will use the "hierarchical" SemVer tags, so you could e.g. subscribe to rancher/k3d-proxy:4 to get v4.x.x images for the proxy container +* Improved DroneCI Pipeline for Multiarch Images and SemVer Tags (#712) + * **Important**: New images will not have the `v` prefix in the tag anymore! + * but now real releases will use the "hierarchical" SemVer tags, so you could e.g. subscribe to rancher/k3d-proxy:4 to get v4.x.x images for the proxy container ## Fixes -- clusterCreate: do not override hostIP if hostPort is missing (#693, @lukaszo) -- imageImport: import all listed images, not only the first one (#701, @mszostok) -- clusterCreate: when memory constraints are set, only pull the image used for checking the edac folder, if it's not present on the machine -- fix: update k3d-tools dependencies and use API Version Negotiation, so it still works with older versions of the Docker Engine (#679) +* clusterCreate: do not override hostIP if hostPort is missing (#693, @lukaszo) +* imageImport: import all listed images, not only the first one (#701, @mszostok) +* clusterCreate: when memory constraints are set, only pull the image used for checking the edac folder, if it's not present on the machine +* fix: update k3d-tools dependencies and use API Version Negotiation, so it still works with older versions of the Docker Engine (#679) ### Misc -- install script: add darwin/arm64 support (#676, @colelawrence) -- docs: fix go install command (#677, @Rots) -- docs: add project overview () (#680) +* install script: add darwin/arm64 support (#676, @colelawrence) +* docs: fix go install command (#677, @Rots) +* docs: add project overview () (#680) ## v4.4.7 ### Features / Enhancements -- new flag: `k3d image import --keep-tools` to not delete the tools node container after importing the image(s) (#672) -- improve image name handling when importing images (#653, @cimnine) - - normalize image names internally, e.g. strip prefixes that docker adds, but that break the process - - see for more info +* new flag: `k3d image import --keep-tools` to not delete the tools node container after importing the image(s) (#672) +* improve image name handling when importing images (#653, @cimnine) + * normalize image names internally, e.g. strip prefixes that docker adds, but that break the process + * see for more info ### Fixes -- Use default gateway, when bridge network doesn't have it (#666, @kuritka) -- Start an existing, but not running tools node to re-use it when importing an image (#672) +* Use default gateway, when bridge network doesn't have it (#666, @kuritka) +* Start an existing, but not running tools node to re-use it when importing an image (#672) ### Misc -- deps: switching back to upstream viper including the StringArray fix -- docs: reference to "nolar/setup-k3d-k3s" step for GitHub Actions (#668, @nolar) -- docs: updated and simplified CUDA guide (#662, @vainkop) (#669) +* deps: switching back to upstream viper including the StringArray fix +* docs: reference to "nolar/setup-k3d-k3s" step for GitHub Actions (#668, @nolar) +* docs: updated and simplified CUDA guide (#662, @vainkop) (#669) ## v4.4.6 ### Fixes -- fix an issue where the cluster creation would stall waiting for the `starting worker processes` log message from the loadbalancer/serverlb - - this was likely caused by a rounding issue when asking docker to get the container logs starting at a specific timestamp - - we now drop subsecond precision for this to avoid the rounding issue, which was confirmed to work - - see issues #592 & #621 +* fix an issue where the cluster creation would stall waiting for the `starting worker processes` log message from the loadbalancer/serverlb + * this was likely caused by a rounding issue when asking docker to get the container logs starting at a specific timestamp + * we now drop subsecond precision for this to avoid the rounding issue, which was confirmed to work + * see issues #592 & #621 ### Misc -- to debug the issue mentioned above, we introduced a new environment variable `K3D_LOG_NODE_WAIT_LOGS`, which can be set to a list of node roles (e.g. `K3D_LOG_NODE_WAIT_LOGS=loadbalancer,agent`) to output the container logs that k3d inspects +* to debug the issue mentioned above, we introduced a new environment variable `K3D_LOG_NODE_WAIT_LOGS`, which can be set to a list of node roles (e.g. `K3D_LOG_NODE_WAIT_LOGS=loadbalancer,agent`) to output the container logs that k3d inspects ## v4.4.5 ### Fixes -- overall: use the getDockerClient helper function everywhere to e.g. support docker via ssh everywhere -- nodeCreate: do not copy meminfo/edac volume mounts from existing nodes, to avoid conflicts with generated mounts -- kubeconfig: fix file handling on windows (#626 + #628, @dragonflylee) +* overall: use the getDockerClient helper function everywhere to e.g. support docker via ssh everywhere +* nodeCreate: do not copy meminfo/edac volume mounts from existing nodes, to avoid conflicts with generated mounts +* kubeconfig: fix file handling on windows (#626 + #628, @dragonflylee) ### Misc -- docs: add [FAQ entry](https://k3d.io/faq/faq/#nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied) on nf_conntrack_max: permission denied issue from kube-proxy (#607) -- docs: cleanup, fix formatting, etc. -- license: update to include 2021 in time range -- docs: link to AutoK3s (#614, @JacieChao) -- tests/e2e: update the list of tested k3s versions +* docs: add [FAQ entry](https://k3d.io/faq/faq/#nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied) on nf_conntrack_max: permission denied issue from kube-proxy (#607) +* docs: cleanup, fix formatting, etc. +* license: update to include 2021 in time range +* docs: link to AutoK3s (#614, @JacieChao) +* tests/e2e: update the list of tested k3s versions ## v4.4.4 ### Enhancements -- nodes created via `k3d node create` now inherit the registry config from existing nodes (if there is any) (#597) -- the cgroupv2 hotfix (custom entrypoint script) is now enabled by default (#603) - - disable by setting the environment variable `K3D_FIX_CGROUPV2=false` +* nodes created via `k3d node create` now inherit the registry config from existing nodes (if there is any) (#597) +* the cgroupv2 hotfix (custom entrypoint script) is now enabled by default (#603) + * disable by setting the environment variable `K3D_FIX_CGROUPV2=false` ### Fixes -- fix using networks without IPAM config (e.g. `host`) +* fix using networks without IPAM config (e.g. `host`) ### Misc -- docs: edit links on k3d.io now point to the correct branch (`main`) -- docs: new FAQ entry on spurious PID entries when using shared mounts (#609, @leelavg) +* docs: edit links on k3d.io now point to the correct branch (`main`) +* docs: new FAQ entry on spurious PID entries when using shared mounts (#609, @leelavg) ## v4.4.3 ### Highlights -- cgroupv2 support: to properly work on cgroupv2 systems, k3s has to move all the processes from the root cgroup to a new /init cgroup and enable subtree_control - - this is going to be included in the k3s agent code directly () - - for now we're overriding the container entrypoint with a script that does this (#579, compare ) - - thanks a lot for all the input and support @AkihiroSuda - - **Usage**: set the environment variable `K3D_FIX_CGROUPV2` to a `true` value before/when creating a cluster with k3d - - e.g. `export K3D_FIX_CGROUPV2=1` +* cgroupv2 support: to properly work on cgroupv2 systems, k3s has to move all the processes from the root cgroup to a new /init cgroup and enable subtree_control + * this is going to be included in the k3s agent code directly () + * for now we're overriding the container entrypoint with a script that does this (#579, compare ) + * thanks a lot for all the input and support @AkihiroSuda + * **Usage**: set the environment variable `K3D_FIX_CGROUPV2` to a `true` value before/when creating a cluster with k3d + * e.g. `export K3D_FIX_CGROUPV2=1` ### Fixes -- fix: docker volume not mountable due to validation failure - - was not able to mount named volume on windows as we're checking for `:` meant for drive-letters and k3d separators +* fix: docker volume not mountable due to validation failure + * was not able to mount named volume on windows as we're checking for `:` meant for drive-letters and k3d separators ### Misc -- fix create command's flags typo (#568, @Jason-ZW) +* fix create command's flags typo (#568, @Jason-ZW) ## v4.4.2 ### Fixes -- k3d-proxy: rename udp upstreams to avoid collisions/duplicates (#564) +* k3d-proxy: rename udp upstreams to avoid collisions/duplicates (#564) ### Features -- add *hidden* command `k3d runtime-info` used for debugging (#553) - - this comes with some additions on package/runtime level -- add *experimental* `--subnet` flag to get some k3d IPAM to ensure that server nodes keep static IPs across restarts (#560) +* add *hidden* command `k3d runtime-info` used for debugging (#553) + * this comes with some additions on package/runtime level +* add *experimental* `--subnet` flag to get some k3d IPAM to ensure that server nodes keep static IPs across restarts (#560) ### Misc -- docs: fix typo (#556, @gcalmettes) -- docs: fix typo (#561, @alechartung) -- ci/drone: pre-release on `-dev.X` tags -- ci/drone: always build no matter the branch name (just not release) -- docs: add automatic command tree generation via cobra (#562) -- makefile: use `go env gopath` as install target for tools (as per #445) -- JSONSchema: add some examples and defaults (now also available via in your IDE) +* docs: fix typo (#556, @gcalmettes) +* docs: fix typo (#561, @alechartung) +* ci/drone: pre-release on `-dev.X` tags +* ci/drone: always build no matter the branch name (just not release) +* docs: add automatic command tree generation via cobra (#562) +* makefile: use `go env gopath` as install target for tools (as per #445) +* JSONSchema: add some examples and defaults (now also available via in your IDE) ## v4.4.1 ### Fixes -- use viper fork that contains a fix to make cobra's `StringArray` flags work properly - - this fixes the issue, that flag values containing commas got split (because we had to use `StringSlice` type flags) - - this is to be changed back to upstream viper as soon as (or a similar fix) got merged +* use viper fork that contains a fix to make cobra's `StringArray` flags work properly + * this fixes the issue, that flag values containing commas got split (because we had to use `StringSlice` type flags) + * this is to be changed back to upstream viper as soon as (or a similar fix) got merged ## v4.4.0 ### Features / Enhancements -- Support for Memory Limits using e.g. `--servers-memory 1g` or `--agents-memory 1.5g` (#494, @konradmalik) - - enabled by providing fake `meminfo` files +* Support for Memory Limits using e.g. `--servers-memory 1g` or `--agents-memory 1.5g` (#494, @konradmalik) + * enabled by providing fake `meminfo` files ### Fixes -- fix absolute paths in volume mounts on Windows (#510, @markrexwinkel) +* fix absolute paths in volume mounts on Windows (#510, @markrexwinkel) ### Documentation -- clarify registry names in docs and help text -- add usage section about config file (#534) -- add FAQ entry on certificate error when running behind corporate proxy -- add MacPorts install instructions (#539, @herbygillot) -- Heal Shruggie: Replace amputated arm (#540, @claycooper) +* clarify registry names in docs and help text +* add usage section about config file (#534) +* add FAQ entry on certificate error when running behind corporate proxy +* add MacPorts install instructions (#539, @herbygillot) +* Heal Shruggie: Replace amputated arm (#540, @claycooper) ## v4.3.0 ### Features / Enhancements -- Use Go 1.16 - - update dependencies, including kubernetes, docker, containerd and more - - add `darwin/arm64` (Apple Silicon, M1) build target (#530) - - use the new `//go:embed` feature to directly embed the jsonschema in the binary (#529) -- Add a status column to `k3d registry list` output (#496, @ebr) -- Allow non-prefixed (i.e. without `k3d-` prefix) user input when fetching resources (e.g. `k3d node get mycluster-server-0` would return successfully) +* Use Go 1.16 + * update dependencies, including kubernetes, docker, containerd and more + * add `darwin/arm64` (Apple Silicon, M1) build target (#530) + * use the new `//go:embed` feature to directly embed the jsonschema in the binary (#529) +* Add a status column to `k3d registry list` output (#496, @ebr) +* Allow non-prefixed (i.e. without `k3d-` prefix) user input when fetching resources (e.g. `k3d node get mycluster-server-0` would return successfully) ### Fixes -- Allow absolute paths for volumes on Windows (#510, @markrexwinkel) -- fix nil-pointer exception in case of non-existent IPAM network config -- Properly handle combinations of host/hostIP in kubeAPI settings reflected in the kubeconfig (#500, @fabricev) +* Allow absolute paths for volumes on Windows (#510, @markrexwinkel) +* fix nil-pointer exception in case of non-existent IPAM network config +* Properly handle combinations of host/hostIP in kubeAPI settings reflected in the kubeconfig (#500, @fabricev) ### Misc -- docs: fix typo in stop command help text (#513, @searsaw) -- ci/ghaction: AUR (pre-)release now on Ubuntu 20.04 and latest archlinux image -- REMOVE incomplete and unused `containerd` runtime from codebase, as it was causing issues to build for windows and hasn't made any progress in quite some time now +* docs: fix typo in stop command help text (#513, @searsaw) +* ci/ghaction: AUR (pre-)release now on Ubuntu 20.04 and latest archlinux image +* REMOVE incomplete and unused `containerd` runtime from codebase, as it was causing issues to build for windows and hasn't made any progress in quite some time now ## v4.2.0 ### Features / Enhancements -- add processing step for cluster config, to configure it e.g. for hostnetwork mode (#477, @konradmalik) -- allow proxying UDP ports via the load balancer (#488, @k0da) +* add processing step for cluster config, to configure it e.g. for hostnetwork mode (#477, @konradmalik) +* allow proxying UDP ports via the load balancer (#488, @k0da) ### Fixes -- fix usage of `DOCKER_HOST` env var for Kubeconfig server ref (trim port) -- fix error when trying to attach the same node (e.g. registry) to the same network twice (#486, @kuritka) -- fix Kube-API settings in configg file got overwritten (#490, @dtomasi) +* fix usage of `DOCKER_HOST` env var for Kubeconfig server ref (trim port) +* fix error when trying to attach the same node (e.g. registry) to the same network twice (#486, @kuritka) +* fix Kube-API settings in configg file got overwritten (#490, @dtomasi) ### Misc -- add `k3d.version` label to created resources -- add Pull-Request template -- docs: add hint on minimal requirements for multi-server clusters (#481, @Filius-Patris) +* add `k3d.version` label to created resources +* add Pull-Request template +* docs: add hint on minimal requirements for multi-server clusters (#481, @Filius-Patris) ## v4.1.1 ### Fixes -- fix: `--k3s-server-arg` and `--k3s-agent-arg` didn't work (Viper StringArray incompatibility) (#482) +* fix: `--k3s-server-arg` and `--k3s-agent-arg` didn't work (Viper StringArray incompatibility) (#482) ## v4.1.0 @@ -434,62 +481,62 @@ The demo repository has also been updated to work with k3d v5: `kubeAPI` - - `options.k3d.noRollback` -> `options.k3d.disableRollback` - - `options.k3d.prepDisableHostIPInjection` -> `options.k3d.disableHostIPInjection` +* :snake: use [viper](https://github.com/spf13/viper) for configuration management + * takes over the job of properly fetching and merging config options from + * CLI arguments/flags + * environment variables + * config file + * this also fixes some issues with using the config file (like cobra defaults overriding config file values) +* :heavy_check_mark: add JSON-Schema validation for the `Simple` config file schema +* :new: config version `k3d.io/v1alpha2` (some naming changes) + * `exposeAPI` -> `kubeAPI` + * `options.k3d.noRollback` -> `options.k3d.disableRollback` + * `options.k3d.prepDisableHostIPInjection` -> `options.k3d.disableHostIPInjection` #### :computer: Docker over SSH -- Support Docker over SSH (#324, @ekristen & @inercia) +* Support Docker over SSH (#324, @ekristen & @inercia) ### Features & Enhancements -- add root flag `--timestamps` to enable timestamped logs -- improved multi-server cluster support (#467) - - log a warning, if one tries to create a cluster with only 2 nodes (no majority possible, no fault tolerance) - - revamped cluster start procedure: init-node, sorted servers, agents, helpers - - different log messages per role and start-place (that we wait for to consider a node to be ready) - - module: `NodeStartOpts` now accept a `ReadyLogMessage` and `NodeState` now takes a `Started` timestamp string +* add root flag `--timestamps` to enable timestamped logs +* improved multi-server cluster support (#467) + * log a warning, if one tries to create a cluster with only 2 nodes (no majority possible, no fault tolerance) + * revamped cluster start procedure: init-node, sorted servers, agents, helpers + * different log messages per role and start-place (that we wait for to consider a node to be ready) + * module: `NodeStartOpts` now accept a `ReadyLogMessage` and `NodeState` now takes a `Started` timestamp string ### Fixes -- do not ignore `--no-hostip` flag and don't inject hostip if `--network=host` (#471, @konradmalik) -- fix: `--no-lb` ignored -- fix: print error cause when serverlb fails to start +* do not ignore `--no-hostip` flag and don't inject hostip if `--network=host` (#471, @konradmalik) +* fix: `--no-lb` ignored +* fix: print error cause when serverlb fails to start ### Misc -- tests/e2e: add config override test -- tests/e2e: add multi server start-stop cycle test -- tests/e2e: improved logs with stage and test details. -- builds&tests: use Docker 20.10 and BuildKit everywhere -- :memo: docs: add (GitHub Action) as a related project (#476, @kuritka) +* tests/e2e: add config override test +* tests/e2e: add multi server start-stop cycle test +* tests/e2e: improved logs with stage and test details. +* builds&tests: use Docker 20.10 and BuildKit everywhere +* :memo: docs: add (GitHub Action) as a related project (#476, @kuritka) ### Tested with -- E2E Tests ran with k3s versions - - v1.17.17-k3s1 (see Known Issues below) - - v1.18.15-k3s1 (see Known Issues below) - - v1.19.7-k3s1 - - v1.20.2-k3s1 +* E2E Tests ran with k3s versions + * v1.17.17-k3s1 (see Known Issues below) + * v1.18.15-k3s1 (see Known Issues below) + * v1.19.7-k3s1 + * v1.20.2-k3s1 ### Known Issues -- automatic multi-server cluster restarts tend to fail with k3s versions v1.17.x & v1.18.x and probably earlier versions (using dqlite) -- Using Viper brings us lots of nice features, but also one problem: - - We had to switch StringArray flags to StringSlice flags, which - - allow to use multiple flag values comma-separated in a single flag, but also - - split flag values that contain a comma into separate parts (and we cannot handle issues that arise due to this) - - so if you rely on commas in your flag values (e.g. for `--env X=a,b,c`), please consider filing an issue or supporting and - - `--env X=a,b,c` would be treated the same as `--env X=a`, `--env b`, `--env c` +* automatic multi-server cluster restarts tend to fail with k3s versions v1.17.x & v1.18.x and probably earlier versions (using dqlite) +* Using Viper brings us lots of nice features, but also one problem: + * We had to switch StringArray flags to StringSlice flags, which + * allow to use multiple flag values comma-separated in a single flag, but also + * split flag values that contain a comma into separate parts (and we cannot handle issues that arise due to this) + * so if you rely on commas in your flag values (e.g. for `--env X=a,b,c`), please consider filing an issue or supporting and + * `--env X=a,b,c` would be treated the same as `--env X=a`, `--env b`, `--env c` ## v4.0.0 @@ -499,41 +546,41 @@ The demo repository has also been updated to work with k3d v5: ClusterConfig -> Cluster + Opts +* Module is now on `github.com/rancher/k3d/v4` due to lots of breaking changes +* `pkg/cluster` is now `pkg/client` +* `ClusterCreate` and `NodeCreate` don't start the entities (containers) anymore + * `ClusterRun` and `NodeRun` orchestrate the new Create and Start functionality +* `NodeDelete`/`ClusterDelete` now take an additional `NodeDeleteOpts`/`ClusterDeleteOpts` struct to toggle specific steps +* NodeSpec now features a list of networks (required for registries) +* New config flow: CLIConfig (SimpleConfig) -> ClusterConfig -> Cluster + Opts #### CLI -- Some flags changed to also use `noun-action` syntax - - e.g. `--switch-context --update-default-kubeconfig` -> `--kubeconfig-switch-context --kubeconfig-update-default` - - this eases grouping and visibility +* Some flags changed to also use `noun-action` syntax + * e.g. `--switch-context --update-default-kubeconfig` -> `--kubeconfig-switch-context --kubeconfig-update-default` + * this eases grouping and visibility ### Changes #### Features -- **Registry Support** - - k3d-managed registry like we had it in k3d v1.x - - Option 1: default settings, paired with cluster creation - - `k3d cluster create --registry-create` -> New registry for that cluster - - `k3d cluster create --registry-use` -> Re-use existing registry - - Option 2: customized, managed stand-alone - - `k3d registry [create/start/stop/delete]` - - Check the documentation, help text and tutorials for more details - - Communicate managed registry using the LocalRegistryHostingV1 spec from [KEP-1755](https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md) - - interesting especially for tools that reload images, like Tilt or Skaffold - -- **Config File Support** - - Put all your CLI-Arguments/Flags into a more readable config file and re-use it everywhere (keep it in your repo) - - Note: this is not always a 1:1 matching in naming/syntax/semantics - - `k3d cluster create --config myconfig.yaml` +* **Registry Support** + * k3d-managed registry like we had it in k3d v1.x + * Option 1: default settings, paired with cluster creation + * `k3d cluster create --registry-create` -> New registry for that cluster + * `k3d cluster create --registry-use` -> Re-use existing registry + * Option 2: customized, managed stand-alone + * `k3d registry [create/start/stop/delete]` + * Check the documentation, help text and tutorials for more details + * Communicate managed registry using the LocalRegistryHostingV1 spec from [KEP-1755](https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md) + * interesting especially for tools that reload images, like Tilt or Skaffold + +* **Config File Support** + * Put all your CLI-Arguments/Flags into a more readable config file and re-use it everywhere (keep it in your repo) + * Note: this is not always a 1:1 matching in naming/syntax/semantics + * `k3d cluster create --config myconfig.yaml` ```yaml apiVersion: k3d.io/v1alpha1 @@ -547,30 +594,30 @@ The demo repository has also been updated to work with k3d v5: ) + +#### K3s + +* Channel v1.23 +* Channel v1.22 + +**Expected to Fail** with the following versions: + +* <= v1.18 (due to not included, but expected CoreDNS in K3s)