Container
The VyOS container implementation is based on Podman as a daemonless container engine.
Configuration
Sets the image name in the hub registry
set container name mysql-server image mysql:8.0
If a registry is not specified, Docker.io will be used as the container
registry unless an alternative registry is specified using
set container registry <name> or the registry is included
in the image name
set container name mysql-server image quay.io/mysql:8.0
Override the default entrypoint from the image for a container.
Override the default command from the image for a container.
The container and the host share the same process namespace. This means that processes running on the host are visible inside the container, and processes inside the container are visible on the host.
The command translates to “–pid host” when the container is created.
The container and the host share the same cgroup namespace. The container sees the host’s cgroup hierarchy instead of a namespaced view of its own.
The command translates to “–cgroupns host” when the container is created.
Allow host networking in a container. The network stack of the container is not isolated from the host and will use the host IP.
The command translates to “–net host” when the container is created.
Note
allow-host-networks cannot be used with network
Attaches a user-defined network to a container. Only one network must be specified and must already exist.
Optionally set a specific static IPv4 or IPv6 address for the container. This address must be within the named network prefix.
Note
The first IP in the container network is reserved by the engine and cannot be used
Set a specific MAC address for the container interface.
Defaults to auto, which generates a random address for the container.
set container name coredns network NET01 mac '00:53:00:12:34:56'
Optionally set a custom name server. If a container network is used with DNS enabled, this setting will not have any effect.
Add custom environment variables. Multiple environment variables are allowed. The following commands translate to “-e key=value” when the container is created.
set container name mysql-server environment MYSQL_DATABASE value 'zabbix'
set container name mysql-server environment MYSQL_USER value 'zabbix'
set container name mysql-server environment MYSQL_PASSWORD value 'zabbix_pwd'
set container name mysql-server environment MYSQL_ROOT_PASSWORD value 'root_pwd'
Publish a port for the container.
source: Port on the host
destination: Port on the container
set container name zabbix-web-nginx-mysql port http source 80
set container name zabbix-web-nginx-mysql port http destination 8080
set container name zabbix-web-nginx-mysql port http protocol tcp
Note
Port publishing cannot be used with network. For this purpose, a workaround using destination NAT and static IP assignment for the container is available.
Mount a volume into the container.
source: Path on the host, place it inside
/configto preserve during upgradesdestination: Path inside the container
set container name coredns volume 'corefile' source /config/coredns/Corefile
set container name coredns volume 'corefile' destination /etc/Corefile
Volume is either mounted as rw (read-write - default) or ro (read-only)
Control how mount and unmount events propagate between the volume on
the host and the mount inside the container. Default is rprivate.
shared: Events propagate in both directions. A mount made below the volume on the host shows up inside the container, and one made inside the container shows up on the host
slave: Events propagate one way only, from the host into the container but not back
private: No events propagate into or out of the mount
rshared, rslave, rprivate: The same, applied recursively to nested mount points as well
Note
The propagation of the host source mount limits what takes effect here:
shared requires the source mount to be shared, and slave
requires it to be shared or slave. Check it on the host with
findmnt -o TARGET,PROPAGATION \<directory\>.
Mount a tmpfs (ramdisk) filesystem to the given path within the container.
Size in MB for tmpfs filesystem, maximum size is 64GB or 50% of the systems total available memory.
Set the restart behavior of the container.
no: Do not restart containers on exit
on-failure: Restart containers when they exit with a non-zero exit code, retrying indefinitely (default)
always: Restart containers when they exit, regardless of status, retrying indefinitely
Time to wait for the container to shut down on its own before it is killed.
Valid values are 1 to 60 seconds, the default is 10 seconds.
This specifies the number of CPU resources the container can use.
Default is 0 for unlimited. For example, 1.25 limits the container to use up to 1.25 cores worth of CPU time. This can be a decimal number with up to three decimal places.
The command translates to “–cpus=<num>” when the container is created.
Constrain the memory available to the container.
Default is 512 MB. Use 0 MB for unlimited memory.
Size of the shared memory mounted at /dev/shm inside the container.
Databases and browsers are typical workloads that need more than the
default.
Default is 64 MB, the maximum is 8192 MB. Use 0 MB for unlimited shared memory.
Add a host device to the container.
source: Device on the router itself
destination: Device inside the container
Set container capabilities or permissions.
net-admin: Network operations (interface, firewall, routing tables)
net-bind-service: Bind a socket to privileged ports (port numbers less than 1024)
net-raw: Permission to create raw network sockets
chown: Permission to set file UIDs and GIDs
mknod: Permission to create special files
setpcap: Capability sets (from bounded or inherited set)
sys-admin: Administration operations (quotactl, mount, sethostname, setdomainname)
sys-module: Load, unload and delete kernel modules
sys-nice: Permission to set process nice value
sys-rawio: Permission to perform raw I/O operations
sys-time: Permission to set system clock
Set container sysctl values.
The subset of possible parameters are:
Kernel Parameters: kernel.msgmax, kernel.msgmnb, kernel.msgmni, kernel.sem, kernel.shmall, kernel.shmmax, kernel.shmmni, kernel.shm_rmid_forced
Parameters beginning with fs.mqueue.*
Parameters beginning with net.* (only if user-defined network is used)
Container Health checks
By default, no health checks are run, even when defined by the image.
Default health check is run for the container if defined by the image.
Override the default health check command from the image for a container.
Override the default health-check interval. For example: 60
Override the default health-check timeout. For example: 10
Container Networks
Note
If you use Global State Policies in your Firewall with container networks, you need an exception for ARP traffic on bridge interfaces. Otherwise, ARP will fail and the container will not be reachable.
Define IPv4 and/or IPv6 prefix for a given network name. Both IPv4 and IPv6 can be used in parallel.
Attach containers on this network to an internal bridge on the host. They reach the outside through NAT, so they are not directly reachable from the parent network without port mapping.
This is the default when no network type is configured.
Attach this network directly to the given parent interface using the MACVLAN driver. Containers appear on the parent network with their own MAC address and are directly reachable, rather than being placed behind a host bridge and NAT.
The parent interface must already exist. It can be an ethernet, bonding or bridge interface, optionally a VLAN sub-interface of one.
Note
MACVLAN networks cannot be assigned to a VRF.
bridge: Containers act as separate hosts on the parent network
private: Containers on the same parent interface cannot communicate with each other, even if the external switch supports hairpin forwarding
vepa: Containers send all traffic through the parent switch for forwarding
Both parent and mode are required for a MACVLAN network.
set container network NET01 prefix '192.0.2.0/24'
set container network NET01 type macvlan parent 'eth1'
set container network NET01 type macvlan mode 'bridge'
Configure MTU for a given network. It is the size (in bytes) of the largest ethernet frame sent on this link.
Disable Domain Name System (DNS) plugin for this network.
Container Registry
Adds registry to list of unqualified-search-registries. By default, for any image that does not include the registry in the image name, VyOS will use docker.io and quay.io as the container registry.
Some container registries require credentials to be used.
Credentials can be defined here and will only be used when adding a container image to the system.
Allow registry access over unencrypted HTTP or TLS connections with untrusted certificates.
Registry mirror, use (host-name|address)[:port][/path].
If you have mirror http://192.168.1.1:8080 for docker.io, you can use docker.io/some/repo or run podman pull docker.io/some/repo
set container registry docker.io mirror address 192.168.1.1
set container registry docker.io mirror port 8080
set container registry docker.io insecure
If http://192.168.1.1:8080 is your own registry, you can use 192.168.1.1:8080/some/repo or run podman pull 192.168.1.1:8080/some/repo
set container registry 192.168.1.1:8080 insecure
Log Configuration
Operation Commands
Delete a particular container image based on it’s image ID. You can also delete all container images at once.
You can not delete a container image if it has more then one tag
assigned, this is why there is a force option to pass down to
the container image to also remove those images.
Container Storage
Container images and the writable layers of all containers are stored on
the persistence partition below
/usr/lib/live/mount/persistence/container/storage. This storage is
shared by all installed VyOS images, so containers survive a system
upgrade.
Podman uses the overlay storage driver. On an installed system, VyOS
uses the in-kernel overlayfs. fuse-overlayfs, a userspace
implementation that runs one helper process per container and is
noticeably slower for I/O-heavy workloads such as databases, is used
only if the storage sits on an overlayfs itself, which is the case on a
live-booted system.
Older VyOS releases used fuse-overlayfs on every system. Container
storage created by those releases keeps using fuse-overlayfs after an
upgrade, because its image layers store file deletions in a format the
in-kernel overlayfs does not understand. Podman detects this
automatically, so no action is required.
To check which implementation is in use, look for fuse-overlayfs
processes while containers are running:
vyos@vyos:~$ pgrep -a fuse-overlayfs
No output means the in-kernel overlayfs is used.
Migrate to In-Kernel Overlayfs
Existing container storage can be moved to the in-kernel overlayfs by resetting it and pulling all images again.
Warning
Resetting the storage deletes all container images, containers and
volumes managed by Podman. Data in volumes mapped to a host path using
set container name <name> volume is not affected. Make sure all images
can be pulled again, or export them first using podman image save.
Note the images in use:
vyos@vyos:~$ show container image
Stop all containers. Repeat for every configured container:
vyos@vyos:~$ sudo systemctl stop vyos-container-<name>
Reset the container storage:
vyos@vyos:~$ sudo podman system reset --force
Pull every image again:
vyos@vyos:~$ add container image <image>
Reboot the system. This recreates the container networks and starts all containers on the new storage.
Afterwards, pgrep -a fuse-overlayfs should not return any process.
Example Configurations
For the sake of demonstration, example #1 in the official documentation is adapted to the declarative VyOS CLI syntax for Zabbix.
set container network zabbix prefix 172.20.0.0/16
set container network zabbix description 'Network for Zabbix component containers'
set container name mysql-server image mysql:8.0
set container name mysql-server network zabbix address 172.20.0.10
set container name mysql-server environment 'MYSQL_DATABASE' value 'zabbix'
set container name mysql-server environment 'MYSQL_USER' value 'zabbix'
set container name mysql-server environment 'MYSQL_PASSWORD' value 'zabbix_pwd'
set container name mysql-server environment 'MYSQL_ROOT_PASSWORD' value 'root_pwd'
set container name zabbix-java-gateway image zabbix/zabbix-java-gateway:alpine-5.2-latest
set container name zabbix-java-gateway network zabbix address 172.20.0.11
set container name zabbix-server-mysql image zabbix/zabbix-server-mysql:alpine-5.2-latest
set container name zabbix-server-mysql network zabbix address 172.20.0.12
set container name zabbix-server-mysql environment 'DB_SERVER_HOST' value 'mysql-server'
set container name zabbix-server-mysql environment 'MYSQL_DATABASE' value 'zabbix'
set container name zabbix-server-mysql environment 'MYSQL_USER' value 'zabbix'
set container name zabbix-server-mysql environment 'MYSQL_PASSWORD' value 'zabbix_pwd'
set container name zabbix-server-mysql environment 'MYSQL_ROOT_PASSWORD' value 'root_pwd'
set container name zabbix-server-mysql environment 'ZBX_JAVAGATEWAY' value 'zabbix-java-gateway'
set nat destination rule 100 destination port 10051
set nat destination rule 100 protocol tcp
set nat destination rule 100 translation address 172.20.0.12
set container name zabbix-web-nginx-mysql image zabbix/zabbix-web-nginx-mysql:alpine-5.2-latest
set container name zabbix-web-nginx-mysql network zabbix address 172.20.0.13
set container name zabbix-web-nginx-mysql environment 'MYSQL_DATABASE' value 'zabbix'
set container name zabbix-web-nginx-mysql environment 'ZBX_SERVER_HOST' value 'zabbix-server-mysql'
set container name zabbix-web-nginx-mysql environment 'DB_SERVER_HOST' value 'mysql-server'
set container name zabbix-web-nginx-mysql environment 'MYSQL_USER' value 'zabbix'
set container name zabbix-web-nginx-mysql environment 'MYSQL_PASSWORD' value 'zabbix_pwd'
set container name zabbix-web-nginx-mysql environment 'MYSQL_ROOT_PASSWORD' value 'root_pwd'
set nat destination rule 101 destination port 80
set nat destination rule 101 protocol tcp
set nat destination rule 101 translation address 172.20.0.13
set nat destination rule 101 translation port 8080