Software problem: Deleting Btrfs subvolumes

Internet is no longer full of Btrfs horror stories so I decided to use Btrfs when I reinstalled my desktop computer with NixOS. It has some nice features, like subvolumes, compression and cooperation with Podman. I still prefer ZFS but due to its weird license it’s not included in Linux and one can fall into trouble when using it as the root file system and something goes wrong with a kernel module upgrade (as happened to me couple of times in the past). Btrfs doesn’t have this problem so it’s preferable for me on desktop computers.

Now I experienced the first problem with Btrfs. When I tried to remove some containers and prune Podman images, I got complaints about files that couldn’t be deleted from the storage. This has led me to the fact that Btrfs permits non-root users to create subvolumes but not to delete them:

$ btrfs subvolume create $HOME/test
Create subvolume '/home/pdm/test'

$ btrfs subvolume delete $HOME/test
WARNING: cannot read default subvolume id: Operation not permitted
Delete subvolume (no-commit): '/home/pdm/test'
ERROR: Could not destroy subvolume/snapshot: Operation not permitted
WARNING: deletion failed with EPERM, send may be in progress

Huh? At least root can still delete the subvolume:

$ sudo btrfs subvolume delete $HOME/test
Delete subvolume (no-commit): '/home/pdm/test'

And removing the volume by simply removing its directory also works, even for a non-root user. But it looks like Podman wants, not surprisingly, delete the volume. I’m not the first one being hit by this problem. It should be fixed but it still doesn’t work for me.

There is a mount option user_subvol_rm_allowed, unfortunately disabled by default, mostly for legacy reasons. After I had added it to my / and /home subvolumes, deletion of subvolumes started to work.

But not so Podman. I was no longer able to create or rebuild containers. For example:

$ podman run -it --rm debian
Error: unlinkat /home/pdm/.local/share/containers/storage/btrfs/subvolumes/6a7bcf44cc576f9ba66b978fc7472380ba0a8e5db318d547f6484acd3e49995e/bin: permission denied

Should I delete the subvolume? Let’s try:

$ btrfs subvol delete /home/pdm/.local/share/containers/storage/btrfs/subvolumes/6a7bcf44cc576f9ba66b978fc7472380ba0a8e5db318d547f6484acd3e49995e
WARNING: cannot read default subvolume id: Operation not permitted
Delete subvolume (no-commit): '/home/pdm/.local/share/containers/storage/btrfs/subvolumes/6a7bcf44cc576f9ba66b978fc7472380ba0a8e5db318d547f6484acd3e49995e'
ERROR: Could not destroy subvolume/snapshot: Permission denied

OK, still not all right, so I removed the subvolume as root. But then:

$ podman run -it --rm debian
Error: stat /home/pdm/.local/share/containers/storage/btrfs/subvolumes/6a7bcf44cc576f9ba66b978fc7472380ba0a8e5db318d547f6484acd3e49995e: no such file or directory

Oops. It looked like the most efficient way to get rid of the mess was to remove all the containers, all the subvolumes and to rebuild the containers again (fortunately, I haven’t had anything valuable in them yet).

After some more trouble (even podman system reset hadn’t worked), I ended up with removing ~/.local/share/containers/ completely and Podman finally started to work.

OK, mostly problems of category 1. The lesson learned is to always use user_subvol_rm_allowed with Btrfs.

Software problem: Compiling Emacs vterm module on NixOS

I started using NixOS again some time ago. The fact that NixOS doesn’t use Filesystem Hierarchy Standard is sometimes a big source of pain and complications. One non-obvious thing for beginners is how to compile a C program.

One C program that is very useful and must be compiled locally is emacs-libvterm C module, providing a visual terminal for Emacs. There are several good reasons to use emacs-libvterm:

  • It’s sometimes useful to run visual terminal programs.
  • Many command line programs don’t work well with line terminals.
  • If a command line program uses its own command line interface, it’s occasionally useful to take advantage of it. Additionally, Eshell command line editing doesn’t work with programs that enter their own command line interfaces.
  • External, non-Emacs, terminal emulators work well but their capabilities are quite limited and they are not integrated with Emacs.
  • Built-in Emacs visual terminal emulators (term, ansi-term) don’t work very well and are unusable with many applications and utilities.

emacs-libvterm solves all the problems. It works, it can use all the Emacs editing capabilities and it is integrated with Emacs. On standard systems, emacs-libvterm compiles its module automatically, as long as libvterm development package is installed. But it’s more difficult on NixOS.

The first problem is how to compile a C program on NixOS. As far as I understand it, nix-shell environment must be used. The second problem is that libvterm package in NixOS contains something old and apparently unmaintained and libvterm-neovim must be used instead. It took me quite long to realize it, I wasted a lot of time investigating problems related to libvterm and possibly other broken NixOS packages. Which was a problem of category 1, my ignorance, combined with a problem of category 2, broken obsolete packages in the distribution.

So how to compile Emacs vterm module on NixOS? Assuming gcc and gnumake packages are already installed on the system, it’s needed to create the following shell.nix file:

with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "emacsenv";
  nativeBuildInputs = [ cmake ];
  buildInputs = [ libvterm-neovim ];
}

Then nix-shell is run in the same directory. When Emacs is started in the given nix-shell environment, emacs-libvtem compiles and installs fine.

Software problem: Music players on Android

One might thing that making a program that plays music stored locally on a computer shouldn’t be that difficult. It must be able to find music files, to play them correctly, to play songs in the correct order, and provide a decent user interface, with some search capabilities. Considering that the most difficult tasks can be outsourced to libraries, there should be no special challenge. Yet all the music players I’ve tried on desktop computers or mobile devices suffered from significant problems of various kinds.

On Android, almost all the music players (or at least those from F-Droid) cannot read tags from music files correctly. That means some tags containing non-ASCII characters are read incorrectly, resulting in displaying album/artist/song names garbled and, what is worse, arbitrarily splitting songs of a single album to two sets, one with a garbled album name and one with the right album name. Additionally, the track numbers may be interpreted incorrectly, resulting in a wrong order of the songs. It looks like almost all Android music players use some common, broken Android library.

A special exception is Vanilla Music, which apparently uses its own, working tag reading library. But even this player is not without problems. It has already happened to me in the past that Vanilla Music couldn’t find new songs uploaded to a phone, whatever I tried, including rescanning the whole media library and reboots. And now I experienced another problem. This time, Vanilla Music search function stopped working, it couldn’t find anything in the displayed lists of artists, albums and songs. After rescanning the whole media library (which takes quite a lot of time), all the lists were split in two parts, sorted separately of each other (i.e. A … Z A … Z instead of A … Z) and one of them being searchable while the other one not. So I ended up with split lists and only one part of each searchable. After more rescanning and reboot attempts, I gave up.

A possible explanation could be that Vanilla Music got damaged by update of my phone from Android 10 to Android 11. So I uninstalled Vanilla Music, installed it again and it started working. Apparently a problem of category 3, a bug somewhere. The lesson learned is that whenever I experience problems with Vanilla Music database, I should reinstall the application instead of attempting to fix its database.

Software problem: Missing virt-manager icons

virt-manager on my NixOS system was missing icons on most of its buttons, making the application handling somewhat difficult. A little stupid annoying problem, apparently of category 3, has wasted several hours of my life.

I reported it as a NixOS bug. NixOS people responded quickly and were helpful so after a couple of days I’ve found out the problem can be remedied by installing humanity-icon-theme package. Hopefully virt-manager NixOS package gets fixed so that users won’t have to deal with this problem again.

Debian builds with Podman

Since I no longer have Debian on my desktop computer, I had to arrange an environment for building Debian packages. I could use a virtual machine running Debian for that purpose but this would have several disadvantages:

  • It would consume unnecessarily much resources.
  • I would have to use a nested build environment such as cowbuilder anyway.
  • I would have to synchronize between my working directories and the virtual machine.

Since I’m moving to Podman, I created a Podman environment for clean Debian package builds. My Containerfile looks like this:

FROM debian:sid
COPY sources.list /etc/apt/
RUN apt-get update && \
    apt-get -y dist-upgrade && \
    apt-get -y install build-essential sudo && \
    apt-get clean
RUN adduser --disabled-password --gecos '' build
COPY build.sh /

sources.list is a replacement of /etc/apt/sources.list redirecting downloads to the apt proxy I use for all my Debian machines to save bandwidth. build user is used to build the packages (to prevent installing files to e.g. /usr by mistake). sudo is needed to run commands as build user, with an available tty (this is a difference against su) to make gpg password prompt happy.

build.sh is a script that takes a *.dsc file as its argument and performs the build inside the container:

#!/bin/sh -ex

dscfile="$1"
if [ -z "$dscfile" ] || [ -n "$2" ]; then
    echo "usage: $0 DSC-FILE"
    exit 1
fi
dscfile=$(basename $dscfile)

user=build
# Directory to look the provided *.dsc file in:
packagedir=/debian/packages
# Directory to put the built package to:
destdir=/debian/build
export DEB_BUILD_OPTIONS='parallel=8'

cd /home/$user
# Set up signing, the key to be used and pinentry mode to make
# password prompt working in the container:
cp -a /root/.gnupg .
cat >.gnupg/gpg.conf <<EOF
default-key ...id-of-the-key...
pinentry-mode loopback
EOF
chown -R build:build .gnupg
# Unpack the sources and install build dependencies:
sudo -u $user dpkg-source -x $packagedir/$dscfile
cd ${dscfile%%_*}-*
apt-get -y build-dep .
# Build the package:
sudo -u $user dpkg-buildpackage --changes-option=-S
# Set reasonable owner and group on the host and move the built files
# to the destination directory:
cd ..
chown root:root *
mv *.buildinfo *.changes *.deb *.dsc *.tar.* $destdir/

Now the container image can be built:

podman build -t debian-build .

The last thing needed is a script to run the build from the host, in a container named debian-build and with host package and destination directories in $HOME/debian/:

#!/bin/sh

podman rm -fi debian-build
podman run -it --name debian-build \
       -v $HOME/debian:/debian \
       -v $HOME/.gnupg:/root/.gnupg \
       debian-build /build.sh $*

Software problem: Lost network in a Podman container

I use Podman for running various stuff. It’s sometimes challenging due to problems of all categories. One special problem I experienced on one of my computers was that if a container had stopped and then was started again, it lost its network interface, only loopback interface remained available.

I searched the web and hoped it could be related to this bug. So it looked worth trying the newest Podman version. Installing a package from a development version is usually not a thing one wants to do in stable installations of Linux distributions. But on NixOS it should be harmless, so I took the exercise.

I’ve found out it’s quite easy. nixos-unstable channel must be added:

sudo nix-channel --add https://nixos.org/channels/nixos-unstable nixos-unstable
sudo nix-channel --update

And then it must be specified in /etc/nixos/configuration.nix (or wherever systemPackages are defined) and enabled for the given package:

{ config, pkgs, ... }:  
let
  unstable = import <nixos-unstable> { };
in
{
  …
  environment.systemPackages = with pkgs; [
    unstable.podman …
  ];
  …
}

So I got the latest and greatest Podman 3.2.2. Unfortunately it didn’t fix the problem.

Hmm, what now? How to find out what’s happening behind? There was no obvious error in journal and it’s not easy to debug complex tools composed of multiple components.

I could notice that slirp4netns process was running on the first container run but not on the subsequent runs. So it could be still related to the bug above. What if I restart the container with podman restart? The network was still there. OK, so how about running the container in the background and stopping and starting it using podman stop/start? It still worked. Let’s attach to the container using podman attach and exit from there. After starting the container again, the network was gone. I see.

Looks like a bug but how is it possible that nobody has noticed such a relatively noticeable problem? Probably because I couldn’t reproduce the problem on CentOS and Debian, it was present only on NixOS. Searching the web again with more specific keywords didn’t help. But stopping and starting the container again using podman restart brings the network back, which is a sufficient workaround.

I reported the behavior as a NixOS bug. I’m not sure about the software problem category, it may be any of 1, 2, 3. Let’s see what happens with the bug report.

Moving to Podman

Until recently, I used to use chroot, schroot and LXC to run my nested Linux environments. I’ve never used Docker, which has always looked like a single-purpose and quite limited tool to me. I started looking more deeply at Podman last year. I realized that this kind of technology can be mature enough now to be generally useful.

It’s important to set up the right storage driver, before containers are created in the storage, otherwise Podman would be a huge resource hog in the default setup. The best is to use Btrfs or ZFS file system, then Podman can take advantage of easy and lightweight file system cloning and works well. The storage driver to use can be configured (for a non-root user) by creating ~/.config/containers/storage.conf file with the following content (for Btrfs):

[storage]
driver = "btrfs"

I’ve been trying to port some of my environments (even from virtual machines when possible) to Podman last months. It’s not without problems and it’s sometimes hacky but the benefit is that I can unify my nested environments using a single tool. Podman looks like an active project that’s hopefully not going to be abandoned anytime soon so it’s worth to invest into learning and using it.

I’ve already experienced switching from one container environment to another one in the past, when I switched from Linux-VServer to LXC. At the time, I was annoyed by missing Linux-VServer features in LXC (while LXC brought little advantages over Linux-VServer beyond being supported in kernel without patches, which was unfortunately important enough) but I could survive it. It’s different with Podman. Podman provides a lot of features although it is not that easy to use and is apparently more prone to bugs due to its complexity.

Podman has been looking promising and usable for my needs so far. I’m not yet done with conversions (partly because Podman is not yet available in Debian stable and partly following the golden rule of not touching things that work and must work) but I don’t think I’ll have to return back to (s)chroot and LXC.

Using Podman has the following advantages for me:

  • Podman works on images, which are provided for all the major distributions (not every Linux distribution has an equivalent of debootstrap to have an easy way to install it in a chroot environment).
  • Almost everything can be done from the command line, there is no need to edit configuration files.
  • Many actions can be run under a normal user, without root privileges, while still having root rights inside the container.
  • Cooperation with systemd (although not without flaws).
  • There is a lot of things that can be done with Podman.

And the following disadvantages:

  • Podman can work only on images so it’s not possible to run it simply on an unpacked directory, the other tools are still needed for that purpose.
  • Everything must be done from the command line using various commands, which is not so easy as editing a single schroot or LXC configuration file.
  • AFAIK the only official way to change a configuration of a container is to commit its image, which is an expensive operation, and to create a new container from the new image. Again, it’s simpler in schroot or LXC: Editing the configuration file and restarting the container, leaving the container file system untouched.
  • There is no Podman in current Debian stable and installing it manually is a non-trivial action.
  • It’s not very transparent, thinks are hidden behind several different components, internal arrangements of data directories and configuration, and numerous man pages (oh, good manuals are rare these days).

Software problem: KDE Wallet

When I reinstalled my desktop computer, KDE started asking me for a wallet password. Each time a short while after I started my web browser. It didn’t look all right:

  • I don’t think I’ve ever set a wallet password and I deleted whole KDE configuration after I had reinstalled the computer.
  • I couldn’t see any reason why my web browser should access the wallet just after starting.

The bogus password prompt kept popping up every day and it got harder to ignore it. I had two options:

  1. Disable the wallet in KDE settings.
  2. Fix the wallet.

Option 1. would be easier and solving the problem immediately. However I opted for 2. (problem of category 1, user curiosity and naivety) because using the wallet could be useful and I also wondered what tries to access the wallet and why.

It seems KDE Wallet has no usable documentation (another problem of category 1, trying to use software that needs documentation and doesn’t have one), which made the process even more cumbersome than needed.

The first step to move on was to get rid of the wallet I couldn’t access because it asked for a password that I’ve never set. I’ve found out it can be done in KDE wallet manager where it’s possible to delete the wallet. OK, done.

Now the followup step was to create a new wallet. When I was prompted about the wallet type, I opted for GPG encryption. Then I pressed Next button and got an error dialog with the following message:

Seems that your system has no keys suitable for encryption. Please set-up at least one encryption key, then try again.

What? I have a GnuPG encryption key, so what’s the problem?

Looking around, I’ve discovered there is an application called kgpg. At the first run, it didn’t look promising. But at the second run kgpg provided a somewhat better error message:

An error occurred while scanning your keyring:

gpg: Oops: keyid_from_fingerprint: no pubkey

Indeed, gpg reported the same error from a command line when listing my keys:

$ gpg -K
gpg: Oops: keyid_from_fingerprint: no pubkey
…

But it still listed the keys, so what was the problem? I verified that I can encrypt and decrypt files using my default key. After a lot of experiments with gpg commands, I came to a conclusion that my more than a decade old ~/.gnupg directory is no longer all right and perhaps got damaged when it had been converted from gpg version 1 to version 2. So I exported my private keys, deleted ~/.gnupg and imported my previously exported keys. The wallet still insisted on its error but kgpg started working. I went successfully through the kgpg wizard and set everything needed, but the wallet error still remained.

Maybe the problem could be that the wallet doesn’t understand my old keys. A question is why when kgpg can see them fine. Nevertheless it’s not a good idea to use my personal key in KDE so I generated a new key pair. Then I could finally create a new wallet using that key. And only now, too late, KDE Wallet provided the reason why it couldn’t see my old key — it wasn’t set as ultimately trusted.

The seemingly innocent KDE Wallet password dialog popping up in my new installation has led to quite a lot of wasted time, apparently due to problems of categories 2 and 3. The positive effects of all of it are that I have cleaned up my ~/.gnupg (way too much — I’ll have to import some of my old stuff) and that I can use KDE Wallet now if it appears any useful.

Software problem: Wahoo mobile application

I wanted to make a bike trip, planned it on komoot and, just before the trip, tried to synchronize the route to my Wahoo ELEMNT. It didn’t work, the Wahoo device couldn’t connect to WiFi. WiFi problems with Wahoo are not uncommon so I tried to use Wahoo mobile application to download my route.

The Wahoo application generally works but this time it welcomed me with the following screen:

image-wahoo-application.jpg

What?! Why an account is needed to do things like upgrading firmware, changing my device settings (which can be done only using the application, not in the device itself), seeing incoming phone calls, or fetching routes from komoot? It must be either company stupidity or a bad intent, clearly software problem of category 2.

Nevertheless, what could I do at the moment? Creating an account requires agreeing to complicated privacy terms that, to my understanding, permit Wahoo to gather anything. A missing summary explaining the terms in an easy to understand way is probably a GDPR violation and stops me from trusting the company. And stopped me from creating the account at the moment. My primary concern is that I don’t want to upload my recorded data anywhere. While Wahoo could technically already get them using their application before, maybe the whole thing with the account is to get my explicit consent to do that. No, Wahoo, no.

BTW what’s the “seamless ELEMNT Experience” they promise thanks to the account? For example that data can be transferred 100% wirelessly, without the need to connect the device via cables. What a bullshit, don’t they realize that the device cannot be charged wirelessly? I have to connect the device to my computer anyway in order to charge it and then it’s easier to transfer the data using the cable connection than using a mobile application and some web application.

Now back to the planned route, how to transfer it from komoot? komoot allows to download routes, but only without embedded navigation instructions, which is not very useful. Another software problem of category 2, software deficiency.

I couldn’t re-plan the route on my computer in BRouter because I had reinstalled my computer recently and BRouter wasn’t running there yet. I admit, relying on proprietary SaaS is a problem of category 1, user stupidity. Well, it wasn’t a trip into unknown and I decided to ride without navigation, which was OK after all.

But what’s the long term solution? I’m against throwing away usable devices and replacing them with “better” ones, for multiple reasons. So I’ll probably create the account, to be able to use the device in future, and I’ll connect the device with the application only when all the recorded tracks are deleted from it. I hope that Wahoo won’t remove the ability to transfer and delete the tracks privately with a future firmware update (I’ll try to avoid firmware updates if possible). And I’ll ask Wahoo from time to time to tell me what data about me they keep, to be sure. GDPR is a very useful tool to deal with such companies.

I made a mistake when I bought a Polar device (their heart monitors are good but their other devices really suck and are miles behind the competition1). Now I can see I’ve made another mistake when I bought a Wahoo device, I’ll never buy anything from Wahoo again. Instead of trying to save money, I should buy a Garmin device, which is the only remaining option. Garmin is also not anything great, considering they produce and use proprietary FIT formats without freely accessible documentation, but at least last time I checked, it was possible to use Garmin devices, unlike Wahoo or Polar, without any application. Can we still hope that one day there will come a company making sport devices respecting user freedoms? Looking at e.g. the smartphone market, I would say that no. Most people are happy with what we have.

Footnotes:

1

It’s unbelievable that one can still make a device that can be configured only using a proprietary Windows application, doesn’t have enough memory to store data from a multi-day bike trip and can transfer recorded data only using the Windows application or, ehm, a cloud account. The primitive and confusing handling is another problem but maybe some users like it.

Software problem: Emacs windows not properly maximized

I don’t like wasting screen space so the very first configuration action I do in newly installed desktop environments is making all windows maximized by default.

After I had reinstalled my computer, I experienced a problem that new Emacs windows (or frames, in Emacs terminology) created after Emacs had started were not fully vertically maximized, being about two lines shorter than they should. It can be easily remedied by unmaximizing and then maximizing them again but since I use lots of Emacs windows it’s quite annoying to do that all the time.

Some initial observations were:

  • The problem happened only for Emacs windows, not for other applications.
  • The same Emacs version running on a different computer with a different KDE version and on the same monitor worked fine.

This was weird – is it a bug in Emacs or in KDE or in combination of both? Or was it an environment issue? I also suspected Emacs could get confused by a font change during startup or some other configuration change. But running emacs -q proved the problem happens also in the default configuration.

Searching the web also didn’t help. Having no better choice, I tried miscellaneous actions in Emacs until I’ve discovered that the following Emacs setting fixes the problem:

(setq frame-resize-pixelwise t)

Which doesn’t make much sense because:

  • The initial Emacs window is all right.
  • The additional Emacs windows were shorter by more than one line.
  • When a short window was unmaximized and maximized again, it got properly maximized.

So it looks like a bug, i.e. software problem of category 3. It’s hard to say where though and it may be bound to special environment properties so it would probably require a lot of time trying to report it. The workaround is good enough and I like the given setting anyway.