Software problem: Compiling Emacs vterm module on NixOS

I started using NixOS again some time ago. The fact that NixOS doesn’t use Filesystem Hierarchy Standard is sometimes a big source of pain and complications. One non-obvious thing for beginners is how to compile a C program.

One C program that is very useful and must be compiled locally is emacs-libvterm C module, providing a visual terminal for Emacs. There are several good reasons to use emacs-libvterm:

  • It’s sometimes useful to run visual terminal programs.
  • Many command line programs don’t work well with line terminals.
  • If a command line program uses its own command line interface, it’s occasionally useful to take advantage of it. Additionally, Eshell command line editing doesn’t work with programs that enter their own command line interfaces.
  • External, non-Emacs, terminal emulators work well but their capabilities are quite limited and they are not integrated with Emacs.
  • Built-in Emacs visual terminal emulators (term, ansi-term) don’t work very well and are unusable with many applications and utilities.

emacs-libvterm solves all the problems. It works, it can use all the Emacs editing capabilities and it is integrated with Emacs. On standard systems, emacs-libvterm compiles its module automatically, as long as libvterm development package is installed. But it’s more difficult on NixOS.

The first problem is how to compile a C program on NixOS. As far as I understand it, nix-shell environment must be used. The second problem is that libvterm package in NixOS contains something old and apparently unmaintained and libvterm-neovim must be used instead. It took me quite long to realize it, I wasted a lot of time investigating problems related to libvterm and possibly other broken NixOS packages. Which was a problem of category 1, my ignorance, combined with a problem of category 2, broken obsolete packages in the distribution.

So how to compile Emacs vterm module on NixOS? Assuming gcc and gnumake packages are already installed on the system, it’s needed to create the following shell.nix file:

with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "emacsenv";
  nativeBuildInputs = [ cmake ];
  buildInputs = [ libvterm-neovim ];

Then nix-shell is run in the same directory. When Emacs is started in the given nix-shell environment, emacs-libvtem compiles and installs fine.

Software problem: Emacs windows not properly maximized

I don’t like wasting screen space so the very first configuration action I do in newly installed desktop environments is making all windows maximized by default.

After I had reinstalled my computer, I experienced a problem that new Emacs windows (or frames, in Emacs terminology) created after Emacs had started were not fully vertically maximized, being about two lines shorter than they should. It can be easily remedied by unmaximizing and then maximizing them again but since I use lots of Emacs windows it’s quite annoying to do that all the time.

Some initial observations were:

  • The problem happened only for Emacs windows, not for other applications.
  • The same Emacs version running on a different computer with a different KDE version and on the same monitor worked fine.

This was weird – is it a bug in Emacs or in KDE or in combination of both? Or was it an environment issue? I also suspected Emacs could get confused by a font change during startup or some other configuration change. But running emacs -q proved the problem happens also in the default configuration.

Searching the web also didn’t help. Having no better choice, I tried miscellaneous actions in Emacs until I’ve discovered that the following Emacs setting fixes the problem:

(setq frame-resize-pixelwise t)

Which doesn’t make much sense because:

  • The initial Emacs window is all right.
  • The additional Emacs windows were shorter by more than one line.
  • When a short window was unmaximized and maximized again, it got properly maximized.

So it looks like a bug, i.e. software problem of category 3. It’s hard to say where though and it may be bound to special environment properties so it would probably require a lot of time trying to report it. The workaround is good enough and I like the given setting anyway.

Emacs workspaces

I use Emacs for everything that can be done with it reasonably. I often run in a single Emacs session Gnus, Org agendas, command line actions, IRC, several development projects, etc. The question is how to switch between all those activities efficiently.

During the years, I tried several workspace, perspective or screen (whatever they are called) Emacs add-on packages. But I have never been completely satisfied with any of them. There have always been bugs, limitations and/or discomfort.

Then I applied the principle of simplicity: Use the simplest thing that may work. And the thing is Emacs frames. As simple as that, I’ve been using my workspaces based on Emacs frames for almost two years now and it has been working to my satisfaction so far.

The frames provide two crucial facilities:

  • Separate dynamic window configurations and switching between them easily. Frames provide that by definition. The only problem is that your window manager must be able to manage a lot of Emacs frames reasonably. But any civilized window manager should be able to do that and e.g. KDE is.
  • Switching to workspace specific buffers. Frames do that by default — at least with Ido, recently used buffers in the given frame are offered first when switching buffers. Unlike in some other workspace solutions, the buffers offered are not limited just to the given workspace, which is a killer feature. It’s much easier not to care about assignments of buffers to workspaces and to simply rely on the recent buffers approach.

Of course, C-x 5 key bindings are not enough to work with workspaces efficiently. I implemented my own workspace management mechanism, based on my wrappers around the previous solutions I used and described below.

First, it’s useful to track the last frame because returning to the last workspace is a very common action:

(defvar my-last-frame nil)

Let’s track last workspaces too:

(defvar my-last-workspaces '())

Frame switching may not work smoothly in some window environments, so let’s make a custom function to select a frame and to store the workspace to the variables defined above:

(defun my-select-frame (frame)
  (unless (eq frame (selected-frame))
    (setq my-last-frame (selected-frame)))
  (raise-frame frame)
  (x-focus-frame frame)
  (select-frame frame)
  (setq my-last-workspaces (cons frame (remove frame my-last-workspaces))))

I give my workspaces names, stored and displayed in frame titles. Let’s define functions retrieving workspace names:

(defun my-workspace-name (&optional frame)
  (let ((frame (or frame (selected-frame))))
    (or (frame-parameter frame 'title)
        (frame-parameter frame 'name))))

(defun my-all-workspace-names ()
  (setq my-last-workspaces (cl-delete-if-not #'frame-live-p my-last-workspaces))
  (mapcar 'my-workspace-name
          (delete-dups (append my-last-workspaces (frame-list)))))

Now we can switch among workspaces:

(defun my-workspace-switch (name)
  (interactive "sWorkspace: ")
  (let ((frame-list (frame-list))
        (frame nil)
        (new nil))
    (while (and (not frame) frame-list)
      (if (string= (my-workspace-name (car frame-list)) name)
          (setq frame (car frame-list))
        (setq frame-list (cdr frame-list))))
    (unless frame
      (setq frame (make-frame `((title . ,name))))
      (setq new t))
    (my-select-frame frame)

(defun my-last-workspace ()
  (if (and my-last-frame (frame-live-p my-last-frame))
      (my-select-frame my-last-frame)
    (my-workspace-switch (read-string "Switch to workspace: "))))

As you can see, it’s possible to switch to an already present workspace or to a new workspace, using the same command. But typing the workspace name each time is cumbersome, let’s make it easier:

(defun my-switch-to-workspace (&optional key)
  (unless key
    (setq key (logand last-command-event 255)))
  (let* ((prefix (char-to-string key))
         (current-name (my-workspace-name))
         ;; This will be explained later:
         (predefined (cl-assoc prefix my-workspaces :test #'string-prefix-p)))
    (if predefined
        (cl-destructuring-bind (name new-fn refresh-fn) predefined
          (if (equal current-name name)
              (when refresh-fn
                (funcall refresh-fn))
            (when (and (my-workspace-switch name)
              (funcall new-fn))))
      (let* ((candidates
              (remove current-name
                       #'(lambda (n) (string-prefix-p prefix n))
             (n (length candidates)))
         ((= n 0)
           (read-from-minibuffer "Switch to workspace: ")))
         ((= n 1)
           (car candidates)))
           (completing-read "Switch to workspace: " candidates nil t prefix))))))))

(global-set-key (kbd "<s-return>") 'my-last-workspace)
(dotimes (i 26)
  (global-set-key (kbd (format "s-%c" (+ ?a i))) 'my-switch-to-workspace))

This implements a single key workspace switching and assigns it to key bindings. I use a–z letters, as seen above, together with Super modifier to switch between workspaces. I name my workspaces in such a way that the name of each of them preferably starts with a different letter, making this mechanism work very well. I also define C-c z binding to be able to switch among workspaces when Super key happens not working:

(defun my-switch-to-workspace-key (key)
  (interactive "c")
  (my-switch-to-workspace key))  
(global-set-key (kbd "C-c z") 'my-switch-to-workspace-key)

As there are Emacs applications I run often, in their own workspaces, it’s useful to have predefined workspaces:

(defvar my-workspaces
  ;; name
  ;; new-fn
  ;; refresh-fn
     (lambda ()
       (require 'bookmark)
       (unless bookmark-alist
       (bookmark-jump "init.d"))) 
     (lambda ()
       (let ((group-buffer (get-buffer "*Group*")))
         (if group-buffer
             (switch-to-buffer group-buffer)
           (cd "~")))))
     (lambda ()
        (lambda ()
          (ido-buffer-internal ido-default-buffer-method nil nil nil "eshell:")))))))

Each entry in this list defines a workspace name, the function to run when the workspace is created and the function to run when switching to the workspace I’m already in (a “refresh” function), which is especially useful to reset the workspace to its default state. The corresponding part in my-switch-to-workspace above takes care of the predefined workspaces.

One last bit is handling workspaces containing development projects. It’s natural to name them after their source directories but that could quickly lead to initial letter collisions with other workspaces. I dedicated a special x: prefix to those workspaces. I use Projectile to manage my source directories and integrate it with my workspaces:

(defconst my-projectile-workspace-prefix "x:")

(defun my-projectile-switch-project-action ()
  (when (my-workspace-switch (concat my-projectile-workspace-prefix
(setq projectile-switch-project-action 'my-projectile-switch-project-action)  

When I open a development project using Projectile, it automatically switches to its workspace, names it accordingly and opens Magit for it. Then I can switch to the project using s-x key. If I work on multiple development projects simultaneously, my-switch-to-workspace offers the last one used by default to avoid using completion all the time.

As you can see, utilizing Emacs frames as workspaces is easy and powerful. I don’t need to use or make special add-ons to have workspaces, Emacs frames are good enough and they can be customized as needed relatively easily.

speechd-el 2.11 released

speechd-el 2.11 release introduces index marking feature, i.e. moving the cursor as the text of a buffer is read. This feature has been planned for more than 10 years but due to various circumstances, its implementation, although not that difficult, has been delayed. Now it is finally implemented although it may need further improvements in future if performance problems are experienced in practice. Sending long texts to Speech Dispatcher in chunks may be one of useful improvements.

speechd-el was designed as a project requiring low maintenance that needn’t be updated with each new Emacs version. Time has proved it is indeed so and speechd-el rarely requires updates of its code base. However, speechd-el has been here long enough to experience some conceptual changes in Emacs. During the time, Emacs has deprecated cl library in favor of cl-lib, improved the function advising mechanism and introduced a true lexical let. 2.11 version adjusts to those changes and, together with some other fixes, removed all compilation warnings.

So speechd-el is now in a clean state and the last long-planned feature has been implemented. I currently don’t plan any further changes to speechd-el unless there are bug reports, feature requests or patches from its users.

Capturing Web content from Firefox to Org

Emacs is a powerful tool but it’s better to use other means for Web browsing, such as Firefox. Now the question is how to transfer pieces of Web content from Firefox to Org mode. Org mode already provides means for communication with external applications. org-protocol.el is a general mechanism for importing information to Org mode via emacsclient, but its setup is not instant and I hadn’t bothered to configure it until I met org-protocol-capture-html. The screenshot of the captured content converted to Org markup was irresistible so I decided to give capturing Web content another try.

That attempt reminded me that I hadn’t ranted about software setup and bugs for quite long time here. I’m not going to fix that now, it suffices to say that utilizing a relatively simple function shouldn’t require advanced technical knowledge and/or several hours of googling and experimenting; I really can’t imagine how a non-advanced user could get that thing run without losing his patience at early stages of the process. Well, so I’ll try to make a summary of how I got it working.

Emacs part

I assume you already can use Org mode and emacsclient.

Plain text capture

Add org-protocol to org-modules variable.

Define entry for capturing Web content in org-capture-templates variable, e.g.:

(add-to-list 'org-capture-templates
             ("w" "Web site" entry (file "~/org/")
              "* %?\n%c\n%:initial"))

Of course, this is just an example. Look at org-protocol.el documentation for another example and for explanation what %:initial means.

If you’d like to use a letter different from w for the template, you can do so but you must replace it in Firefox bookmarklets and helpers below. See also org-protocol-default-template-key variable.

Capture with HTML conversion

First, configure plain text capture as described above. Then fetch org-protocol-capture-html.el from its home page and put it into your site-lisp directory. Add the following lines to your ~/.emacs or other Emacs initialization file:

(require 'org-protocol)
(require 'org-protocol-capture-html)

Note that org-protocol must be already loaded at the time org-protocol-capture-html is loaded, otherwise the corresponding subprotocol won’t be registered.

Firefox part

The easy way

Install Org-capture for Firefox. It allows capturing content without the need to register org-protocol: handler in Firefox. However, org-protocol-capture-html or other custom captures won’t work this way.

The advanced way

The following installs universal capturing mechanism via org-protocol: handler in Firefox. It works independently (with or without it) of Org-capture Firefox extension mentioned above.

Register org-protocol: handler as described in MozillaZine Knowledge Base (replace foo with org-protocol). One important thing they forgot to emphasize is that you must use real link to invoke the application dialog, typing org-protocol:something into the address bar doesn’t work. For your convenience, I provide an org-protocol link here. Select something like /usr/bin/emacsclient in the Firefox dialog as the application handling org-protocol.

Then define your capturing bookmarklets. If you don’t have Bookmarks Toolbar enabled, enable it by right clicking on a Firefox toolbar and selecting Bookmarks Toolbar. Then create new bookmark in Bookmarks Toolbar section and insert the following code as its URL:


This is for plain text capture. If you want HTML capture, define another toolbar bookmark and use the code from org-protocol-capture-html home page (it’s also available in the introductory comments in org-protocol-capture-html.el) as its URL. Just make sure that:

  • The bookmark URL starts with javascript:.
  • Pandoc is installed.

Now you can (optionally) select part of an HTML page and press one of the newly created bookmark buttons in Bookmarks Toolbar. If everything is set up correctly, the selected part of the page (or just page URL and title if nothing is selected) should appear in your Emacs capture buffer.

Getting rid of Bookmarks Toolbar

If you don’t use Bookmarks Toolbar in Firefox, you probably don’t want to waste screen space on it just for Org capture bookmarks. The remedy is easy, invoke Firefox Customize and drag Bookmarks Toolbar to another place. Alternatively, you can use Custom Buttons Firefox extension.


Some Web pages can’t be captured, I don’t know why. I have more important things to do than playing with Org and Firefox further.

Another useful Org related Firefox extension is Copy as Org-mode. It doesn’t capture content via org-protocol but can copy some objects such as page or link URLs to kill ring, in the Org format. This is what I used to insert links into this article! It’s easier with this nice helper than performing all the copy&paste&edit by hand.

20 years with Emacs

IIRC, it was about 20 years ago when I ran Emacs for the first time. I originally used it on a System V system as a civilized alternative to vi editor. But during the time I became an Emacs power user, learned Elisp and started to use Emacs as my primary working environment.

Emacs, as a truly extensible text editor, is an excellent environment for implementing text handling applications (and not only them). I use it as a planner and information storage system (including things like flash card training), mail and news reader, powerful command line processor (better than readline which is limited by terminal capabilities) and command line debugger interface, file manager, interface to Unix tools, git interface, Common Lisp IDE, remote file handler (no need to run a separate Emacs instance on the remote machine), Jabber client, music player, etc.

Emacs applications use to be very powerful. I don’t use the above mentioned applications because they run in Emacs but because they work better for me than anything else I’ve tried. There are several reasons why it is so. Emacs applications often implement just user interfaces to standard tools such as ls, grep, ssh, git or debuggers without duplicating their functionality. Most user activities are based on some sort of text processing regardless the graphical elements of their user interface and Emacs is strong in text processing. Emacs provides high level programming environment with powerful and simple to use customization features. Emacs makes a uniform platform encouraging code sharing among its applications. Emacs is used primarily by power users so it can focus on usability rather than on handling ignorance. And finally Emacs attracts creative and skilful programmers. All these features together make Emacs a unique development platform.

One notable missing feature in Emacs is a Web browser. Emacs didn’t catch up with Web progress and there is no generally useable Emacs Web browser. But hopefully Emacs will be able to build something based on interaction with an extensible Web browser environment using XEmbed or so in future (some experiments with Uzbl have already been done).

An important Emacs feature is its stability. When I first started to use Emacs, after using buggy MS-DOS IDEs, it seemed to me there are no bugs in Emacs. Of course, Emacs is not perfect and there have always been bugs in it, but it took a few years before I actually experienced an Emacs bug for the first time. Now, when I use Emacs for many activities, I meet bugs more often but their number and importance is still acceptable, in comparison to applications I had to abandon because they started to expose unacceptable functional or usability problems. But stability is not only about bugs, it’s also about persistent consistency, code quality and documentation; Emacs has always been strong in these areas.

I use to be quite critical to software, for good reasons. But my twenty years with Emacs were happy. Since I started to use Emacs I’ve always been mostly satisfied with it and I’ve never had to look for a better alternative. I consider Emacs being an excellent demonstration of how free software, free of direct commercial pressures and combined with a community of excellent software developers, can successfully survive and continuously evolve for decades. I assume Emacs will continue to grow in future and will remain my primary working tool, perhaps forever.

Getting organized

I’ve never been successful in keeping really useful and up-to-date diaries and todo lists. Well, one can think: “If it’s really important I won’t forget about it and the other things don’t matter much.” But if nothing else then family life can cause semi-chaos making really difficult to get anything done. Nevertheless an important change has happened to me last year.

I used to use KDE PIM suite as my software organizer (indeed, there are some important things one can forget of). Last summer I had to upgrade my desktop and got exposed to the infamous KDE 4. I hadn’t been very comfortable with KDE PIM before the upgrade and it got only worse afterwards. I think I needn’t explain much, KDE PIM simply wasn’t among the few excellent and already completely working KDE 4 applications (believe or not there are some).

By chance I met Emacs Org-mode at that time. This wasn’t for the first time I’ve seen it. But previously either Org-mode wasn’t mature or I didn’t really understand its concepts (perhaps both). This time I found a mature product with interesting concepts and features. After some exploration I decided to move from KDE PIM to Org-mode.

What’s so good about Org-mode? The basic idea of simplicity and flexibility. You can just write down anything that comes to your mind and put it into plain text files using simple markup. And then you can use available tools to manage all the information. Easy, isn’t it? Actually it may not be that easy, it requires some mental change when abandoning GUI tools designed to drive you and moving to a plain text organizer designed to be driven by you. But once you get it you’ll start to like it a lot.

You can simply start writing anything you want in your editor without having to use menu commands and to navigate through fields and buttons of predefined dialogs – you define how information is organized. You can keep related information in a single place – no need to split a journal entry containing a calendar item, a short todo list, a contact and some note into five different locations in five different applications. You can structure and organize your information as you like, it can be anything from a single big file to a directory tree containing many different files. In any case you can use the presentation tools to retrieve and combine required information. And you can customize it as you like, it’s Emacs after all.

In the result KDE PIM wasn’t the only set of applications I abandoned in favor of Org-mode. Think, you can put a lot of things into plain text files: notes, tasks, events, anniversaries, journal, bookmarks, passwords, contacts, vocabularies, clocking, documents, source code, commands, ideas, this blog entry, etc. There are many contributions to Org-mode, putting some kinds of specialized applications into obsolescense due to their limitations, low flexibility and lack of integration.

A very important part of Org-mode is its excellent documentation. When you’ve got a flexible software, good documentation is essential because the software is much about general concepts and personal preferences. It’s interesting to see that e.g. KDE PIM pages present just feature lists and completely omit the important areas. Org-mode is equipped with a complete user manual, tutorials, presentations, FAQ and other documentation that help you learn how to manage things. It’s software with intelligent community around it.

I can confirm the Org-mode way works. I was able to reasonably organize my things soon after I had begun to start using Org-mode and this is for the first time I’m able to keep my agenda in order. Org-mode has impressed me a lot. Not because it’s perfect (it’s not) or because of its feature set (it offers much more than I need while some things are missing), but because of its excellent concepts that solve the right problem, in a right way and for a useful purpose.

Well, this is not an introduction to Org-mode nor a feature overview. If you’d like to know what Org-mode actually is, look at Org-mode home page where you can find explanation of its concepts and everything else about it.

Fonts in Emacs 23.2

When my Emacs was upgraded from 23.1 to 23.2, I got negatively surprised: It changed font rendering, making fonts significantly less readable. I quickly identified the corresponding NEWS entry:

*** On X11, Emacs reacts to Xft changes made by configuration tools,
via the XSETTINGS mechanism.  This includes antialias, hinting,
hintstyle, RGBA, DPI and lcdfilter changes.

But I didn’t know anything about Xft and XSETTINGS. I started to look for a quick solution and tried to install and use common desktop configuration tools such as gnome-control-center, systemsettings and xfce4-settings. I’ve found that font adjustments apply in Emacs only when using the GNOME tools. So running gnome-settings-daemon was necessary and it was very annoying: It changes keyboard layout on startup and after switching back to X from console, breaks multimedia keys, pops up annoying dialogs on various occasions, etc. and I couldn’t find a way to prevent all the odd behavior.

Now I’ve finally got rid of it. I discovered Xft font properties can be set simply via X resources. After some testing and elaboration I ended up with the following new lines in my ~/.Xresources:

Xft.antialias: 1
Xft.hinting:   1
Xft.hintstyle: hintslight
Xft.lcdfilter: lcdnone
Xft.rgba:	 none

Now fonts in Emacs look fine again and gnome-settings-daemon is no longer present on my computer.

Tramping in Emacs

Why to use separate terminal applications for remote connections when one can use Emacs? Why to run remote Emacs of different version, in different configuration, with echo delays, without your customizations, etc.? Why to pollute your desktop with another window and why to switch between it and Emacs all the time? Why to run multiple editors?

The only reason I didn’t simply use Tramp for remote connections was lack of command line completion. But that can be done with Tramp too, one just must configure it. rlogin.el doesn’t seem to be directly useful for that purpose, but it’s useful for getting a clue. So I wrote a little hack:

(defun my-shell-directory-tracker (command)
  (when (string-match "^ssh " command)
    (make-local-variable 'comint-file-name-prefix)
    (let ((host (comint-substitute-in-file-name
                 (comint-arguments command nil nil))))
      (set (make-local-variable 'comint-file-name-prefix) (format "/%s:" host))
      (set (make-local-variable 'comint-process-echoes) t)
      (cd-absolute (format "%s~" comint-file-name-prefix))
      (rename-buffer (format "*shell:%s*" host)))))
(add-hook 'comint-input-filter-functions 'my-shell-directory-tracker)

Now I can enjoy remote editing (both editing files and editing command lines) quite comfortably and without echo delays.

This hack is not perfect, but it has good usefulness/effort ratio. If you can improve it or know about a better solution, please tell me.