Capturing Web content from Firefox to Org

Emacs is a powerful tool but it’s better to use other means for Web browsing, such as Firefox. Now the question is how to transfer pieces of Web content from Firefox to Org mode. Org mode already provides means for communication with external applications. org-protocol.el is a general mechanism for importing information to Org mode via emacsclient, but its setup is not instant and I hadn’t bothered to configure it until I met org-protocol-capture-html. The screenshot of the captured content converted to Org markup was irresistible so I decided to give capturing Web content another try.

That attempt reminded me that I hadn’t ranted about software setup and bugs for quite long time here. I’m not going to fix that now, it suffices to say that utilizing a relatively simple function shouldn’t require advanced technical knowledge and/or several hours of googling and experimenting; I really can’t imagine how a non-advanced user could get that thing run without losing his patience at early stages of the process. Well, so I’ll try to make a summary of how I got it working.

Emacs part

I assume you already can use Org mode and emacsclient.

Plain text capture

Add org-protocol to org-modules variable.

Define entry for capturing Web content in org-capture-templates variable, e.g.:

(add-to-list 'org-capture-templates
             ("w" "Web site" entry (file "~/org/notes.org")
              "* %?\n%c\n%:initial"))

Of course, this is just an example. Look at org-protocol.el documentation for another example and for explanation what %:initial means.

If you’d like to use a letter different from w for the template, you can do so but you must replace it in Firefox bookmarklets and helpers below. See also org-protocol-default-template-key variable.

Capture with HTML conversion

First, configure plain text capture as described above. Then fetch org-protocol-capture-html.el from its home page and put it into your site-lisp directory. Add the following lines to your ~/.emacs or other Emacs initialization file:

(require 'org-protocol)
(require 'org-protocol-capture-html)

Note that org-protocol must be already loaded at the time org-protocol-capture-html is loaded, otherwise the corresponding subprotocol won’t be registered.

Firefox part

The easy way

Install Org-capture for Firefox. It allows capturing content without the need to register org-protocol: handler in Firefox. However, org-protocol-capture-html or other custom captures won’t work this way.

The advanced way

The following installs universal capturing mechanism via org-protocol: handler in Firefox. It works independently (with or without it) of Org-capture Firefox extension mentioned above.

Register org-protocol: handler as described in MozillaZine Knowledge Base (replace foo with org-protocol). One important thing they forgot to emphasize is that you must use real link to invoke the application dialog, typing org-protocol:something into the address bar doesn’t work. For your convenience, I provide an org-protocol link here. Select something like /usr/bin/emacsclient in the Firefox dialog as the application handling org-protocol.

Then define your capturing bookmarklets. If you don’t have Bookmarks Toolbar enabled, enable it by right clicking on a Firefox toolbar and selecting Bookmarks Toolbar. Then create new bookmark in Bookmarks Toolbar section and insert the following code as its URL:

javascript:location.href='org-protocol:/capture:/w/'+encodeURIComponent(location.href)+'/'+encodeURIComponent(document.title)+'/'+encodeURIComponent(window.getSelection())

This is for plain text capture. If you want HTML capture, define another toolbar bookmark and use the code from org-protocol-capture-html home page (it’s also available in the introductory comments in org-protocol-capture-html.el) as its URL. Just make sure that:

  • The bookmark URL starts with javascript:.
  • Pandoc is installed.

Now you can (optionally) select part of an HTML page and press one of the newly created bookmark buttons in Bookmarks Toolbar. If everything is set up correctly, the selected part of the page (or just page URL and title if nothing is selected) should appear in your Emacs capture buffer.

Getting rid of Bookmarks Toolbar

If you don’t use Bookmarks Toolbar in Firefox, you probably don’t want to waste screen space on it just for Org capture bookmarks. The remedy is easy, invoke Firefox Customize and drag Bookmarks Toolbar to another place. Alternatively, you can use Custom Buttons Firefox extension.

Notes

Some Web pages can’t be captured, I don’t know why. I have more important things to do than playing with Org and Firefox further.

Another useful Org related Firefox extension is Copy as Org-mode. It doesn’t capture content via org-protocol but can copy some objects such as page or link URLs to kill ring, in the Org format. This is what I used to insert links into this article! It’s easier with this nice helper than performing all the copy&paste&edit by hand.

20 years with Emacs

IIRC, it was about 20 years ago when I ran Emacs for the first time. I originally used it on a System V system as a civilized alternative to vi editor. But during the time I became an Emacs power user, learned Elisp and started to use Emacs as my primary working environment.

Emacs, as a truly extensible text editor, is an excellent environment for implementing text handling applications (and not only them). I use it as a planner and information storage system (including things like flash card training), mail and news reader, powerful command line processor (better than readline which is limited by terminal capabilities) and command line debugger interface, file manager, interface to Unix tools, git interface, Common Lisp IDE, remote file handler (no need to run a separate Emacs instance on the remote machine), Jabber client, music player, etc.

Emacs applications use to be very powerful. I don’t use the above mentioned applications because they run in Emacs but because they work better for me than anything else I’ve tried. There are several reasons why it is so. Emacs applications often implement just user interfaces to standard tools such as ls, grep, ssh, git or debuggers without duplicating their functionality. Most user activities are based on some sort of text processing regardless the graphical elements of their user interface and Emacs is strong in text processing. Emacs provides high level programming environment with powerful and simple to use customization features. Emacs makes a uniform platform encouraging code sharing among its applications. Emacs is used primarily by power users so it can focus on usability rather than on handling ignorance. And finally Emacs attracts creative and skilful programmers. All these features together make Emacs a unique development platform.

One notable missing feature in Emacs is a Web browser. Emacs didn’t catch up with Web progress and there is no generally useable Emacs Web browser. But hopefully Emacs will be able to build something based on interaction with an extensible Web browser environment using XEmbed or so in future (some experiments with Uzbl have already been done).

An important Emacs feature is its stability. When I first started to use Emacs, after using buggy MS-DOS IDEs, it seemed to me there are no bugs in Emacs. Of course, Emacs is not perfect and there have always been bugs in it, but it took a few years before I actually experienced an Emacs bug for the first time. Now, when I use Emacs for many activities, I meet bugs more often but their number and importance is still acceptable, in comparison to applications I had to abandon because they started to expose unacceptable functional or usability problems. But stability is not only about bugs, it’s also about persistent consistency, code quality and documentation; Emacs has always been strong in these areas.

I use to be quite critical to software, for good reasons. But my twenty years with Emacs were happy. Since I started to use Emacs I’ve always been mostly satisfied with it and I’ve never had to look for a better alternative. I consider Emacs being an excellent demonstration of how free software, free of direct commercial pressures and combined with a community of excellent software developers, can successfully survive and continuously evolve for decades. I assume Emacs will continue to grow in future and will remain my primary working tool, perhaps forever.

Getting organized

I’ve never been successful in keeping really useful and up-to-date diaries and todo lists. Well, one can think: “If it’s really important I won’t forget about it and the other things don’t matter much.” But if nothing else then family life can cause semi-chaos making really difficult to get anything done. Nevertheless an important change has happened to me last year.

I used to use KDE PIM suite as my software organizer (indeed, there are some important things one can forget of). Last summer I had to upgrade my desktop and got exposed to the infamous KDE 4. I hadn’t been very comfortable with KDE PIM before the upgrade and it got only worse afterwards. I think I needn’t explain much, KDE PIM simply wasn’t among the few excellent and already completely working KDE 4 applications (believe or not there are some).

By chance I met Emacs Org-mode at that time. This wasn’t for the first time I’ve seen it. But previously either Org-mode wasn’t mature or I didn’t really understand its concepts (perhaps both). This time I found a mature product with interesting concepts and features. After some exploration I decided to move from KDE PIM to Org-mode.

What’s so good about Org-mode? The basic idea of simplicity and flexibility. You can just write down anything that comes to your mind and put it into plain text files using simple markup. And then you can use available tools to manage all the information. Easy, isn’t it? Actually it may not be that easy, it requires some mental change when abandoning GUI tools designed to drive you and moving to a plain text organizer designed to be driven by you. But once you get it you’ll start to like it a lot.

You can simply start writing anything you want in your editor without having to use menu commands and to navigate through fields and buttons of predefined dialogs – you define how information is organized. You can keep related information in a single place – no need to split a journal entry containing a calendar item, a short todo list, a contact and some note into five different locations in five different applications. You can structure and organize your information as you like, it can be anything from a single big file to a directory tree containing many different files. In any case you can use the presentation tools to retrieve and combine required information. And you can customize it as you like, it’s Emacs after all.

In the result KDE PIM wasn’t the only set of applications I abandoned in favor of Org-mode. Think, you can put a lot of things into plain text files: notes, tasks, events, anniversaries, journal, bookmarks, passwords, contacts, vocabularies, clocking, documents, source code, commands, ideas, this blog entry, etc. There are many contributions to Org-mode, putting some kinds of specialized applications into obsolescense due to their limitations, low flexibility and lack of integration.

A very important part of Org-mode is its excellent documentation. When you’ve got a flexible software, good documentation is essential because the software is much about general concepts and personal preferences. It’s interesting to see that e.g. KDE PIM pages present just feature lists and completely omit the important areas. Org-mode is equipped with a complete user manual, tutorials, presentations, FAQ and other documentation that help you learn how to manage things. It’s software with intelligent community around it.

I can confirm the Org-mode way works. I was able to reasonably organize my things soon after I had begun to start using Org-mode and this is for the first time I’m able to keep my agenda in order. Org-mode has impressed me a lot. Not because it’s perfect (it’s not) or because of its feature set (it offers much more than I need while some things are missing), but because of its excellent concepts that solve the right problem, in a right way and for a useful purpose.

Well, this is not an introduction to Org-mode nor a feature overview. If you’d like to know what Org-mode actually is, look at Org-mode home page where you can find explanation of its concepts and everything else about it.

Fonts in Emacs 23.2

When my Emacs was upgraded from 23.1 to 23.2, I got negatively surprised: It changed font rendering, making fonts significantly less readable. I quickly identified the corresponding NEWS entry:

*** On X11, Emacs reacts to Xft changes made by configuration tools,
via the XSETTINGS mechanism.  This includes antialias, hinting,
hintstyle, RGBA, DPI and lcdfilter changes.

But I didn’t know anything about Xft and XSETTINGS. I started to look for a quick solution and tried to install and use common desktop configuration tools such as gnome-control-center, systemsettings and xfce4-settings. I’ve found that font adjustments apply in Emacs only when using the GNOME tools. So running gnome-settings-daemon was necessary and it was very annoying: It changes keyboard layout on startup and after switching back to X from console, breaks multimedia keys, pops up annoying dialogs on various occasions, etc. and I couldn’t find a way to prevent all the odd behavior.

Now I’ve finally got rid of it. I discovered Xft font properties can be set simply via X resources. After some testing and elaboration I ended up with the following new lines in my ~/.Xresources:

Xft.antialias: 1
Xft.hinting:   1
Xft.hintstyle: hintslight
Xft.lcdfilter: lcdnone
Xft.rgba:	 none

Now fonts in Emacs look fine again and gnome-settings-daemon is no longer present on my computer.

Tramping in Emacs

Why to use separate terminal applications for remote connections when one can use Emacs? Why to run remote Emacs of different version, in different configuration, with echo delays, without your customizations, etc.? Why to pollute your desktop with another window and why to switch between it and Emacs all the time? Why to run multiple editors?

The only reason I didn’t simply use Tramp for remote connections was lack of command line completion. But that can be done with Tramp too, one just must configure it. rlogin.el doesn’t seem to be directly useful for that purpose, but it’s useful for getting a clue. So I wrote a little hack:

(defun my-shell-directory-tracker (command)
  (when (string-match "^ssh " command)
    (make-local-variable 'comint-file-name-prefix)
    (let ((host (comint-substitute-in-file-name
                 (comint-arguments command nil nil))))
      (set (make-local-variable 'comint-file-name-prefix) (format "/%s:" host))
      (set (make-local-variable 'comint-process-echoes) t)
      (cd-absolute (format "%s~" comint-file-name-prefix))
      (rename-buffer (format "*shell:%s*" host)))))
(add-hook 'comint-input-filter-functions 'my-shell-directory-tracker)

Now I can enjoy remote editing (both editing files and editing command lines) quite comfortably and without echo delays.

This hack is not perfect, but it has good usefulness/effort ratio. If you can improve it or know about a better solution, please tell me.