Capturing Web content from Firefox to Org

Emacs is a powerful tool but it’s better to use other means for Web browsing, such as Firefox. Now the question is how to transfer pieces of Web content from Firefox to Org mode. Org mode already provides means for communication with external applications. org-protocol.el is a general mechanism for importing information to Org mode via emacsclient, but its setup is not instant and I hadn’t bothered to configure it until I met org-protocol-capture-html. The screenshot of the captured content converted to Org markup was irresistible so I decided to give capturing Web content another try.

That attempt reminded me that I hadn’t ranted about software setup and bugs for quite long time here. I’m not going to fix that now, it suffices to say that utilizing a relatively simple function shouldn’t require advanced technical knowledge and/or several hours of googling and experimenting; I really can’t imagine how a non-advanced user could get that thing run without losing his patience at early stages of the process. Well, so I’ll try to make a summary of how I got it working.

Emacs part

I assume you already can use Org mode and emacsclient.

Plain text capture

Add org-protocol to org-modules variable.

Define entry for capturing Web content in org-capture-templates variable, e.g.:

(add-to-list 'org-capture-templates
             ("w" "Web site" entry (file "~/org/notes.org")
              "* %?\n%c\n%:initial"))

Of course, this is just an example. Look at org-protocol.el documentation for another example and for explanation what %:initial means.

If you’d like to use a letter different from w for the template, you can do so but you must replace it in Firefox bookmarklets and helpers below. See also org-protocol-default-template-key variable.

Capture with HTML conversion

First, configure plain text capture as described above. Then fetch org-protocol-capture-html.el from its home page and put it into your site-lisp directory. Add the following lines to your ~/.emacs or other Emacs initialization file:

(require 'org-protocol)
(require 'org-protocol-capture-html)

Note that org-protocol must be already loaded at the time org-protocol-capture-html is loaded, otherwise the corresponding subprotocol won’t be registered.

Firefox part

The easy way

Install Org-capture for Firefox. It allows capturing content without the need to register org-protocol: handler in Firefox. However, org-protocol-capture-html or other custom captures won’t work this way.

The advanced way

The following installs universal capturing mechanism via org-protocol: handler in Firefox. It works independently (with or without it) of Org-capture Firefox extension mentioned above.

Register org-protocol: handler as described in MozillaZine Knowledge Base (replace foo with org-protocol). One important thing they forgot to emphasize is that you must use real link to invoke the application dialog, typing org-protocol:something into the address bar doesn’t work. For your convenience, I provide an org-protocol link here. Select something like /usr/bin/emacsclient in the Firefox dialog as the application handling org-protocol.

Then define your capturing bookmarklets. If you don’t have Bookmarks Toolbar enabled, enable it by right clicking on a Firefox toolbar and selecting Bookmarks Toolbar. Then create new bookmark in Bookmarks Toolbar section and insert the following code as its URL:

javascript:location.href='org-protocol:/capture:/w/'+encodeURIComponent(location.href)+'/'+encodeURIComponent(document.title)+'/'+encodeURIComponent(window.getSelection())

This is for plain text capture. If you want HTML capture, define another toolbar bookmark and use the code from org-protocol-capture-html home page (it’s also available in the introductory comments in org-protocol-capture-html.el) as its URL. Just make sure that:

  • The bookmark URL starts with javascript:.
  • Pandoc is installed.

Now you can (optionally) select part of an HTML page and press one of the newly created bookmark buttons in Bookmarks Toolbar. If everything is set up correctly, the selected part of the page (or just page URL and title if nothing is selected) should appear in your Emacs capture buffer.

Getting rid of Bookmarks Toolbar

If you don’t use Bookmarks Toolbar in Firefox, you probably don’t want to waste screen space on it just for Org capture bookmarks. The remedy is easy, invoke Firefox Customize and drag Bookmarks Toolbar to another place. Alternatively, you can use Custom Buttons Firefox extension.

Notes

Some Web pages can’t be captured, I don’t know why. I have more important things to do than playing with Org and Firefox further.

Another useful Org related Firefox extension is Copy as Org-mode. It doesn’t capture content via org-protocol but can copy some objects such as page or link URLs to kill ring, in the Org format. This is what I used to insert links into this article! It’s easier with this nice helper than performing all the copy&paste&edit by hand.

Privacy Badger

Privacy Badger 1.0 has been released some time ago. If I could install a single Firefox extension only, it would be this one. It’s very easy to use, effective (AFAICT) and doesn’t cause problems; much better than the other privacy extensions I used (although they can still be useful for special purposes). I can recommended using it, as well as reading the FAQ and supporting the Electronic Frontier Foundation (EFF). BTW, another very useful web browser extension by EFF is HTTPS Everywhere. Good work, EFF! Looking forward an Android version as there is currently very little one can do to protect his privacy in Firefox for Android.

How Internet Explorer supported free software

Internet Explorer, the nightmare of all web developers and many Internet users, the vendor lock-in application of the worst kind, has significantly contributed to the advancement of free software. How is it possible that such a product could help us?

When Internet Explorer appeared all the (at the time) modern and popular web browsers were proprietary products widely used by most people, including free software users. The leading web browser, Netscape, was another such product, distributed for free. That free beer was so popular that there was little demand for a free alternative and the presence and monopoly of Netscape effectively blocked development of a modern free web browser. That was a big problem: The Web, an emerging universal platform for information interchange, was here without any good free software to access it. It was ruled by a proprietary browser of unclear qualities, e.g. Netscape was suspected of leaking private information (BTW, the suspicion was probably false 20 years ago while the private information is routinely leaked these days through web sites and services; see IceCat for partial fixes). And then Internet Explorer started to grow and initiated the war of browsers.

It had taken some time before Netscape realized that their browser couldn’t compete against the universal desktop computer monopolist. At one time the numbers became more than clear and Netscape completely lost its market. It happens sometimes that companies finally, although quite late, recognize their mistakes and manage to help saving their works before they bankrupt (see Sun’s Java for another example). So did Netscape and the Mozilla project was founded. The project was successful, led to the development of Firefox and so we have a modern free and working web browser today. If Internet Explorer didn’t destroy Netscape market, the development of a competitive free web browser could be delayed by many more years.

So we can celebrate the end of Internet Explorer development twice: First for getting rid of the horror and second for helping us to get a good free web browser. It’s a good illustration how bad things can, contrary to their intentions, do something good. Turning bad things into making good is a very important general principle. This should be part of our strategy when making things better.

Debian GNU/Linux on Android

Android tablets are useful portable devices. Unfortunately Android makes them less useful than they could be. There are several problems with Android, one of them being lack of applications. While there are free Android applications for many common tasks, they are typically not suitable for serious work and many applications are completely missing, most notably (real) Emacs.

But it’s possible to partially overcome the problem by installing GNU/Linux in a chroot environment on Android, making an Android tablet a poor man’s substitute for a laptop on occasional traveling. Here’s what I’ve found about running Debian GNU/Linux on Android after some experiments and testing a couple of alternatives:

  • Get a Bluetooth or other hardware keyboard. The software keyboards take screen space, are not suitable for touch typing and are especially uncomfortable to use with X.
  • Install XSDL X server, available also on Google Play. The other Android X Server, available on F-Droid, doesn’t work very well, nor the free VNC servers I tried.
  • Install GNURoot, together with one of its root images, available from Google Play. GNURoot environment works very well for many tasks, as long as you don’t need real root access.
  • Start the X server, start GNURoot, install your favorite Debian packages, set DISPLAY to localhost:0, run your favorite window manager and the applications you need. The X environment is slow but useable.
  • If you need real root access, you can use Lil’ Debi, available on F-Droid. But there are some limitations: Your Android must be rooted (obviously), the file system size is limited to about 2 GB and I couldn’t connect to the X server under user other than root.

Installing an alternative ROM on Google Nexus 10

Enough is enough. I don’t like how the stock Android installation treats its users and I finally decided to replace it on my Google Nexus 10 with something else. The reasonable choices were CyanogenMod, AOKP and OmniROM. I decided to install AOKP.

Installation instructions looked like installing an alternative ROM on Nexus is a completely easy (and safe) process. It wasn’t that easy though, there is some missing and misleading information in those installation instructions. Due to it, I got a bit scared about three times during the process and had to search the Web for solutions. Thanks to all those kind people who post instructions, answers and solutions I was able to complete the installation successfully. So here is another short guide summarizing how to install an alternative ROM on current (4.4) stock Nexus 10, based on my own experience:

  • Accept that by following these or other similar instructions you can brick your device, lose warranty, etc.
  • Backup everything on your tablet you want to retain. All data may get deleted during the installation.
  • Install Android SDK on your PC. You need it to get working adb and fastboot.
  • On your PC, download TWRP recovery image for your device. You can use ClockworkMod as well, I chose TWRP so that I could contingently install OmniROM without replacing the recovery again.
  • Download the desired ROM and Google applications (if you want them) and put them to /sdcard/ directory on your tablet.
  • Enable developer’s setting on the tablet: Go to settings, device information and tap several times on the build number, until you see the message about being a developer.
  • Go back in settings, enter the developer’s settings and enable USB debugging.
  • Connect the tablet to your PC via a USB cable. The tablet should indicate that you’re in USB developer’s mode.
  • Check that sudo adb devices (use adb binary from Android SDK) shows your tablet on your PC. Note it’s important to run adb as root for the first time. It starts a server process on the first invocation and if that starts under an ordinary user, adb functions may not work even under root later.
  • If there is something strange in adb devices output, such as no entry at all, question marks or a message about permissions, then there is something wrong with your setup.
  • Run adb reboot bootloader. The tablet reboots and enters bootloader menu.
  • Run sudo fastboot oem unlock (use fastboot binary from Android SDK). The tablet should reboot, with unlocked lock displayed under “Google”.
  • Perhaps the tablet gets stuck on the boot. It looks scary, but it can (hopefully) be fixed following the next instructions. If the tablet boots normally, skip a few next instructions and continue with “re-enable USB debugging”.
  • Long press Power + Volume Up + Volume Down to restart the tablet and to enter the boot menu.
  • Select “recovery mode” using the volume buttons and press Power. An Android robot with a triangle appears after a short while.
  • Shortly tap Power + Volume Up (the volume key on the right). A menu appears.
  • Select “wipe data” and wait until it finishes.
  • Select “reboot” and the tablet should reboot fine now. Your data is lost (no problem as that will happen during the installation anyway) and the tablet must be set up as when it was started for the first time after you had bought it.
  • Re-enable USB debugging in developer’s settings.
  • Run adb reboot bootloader to enter the bootloader.
  • Run sudo fastboot flash recovery RECOVERY-IMAGE.zip (if you decided to use TWRP then RECOVERY-IMAGE should be something like openrecovery-twrp-*-manta.img).
  • Select “power off” on the tablet.
  • Long press Power + Volume Up + Volume Down to power the tablet on and to enter the boot menu. Don’t start the tablet normally otherwise the installed recovery may get replaced with the original one during the boot.
  • Select “recovery mode” and you should be in TWRP now.
  • Perform “Backup” in TWRP to backup the original system.
  • Perform “Wipe” in TWRP.
  • If your installation .zip files are not in /sdcard, you can upload them now using adb push YOUR-DOWNLOADED-ROM-FILE.zip /sdcard/ from PC.
  • Perform “Install” in TWRP for your ROM (and Google applications, if you want them) zip files.
  • Perform “Reboot” in TWRP.
  • Hopefully the new system starts now and you are freed from the Google restrictions.

Many thanks to all those who develop alternative Android installations with improved user freedom!

Really afraid of Mozilla

A week after the International Day Against DRM, Mozilla has announced that it accepts the W3C DRM proposal and will actively support DRM on the Web. So we’ve lost another battle against DRM. All major Web browsers now officially accept DRM on the Web.

I find the following Mozilla explanation scary: “Every other major browser vendor has already implemented EME, and as it becomes the norm, we want to avoid the possibility that people will be unable to access key Internet content such as streaming Hollywood movies via Firefox. We also do not want to create a situation where Firefox users must use other browsers for key Internet activities.”

I don’t consider Hollywood movies being “key Internet content” and I don’t want to access any DRM content in Firefox. Whenever I need to use DRM or a similar broken system, I do so on platforms where privacy and security are already compromised (such as Google Android or proprietary operating systems).

So what’s the problem when using DRM in Firefox is going to be voluntary? For first, Hollywood demanded to put control on the users and the W3C and all the creators of major Web browsers have accepted that. For second, Mozilla has recently demonstrated several times that freedom is not very high among its priorities: It promotes non-free add-ons (without informing users about their licenses on mobile platforms), it failed to resist the outrage of political lobbyists and now it supports DRM. We all know the situations when we have to make hard choices and make decisions we are not proud of, because of our cowardice, lack of alternative choices or because we think we do the least evil. It’s a dangerous game we don’t want to play but we can’t avoid it. And it can lead us to crossing all limits.

When Mozilla says that “every other major browser has already done that and it becomes the norm” I’m alerted. The fact is that every other major browser is designed to serve interests of its creator and not the interests of the users. This is the primary reason I use Firefox and not another major browser. But when Mozilla conforms to what other major browsers do and what is “the norm” then what’s the difference? I have to ask some serious questions such as whether Mozilla will sell users’ data or whether it will close some parts of its source code or will switch to a hostile license when that looks like an effective way to fulfill “interests of Firefox users” and to save the market share. My answer is such things may happen. So I can’t trust Mozilla anymore. (As a precaution I disabled sending Firefox reports to Mozilla from my mobile devices and of course I use my own Firefox sync server instead of the one provided by Mozilla.)

Retracting from certain principles typically doesn’t bring the expected good. When my country was sacrificed to Hitler, it was applauded as saving the world peace. The next year World War II has started. When Netscape offered its proprietary Web browser as free meal, users have accepted it instead of making a free Web browser. Some years later Netscape has lost the war of browsers and turning to the Mozilla project has saved us from the worst in the last minute. The years of Netscape dominance had nevertheless made serious damages to free software which I hoped were recently finally recovered. But I was mistaken, we actually tightly depend on a single free software product apparently ready to sacrifice the principles of freedom just to conform to “the norms” and retain some users. Maybe Mozilla changes its attitude and instead of making definite steps against user freedom and vague claims about supporting it it will act in the opposite way. But maybe things get worse and we must be prepared for that. It’s time to start preparing for the worst, i.e. loosing the last free major Web browser.

We need a major free Web browser, sufficiently separated from direct commercial interests and with high respect to users’ freedom and privacy. We should start with properly maintaining a freedom aware branch of Firefox (perhaps the GNU IceCat project, the GNU version of Firefox, might be a good start). In case more bad things happen with Mozilla we should be ready to start a complete Firefox fork. Otherwise we may end up with a completely proprietary Web.

Afraid of Mozilla

So Brendan Eich has resigned and has left Mozilla. It was apparently the only reasonable option he had which is a very bad news. (If you don’t know what it is about, look for Brendan Eich Mozilla CEO in your favorite search engine.)

The hatred and bigotry demonstrated in this case is scaring. AFAIK Brendan Eich did nothing wrong and didn’t discriminate anybody. He is just incompatible with certain ideology. I don’t know whether people from Mozilla have joined the attacks or not but Mozilla as a whole has definitely failed to resist the pressure. As a citizen of a country that was under single ideology a few decades ago and anybody incompatible with it was a public enemy I’m especially sensitive to such dealing with people. It’s really scaring.

While I don’t think there is something like same-sex marriage I can understand that some people think otherwise and we can explain our views and arguments to each other and discuss how our society should be organized. There may be bigots on both sides but I believe most people can behave reasonably. I can’t see what such views have to do with the role of Mozilla CEO.

Dear Mozilla, I little care about whether your CEO is for or against same-sex marriage. I’m much more concerned about other facts. You make the only and last big free Web browser which is a very important mission. I’m disgusted that political screening is part of the process. But not only that. I’m also very disappointed that the mobile version of Firefox doesn’t display licensing information about the browser extensions and about the applications at the Firefox market. You mix free and non-free software without helping the users to distinguish. I’m not sure you do enough against making DRM part of official Web standards. I’m afraid you divert from open Web and freedom and you may, directly or indirectly, help the movements opposing them. Then we may lose even the last free Web browser.

The resignation of Brendan Eich was a failure of our society and of the Mozilla project and perhaps a loss for both. Yes, we must do better.