cosine.blue

Emacs Keeps Surprising Me

greg@cosine.blue (Gregory Chamberlain) — Sun, 12 Sep 2021 00:00:00 +0000

Emacs keeps surprising me. Though I've been using it regularly for about two years now, I am still discovering features and nuances that have been right under my nose since day one.

I'm not talking about writing your own Elisp. I'm not even talking about third-party packages. I'm talking about stock, built-in, everyday functionality that is right there when you open GNU Emacs for the first time, wondering what all the fuss is about.

There is so much there. It's overwhelming. Really, it's insane. Stuff you didn't think existed, stuff you didn't know was possible, stuff you don't expect to find, stuff you didn't think you needed, stuff you really really needed that one time but didn't know about or even think to look for, and stuff you always suspected might exist but hadn't looked into yet.

Sometimes it's a default key binding, however obscure, that has been waiting patiently under your fingers, hidden beneath the keyboard all this time, never before invoked (except perhaps by mistake). Sometimes it's a minor mode, or a unique tool for editing, or just a subtlety in the behaviour of a command you've already been using for months.

Most of these things are documented thoroughly in the manual, so it's not like they're deliberately hidden from us. It's just that there is such an abundance of features built into Emacs that they effectively must compete for our attention.

Inserting special characters

Recently I finally got around to looking in more detail at a feature I only vaguely knew of: input methods. I do not know why I waited until now to look into it but, as someone who is interested in languages, it was curiosity that eventually led me there.

Now, there are a couple of sections in the manual about input methods (22.3 and 22.4), and I'm sure they're great, but that was not how I went about learning this particular feature. No, I just dove straight in. What little I did know of input methods was only because I inadvertently hit C-\ from time to time, invoking toggle-input-method. I knew it had something to do with internationalisation but I had always hit C-g to get out of it and continue with whatever highly important task I was doing (such as M-x tetris). This time I hit TAB instead, revealing a screenful or two of completion candidates.

Among them were czech, arabic, japanese and many other languages and their related input methods. No surprises there. What caught my eye, though, were TeX and sgml. These are not spoken languages. These are methods for inserting all kinds of special characters—obscure diacritical marks, quote marks, superscript and subscript mathematical symbols, playing card suits, smileys, various shapes and arrows, and so on.

Input methods allow you to insert characters that you can’t type by typing a sequence of characters that you can type. For example, using the spanish-postfix input method, typing n~ produces ñ. C-h C-\ (describe-input-method) shows you all the sequences for a given input method.

Until now I had been using C-x 8 RET (insert-char) to insert special characters by name, if I could guess the name (was it SMALL LATIN LETTER ETH or LATIN SMALL LETTER ETH?). Suddenly I had a much smoother, less cumbersome way of doing the same thing.

This is a feature you can just use. There’s no Lisp to write or variables to tweak to make it usable. There’s no external package you have to install from MELPA. You hit C-\, choose your input method, and start writing.

My last article was about IPA (the International Phonetic Alphabet) and I had to write a lot of phonetic symbols like ə and ʧ. There I was going through Wiktionary, painstakingly copy-pasting symbols into Emacs while its ipa input method was right under my nose the whole time.

A few days later I discovered it. That’s what prompted me to write this article.

View and edit lines that match a pattern

grep is an old-school UNIX program for printing only the lines of a file that match a regular expression.

Emacs has a command called occur (M-s o), which does a similar thing for Emacs buffers. The matching lines are shown in the *Occur* buffer, which can be used to jump to the location in the source buffer where that match occurs.

What’s really cool is that you can edit the *Occur* buffer and any changes made there are reflected in the source buffer. You just hit e to begin editing and C-c C-c to apply the changes when you’re done. In that sense, it’s a bit like C-x n n (narrow-to-region) but for “multiple regions,” if you like.

As an example, let’s say I wanted to reword some of the headings in this article. I’m writing it in Markdown, so each heading is on its own line beginning with #. So I begin with M-s o ^# RET to have all the headings reproduced in the *Occur* buffer—not unlike a table of contents. In that buffer, I can use o or C-o to jump to that section in my article. I can also hit e to begin editing them. When I’m happy with my reworded headings, I can hit C-c C-c to apply the changes.

Occur is a built-in package. M-s o is a default global key binding. It’s right there from day one. No external packages or Elisp to copy and paste into your init file. Just vanilla Emacs.

Visualisation of whitespace

The annoying thing about editing whitespace is that you can’t see it. I mean, that’s kind of the point, but sometimes you need to know exactly where it is and how much of it is there. This is especially true for some programming languages where whitespace and indentation are significant to the compiler or interpretter.

The Emacs developers know this pain all too well, and that’s why we have whitespace-mode. It makes it abundantly clear where there are spaces, tabs and newline characters by highlighting them and denoting them with certain symbols—a middle dot (·) for spaces, a double chevron for tabs (»), a dollar sign ($) for newlines. The symbols shown can be changed or removed, of course—see whitespace-style and whitespace-display-mappings.

Trailing whitespace is made particularly obvious because it is highlighted in red. Lines longer than whitespace-line-column characters (80 by default) are highlighted in yellow. These colours may vary by theme.

I wouldn’t recommend keeping whitespace mode on at all times because it makes the rest of the text quite hard to read. I only enable it for a moment while I check for stray whitespace, or sometimes when editing tab-separated data.

Display buffer continuously over several windows

When I’m learning (or writing) a new piece of music, I like to have the notes (chords, tabs, lyrics, etc.) for the entire song shown at one time, if possible. This way I can play through it from start to finish without having to pause for scrolling.

Usually I set it up in a column layout to make the most of the horizontal screen space. Emacs is perfect for this because of its windowing capabilities combined with a minor mode called follow-mode. Follow mode shows adjacent screenfuls of the same buffer across multiple windows, like pages in a book.

It’s hard to explain, so maybe it’s best to try it for yourself—for example, by visiting this very page with M-x eww. Create a side-by-side split using C-x 3 (split-window-right), then do M-x follow-mode and try scrolling around. The window on the right shows the part of the buffer that immediately follows the part in the window on the left.

Undo only within a region

This feature is phenomenal and it’s actually enabled already because it’s just a nuance of the undo command. I don’t know of any other text editor that can do this. I believe Vim has “time travel” history which is cool in its own right, but not quite the same.

Emacs allows you to undo changes that happened in a particular region of the buffer, without undoing changes elsewhere across it. Typically that region will be a range of lines but it doesn’t have to be.

All you do is select the region of text in which the changes occured, then invoke undo as normal (e.g. C-/).

Again, this is something that was right under my nose and for months I had no idea Emacs could do this—I had never even wondered if it was possible; it simply didn’t occur to me.

How to Pronounce Any English Word

greg@cosine.blue (Gregory Chamberlain) — Tue, 03 Aug 2021 00:00:00 +0000

We’ve all been there. You’re reading something, maybe out loud, maybe even in front of an audience, and you’re met with some word whose spelling is so irregular and confusing as to incite fear in the hearts of native speakers and language learners alike. Maybe it has an unreasonable combination of vowels or some weird consonant cluster that looks to be one dollar sign short of being a fairly secure email password.

With a raised eyebrow and a rising tone, you make a guess or two based loosely on an ever-decaying recollection of primary school heuristics and rules that are supposed to be broken regularly. English spelling is too fickle for such guesswork!

Towards the end of secondary school and beginning of sixth form college, I began to take an interest in English phonology—the study of how our language arranges sounds. While I didn’t exactly go on to become a linguist, I did come out of it with one little skill that I want to share with you today. It’s not too difficult to learn and and it solves this problem. I use it all the time.

The International Phonetic Alphabet

Anyone who's flicked through a dictionary knows about the funny symbols that describe exactly how words are spoken, but few endeavour to study them.

While traditional printed dictionaries vary in their choice of phonetic alphabet, the de facto dictionary of the Internet, Wiktionary, uses the IPA. And it tends to have an IPA transcription for just about any English word you’ll come across. These pieces of phonetic notation tell us exactly what sounds to make and in what order, as well as other things like which syllables should be stressed.

Of course, many words are pronounced differently throughout English-speaking parts of the world. In these cases Wiktionary usually lists the main ones—US, Canada, Australia, New Zealand, UK—and sometimes others; the entry for about has transcriptions for England, Scotland, Ireland, Canada, Virginia and elsewhere in the US.

Whenever I’m reading and I come across an apparently unpronouncable word, I quickly punch it into Wiktionary and I’m almost always presented with an IPA transcription to fill that gap in my knowledge. If you use DuckDuckGo, try adding the !wt bang to redirect your search to Wiktionary. After reading the IPA transcription, I always try to say it out loud to myself to be sure, and voilà! Word learnt.

The IPA is well suited to English, and learning enough to read transcriptions is not too hard. 16 of the 23 consonant letters are pronounced exactly how you’d expect, and there aren’t more than a dozen or so vowel letters to learn.

Most IPA transcriptions on Wiktionary are what we call phonemic transcriptions, so they are delimited by slashes, like /bluː/ for the word blue. IPA notation can be written more precisely in the form of phonetic transcriptions, delimited by square brackets, but we won’t be needing that level of detail.

Consonants

As I said, most consonants are exactly what you’d expect: /p/, /b/, /t/, /d/, /k/, /ɡ/, /f/, /v/, /s/, /z/, /h/, /m/, /n/, /w/, /r/, and /l/ are pronounced as they typically would be in English.

Keep in mind that /ɡ/ represents a hard ‘g’ (as in bag) whereas a soft ‘g’ (as in badge) is written as /dʒ/, so gadget is transcribed as /ˈɡædʒɪt/, for example. This is because the soft ‘g’ is really two phonemes smooshed together—/d/ and /ʒ/ where /ʒ/ represents the sound of s in measure (/ˈmɛʒ.ə/) or si in vision (/ˈvɪ.ʒ(ə)n/). We often write them with a curved line over the top to join them together, like /d͡ʒ/ in /ˈɡæd͡ʒɪt/ (gadget).

Next we have /ʃ/ which is the sh in fish (/fɪʃ/). Analogous to /d͡ʒ/ we have /t/ and /ʃ/ smooshed together to form the sound of tch in hitch (/hɪt͡ʃ/) or ch in choice (/t͡ʃɔɪs/).

IPA really only has 4 other consonants that may not be immediately obvious:

/θ/ representing the th in thanks (/θæŋks/)
/ð/ representing the th in then (/ðɛn/)
/ŋ/ representing the ng in thing (/θɪŋ/) or the n in think (/θɪŋk/)
/j/ representing the y in yes (/jɛs/)

Vowels

Unfortunately, the IPA letters for vowels are not quite as straightforward as for consonants. To make matters worse, pronunciation of vowel sounds varies wildly between dialects.

Below are the vowels which are basically the same for RP (a kind of “standard” British English), US and Canada:

/ɑ/ as in father	/u/ as in soon
/æ/ as in trap	/ə/ as in comma
/ɪ/ as in kit	/eɪ/ as in face
/ʌ/ as in strut	/aɪ/ as in price
/ʊ/ as in foot	/ɔɪ/ as in boy
/ɛ/ as in bed	/aʊ/ as in mouth
/i/ as in happy

For RP speakers, /ɒ/ is the vowel that occurs in words like lot and stop. The same is true for some Canadian, Australian and New Zealand speakers. Canadian speakers also use /ɒ/ for words like law and caught and some use it for father and palm too.

The vowel in words like know and hope are transcribed as /əʊ/ for RP and /oʊ/ for US and Canada.

Finally we have /ɔ/ which is found in words like law and caught in RP and some US dialects.

Particularly in British, Australian and New Zealand dialects you’ll often see a triangular colon following some vowels, like /ɑː/ in /ˈfɑːðə(r)/ (father) or /uː/ in /ʃuːz/ (shoes). This symbol just means that the preceding vowel is elongated but honestly you can usually ignore it.

There are a few other vowels like /ɜː/ (RP) or /ɜr/ (US) as in /nɜːs/ (RP) or /nɜrs/ (US) (nurse) but these are quite rare.

Stress

English is one language where stress is semantically significant—for example, what’s the difference between /ˈkɒnkriːt/ and /kɒnˈkriːt/? These are both ways of saying concrete, but the first is a building material and the second is an adjective meaning “not abstract.”

Stress is indicated with a stress mark (ˈ) placed before the syllable that should be stressed. It looks an awful lot like a straight apostrophe (') but I promise it’s not. Sometimes you’ll see the same mark written below the baseline (ˌ), indicating secondary stress, as in /ˌɛd͡ʒʊˈkeɪʃn/ (education) or /ˌnɒtwɪðˈstændɪŋ/ (notwithstanding).

Where to go from here

Beside each IPA transcription on English Wiktionary you’ll find a link labelled “KEY” to an appendix page called English pronunciation which features a more comprehensive and detailed overview of the IPA symbols used on the site. This is great if you need a quick reminder. It’s also great if you need to know more about Australian and New Zealand vowels—I haven’t been able to cover those in great detail here since they differ considerably from RP.

Of course, using Wiktionary requires a computer and an Internet connection and everything, so if you want to do this without all that you’ll need an offline or paper dictionary that features IPA, such as the Oxford English Dictionary.

Declarative Package Configuration In 5 Lines of Emacs Lisp

greg@cosine.blue (Gregory Chamberlain) — Fri, 02 Apr 2021 00:00:00 +0000

GNU Emacs is a real phenomenon of computing, not least thanks to the wealth of free/libre packages available.

Thing is, most of these packages are more like software libraries than programmes, having little to no effect out of the box; they provide all the Lego bricks, but leave it to the user to put them together. And like Lego, putting the bricks together is the fun part! The way we do that is by writing bits of Emacs Lisp.

For example, here are a few lines of Elisp you could write to configure one of my favourite packages, dired-subtree.

(setq dired-subtree-use-backgrounds nil)
(let ((map dired-mode-map))
  (define-key map (kbd "TAB") #'dired-subtree-cycle)
  (define-key map (kbd "M-^") #'dired-subtree-remove)))

However, there is a problem. What happens when you write a bit of Elisp to configure a certain package, but that package is not (yet) installed? You often get errors about undefined commands, files not found, and so on.

Conditional configuration

One way around this is to put package-specific configuration behind a conditional.

(when (package-installed-p 'dired-subtree)
  (setq dired-subtree-use-backgrounds nil)
  (let ((map dired-mode-map))
    (define-key map (kbd "TAB") #'dired-subtree-cycle)
    (define-key map (kbd "M-^") #'dired-subtree-remove)))

This works, but what if you want to then go ahead and install all packages for which you have a conditional configuration? There is no record of the names of those packages.

You could scan through your user-init-file by eye and manually install each package by hand using M-x package-install pkg RET. But that only gets more laborious the more packages you want to have.

Installing packages programmatically

A more viable solution is to use a macro that does a similar thing and collects the package names into a list variable at the same time. That way, you could programmatically install all such packages by iterating over the list.

As it happens, such a variable is already defined and documented by package.el and it goes by the name package-selected-packages. What’s more, there exists a command M-x package-install-selected-packages that asks the user for approval before installing them all. Making use of this variable, our job becomes very easy.

Each time this macro is called with the name of some package pkg, we need to do three things:

Add pkg to package-selected-packages;
Try to require the package feature;
If successful, evaluate the body using progn.

Reproduced below is just 5 (logical) lines of Emacs Lisp sufficient to define this macro. I’ve named it with-package, analogous to the likes of with-current-buffer, with-selected-window, with-editor, etc.

(defmacro with-package (package &rest body)
  "Add PACKAGE to ‘package-selected-packages’, then
attempt to ‘require’ PACKAGE and, if successful, 
evaluate BODY."
  (declare (indent 1))
  `(and (add-to-list 'package-selected-packages ,package)
        (require ,package nil 'noerror)
        (progn ,@body)))

The declare expression is related to code formatting and is not strictly necessary for the macro to work. Its purpose is to inform Emacs that the body of the macro call should be indented by an additional character.

Here’s our dired-subtree example from above, this time in with-package form.

(with-package 'dired-subtree
  (setq dired-subtree-use-backgrounds nil)
  (let ((map dired-mode-map))
    (define-key map (kbd "TAB") #'dired-subtree-cycle)
    (define-key map (kbd "M-^") #'dired-subtree-remove)))

Note that, even though this is a macro, the name of the package should be quoted. Otherwise Emacs treats it as a variable and tries to find its value, which is not necessarily defined.

External solutions

All this is to neglect the elephant in the room, a highly popular package by the name of use-package which provides a macro of the same name that does what my little with-package macro does and lots more. For a long time and until recently, my user-init-file was scattered with calls to use-package for the external packages I employed.

I even had a little snippet near the top that would install the use-package package if it wasn’t present already. This practice is almost as popular as the package itself, it seems.

There are several reasons why I ditched use-package in favour of with-package:

There is a way to install all packages for which a use-package form has been evaluated, but it does not ask interactively for consent from the user before doing so. Rather, the packages are installed at startup without prompt nor warning.

use-package is a mighty big package that does way more than I need it to do. Simpler is usually better when you can get away with it.

Among its more advanced features is a way to selectively defer the loading of packages until they’re needed, shortening startup times. That sounds great, but personally I’m happy with my startup time as it is.

The last reason is that, honestly, I was rather pleased with myself after writing this macro. And if you ask me, DIY programming is what Emacs is all about.

So yeah, a 5 line Lisp macro may be all you need to take your init.el to the next level. If this post has inspired you to rip apart your carefully crafted Emacs configuration as I did mine, do send an email to greg@cosine.blue letting me know.

I ❤ Emacs!

The wide-reaching applications of GNU Emacs have slowly invaded my workflow over the past year and a half, and I have much more to write about it. To find out when new Emacs related content emerges on cosine.blue, subscribe via RSS or follow me on Mastodon.

Vanilla Firefox Keyboard Mastery

greg@cosine.blue (Gregory Chamberlain) — Sun, 06 Dec 2020 00:00:00 +0000

All software packages mentioned in this article are free as in freedom. There is no need to be using proprietary browsers these days.

Is it worth the time?

Last weekend I carved out half an hour to read Mozilla’s support page titled Keyboard shortcuts - Perform common Firefox tasks quickly. Sounds boring I know. But it was time well spent and my only regret is not having read it sooner (like 5 years sooner).

Many of the key combinations described therein were familiar friends of mine, but with others I had no acquaintance. I won’t lie to you: a fair number of them are worthless. But others are essential for keyboard-only browsing.

Over the last decade I must have spent ten thousand hours using a web browser. And I don’t think I’m an outlier here. The Web has steamrolled into the twenty-first century and for better or worse become the ubiquitous interface through which the world now wants us to read and write, to listen and speak and to otherwise occupy ourselves.

Let’s say you have to use a web browser just once a day on average for the next 5 years. According to the chart below, even if you can shave off 1 measly second from this daily obligation, it is worth your time to set aside up to 30 minutes today to achieve that daily time saving.

From xkcd: Is It Worth the Time? (CC BY-NC 2.5).

It takes half a second to move your hand from the typing position to the mouse (or touch pad or whatever) and another half a second to point your cursor at the right spot. Learning a few keyboard shortcuts will surely shave off a few seconds every now and then.

You’ve already wasted 2 minutes reading this page and another 3 staring at that incredibly insightful chart so you’d better get cracking.

I’ve outlined below a few of my favourite shortcuts that comprise my general strategy for keyboard-only navigation, which works better on some sites than others. A good site to practise on is Wikipedia (or, you know, this one). In particular, we will eventually get to the apostrophe key ' which enables lightning fast link following and makes the whole experience much smoother.

You won’t need to go digging through any settings or configuration files to try these; they are built into Firefox and enabled by default.

Learning to use the keyboard as much as possible can be really beneficial to your efficiency and ergonomics when using a computer because it is profoundly more expressive and reliable than any pointing device. If you’re looking to become more proficient with keyboard-driven software but not ready to learn something like Vim or Emacs then this article might be a good stepping stone.

Mozilla have made some increasingly questionable decisions lately, but there are a few privacy-respecting alternatives such as GNU Icecat and Tor Browser which are based on Firefox and have all the same keyboard shortcuts.

The basics

Obviously you can scroll the page screenful-by-screenful using PgUp and PgDn. When the page is in focus, Space and Shift + Space do the same and are more accessible from the home position on full size keyboards. The vertical arrows keys ↑ ↓ are best reserved for making fine adjustments to the viewport. For jumping to the beginning and end of the page, use Home and End.

Ctrl + Tab and its Shifted equivalent are not unheard of even among novice computer users, but holding Alt and hitting 1–8 to immediately switch to the corresponding tab will surely impress on-lookers. If you need to switch quickly between adjacent tabs, try Ctrl + PgUp and Ctrl + PgDn.

You probably know Ctrl + w closes the current tab but did you know Ctrl + Shift + W closes the current window? Also essential are Ctrl + t to open a new tab and crucially Ctrl + Shift + T to restore that tab you really wish you hadn’t closed.

Continuing the Shift key theme, I think most people are familiar with Ctrl + r to refresh the page—but hold Shift to clean the cache while doing so.

Of course there are shortcuts like Ctrl + h and Ctrl + b to open the history and bookmarks sidebars, but in practice I rarely use these—instead I use Ctrl + l or Alt + d to bring the address bar into focus and begin searching through history and bookmarks right there. Ctrl + d, however, is really handy for quickly throwing the current page into your bookmarks list—you can then Tab and Shift + Tab around the little dialogue window to make adjustments.

As with most graphical applications, you can generally ‘tab around’ between UI elements and hyperlinks with Tab and Shift + Tab. This is a cumbersome and frustrating way of getting around but it is often necessary. It can be tempting to rely on this in Firefox but think of it as a last resort. In the next section we will learn some faster techniques.

Quick find and navigation

You may know you can hold Alt and use the horizontal arrow keys ← → to go back and forward. Make sure you’re comfortable using them as they’ll become somewhat important later.

Everybody knows Ctrl + f launches the familiar find in page rigmarole, and there is certainly a time and a place for its case sensitive and whole word matching. But in most cases all you need is quick find. Hit forward slash / to begin. It works much the same but is less annoying because it soon gets out of your way; you can still hold Ctrl and hit g and Shift + G to jump between matches, but when you do something else like press Tab or Enter, or even just pause for a few moments, the bar slides away and page focus is restored. I often use this to get back to where I was reading by quick-searching for the short string of words I last read. You can always upgrade to the fully-featured find-in-page mode by hitting Ctrl + f.

By the way, wherever you see an underlined letter in a menu, that means you can hold Alt and hit the corresponding key to activate it. Try toggling Whole Words matching in the find-in-page bar using Alt + w

Finally we are ready to meet the apostrophe key '. This puts you into the same kind of quick search prompt, but results are limited to links only. This is magical. By using Ctrl + g and Ctrl + Shift + G to jump between matches and Enter to follow them, interspersed with the Alt plus arrow key combos mentioned earlier, you can zip around between web pages at light speed with a modicum of keystrokes.

If you follow the wrong link and react before it loads, quickly reach for Esc to cancel it. If you’re too late, remember there’s always Alt + ← to go back.

When following a focused link with Enter, you can hold Ctrl to make it open in a new tab. It won’t switch to that new tab immediately though; to do that use Ctrl + Shift + Enter instead.

These key commands are great and all, but some rely on page focus; when the address bar or other UI element has keyboard focus, shortcuts like / and ' become inaccessible. Well, get used to hitting F6 to bring focus back to the page itself—you should see a thin black and white dotted rectangle surrounding the page viewport to indicate page focus. Frustratingly F6 actually toggles focus between page and address bar so you can’t always be certain you’ll end up with one or the other without having first hit, say, Ctrl + l or looking for focus clues afterwards.

A word for the wizards

Now, an elite minority among you will no doubt be laughing away from the comfort of one of those Vi-like modal keyboard layer add-ons that grant you web browsing super powers and sing and dance and brew your coffee at the same time. They’re fantastic and I’ve been there. There are even standalone browsers designed around this idea, and they certainly can be worth the time investment. See add-ons Krabby and Vimium; standalone browsers Qutebrowser, Luakit, Nyxt Browser, Vimb, even surf with tabbed and the right combination of patches; and many others. Also worth mentioning are terminal and text-based browsers such as lynx, w3m and EWW. But there are reasons to (and times when you have no choice but to) revert to the bog standard, vanilla, de facto tool.

Also, those keyboard-centric browsers have steep learning curves. They can be dauntingly unfamiliar to new users and behave in unexpected ways to begin with. On the other hand, picking up some of the keyboard-driven techniques outlined above doesn’t mean throwing everything you know about your browser out the window.

Rudimentary Parsing with Idiomatic POSIX Shell

greg@cosine.blue (Gregory Chamberlain) — Tue, 08 Sep 2020 00:00:00 +0000

POSIX shell idioms

You sit down before your obnoxiously clacky mechanical keyboard and bang out the familiar #!/bin/sh with that precious sense of optimism like writing the first words in a new notebook: the margins are aligned and you begin to think that your handwriting isn’t so bad after all. Soon enough, though, it’s riddled with scribbled out paragraphs and arrows that swing around the pages.

Maintaining safe and bug-free shell scripts can become troublesome as they grow in complexity, and limiting one’s craft only to those few blessed features defined by POSIX only amplifies that.

However, what is lost in flexibility is gained in portability. With some casual experimentation and a handful of powerful idioms, perhaps the purely POSIX shell language can prove more elegant and expressive than it is given credit for.

What follows are three ways to use one such idiom: the read(1)-while loop.

The read(1) builtin

Built into any POSIX compatible shell is the read(1) command.

read [-r] var…

In its most basic usage, it can be used to gather a line of input from the user at the terminal; that might be to answer a yes/no prompt, or to provide a text string. But more versatile is its ability to parse text files in a rudimentary but reliable way.

The read(1) command always consumes exactly one line of text—that’ll be from standard input or whatever you pipe or redirect into it. When it hits EOF it fails. EOF stands for end-of-file and marks the end of a file or data stream. So, the general pattern is to call read(1) repeatedly until it fails.

Below is a really simple example which simply prints each line read. The entire content of each line, except the terminating newline character, is assigned to the line variable. We set the IFS to the empty string to prevent field splitting. It’s conventional to call the variable line, but really you can call it anything you like.

while IFS= read -r line
do printf '%s\n' "$line"
done

Unless you know what you’re doing, you probably want to pass the -r option; its effect is described in the POSIX Programmer’s Manual. Beware that read(1) still fails if a non-empty line is missing its newline character before EOF, so the loop will not be entered for that final line. The solution is to check for emptiness in the event that read fails

while IFS= read -r line || [ "$line" ]
do …
done

but you may instead prefer to assume that the input is well formed—garbage in, garbage out.

Case study 1 of 3: Linewise filtering

Let’s say we want to remove lines that begin with a hash sign (#)—something akin to grep -v ^\#. This can be achieved with the #* glob pattern in a case statement.

while IFS= read -r line
do
    case $line in
        \#*) ;;
        *) printf '%s\n' "$line"
    esac
done

The backslash escape in \#* is necessary to protect the hash sign from becoming a shell comment. Pattern matching each line like this is a powerful way to interpret file contents. Let’s also ignore leading white space:

while IFS= read -r line
do
    case ${line#${line%%[![:space:]]*}} in
        \#*) ;;
        *) printf '%s\n' "$line"
    esac
done <<EOF
one
# two
  # three
  four
EOF

The precise workings of this parameter expansion wizardry is described in Dylan Arap’s excellent pure-sh-bible, but understanding it is not necessary going forward—just know that it expands to the value of line stripped of any leading white space. This time I fed it a here document, resulting in the following two lines being printed:

one
  four

Case study 2 of 3: Key-value pairs

Let’s now try to parse a file consisting of lines of the form key=value where key contains no equals signs and value can be anything at all.

For this we set the IFS to the equals sign and pass two variable names as arguments to read(1).

while IFS='=' read -r key value
do printf 'Key ‘%s’ has value ‘%s’.\n' "$key" "$value"
done <<EOF
name=banana
type=fruit
colour=yellow
EOF

Setting IFS='=' tells read(1) to split the line into fields at every equals sign. Since we named only two variables (key and value) to assign fields to, any fields right of the second are merged into value. In other words, all characters after the first equals sign are considered part of the value, so no quoting or escaping is necessary.

Running the above code results in the following output:

Key ‘name’ has value ‘banana’.
Key ‘type’ has value ‘fruit’.
Key ‘colour’ has value ‘yellow’.

Case study 3 of 3: User’s full name

Since the original Unix the fifth field of a user’s passwd(5) record has been the GECOS field where users’ contact information is stored. Here’s mine:

$ grep ^$USER: /etc/passwd
greg:x:1000:1000:Gregory Chamberlain,,,:/home/greg:/bin/bash

Among the colon-separated fields, you can see that my full name is the first of four comma-separated values within the GECOS field. Many desktop operating systems ask for your full name during installation or when adding a new user and that’s where it’s stored. If not, try chfn(1) or just (carefully!) edit the /etc/passwd directly as root.

Assigning the first field to user and the fifth to gecos, we use underscores in place of the other fields which we don’t care about.

while IFS=: read -r user _ _ _ gecos _
do [ x"$user" = x"$USER" ] && name="${gecos%%,*}"
done < /etc/passwd

Don’t be tempted to move the < /etc/passwd file redirection closer to the read statement; it needs to feed into the while loop itself so that it progresses over successive lines.

After reading each line, the resulting values of user and USER are tested for equality. It’s important to protect arbitrary strings from [(1) options parsing by prefixing each with any ol’ character, in this case x—otherwise $user could expand to something like -z and cause an error.

[ x"$user" = x"$USER" ]

If the two match, then we know we are looking at the right line and so we begin parsing the gecos string for a name:

name="${gecos%%,*}"

We can’t use the same read(1) trick again because we already absorbed the line. Anyway, we’re only interested in the first field so we can just use a greedy suffix pattern in the expansion of gecos. You can read about Parameter expansion in dash(1) for the details but in short %%,* means remove everything after the first comma.

Below is a short script that illustrates how this could be integrated into a larger program. I’ve also thrown in a read(1) command that prompts interactively for a name if one is not found, demonstrating how read(1) can be used outside of a loop as well.

#!/bin/sh
while IFS=: read -r user _ _ _ gecos _
do [ x"$user" = x"$USER" ] && name="${gecos%%,*}"
done < /etc/passwd

if [ "$name" ]
then
    printf 'Your real name is %s.\n' "$name"
else
    printf 'What’s your name? '
    IFS= read -r name
    printf 'Hello %s!\n' "$name"
fi

Reasons not to use Bash

Most prominently, Bash scripts are not portable. From Drew Devaults’s Introduction to POSIX shell:

Any shell that utilizes features specific to Bash are not portable, which means you cannot take them with you to any other system. […] This is bad if your users wish to utilize your software anywhere other than GNU/Linux. If your build tooling utilizes bashisms, your software will not build on anything but GNU/Linux. If you ship runtime scripts that use bashisms, your software will not run on anything but GNU/Linux.

He goes on to argue that

you should stick to POSIX shell for your personal scripts, too. You might not care now, but when you feel like flirting with other Unicies you’ll thank me when all of your scripts work.

Also, Bash is monstrously complex; even its man page confesses it’s too big and too slow. And let’s not forget Shellshock, the arbitrary code execution vulnerability in Bash responsible for millions of attacks on web-facing servers.

How I Merged 36 Spreadsheets In 2 Minutes—LibreOffice on the Command-Line

greg@cosine.blue (Gregory Chamberlain) — Sat, 28 Mar 2020 00:00:00 +0000

A troublesome task

In a recent project, I found myself tasked with the laborious feat of painstakingly joining together tables, hundreds of thousands of rows each, from several years of new-file-every-month spreadsheets—by hand.

At that moment, my human eyeballs were paused hovering, mouse hopelessly scrolling past millions of cells; a bleak air beset the white walls of my otherwise cozy student flat on that mild March morning.

I soon gathered up my pessimism until it amounted to what you might call a stubborn sort of frustration-fueled optimism and looked for a better way.

A simple solution

Naturally, I reached for my office suite’s documentation. Although LibreOffice is primarily a graphical application, its little-known command-line features can be life saving. A leisurely perusal of its man page armed me with the --convert-to option, which does what you’re thinking.

libreoffice --convert-to csv ./*.ods

Some ten billion CPU cycles later—a few seconds in mammal time—I had trivialised a daunting task that otherwise would have threatened my lunch break. My thirty-something spreadsheets were now thirty-something CSV files, leaving only to concatenate them.

Of course, CSV is not a spreadsheet format. Any formulas, comments, charts and other media would be lost in this process. Fortunately for me, these spreadsheets were just holding raw data.

Each had its first five rows reserved for useless metadata (title and time period), and the sixth for table headings; the remainder were the dreaded data. Using sed(1) I was able to extract only the sixth row and concatenate the data of all CSV files onto that, writing it to result.csv.

(
    sed -n 6p ./*.csv      # Print the table headings line;
    for i in ./*.csv; do   # For each CSV file,
        sed 1,6d "$i"      # print its contents except lines 1-6;
    done
) > result.csv             # Write that to result.csv.

With just a few simple commands, I had a single well-formed CSV file holding all the data, ready to be imported into LibreOffice or indeed into R or Python or what have you. I needn’t have even fumbled for the mouse.

Potential complications

Throughout that process, I was working under the assumption that each spreadsheet was written the same way—five rows of information I don’t care about, the same 10 or so table headings on the sixth row followed by data and only data as far as the eye can scroll.

The presence of stray cells or breaking changes to the overall design or nature of data gatherment in the past few years could have rendered my monstrous CSV a totally meaningless muddle of words and numbers riddled with silly mistakes (like many of my assignments).

In an effort to justify this assumption, I did poke around in some of the spreadsheets at regular intervals until I was confident it was all in the same format.

More terminal trickery

There are other things LibreOffice can do programmatically. For example, you could print a huge batch of documents to PDF

--print-to-file [--printer-name printer_name] [--outdir output_dir] file…

Batch print files to file. If --printer-name is not specified the default printer is used. If --outdir is not specified then the current working directory is used as the output directory for the converted files.

Examples:
--print-to-file *.doc
Prints all .doc files to the current working directory using the default printer.

or onto actual paper.

-p file…

Prints the given files to the default printer and ends. The splash screen does not appear.

LibreOffice also has a flexible macro system for manipulating spreadsheets and text documents with Java, Python, JavaScript, or its own flavour of Basic—although documentation seems limited.

While the --convert-to option may not be groundbreaking, and there may well be other viable solutions, I was pleasantly surprised to find such functionality in an office suite. I’d be interested to hear of praiseworthy command-line support in other graphical applications; send me an email via greg@cosine.blue if you know of any.

The Vim-Inspired Editor with a Linguistic Twist

greg@cosine.blue (Gregory Chamberlain) — Fri, 06 Sep 2019 00:00:00 +0000

What is Kakoune?

Kakoune (/kə'kuːn/) describes itself as a code editor heavily inspired by Vim, and like the venerable vi(1) and its successor, its internal model interprets the user’s keystrokes like utterances of a sort-of text-editing language, but with a certain linguistic twist (more on that later). Available—for reasons that will later become clear—only on Unix-like systems, it was written in C++ by Maxime Coste, has a clean code base and runs like the wind in a terminal. It is released under the Unlicense, which is compatible with the GPL.

Skip ahead to Using Kakoune if you just want to get stuck in right way.

As a long time Vim user, Kakoune quickly piqued my interest and before long I was convinced that it was the editor for me, and not just because it’s incredibly obscure. Its grammatical reshuffle and wise design principles put it above the rest for me, and I’m starting to think it’s more than just a comfy nook in which to hide from the large and looming cold hand of Emacs, the curious GNUisance (sorry) that swings periodically over my shoulder whispering we have Org mode! And Lisp! Maybe next year, Emacs. I really want to like you.

But that’s an essay for another day.

Here I endeavour to introduce the reader to Kakoune and present my thoughts on its merits and misfits in the context of a terminal-based, keyboard-driven workflow. I make frequent comparisons to Vim, and the reader would benefit from some experience with a modal editor such thereas. That said, those uninitiated in modal editors might find that Kakoune’s forgiving mechanics place it among the easier to learn.

Do we really need another Vim clone?

This is neither a fork nor clone.

In an effort to provide incremental results and interactivity (words whose meaning became clear once I met the s key), the selection-oriented code editor takes a bold step away from some of the Vim interactions familiar to many of us, while remaining competitive keystroke-for-keystroke, millisecond-for-precious-millisecond.

Its author cites the Unix philosophy as one of Kakoune’s virtues, and certainly I would agree that it trumps Vim in adherence thereto (examples of this to come); and that is not to say it lacks enough features. Indeed it has built-in wrappers around Git, grep, make, and man, works with ctags, conforming linters and other things.

Screencast: browsing man pages in Kakoune

Vim is difficult to extend, not least because of the idiosyncratic train wreck of a language that is VimL (or Vim Script, or whatever). Conversely, Kakoune stands by its decision not to complicate its language with flow control and whatnot, and instead relies on languages we already know through its %sh{…} blocks. And although Kakoune is very usable out-of-the-box, one look at the plugins page is enough to make any purist cave.

You can even whip some Perl or Ruby or anything you like into an %sh{…} block; in fact—that’s how some of the more complex plugins are built.

So what’s the twist?

Whereas vi’s keystroke language follows verb-object order, Kakoune inverts that by following object-verb order. In real terms, that means you make a selection (object) before deciding what to do (verb) with it. The object might be a character, word, sentence, paragraph, parenthetical, regular expression, you name it; the verb might be delete, yank (copy), change, indent, or even transformative operations like lint, format, uppercase, etc. In Kakoune, it is with this reversed grammar, this postfix notation, that you interactively sweep up a group or groups of characters before acting on them. That way if your object isn’t quite right, you can immediately correct it without having to undo and redo your verb.

I like to think Kakoune’s keystrokes as a text editing language model, in comparison to that of vi, is what linguists would call an anastrophe—specifically an object-verb inversion.

Photo by JJ Cadiz, CC BY-SA 3.0. Subject-object-verb order can be seen in this anastrophic idiom, one swallow does not a summer make.

To English speakers, it might seem unnatural to place the verb last, but about 45% of documented natural languages have subject-object-verb order, including Latin, Ancient Greek, Japanese and Korean.

Isn’t that just Vim’s visual mode?

Well, yes, but Vim’s visual mode is a malformed afterthought in comparison, and you have to hit v every time you want to use it. Kakoune is visual mode done right, but not just that; it permits—nay thrives on—multiple selections (thousands if you wish but typically just a few) and provides some really neat tools to narrow them down incrementally, as we’ll see later.

This paradigm shift necessitated a revised set of key mappings, which Kakoune implements rather well if you don’t mind using Alt. Holding Shift means causes the usual motion keys to extend the selection, so unlike in Vim, traversing whitespace-separated words means holding Alt instead. And what about text objects? How do you select in and around words, parenthetical pairs, paragraphs, and so on? In Vim you begin by hitting v to enter visual mode followed by a or i. In Kakoune, however, normal mode is visual mode so instead Alt + a and Alt + i are used to avoid conflict with the normal meanings of a and i (append and insert).

Keystroke idioms in Kakoune and compared to Vim.
English	Vim	Kakoune
Delete word	`dw`	`wd`
Delete up to whitespace	`dW`	`<a-w>d`
Delete inner word	`diw`	`<a-i>wd`
Delete back four words	`d4b`	`BBBBd or4Bd`
Change to end of word	`ce`	`ec`
Yank a sentence	`yas`	`<a-a>sy`
Change to next occurrence of ‘x’	`ctx`	`txc`
Select to next occurrence of ‘x’	`vtx`	`tx`
Indent three lines down	`3>>`	`xXX>`
Unindent within braces	`<i}`	`<a-i>}<`

Admittedly, from the table above, my new favourite editor looks to be more complicated than Vim. Perhaps that’s true in some sense, but Kakoune’s strengths lie in its forgiving interactivity and incremental results, which are best experienced first hand.

Using Kakoune

At this point, I encourage the reader to play around with Kakoune. Have fun with it! Install it with your package manager or build it from source. Skim over the manual page for kak(1) and open a file in the terminal like so

kak example.cc

Make sure your terminal emulator supports alt key combinations and isn’t overriding them.

Have a look through :doc faq, :doc keys (then :doc keymap for the cheat sheet), and :doc commands, experimenting as you go. That said, the built-in help system is verbose and won’t exactly hold your hand. The Kakoune wiki is more navigable, and this walkthrough is particularly helpful for learning the basics. You can edit files and manipulate buffers with the following commands

:edit <filename>
:buffer <name>
:buffer-next and :buffer-previous

Use :delete-buffer to delete a buffer. Write to disk with :w and quit with :q. As you enter commands, you’ll see an ASCII art Clippy show up in a box with the relevant documentation. Commands are self-documenting, and many have a -docstring argument in their definition which enriches these info boxes.

Depending on your terminal colours, things might be unreadable. If so, or otherwise, pick a colour scheme using the :colorscheme <name> command. Try not to spend too many of your precious Earth seconds fussing over which is best, as I did.

Kakoune shares a good deal of elementary keys with Vim, which eases the learning curve for veterans thereof. Of the lowercase letters, only s, z, x, v and m have completely new jobs. Hitting the a key, i or o or their shifted equivalents will enter insert mode in much the same way as in vi; Shift + Q and q handle macros; press u to undo (some will rejoice it’s Shift + U, not Ctrl + r, to redo); I could go on. As you’d expect, many motion keys accept counts, as in 4 w to traverse four words.

Motions

Get the hang of using motions—h, j, k, l, w, e, b, f and t for starters—and their majuscule equivalents; hold shift to extend the selection you already have; hit ; to reset the selection range back to the cursor itself. Then of course r, d and c are your destructive verbs, while ~, `, < and > are transformative; i, a, o and Shift + O respect the selection range (try it to see what I mean), as do y (yank/copy) and p (paste). Try combining some of these with Alt and see what happens. Try Alt + i or Alt + a and read the info box to see what you can do with them; same goes for [ and ], and their shifted equivalents { and } which, predictably, extend the selection.

Press g and you’ll see the ‘goto’ menu, which shows you the handful of keys that could complete the key-stroke sequence. Familiarise yourself with some of these.

Go to …	Vim	Kakoune
Top of buffer	`g`, `g`	`g`, `g` or `g`, `k`
End of line	`$`	`g`, `l`
Beginning of line	`0`	`g`, `h`
First non-blank character	`^`	`g`, `i`
Bottom of buffer	`G`	`g`, `j`
End of buffer	`G`, `$`	`g`, `e`
Top of window	`H`	`g`, `t`
Bottom of window	`L`	`g`, `b`
Centre of window	`M`	`g`, `c`
Last (alternate) buffer	`Ctrl` + `^`	`g`, `a`
Path under cursor / in selection	`g`, `f`	`g`, `f`

Same goes for v and the ‘view’ menu, which is for adjusting your viewport into the buffer.

View …	Vim	Kakoune
Centre cursor vertically	`z`, `z`	`v`, `v` or `v`, `c`
Centre cursor horizontally	Nope(*)	`v`, `m`
Cursor on top	`z`, `t`	`v`, `t`
Cursor on bottom	`z`, `b`	`v`, `b`
Scroll left	`z`, `h`	`v`, `h`
Scroll down	`Ctrl` + `e`	`v`, `j`
Scroll up	`Ctrl` + `y`	`v`, `k`
Scroll right	`z`, `l`	`v`, `l`

(*) The closest approximations I can find are z, H and z, L to scroll half a screen width horizontally. Or there’s

set
  sidescrolloff=999

Multiple selections

Pressing x selects a whole line, x again the next instead; or hold shift and see that X sweeps up multiple lines. Here’s where things get cool: sweep up a few lines with X or Alt + i, p and then press Alt + s. That splits the selection linewise into multiple selections. Another way to accrue selections is using Shift + C or Alt + Shift + C. Play around with these yourself. If things get out of hand you can hit Space to reset to one selection.

Aligning columns interactively

This is one case where multiple selections are your best friend. You just make your selections and hit & to align them vertically.

Screencast: aligning columns interactively

Narrowing

Still think multiple selections are a gimmick? So did I. Until I tried some of the narrowing mechanisms. What do I mean by that? Well, let’s say you want to replace all occurrences of ‘sir’ with ‘mate’ inside a particular paragraph. With your cursor situated somewhere within it, you’d press Alt + i, p to select the whole paragraph, then (and this is the good bit)

s, s, i, r, Enter

to select each occurrence of ‘sir’. Now you have multiple selections. Finally you’d type

c, m, a, t, e, Esc

to change each into ‘mate’.

Screencast: Incremental search and replace

To achieve the same in Vim you’d need to specify the line range of the paragraph in an ex command (perhaps with the help of visual mode, i.e. v, i, p), and provide the ‘g’ flag. In Kakoune, everything is achieved with the keyboard-based text-editing language and as a result is more interactive in that it provides incremental results so that you can make corrections on the fly.

You can enter a regular expression (see :doc regex) into the s command, but often simple key-stroke-based interactions thereafter can achieve the same effect while being easier to come up with and adjust as you go. As a contrived example, imagine trying to replace the word that comes immediately after every occurrence of a pattern.

A commonly used technique is to press % which selects the entire buffer. Then you can hit s to select occurrences of some pattern throughout the file. The information in the bottom right keeps count of how many selections you have.

The %, Alt + s combo is useful for operating linewise on entire files. For example,

%, Alt + s, Alt + k, i, n, c, l, u, d, e, Enter

will select all lines that contain the word ‘include’; or using Alt + Shift + K we can do

%, Alt + s, Alt + Shift + K, :, $, Return

to select all lines that don’t end with a colon. We could then do

Alt + k, ^, d, e, f, Enter

to narrow down further to only those that begin with ‘def’. Follow that up with an S key like so:

S, \, ., Enter

to split each selection at every period. As you can see, the narrowing tools are immensely powerful and the possibilities endless. And if you’re still not satisfied, there’s the $ key which filters your selections by piping each one through any program you like and keeping only those for which the program exists successfully—meaning you could always write a little shell script to be the predicate of that filter.

Changes through external programs

Don’t get too excited about $ before you’ve met |. This key is your program-wielding text-processing glue. The idea is that you make your selection(s) in Kakoune, type something of the form

|program args ...<ret>

and watch the magic happen. For each selection, Kakoune feeds it as standard input to the program, and replaces it with the standard output.

<ret> refers to the Enter or Return key. See :doc mapping for more key names like this.

Word count and formatting

Starting with a simple example, you could select the entire buffer with %, then run it through wc(1) to get a word count of your prose:

%|wc -w<ret>

Or pipe it first through the appropriate pandoc(1) command to convert it to plain text, thereby eliding markup from your word count:

%|pandoc -t plain|wc -w<ret>

Or you could select a paragraph and pipe that through to fmt(1) or par(1), which are programs that nicely reflow paragraphs of text from standard input. This is why Kakoune seems to lack important features such as text reflowing; instead it makes it easy to delegate this sort of processing to other programs, thereby adhering a little closer to the Unix philosophy.

Evaluating code within the buffer

If your buffer contained a Python expression, say

factorial = lambda n: 1 if n<1 else n*factorial(n-1)
print(factorial(13) + factorial(16))

then you could select those lines and pipe it into python(1) by typing |python<ret>, leaving you with 20929016908800 as the selection.

Screencast: Piping the selection through Python

The ! key does the same but without piping anything through to the program: it simply copies the output of the program into the buffer. For instance, !ls<ret> will add a list of the contents of the current directory before the selection. !cat example.sh<ret> will add the lines of the file example.sh before the selection.

Inserting Emoji

Use Shift + | to conveniently pipe words like ‘smile’ 😄 and ‘house’ 🏠 to a program that converts stdin to emoji, like this one liner:

#!/bin/sh
xargs -r printf ':%s:' | pandoc -f gfm -t plain

The -r option is a GNU extension.

Maybe Pandoc is overkill here, but you get the point.

Using the system clipboard

Piping to and from xsel(1) or xclip(1) is your interface to the X clipboard. Now you can copy those snippets from Stack Overflow and feed your cargo cult programming habit.

	Primary selection	Clipboard
Copy	`$xsel -i`	`$xsel -bi`
Cut	`\|xsel -i`	`\|xsel -bi`
Paste	`!xsel`	`!xsel -b`

User configuration

Mapping keys

Of course, you can create key mappings for these things to your heart’s content. Nice thing about Kakoune is it has a proper way of keeping user-specific mappings separate from global mappings, thanks to the scope parameter of :map. By default , is the prefix to all user mappings.

As an example, here is a mapping I use all the time:

map global user m ': format;w;make<ret>ga: echo Making<ret>' -docstring 'Format and write the file, then call *make*(1)'

It’s scoped globally because I want this to work everywhere. Now I just hit ,, m to

fix my sloppy formatting (: format)
- the space after the colon prevents this command from being put into Kakoune’s prompt history
write the file (;w)
- the semicolon separates Kakoune commands
run make(1)
- ;make<ret> calls Kakoune’s :make wrapper, which opens a new buffer
- ga goes back to the buffer I was working on
- : echo Making<ret> is just for visual feedback.

Have a look at :doc mapping for the full story. Note that :map is one of those commands that can take a -docstring argument, which becomes part of the self-documenting hints that you see throughout the UI.

Classic line-editing shortcuts like Ctrl + w and Ctrl + u that predate even Unix are missing from Kakoune’s insert mode; those shortcuts do not fit the paradigm that Kakoune implements says the :doc faq, but for me this is one case where practicality beats purity.

See the kakoune-readline plugin.

Persistent configuration

Let’s say you’ve mapped a few keys and set a few options. How do you make them persistent between kak(1) sessions?

Create the file $HOME/.config/kak/kakrc and place your mapping and options in there. This is the analogue to one’s $HOME/.vimrc.

When you start kak(1), before sourcing your kakrc it first sources all *.kak files within your $HOME/.config/kak/autoload/ directory. That means you can put anything—even Git repositories in there (i.e. plugins)—and all non-Kakoune files don’t cause Kakoune any grief.

See kak(1) (man kak) for details on how exactly Kakoune’s run-time files are sourced.

Once your autoload directory exists, however, Kakoune decides not to source the system run-time files under /usr/share/kak/autoload/, so you’ll want to link that system directory symbolically into yours:

    mkdir -p $HOME/.config/kak/autoload/
    ln -s /usr/share/kak/autoload/ $HOME/.config/kak/autoload/

Formatting the entire buffer

The :format command is like doing %, | and specifying some format-fixing program—maybe fmt(1), par(1) or something else. You just need to set the formatcmd option.

set-option window formatcmd par

The window argument specifies the scope of this particular assignment. See :doc options set-option and :doc scope for more explanation.

Also look at autowrap_fmtcmd and autowrap-enable which wrap lines for you while typing, so it doesn’t feel like you’ve reverted to using Notepad.

Lightweight markup

If you’re writing in a lightweight markup language, be it Markdown, ReStructuredText, AsciiDoc, or what have you, it’s nice to have a program go through your writing and fix little formatting mistakes here and there, remove trailing spaces and unnecessary line breaks and so forth. Kakoune’s :format command lets you do this really smoothly without leaving the editor. Just set formatcmd to some command that calls your style linter with any appropriate arguments.

Pandoc can neatly reformat your lazy keyboard-mash markup by telling it that the input and output are of the same kind. ReStructuredText for example,

set-option buffer formatcmd 'pandoc -f rst -t rst --reference-links'

where passing --reference-links is optional of course, but means you can write an inline link and have it converted automatically to a reference at the bottom of the file, which takes care of the otherwise laborious task of managing references, their names and order by hand.

This could become slow for particularly large documents; here I’m using Pandoc to reformat some 600 lines of text and it takes a fraction of a second.

SCSS

In this screencast you can see me fixing a poorly written stylesheet using prettier(1) with the following formatcmd:

set-option window formatcmd 'prettier --parser=scss'

Hooks

You can use hooks to trigger a command when some event happens, such as opening a new file, resizing the window, pressing a key in insert mode, or the user being idle for some length of time. These also have an associated scope. Read :doc hooks to see how they work.

Filetype-specific run commands

Now, what if you want to set an option differently depending on the kind of file you’re editing? You can do that using the WinSetOption hook:

hook <scope> WinSetOption 'filetype=<filetype>' <command>

This hook is triggered whenever the filetype option is set to <filetype>. The command can be any list of commands. So to set a few options that would be something like this:

hook global WinSetOption filetype=(css|scss) %{
    set-option buffer indentwidth 2
    set-option buffer tabstop 2
    set-option buffer scrolloff 5,0
}

This can get repetitive for lots of different filetypes, so we can factor this out using the function filetype-hook as defined below:

define-command filetype-hook -params 2 %{
    hook global WinSetOption 'filetype=(%arg{1})' %arg{2}
}

filetype-hook css|scss %{
    set-option window indentwidth 2
    set-option window tabstop 2
    set-option window scrolloff 5,0
    set-option window formatcmd 'prettier --parser=scss'
}

filetype-hook ruby|python %{
    set-option window indentwidth 4
    set-option window tabstop 4
    set-option window matching_pairs ( ) [ ] { }
}

Should you use Kakoune?

There’s loads more to Kakoune, not least of which are its plugins. But the potential for it to be configured exactly how you like is huge, and it’s very easy to do so. I haven’t even talked about evaluate-commands and %sh{…} blocks here.

Kakoune is so smooth in the way it uses existing programs; take the clipboard for example: why should Kakoune bother with implementing a feature specific to and dependent on X, when it can very easily be done in run-time configuration by piping to and from a program such as xsel? This is a taste of the Unix philosophy, and it’s what makes Kakoune so wonderfully extensible.

But due to the difference in key mappings, existing Vim users will become frustrated in the transition, and Vim veterans will be leaving their beloved configuration behind, putting them back to square one. I can see many Vim users dismissing Kakoune as an interesting idea, but not worth the effort to learn. I mean, even its creator describes it as an experiment for a better code editor.

I have found that learning to use Kakoune’s key stroke language has messed with my muscle memory somewhat when it comes to Vim and crucially Readline, although not as much as I’d anticipated. Still, I now use Emacs bindings in Readline, because they differ enough that I don’t get confused.

GNU Readline is the line-editing library used by countless terminal-based programs.

Update, August 2020: After much experience with all four, my muscle memory can switch smoothly between Emacs, Readline, vi and Kakoune. But YMMV.

This write-up has been mostly positive, but my experience with Kakoune has not been without issue:

I’m having to use Alt all the time now, which can be a bit fiddly to say the least.
There’s just no way to have the cursor turn into a pipe shape while in insert mode, which is a shame because many people rely on that to indicate whether they’re in insert mode.
Insert mode auto wrapping, which uses fold(1), will occasionally create a new line in the middle of a word I’m writing.
Having lots of plugins and a lengthy kakrc can increase startup time, but this true of many if not all text editors.

Guide to Installing the rc Shell with Line-Editing in Linux

greg@cosine.blue (Gregory Chamberlain) — Wed, 26 Jun 2019 00:00:00 +0000

Introduction

This article demonstrates how to download, compile and install Byron Rakitzis’ reimplementation of the rc shell, originally from the Plan 9 system. The first section is a brief history of rc and some of its newer forms, but feel free to skip ahead to the actual guide.

History

Around 10 years or so after creating Unix, the Computing Science Research Center at AT&T Bell Labs developed the Plan 9 operating system, which further riffed on the Unix philosophy. It was only used internally at Bell Labs until the early ’90s when they made it available to universities and businesses. Eventually they released it under an open-source license. Today it’s only really used by hobbyists and people learning about operating systems.

Anyway, rc (short for “run commands”) was the system’s canonical command-line interpreter, and it’s ace. Its syntax is much simpler than that of the established Bourne shell from which Bash and most other contemporary shells derive; and rc’s handling of strings and special characters makes it less error-prone overall.

Many tools from Plan 9, including its fantastic rc shell, have been ported to Unix-like systems under the name Plan 9 from User Space (a.k.a. plan9port). A subset of these programs packaged as 9base is provided by Suckless. The two are probably available from your distribution’s package repository (but keep reading).

An independent project by Byron Rakitzis saw a reimplementation of the rc shell for Unix-like systems. His implementation differs very slightly from the true Plan 9 shell (as ported by plan9port), but before compiling it can be linked with line-editing libraries such as GNU readline, which is why I much prefer it for interactive use.

Install the rc shell

I recommend installing Byron Rakitzis’ reimplementation of rc, which provides such nice features as line editing and tab completions.

Compile from source

Clone the GitHub repository and run the bootstrap script:

git clone https://github.com/rakitzis/rc
cd rc
./bootstrap

This generates an INSTALL file with detailed instructions. Then configure and build like so:

sh configure --with-edit=readline
make
sudo make install

You now have rc installed on your machine. To uninstall, use sudo make uninstall in that same directory.

Start the rc shell

You may need to log out and back in for it to work. Start the shell by typing

rc

or, in order to have rc behave as a login shell, pass the -l flag:

rc -l

As described in the manual (man rc), this tells rc to source $home/.rcrc when it starts. Much like one’s .bashrc, you might like to populate that file with commands that

change your $prompt
change your $path
define functions,
assign environment variables,
do whatever else.

In rc, the home directory is stored in both the lowercase variable $home and the usual uppercase environment variable $HOME. Also, the built-in $path variable is a list of directories that is kept in sync with the usual $PATH environment variable, a colon-separated string of the same directories.

The default prompt is a semicolon, which seems an odd choice. According to the manual,

[t]he reason for this is that it enables an rc user to grab commands from previous lines using a mouse, and to present them to rc for re-interpretation; the semicolon prompt is simply ignored by rc.

To quit the shell, press Ctrl + d or type exit.

Make rc your login shell

When you open your terminal, the first program it runs is your login shell. On most Linux machines, users’ login shells are set to Bash by default. Changing your login shell is easy to do.

Firstly, add rc’s full path to your machine’s list of approved login shells. This must be done as root, as so:

sudo su -c 'which rc >> /etc/shells'

Let’s do this next step in an rc shell, demonstrating its backquote substitution (analogous to the Bourne shell’s command substitution). And we’ll use the built-in chsh utility:

rc
chsh $USER --shell `{ which rc }

That’s it! Next time you log in as the same user, your tty and terminals will start with the rc shell. This has had no effect on other users; so if your rc executable breaks or disappears, you can simply log in as root and change your shell back to Bash or what have you.

Make use of run commands

Many users like to use la as an alias to ls -A. To implement this in our rc, let’s define a function in $home/.rcrc.

Unlike the Bourne shell, rc does not support aliases. Use functions instead! The builtin keyword should be used to avoid infinite recursion.

fn la { ls -A $* }

In short, fn is the keyword for creating functions, and the braces contain the sequence of commands the function shall execute. Arguments of the function are stored in $1, $2, etc., but $* stores the list of all arguments given, which we humbly and helpfully pass straight through to the ls program.

I found that I had to log out and back in for the updated $home/.rcrc to be sourced automatically; starting a new instance of rc was not enough. But we can source the new commands manually with the . built-in, as so:

. $home/.rcrc

or, if we’re already in the home directory, simply

. .rcrc