bash

Here’s for a small change of pace…

I’d like to talk about a tool I’ve all but forgotten I’m even using (and that’s a compliment to its stability and unobtrusiveness).

dzen is a great little application from the folks at suckless. It’s one of those do one thing and do it well types of tools. It’s probably not useful at all for anyone with a bloated –ahem, excuse me– featureful desktop environment or window manager (or both).

In my case, I’m using just XMonad with its beautiful simplicity. This means, of course, that there’s no out-of-the box… anything.

I’ve already covered some of this from an XMonad perspective, so this post is more about dzen’s general usefulness.

Volume

First up, a small visual notification when I adjust my volume:

It fades in (implicitly thanks to xcompmgr) for just a second when I adjust my volume and gives me that nice, unobtrusive indication of the volume level.

The actual volume adjustment can be done in many alsa or oss specific ways; for my implementation, just see the script as it is live. Completely separate of that, however, we can just use dzen to show the notification:

level=$(get_it_from_alsa_or_oss)

# we use a fifo to buffer the repeated commands that are common with 
# volume adjustment
pipe='/tmp/volpipe'

# define some arguments passed to dzen to determine size and color.
dzen_args=( -tw 200 -h 25 -x 50 -y 50 -bg '#101010' )

# similarly for gdbar
gdbar_args=( -w 180 -h 7 -fg '#606060' -bg '#404040' )

# spawn dzen reading from the pipe (unless it's in mid-action already).
if [[ ! -e "$pipe" ]]; then
  mkfifo "$pipe"
  (dzen2 "${dzen_args[@]}" < "$pipe"; rm -f "$pipe") &
fi

# send the text to the fifo (and eventually to dzen). oss reports 
# something like "15.5" on a scale from 0 to 25 so we strip the decimals 
# and send gdbar an optional "upper limit" argument
(echo ${level/.*/} 25 | gdbar "${gdbar_args[@]}"; sleep 1) >> "$pipe"

Pretty easy, and about as light-weight as you can get.

Status bar

Little known fact: you can use the ubiquitous conky to feed a simple statusbar via dzen. This means you can also use dzen escapes in your TEXT block to do cool things:

My statusbar has the following “features”

• Shows CPU/Mem/Network
• The time, of course
• Shows “Now playing” information from MPD
• The music state (playing/paused) can be clicked to toggle it
• The track title, when clicked, will advance

And here’s the conkyrc to achieve it:

background no
out_to_console yes
out_to_x no
override_utf8_locale yes
update_interval 1
total_run_times 0
mpd_host 192.168.0.5
mpd_port 6600

TEXT
[ ^ca(1, mpc toggle)${mpd_status}^ca()

  ${if_mpd_playing}- ${mpd_elapsed}/${mpd_length}$endif ]

  ^fg(\#909090)^ca(1, mpc next)${mpd_title}^ca()^fg() by

  ^fg(\#909090)${mpd_artist}^fg() from

  ^fg(\#909090)${mpd_album}^fg()

  Cpu: ^fg(\#909090)${cpu}%^fg()

  Mem: ^fg(\#909090)${memperc}%^fg()

  Net: ^fg(\#909090)${downspeedf eth0} / ${upspeedf eth0}^fg()

  ${time %a %b %d %H:%M}

The most interesting part is the clickable areas: ^ca( ... )some text^ca() defines an area of “some text” that can be clicked. The two arguments inside the first parens are are “which mouse button” and “what command to run”. Pretty simple and damn convenient.

Then all you’ve got to do is call this from your startup script:

$ conky -c ~/path/to/that | dzen2 -p -other -args

The -p option just means “persist” so the dzen will never close.

Wrap-up

This was just two examples of some uses for a simple “pipe some text in and see it” GUI toolkit – there are plenty others.

This echoes one of the great things about open-source: something like this is so small, so simple, it could never have survived marketing meetings, planning sessions or cost-benefit analyses – but here it is, and I find it oh-so-very-useful.

Anyone who knows me knows I love the shell. I got my “start” in bash and still have a plethora of scripts lying around doing all sorts of useful and fun things for me. Recently, however, I tackled a task that I had attempted many times in shell script always to be met with frustration. How did I finally figure it out? I made it a rake task and did it in ruby.

Ask me last month what I thought the best tool for this job would’ve been, and 99 times out of 100 I would’ve said “shell script”. But guess what, I couldn’t do it – just never worked out. Now, after having written quite a nice little Rakefile, I can say confidently that I wish I had thought to do this sooner – and I hope I’ll think to do it again.

I want to write about this exercise mainly because I found the process to be quite enjoyable. When I needed to do imperative flow control, call system commands, and move things about the file system, I felt no resistance. More importantly, I could use all of the higher-level features to keep the code clear and clean.

And this is not just praise for ruby (though it does a good job), I’m more recommending that when presented with a task that makes sense as a shell script – think for a second if it might not be possible to do in a higher-level language, you might be surprised.

The Problem

I’ve got a repo (as a lot of you probably do) that contains my main dotfiles. It’s a collection of files that are usually scattered throughout my home directory which I’ve centralized into one folder and placed under version control. The normal approach with this is to symlink these files from the central location out into the proper placed under $HOME.

I wanted to automate this process. I wanted to be able to setup a new box by cloning this repo and running a single script. After that script completes, I want as much of my environment as is generally applicable to be fully configured.

The challenges here were that not all of the files in the repo made sense on every machine, some required parent directories to exist and, of course, I had to be careful not to clobber anything already present.

Nothing about this is insurmountable; the (albeit self-imposed) challenge is to do it as simply and maintainably as possible.

Objectify

The interesting thing about this script is what parts are higher level and what parts are not. So first, here are all of the higher-level bits with the scriptier parts left out:

require 'fileutils'

module Dotfiles
  def self.each(&block)
    [
      '.xcolors/jasonwryan.xcolors',
      '.xcolors/zenburn.xcolors',
      '.gitconfig',
      '.gitignore',
      '.htoprc',
      '.dir_colors',
      '.Xdefaults',
      '.zshrc',
      '.oh-my-zsh',
      '.screen',
      '.vim'
    ].each do |file|
      yield Dotfile.new(file)
    end
  end

  class Dotfile
    include FileUtils

    attr_reader :dotfile
    attr_accessor :source, :target

    def initialize(dotfile)
      @dotfile = dotfile
      @source  = File.join(pwd, dotfile)
      @target  = File.join(ENV['HOME'], dotfile)
    end

    def install!

      #
      # ...
      #

    end
  end
end

desc "updates all submodules"
task :submodules do

  #
  # ...
  #

end

desc "installs all dotfiles into the proper places"
task :install => [:submodules] do

  #
  # ...
  #

end

task :default => :install

This shows the pattern I most often follow when scripting in ruby (which is very different than programming in ruby): one, top-level module to hold any script-wide logic or constants as well as classes to represent the data your working with.

With an overall module and a clean API of classes and methods, you provide yourself a useful set of commands above and beyond the flow control and backtick-interpolation you would normally lean on.

You’ll also notice, in that each method, something I’m calling a Parallel Good Decision. I decided to hardcode the list of dotfile paths relative to the repo. This solved a number of problems that were leading to very smelly code. I could’ve used git ls-files or a normal glob-and-blacklist approach, but simply hardcoding this list allows finer control over what files are linked and if they are treated as files or directories.

Had I made this decision in isolation, it might have been enough for me to get that shell script approach working – but I didn’t. For some reason, only when cleaning up everything else and approaching the problem from a (slightly) higher level did I see that a simple list of relative file paths made the most sense here.

Script It Out

Now that the skeleton-slash-library code is in place, we can fill in the gaps:

require 'fileutils'

module Dotfiles
  def self.each(&block)
    # ... 
  end

  class Dotfile
    # ...

    def install!
      puts "--> installing #{dotfile} as #{target}..."
      if File.exists?(target)
        if File.symlink?(target)
          rm target, :verbose => true
        else
          mv target, "#{target}.backup", :verbose => true
        end
      end

      ln_s source, target, :verbose => true
    end
  end
end

desc "updates all submodules"
task :submodules do
  unless system('git submodule update --init --recursive')
    raise 'error initializing submodules'
  end
end

desc "installs all dotfiles into the proper places"
task :install => [:submodules] do
  Dotfiles.each(&:install!)

  vimrc = Dotfiles::Dotfile.new('.vimrc')
  vimrc.source = File.join(ENV['HOME'], '.vim', 'vimrc')
  vimrc.install!
end

task :default => :install

The stuff that’s easy is easy, the stuff that’s hard is easier and overall, the code is very clean and maintainable.

Oh, and I guess it’s nice that it works.

Mairix is a nice little utility for indexing and searching your emails. Its smooth integration with mutt is also a plus.

I used to use native mutt search, but it’s pretty slow. So far, mairix is giving me a good approximation of the google-powered search available in the web interface and it’s damn fast.

As I go through this setup, keep in mind the example config files are designed to work with my overall mutt setup; one which is described in two other posts here and here.

Mairix

First, of course, install mairix:

pacman -S mairix

Then, setup a ~/.mairixrc which defines where your mails are and their type as well as where to store the results and index. Here’s an example:

# where you keep your mail
base=/home/<you>/Mail

# colon separated list of maildirs to index.
#
# I have two accounts each in their own subfolder. the '...' means there 
# are subdirectories to search as well; it's like saying GMail/* and 
# GMX/*
maildir=GMail...:GMX...

# I omit gmail's archive folder so as to pevent duplicate hits
omit=GMail/all_mail

# search results will be copied to base/<this folder> for viewing in 
# mutt
mfolder=mfolder

# and the path to the index itself
database=/home/<you>/Mail/.mairix_database

With that in place, run mairix once to build the initial index. This first run will be slower but in my tests, subsequent rebuilds were almost instant.

At this point, you could actually do some searching right from the commandline:

mairix some search term # search and populate mfolder
mutt -f mfolder         # open it in mutt

This wasn’t the usage I was after however, I’m typically already in mutt when I want to search my mails.

Mutt

My original script for this purpose was pretty simple. It prompted for the search term and ran it. The problem was you then needed a separate keybind to actually view the results.

Thankfully, Scott commented and provided a more advanced script which got around this issue. Many thanks to Scott and whomever wrote the script in the first place.

This version does some manual tty trickery to build its own prompt, read your input, execute the search and open the results. All from just one keybind.

I merged the two scripts together into what you see below. The main changes from Scott’s version are the following:

1. I kept my clear, purge, search method rather than relying on cron to keep the index up to date.
2. I removed the append-search functionality; not my use-case.
3. I removed the <return> from the ^G trap; it was getting executed by mutt and opening the first message in the inbox after a cancelled search.
4. I fixed it so that backspace works properly in the prompt.

So, here it is:

#!/bin/bash

read_from_config() {
  local key="$1" config="$HOME/.mairixrc"

  sed '/^'"$key"'=\([^ ]*\) *.*$/!d; s//\1/g' "$config"
}

read -r base    < <(read_from_config 'base')
read -r mfolder < <(read_from_config 'mfolder')

# prevent rm / further down...
[[ -z "$base$mfolder" ]] && exit 1

searchdir="$base/$mfolder"

set -f                          # disable globbing.
exec < /dev/tty 3>&1 > /dev/tty # restore stdin/stdout to the terminal,
                                # fd 3 goes to mutt's backticks.
saved_tty_settings=$(stty -g)   # save tty settings before modifying
                                # them

# trap <Ctrl-G> to cancel search
trap '
  printf "\r"; tput ed; tput rc
  printf "/" >&3
  stty "$saved_tty_settings"
  exit
' INT TERM

# put the terminal in cooked mode. Set eof to <return> so that pressing
# <return> doesn't move the cursor to the next line. Disable <Ctrl-Z>
stty icanon echo -ctlecho crterase eof '^M' intr '^G' susp ''

set $(stty size) # retrieve the size of the screen
tput sc          # save cursor position
tput cup "$1" 0  # go to last line of the screen
tput ed          # clear and write prompt
tput sgr0
printf 'Mairix search for: '

# read from the terminal. We can't use "read" because, there won't be
# any NL in the input as <return> is eof.
search=$(dd count=1 2>/dev/null)

# clear the folder and execute a fresh search
( rm -rf "$searchdir"
  mairix -p
  mairix $search
) &>/dev/null

# fix the terminal
printf '\r'; tput ed; tput rc
stty "$saved_tty_settings"

# to be executed by mutt when we return
printf "<change-folder-readonly>=$mfolder<return>" >&3

A non-trivial macro provides the interface to the script. It sets a variable called my_cmd to the output of the script, which should be the actual change-folder command, then executes it.

macro generic ,s "<enter-command>set my_cmd = \`$HOME/.mutt/msearch\`<return><enter-command>push \$my_cmd<return>" "search messages"

One downside which I’ve been unable to fix (and believe me, I’ve tried!) is that if you press ^G to cancel a search but you’ve typed a few letters into the prompt, mutt will read those letters as commands (via the push) and execute them.

The only thing I could do is prefix those characters with something. I’ve decided to use /. That makes mutt see it as a normal search which you can execute or ^G again to cancel. Annoying, but better than mutt flailing around executing rando commands…

Power search

I haven’t had the time yet to learn all the tricks, but here are some of the more useful-looking searches from man mairix:

Useful searches

   t:word                             Match word in the To: header.

   c:word                             Match word in the Cc: header.

   f:word                             Match word in the From: header.

   s:word                             Match word in the Subject: header.

   m:word                             Match word in the Message-ID: 

                                      header.

   b:word                             Match word in the message body 
                                      (text or html!)

   d:[start-datespec]-[end-datespec]  Match messages with Date: headers 
                                      lying in the specific range.

Multiple body parts may be grouped together, if a match in any of them 
is sought.

   tc:word  Match word in either the To: or Cc: headers (or both).

   bs:word  Match word in either the Subject: header or the message body 
            (or both).

   The a: search pattern is an abbreviation for tcf:; i.e. match the 
   word in the To:, Cc: or From: headers.  ("a" stands for "address" in 
   this case.)

The "word" argument to the search strings can take various forms.

   ~word        Match messages not containing the word.  

   word1,word2  This matches if both the words are matched in the 
                specified message part.

   word1/word2  This matches if either of the words are matched in the 
                specified message part.

   substring=   Match any word containing substring as a substring

   substring=N  Match any word containing substring, allowing up to N 
                errors in the match.

   ^substring=  Match any word containing substring as a substring, with 
                the requirement that substring occurs at the beginning 
                of the matched word.

Happy searching!

A fairly long time ago, there was a thread on the Arch forums about clearing your pacman cache.

Pacman’s normal -Sc will remove all versions of any packages that are no longer installed and -Scc will clear that plus old versions of packages that are still installed.

The poster wanted a way to run -Scc but also keep the last 1 or 2 versions back from installed. There was no support for this in pacman directly, so a bit of a bash-off ensued.

I wrote a pretty crappy script which I posted there, it laid around in my ~/.bin collecting dust for a while, but I recently rewrote it. I’m pretty proud of the result for its effectiveness and succinctness, so I think it deserves a little discussion.

The methodology of the two versions is the same, but this new version leans heavily on good ol’ unix shell-scripting principles to provide the exact same functionality in way less code, memory, and time.

Approach

The first approach discussed on the thread was to parse filenames for package and version, then do a little sort-grepping to figure out which versions to keep and which versions to discard. This method is fast, but provably inaccurate if a package name contains numbers on the end.

I went a different way.

For each package, pull the .PKGINFO file out of the archive, parse the pkgname and pkgversion variables out of it, then do the same sort-grepping to figure out what to discard.

My first implementation of this algorithm was really bad. I’d parse and write pkgname|pkgversion to a file in /tmp. Then I’d grep unique package names using -m to return at most the number of versions you want to keep (of each package) and store that in another file. I’d then walk those files and rm the packages.

Ick.

Needs moar unix

The aforementioned ugliness, plus some configuration and error checking weighed in at 162 lines of code, used two files, and was dirt slow. I decided to re-attack the problem with a unix mindset.

In a nutshell: write small units that do one thing and communicate via simple text streams.

The first unit this script needs is a parser. It should accept a list of packages (relative file paths) on stdin, parse and output two space-separated values on stdout: name and path. The path will be needed by the next unit down the line, so we need to pass it through.

parse() {
  local package opt

  while read -r package; do
    case "$package" in
      *gz) opt='-qxzf' ;;
      *xz) opt='-qxJf' ;;
    esac

    bsdtar -O $opt "$package" .PKGINFO |\
        awk -v package="$package" '/^pkgname/ { printf("%s %s\n", $3, package) }'
  done
}

11 lines and damn fast. Thank god for bsdtar’s -q option. It tells the extraction to stop after finding the file I’ve requested. Since the .PKGINFO file is usually the first thing in the archive, we barely do any work to get the values.

It’s also done completely in RAM by piping tar directly to awk.

Step two would be the actual pruning. Accept that same space-separated list on stdin and for any package versions beyond the ones we want to keep (the 3 most recent), echo the full path to the package file on stdout.

prune() {
  local name package last_seen='' num_seen=0

  while read -r name package; do
    [[ -n "$last_seen" ]] && [[ "$last_seen" != "$name" ]] && num_seen=0

    num_seen=$((num_seen+1))

    # print full path
    [[ $num_seen -gt $versions_to_keep ]] && readlink -f "$package"

    last_seen="$name"
  done
}

Just watch the list go by and count the number of packages for each name. I’m ensuring that the list is coming in reverse sorted already, so once we see the number of packages we want to keep, any same-named packages after that should be printed.

So simple.

All that’s left is to make that reverse sorted list and pipe it through.

find ./ -maxdepth 1 -type f -name '*.pkg.tar.[gx]z' | LC_ALL='C' sort -r | parse | prune

So the whole script (new version) weighs in at ~30 lines (with whitespace) and I claim it is exactly as feature-rich as the first version.

I know what you’re saying: there’s no definition of the cache, no optional safe-list vs actual-removing behavior, there’s no removing at all!

Well, you’re just not thinking unix.

$ cd /some/cache/of/packages
$ pacprune                  # as a normal user, just print the list that 
                            # should be removed -- totally safe.
$ pacprune | sudo xargs rm  # then do the actual removal

You’re free to get as fancy as you’d like too…

$ archiveit() { sudo mv "$@" ~/pkg_archive/; }
$ pacprune | xargs archiveit

And the only configuration is setting the versions_to_keep variable at the top of the script.

The script can be found in my scripts repo.

For me, any sort of general purpose note taking and/or keeping solution needs to meet only a few requirements:

1. Noting something has to be quick and easy (in a terminal and scriptable)
2. Notes should be available from anywhere… tothecloud!
3. Notes should be searchable

Now, just to clarify – I’m not talking about classroom notes, those things go in note-books. I’m talking about short little blurbs of information I would like to keep and reference at a later time.

Though, I suppose this could work for classroom notes too…

I’m also not talking about reminders, those are the stuff of calendars, not note-keeping apps.

So what’s my solution? What else, Gmail!

Gmail

Setting up gmail as a note keeper/searcher is simple. A note is an email from me with the prefix “Note -” in the subject line. Therefore, it’s easy to setup a label and a filter to funnel note-mails into a defined folder:

From:    me@whatever.com
Subject: ^Note - 

I also add “Skip inbox” and “Mark as read” as part of the rule.

Requirements 2 and 3 done.

Mutt

So if taking a note is done by just sending an email of a particular consistent format, then it’s easy for me to achieve requirement 1 since I use that awesome terminal mail client mutt.

A short bash function gives us uber-simple note taking abilities by handling the boilerplate of a note-mail:

noteit() {
  _have mutt || return 1 # see my dotfiles re: _have

  local subject="$*" message

  [[ -z "$subject" ]] && { echo 'no subject.'; return 1; }

  echo -e "^D to save, ^C to cancel\nNote:\n"

  message="$(cat)"

  [[ -z "$message" ]] && { echo 'no message.'; return 1; }

  # send message
  echo "$message" | mutt -s "Note - $subject" -- pbrisbin@gmail.com

  echo -e "\nnoted.\n"
}

And here’s how that might work out in the day-to-day:

//blue/0/~/ noteit test note
^D to save, ^C to cancel
Note:

This is a test note.

< I pressed ^D here >
noted.

//blue/0/~/

And there it is, ready to be indexed by the almighty google:

With a few mutt macros, I think this could get pretty featureful without a lot of code.

Let me know in the comments if there are any other simple or out-of-the-box note-keeping solutions you know of.

Oh, and before anyone mentions it – no, you can’t take notes without internet when you’re using this approach. I’m ok with that, I understand if you’re not.