Sunday, 24 January 2010

Speaking in Sentences

I have done quite a lot of programming in Clojure over the last couple of days, and have finally moved past the initial, hesitant stage where I would sit for ages trying to work out what I needed to type. I'm no longer looking up the dictionary to translate individual words. Now I'm speaking in sentences.

This is a very important point to reach because it means I have moved beyond the initial frustrating stage where, sat with my fingers on the keyboard in front of Emacs, connected to a repl, I just didn't know what to type. Every little step forward was painful. As the code flows more and more easily from my fingertips, so it becomes more and more satisfying.

Today, for example, I was trying to convert a series of 163 mp3 files into an audiobook on iTunes, consisting of 15 chapters. Since I couldn't find any nice friendly tools on the Mac, I switched over to my Ubuntu PC to do some proper work. Previously I would have written a little Perl script to automate the tasks I wanted to do, but this time I decided to do it in Clojure. And what a joyful experience that turned out to be.

Let me give you an example. I had already, a few years ago, sorted the 163 mp3 files into 15 different folders. The files are an audio recording of a General Semantics seminar given by Alfred Korzybski himself in 1949. Each folder contained the files for individual lectures. The first thing I wanted to do was tidy up the file names to remove spaces. So I kicked off a repl and started to write some code in Emacs.

First I got a handle on all the mp3 files:
(defn has-ext? [ext file]
  "Does the file end with the extension ext"
  (.endsWith (.getName file) (str "." ext)))

(def mp3? (partial has-ext? "mp3"))

(defn mp3s
  (filter mp3? (file-seq (in-dir dir))))
At the repl I was then able to do something like:
(def mp3files (mp3s "/path/to/top/level"))
mp3files then contains a sequence of objects for each of the files. So now I can use this to rename the files:
(defn replace-str
  "Wraps STring replaceAll"
  [pattern in with]
  (.replaceAll in pattern with)

(defn replace-whitespace
  "Replace whitespace in 'in' with 'with'"
  [in with]
  (replace-str " " in with)

(defn rename-file
  "Rename the file using the single arg fn to transform the file name"
  [file fn]
  (.renameTo file ( (.getParentFile file) (fn (.getName file)))))

(defn rename-files
  "Rename all the files in the fseq using fn to transform the name"
  [fseq fn]
  (map #(rename-file %1 fn) fseq)
With these functions, replacing the whitespace was as simple as typing the following in the repl:> (rename-files mp3files #(replace-whitespace %1 "-"))
This kind of dynamic programming is so much more satisfying than trying to write a Perl script that will finally (hopefully) work only once all the code has been written. Clojure lends itself to writing a little bit of code at a time, sketching out your solution and evolving it. Once I had the handle to the, I could play around with them, experiment with how to extract the names etc. It's just an incredibly satisfying way to work.

I'm now beginning to think in Clojure. The next stage is to gain more familiarity with the core and contrib apis. I'm speaking in sentences, now I need to improve on the vocabulary.

Saturday, 23 January 2010

"Seeing" by Jose Saramago

I finished reading "Seeing" by Jose Saramago this morning. What a wonderful book. It is a sequel of sorts to "Blindness", though I didn't realize this until halfway through.

Words to describe this book: funny, intelligent, gripping, angry, sad. The phrase 'biting satire' is too mild. This doesn't so much bite as dismember with ruthless precision. The world Saramago describes doesn't seem too far removed from our own.

I went straight from the coffee shop where I devoured the last 30 odd pages to Waterstones to find more by the same author. I left with 2 in my hand, in eager anticipation of what he will serve up next.


In this blog entry I want to demonstrate the elegance of Clojure by showing how a function I wrote evolved. This is for a little tool that I have written previously in Perl and in Groovy, that I'm now writing it in Clojure. The aim of the function is to recursively find all the pom files in a specific directory.

Here is version 1:
(defn pom-files-under
  "Find all the pom files in the current directory"
  (filter pom-file? (file-seq (File. dir)))

(defn pom-file?
  (and (= "pom.xml" (.getName file))
       (not-under-target? file)))

(defn not-under-target?
  (not (.contains (.getAbsolutePath file) "target")))
This worked, but I wasn't particularly happy with it. There is no checking that the parameter to pom-files-under is actually a directory and the pom-file? method is ugly. So I scrapped that code and started again. Here is version 2:
(defmulti in-dir
  "Abstracts over the concept of a directory, always returing a
that is guaranteed to be a directory"

(defmethod in-dir String [s]
  (in-dir ( s)))

(defmethod in-dir [f]
  (if (.isDirectory f)
    (throw (IllegalArgumentException. "File is not a directory"))))

(defn has-name? [name file]
  (= name (.getName file))

;; partial application of has-name? checking if file name is pom.xml
(def pom? (partial has-name? "pom.xml"))

(defn not-under-target?
  (not (.contains (.getAbsolutePath file) "target")))

(defn find-files
  "Find files in directory that match predicates pred & others"
  [in-dir pred & others]
  (filter pred (file-seq in-dir))

(defn pom-files
  "Find all pom files recursively in directory in-dir"
  (find-files (in-dir dir) pom?)
This seems much more elegant.The in-dir multi-method now guarantees to return a object that represents a directory. And the find-files method now takes in multiple predicates, the idea being that you supply the directory and the predicates, and it returns the files that match all predicates. The only problem is that the find-files method doesn't actually work at this point. I couldn't for the life of me work out how to implement that functionality. And so I posted a plea for help on the Clojure Google Groups board. The advice I got back really opened my eyes to the kind of abstraction possible in Clojure.

The first solution I implemented based on this advice was:
(defn find-files
  "Find files in directory that match predicates pred & others"
  [in-dir pred & others]
  (reduce (fn [xs f] (filter f xs)) (file-seq in-dir) (cons pred others)))
This fixed the find-files function, making it apply all predicates. Then a discussion started about how to abstract this out further, leading to the following comment from Perry Trolard:
I think it's easier to think about combining predicates separately from your file-filtering code.
Then Sean Devlin followed up with this code to combine predicates:
(defn every-pred?
    "Mimics AND"
    [& preds]
    (fn [& args] (every? #(apply % args) preds)))

(defn any-pred?
    "Mimics OR"
    [& preds]
    (fn [& args] (some #(apply % args) preds))) 
I incorporated this into my code, and did a bit more abstracting, leading to this final version (the rest of the code remained the same):
(def target? (partial has-name? "target"))

(defn not-under-target?
  (not (target? (.getParentFile file))))

(defn every-pred?
    "Mimics AND"
    [& preds]
    (fn [& args] (every? #(apply % args) preds))) 

(defn pom-files
  "Find all pom files recursively in directory in-dir"
  (filter (every-pred? pom? not-under-target?) (file-seq (in-dir dir))))
This process of abstracting really opened my eyes to what is possible in Clojure. I love the elegance and expressiveness possible in this language. The only problem is, the more I program in it, the less I want to go back to my day job of programming in Java! It seems so clunky and primitive now!

Friday, 22 January 2010

Setting up the Clojure Classpath for Utility Scripts in Cygwin

I'm beginning to really enjoy programming in Clojure, but it's still a struggle. The best way to improve is to use it every day. And so I decided to configure my environment to make it easy to write scripts in Clojure. Over the last few years, whenever I have learned a new language, I have used it to write little utility scripts for day to day tasks at work. I have scripts written in Bash, Perl, Groovy, and JRuby. I decided it was time to add Clojure to the list.

I use Cygwin at work, so my starting point was the clj Bash script at I used this one:
if [ $# -eq 0 ] ; then
    java -cp $CLASSPATH jline.ConsoleRunner clojure.main --repl
    while [ $# -gt 0 ] ; do
        case "$1" in
            TMPFILE="/tmp/$(basename $0).$$.tmp"
            /bin/echo $2 > $TMPFILE
            ARGS="$ARGS $1"
    java -cp $CLASSPATH clojure.main $ARGS
    if [ "$TMPFILE" != "" ] ; then
        rm $TMPFILE
I had to edit it a bit to make it Windows friendly, but the script basically worked. However, I wasn't satisfied. I didn't want to have to manually edit the script, or an environment variable, simply to add jar files to the classpath. I wanted a simpler, more obvious way. And so I created a directory in my home directory called .clojure-classpath. The contents of the directory are:
$ ls -l
total 24
-rw-r--r-- 1 alexanc mkgroup-l-d 82 Jan 22 11:16 README
lrwxrwxrwx 1 alexanc mkgroup-l-d 92 Jan 22 12:06 clojure-contrib.jar -> /cygdrive/c/m2repository/org/clojure/clojure-contrib/1.1.0-RC3/clojure-contrib-1.1.0-RC3.jar
lrwxrwxrwx 1 alexanc mkgroup-l-d 68 Jan 22 12:06 clojure.jar -> /cygdrive/c/m2repository/org/clojure/clojure/1.1.0/clojure-1.1.0.jar
lrwxrwxrwx 1 alexanc mkgroup-l-d 79 Jan 22 12:42 commons-exec-1.0.1.jar -> /cygdrive/c/m2repository/commons-exec/commons-exec/1.0.1/commons-exec-1.0.1.jar
lrwxrwxrwx 1 alexanc mkgroup-l-d 60 Jan 22 12:06 jline.jar -> /cygdrive/c/m2repository/jline/jline/0.9.94/jline-0.9.94.jar
lrwxrwxrwx 1 alexanc mkgroup-l-d 82 Jan 22 12:08 swank-clojure.jar -> /cygdrive/c/m2repository/swank-clojure/swank-clojure/1.1.0/swank-clojure-1.1.0.jar
I already have a mountain of jar files in my local maven repository so, in order to make them available to Clojure, I simply created symbollic links to them. Then, I modified the clj
Bash script to automajically add all the jar files in this directory to the classpath prior to starting up Clojure. My new, modified version of clj is as follows:
# set up the classpath dynamically. Note this includes the jline jar
for jarfile in `ls -l ~/.clojure-classpath/ | pcol 11`; do
    JAR=`cygpath --windows $jarfile`

if [ $# -eq 0 ] ; then
    stty -icanon min 1 -echo
    java -Djline.terminal=jline.UnixTerminal -cp $CLASSPATH jline.ConsoleRunner clojure.main
    while [ $# -gt 0 ] ; do
        case "$1" in
            TMPFILE="/tmp/$(basename $0).$$.tmp"
            /bin/echo $2 > $TMPFILE
            ARGS="$ARGS $1"
    java -cp $CLASSPATH clojure.main $ARGS
    if [ "$TMPFILE" != "" ] ; then
        rm $TMPFILE
The only non-standard thing in the script is the call to pcol which is a little Perl script I wrote to split a line on whitespace and print the column specified by the number. This modified version of clj:
  1. Lists all the files in the .clojure-classpath directory, and gets the full path from the output of ls.
  2. Converts the path to a Windows friendly path.
  3. Dynamically adds every file to the classpath.
  4. Starts Clojure.
That worked beautifully. Now I can run stand-alone scripts by simply typing: clj [script_name].

The next problem I had to solve was how to edit my scripts in Emacs using the same classpath. Fortunately I knew that I could do this by starting an instance of a Swank-clojure server from within the repl on the command line. Note that the swank-clojure.jar is in the .clojure-classpath directory. The script to start the server is simply:
(require 'swank.swank)
(swank.swank/start-server "nul" :encoding "utf-8" :port 4005)
I can start the server by:
  1. Running: clj start-swank.clj
  2. From a running repl loading start-swank.clj.
  3. Entering the commands directly into the running repl:
user=> (require 'swank.swank)
user=> (swank.swank/start-server "nul" :encoding "utf-8" :port 4005)
Connection opened on local port  4005
Bingo. I can now edit my scripts in Emacs using Slime, with exactly the same classpath that will be used to run them. And since the .clojure-classpath directory contains a symbollic link to the directory I keep the scripts in (not shown in the listing above - I copied that before I added the link to the directory), all the scripts I write will automatically be on the classpath, and hence I can start building up a library of utility functions.

I am delighted with this set up, and can't wait to start writing some utility scripts in Clojure using Emacs.

Monday, 11 January 2010

Documenting Clojure Code

As I begin to write Clojure code, I want to make sure I document it correctly. Looking at some source I can see the following (from
  #^{:author "Laurent Petit (and others)"
     :doc "Functions/macros variants of the ones that can be found in clojure.core 
 (note to other contrib members: feel free to add to this lib)"}
  (:use clojure.contrib.def))

(defn new-by-name
  "Constructs a Java object whose class is specified by a String."
  [class-name & args]
   (clojure.lang.RT/classForName class-name)
   (into-array Object args)))
There is some meta data in the namespace declaration, and a documentation string in the function definition. This is what I should look to emulate. ":doc" must be a standard meta-data label. Yep - page 57 of Programming Clojure confirms it. Others include:
  • :arglists - Parameter info used by doc
  • :doc - Documentation used by doc
  • :file - Source file
  • :line - Source line number
  • :macro - True for macros
  • :name - Local name
  • :ns - Namespace
  • :tag - Expected argument or return type

Sunday, 10 January 2010

Configuring My Development Environment

I'm going for the following configuration:
  1. Use NetBeans to do all the Java / Maven specific stuff.
  2. Use Emacs to do all the Clojure programming.
  3. Use the clojure-maven-plugin to compile the .clj files, and to start the swank server for the project.
If I understand it correctly, this means that I can add dependencies to my pom file, then start up the swank server using "mvn clojure:swank". That will make all the jar files pulled in by Maven available in the Slime repl in Emacs.

I could do all my development in Emacs, but I am unwilling to part with NetBeans. For the pure Java / Maven side, I am very comfortable with NetBeans. This setup gives me the best of both worlds - NetBeans does what it is good at. Emacs does what it is good at.

Let's test this out. I will create a project, add a Jakarta Commons dependency, then try to call it from the repl. Here is the pom:
<project xmlns:xsi="" xmlns="" xsi:schemalocation="">



Some things to note:
  1. I added a repository to pick up the clojure dependency
  2. I had to manually install the swank-clojure dependency to my local repository.
If I then start up the swank server with "mvn clojure:swank", I am able to connect to it from Emacs using M-x slime-connect. Now, am I able to access the Jakarta Commons Lang classes?
user> (org.apache.commons.lang.StringUtils/isBlank " ")
user> (org.apache.commons.lang.StringUtils/isBlank "not blank ")
Yes! That proves the set up.

And does the clojure-maven-plugin compile my .clj files?
neill@korzybski:~/src/mrtj-clojure$ ls target/classes/com/nwalex/mrtj
test$hello_world__3.class  test__init.class  test$loading__6309__auto____1.class
Yes it does.

I am very happy with this. I started down this road yesterday afternoon when I struggled and failed to configure the NetBeans Enclojure plugin to work with Clojure 1.1.0. 1 day later I have a working Clojure development environment integrating NetBeans, Maven, and Emacs. I don't need to mess about with classpaths. Maven will sort all of that out for me when I add dependencies to the pom.

Time to start hacking!

Further Adventures in Emacs

I've decided, if I want to be a Clojure programmer - if I want to be a serious hacker - then I need to learn Emacs. Enough screwing around. I don't need to learn it all at once. If I start using it now, and learn the shortcuts as and when I need them, then in a few months (years?) I should be fairly proficient.
So - I need to set Emacs up on my main Ubuntu PC, on my Mac, and on my Windows work PC. I already tried to set up my work PC using the Emacs Starter Kit, but I couldn't make it work with my company's proxy. I therefore plan to set it up on my Windows PC at home, then send myself a tar ball of the .emacs.d directory for work.

But, first things first. I need to install Emacs + Clojure + Slime on my Ubuntu PC. In a previous post I linked to some articles which described how to set this up. However, I have since discovered that the preferred way is to use the Emacs Starter Kit to install swank-clojure. So that's what I'm going to do. I will document the steps for future reference, and for anyone else who wants to do the same.

From home directory:
  1. Install Emacs: sudo apt-get install emacs23
  2. Get the Emacs Starter Kit: git clone git:// .emacs.d
  3. Run Emacs: emacs &
  4. In Emacs, install swank-clojure: M-x package-list-packages, put an 'I' beside package swank-clojure, and then press 'x' to install.
  5. Run slime: M-x slime
  6. That should prompt to install Clojure. Type 'y' and all should be working.
When I first tried to install this I was getting problems with starting up Slime.
user=> Could not locate swank/swank__init.class or swank/swank.clj on classpath:  (NO_SOURCE_FILE:0)
user=> user=> java.lang.ClassNotFoundException: swank.swank (NO_SOURCE_FILE:0)
user=> user=> nil
java.lang.ClassNotFoundException: swank.swank (NO_SOURCE_FILE:0)
user=> user=>
It turns out this was because I had already manually installed Clojure into my ~/.clojure directory. When I removed the .clojure directory and tried again, it all worked fine. Watching the video at pointed me in the right direction for solving this. I realized that I wasn't being prompted to install Clojure, which suggested to me that I should mask the manually installed version.

Now to try to do the same on Windows...

Update 1: I followed the same process on my Windows PC, but got the following error when installing swank-clojure: "Local variables entry is missing the suffix". A quick Google brought me to Basically the solution was to edit my ~/.emacs.d/package.el file to make the changes indicated in that post. I've placed the edited version that I used at: Use at your own risk.

Update 2: I also made the following changes to make Emacs work better on my work PC. The Emacs Starter Kit will look for a file called [username].el, and load it as part of the initialization process. So I created a file called alexanc.el in .emacs.d and added the following to make it work well with our proxy:
;;; set up the proxy
(setq url-using-proxy t)
(setq url-proxy-services
'(("http" . "our-proxy:8080")))
I'm starting to really enjoy working with Emacs.

Saturday, 9 January 2010

Exporting Code Templates from NetBeans

After upgrading to NetBeans 6.8 I realized I no longer had the custom code templates I had put together. Although NetBeans has functionality to export settings, the code templates are not included. There is an issue raised in Mozilla about this:

Fortunately there is a workaround, as described in this comment:
In fact there is a workaround for this, the codetemplates are stored in an xml file in your userdir - eg.[userdir]/config/Editors/text/x-java/abbreviations.xml. You can copy this file over to your mates' userdir and they should see the same codetemplates in their IDEs.
This worked for me.

Using iPhone Calendar and Remember the Milk

In a previous post I described my Remember the Milk set up. As part of my switch to RTM I moved all the entries from the Calendar app on my iPhone into RTM. This worked very well. The one problem I had was that there was no way to view future tasks easily. I wanted to be able to have a calendar view of upcoming tasks, something I could glance at quickly to determine if I was free at some future date.

The solution for this problem was to use the iCalendar feeds RTM provides to add certains tasks to Google Calendar. Then using Google Sync it was possible to synchronize Google Calendar with the calendar on my iPhone. The net result is that anything tagged with 'appointment' in RTM now automatically appears in my calendar. I don't enter anything directly into the calendar. I simply use the Calendar app on my phone as the calendar view the RTM app lacks.

The one problem I had setting this up was while adding the calendar to Google Calendar. For some reason it just wouldn't import. Other people have had this issue too. It took about 6 attempts to add the iCalendar feed to Google Calendar.

Friday, 1 January 2010

ACL: Chapter 3

Well, the exercises at the end of chapter 3 really forced me to think. I had to really bend my brain to force it to think in functional, rather than imperative terms.

I have decided to try to provide answers to the exercises in the best idiomatic Clojure code that I can manage, including using the clojure-contrib library where possible. I think this is the best way for me to learn to think in Clojure.

Exercise 3 wasn't too bad. It just required a little bit of lateral thinking to implement the required functionality.
; exercise 3
; write a version of union that preserves the order
; of the original lists
(defn new-union [lst1 lst2]
  (distinct (interleave lst1 lst2)))
Exercise 4 really stumped me for a while. It was while doing this exercise that I realized how much work I had to do to think functionally, rather than imperatively. In the end I searched through the api docs to find something that implemented this functionality. I found clojure.contrib.seq-utils/frequencies and looked at the source code, which led me to the reduce function, and hence to the following solution:
; exercise 4
; define a function that takes a list
; and returns a list indicating the number
; of times each element appears, sorted
; from most common element to least
; common
(defn occurrences [lst]
  (sort-by val > (reduce 
                   (fn [counts x]
                     (assoc counts x (inc (get counts x 0))))
In my drive to attempt to write concise, idiomatic Clojure code, making use of libraries, I then re-wrote the function as:
; better version using clojure.contrib.seq-utils/frequencies
(defn better-occurences [lst]
  (sort-by val > (clojure.contrib.seq-utils/frequencies lst)))
That was blood worth sweating. It forced me to hunt through the api documentation, and taught me how to use reduce.

By the time I reached exercise 5 I was beginning to enjoy the power of the sequence library.
; exercise 5
; Suppose the function pos+ takes a list and returns a
; list of each element plus its position
; (pos+ '(7 5 1 4))
; (7 6 3 7)
; Define this function using a) recursion, b) iteration
; c) mapcar (not sure this exists in clojure)

; a) recursion
(defn recursive-pos+ 
  ([lst] (recursive-pos+ lst 0))
  ([lst pos]
    (if (empty? lst)
    (cons (+ pos (first lst)) (recursive-pos+ (rest lst) (inc pos))))))

; b) iteration
(defn iterative-pos+ [lst]
  (loop [result-list () working-list lst pos 0]
    (if (empty? working-list)
      (reverse result-list) ; yuck
      (recur (cons (+ pos (first working-list)) result-list)
        (rest working-list)
        (inc pos)))))

; c) with map - seems to do what mapcar would do
(defn map-pos+ [lst]
  (map (fn [x pos] (+ x pos)) lst (iterate inc 0)))
I just love the elegance of map-pos+.

After all that, exercise 8 came quite easily:
; exercise 8
; define a function that takes a list and prints it
; in dot notation
; (showdots '(a b c))
; (A . (B . (C . NIL)))
(defn showdots [lst]
  (if (empty? lst)
    (str "NIL")
    (str "(" (first lst) " . " (showdots (rest lst)) ")"))) 
As I said above, I've begun to appreciate the difference between thinking in imperative and functional terms. Working through these exercises I feel I am beginning to grok the functional approach. I have a long way to go, but the journey is proving fun.