This post discusses Lisp's metaprogramming facilities via Clojure.
In the last post, we explored some of the basics of programming in Clojure and how its constructs and patterns may differ how you'd program in other languages. Many of these differences arise out of its functional programming paradigm; others stem from Lisp itself (i.e. other functional languagess do not necessarily work the same way). These differences highlight the benefits of functional programming and Lisp for writing code that is simultaneously terser and clearer. In this post, we will explore the metaprogramming facilities of Lisp, which allow you to create code that modifies other code.
We've already covered how the Lisp runtime works, but I left out an important detail that allows for metaprogramming to even work! Recall that the runtime model has a reading step and an evaluation step. The reader turns text into data structures for the evaluator to evaluate/run. However, there is actually another step that goes between them – the macroexpansion step! This transforms data structures that, while valid in some respects, are invalid for the evaluator, for a variety of reasons.
Since macroexpansion comes right after the reader, Lisp data structures are passed in, but they're not quite ready for the evaluator yet. Data structures that get macroexpanded involve a macro call. Macros are special functions that receive Lisp data structures that represent program code and return new data structures that also represent program code. These data structures are the same ones we've already been using – vectors, lists, symbols, etc. – which highlights the homoiconic nature of Lisp. Homoiconicity (homo => same, icon => representation) means that Lisp code can manipulate other Lisp code as data that is in the same representation as non-code data (e.g. lists can be used to represent some ordinary data and Lisp code).
Once the macroexpansion process is finished and new data structures have been created that are valid for the evaluator, they are passed straight in and evaluated. The evaluator doesn't know nor care that these data structures were manipulated during macroexpansion – it's all the same as far as it's concerned!
You've already been using macros in your code whether you knew it or not. A number of very basic but not quite fundamental Lisp features and constructs are implemented with macros. For instance defn, which we use to define a function, is actually a macro that combines def with fn.
(macroexpand '(defn function [x] (println x)))
In the above code, we have a list that defines a function. Note that the list has a ' before it, so it's not evaluated (the function is never really defined). Instead, we pass it to the macroexpander to see what it looks like after macroexpansion. After macroexpansion, defn expands to a def form with the same symbol name that we supplied, but now the function arguments and body are wrapped in an anonymous function (via fn). Cool!
It would have been extremely trivial for the Clojure developers to make defn a built-in language construct rather than a macro that relies on other builtins, but recall that Lisp is intended to be as simple as possible. In the interest of keeping the Clojure language simple and free of unnecessary features, defn was not made a language feature. However, it was incorporated into Clojure's standard library because it's still a very useful construct, it just doesn't need to be in the language itself.
Another common macro is when, which evaluates a body of multiple forms if some test form is truthy. For example
(let [x "foobar"]
(when x
(println x)
(str x " and foobaz")))
when is useful if you need to perform side-effect(s) in addition to yielding some value, since, unlike if, it can take multiple forms for the then form. It has no else form, but in many cases you might not need that. Let's see what happens if we macroexpand when.
(macroexpand '(when x (println x) (str x " and foobar")))
Under the hood, it's just using if paired with a do-form! do just wraps multiple forms and evaluates them all, evaluating to the result of the last form.
Macros are practically essential to Lisp. Technically you could program without them, but it would be quite tedious. Not only are they used to implement many features you would probably consider core to any programming language, but some of the code they produce would be extremely tedious to write by hand.
Now that you have witnessed the incredible power of macros, let's see how you can write your own! We'll start with a very basic one allowing us to write our code in infix rather than prefix notation (e.g. (1 + 2) instead of (+ 1 2)). To do this, our macro will have to deconstruct the list it is given into the function we are invoking and its arguments and rearrange them into prefix notation.
(defmacro infix
[form]
(list (second form) (first form) (last form)))
(macroexpand '(infix (1 + 2)))
Our macro is remarkably simple. It receives a form, which is in infix notation, and yields a new list composed of ``form``'s elements in a different order. list, rather obviously, constructs a list of its elements. It's a very simple function, but here we use it to demonstrate Lisp's homoiconicity – we're using a function that produces a data structure in order to produce code! Incredible!
Another macro we could write would allow us to invert the then and else clauses of an if expression.
(defmacro unless
[test a b]
(list `if (not test) a b))
(macroexpand '(unless true (println "false") (println "true")))
Here we simply return a list where the test form is inverted with not. This allows a programmer to switch the order of their then and else forms. This is actually already implemented in Clojure as if-not, but it's a good example. It also highlights the importance of using constructs like the single quote (') to prevent evaluation. In the above macro, we don't want to evaluate if as a symbol -- it has no value! What we want is for if to appear in the final output list, which it does.
In this post, you got your start with Lisp's metaprogramming capabilities. You wrote code that manipulated code – how cool is that?! We formally demonstrated the concept of homoiconicity and its ramifications for the macro system. Lisp's metaprogramming is arguably its coolest feature and one of the best reasons to use it over another language. It allows you to extend the language however you want so that your code is both clearer and more elegant.
Hopefully by now, having read three entire posts about Lisp, you are very intrigued by it and will continue to learn more about it on your own time, applying the principles of Lisp and functional programming wherever you can, whether in a functional language or not. At the very least, you should've learned what Lisp is and why it's not some obscure old tool but is in fact very useful and powerful and thereby deserving of a high rank among the available programming languages. I have provided more resources that you can look into to learn more about Lisp in the previous posts.