Lecture notes — More Macros and DSLs

Published

2026-08-20 00:00

Keywords

ver. 1.0.0

Where we are

In More on Functional Programming you pushed closures until they became objects, and then decided not to use them — because knowing you can build something is what makes not building it a decision.

This unit does the same thing with macros, at a larger scale. Evolving Clojure Through Macros added constructs one at a time — unless, randomly, assert-true. Here we go from adding constructs to building languages.

It is the last unit of the module, and it ends where Lisp has always pointed: at the idea that when a problem is awkward in your language, you can build a better one for it.

What you will be able to do

recall-the-macro-machinery — Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it.
write-an-anaphoric-macro — Write an anaphoric macro that deliberately introduces a binding the caller can use.
build-a-threading-anaphoric-macro — Build a threading macro that binds each intermediate result.
shift-work-to-compile-time — Move computation from runtime to compile time with a macro.
write-a-macro-generating-macro — Write a macro that writes macros.
design-a-dsl — Design a domain-specific language by decomposing a problem bottom-up.
build-the-classification-dsl — Build a working DSL through the user-classification example.
explain-metalinguistic-abstraction — Explain metalinguistic abstraction and judge when a DSL is worth building.

What we will cover

Anaphoric macros — deliberately introducing a binding named it.
Compile-time computation — moving work out of runtime entirely.
Macro-generating macros — macros whose expansion is a defmacro.
Metalinguistic abstraction — building a language for the problem, then solving it there.

From constructs to languages

Four steps, each larger than the last:

Anaphoric macros — inverting the hygiene rule you were taught
Compile-time computation — which no function can do
Macro-generating macros — for whole families at once
Domain-specific languages — building a small language and solving the problem in it

Learning outcomes

recall-the-macro-machinery: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it.
explain-metalinguistic-abstraction: Explain metalinguistic abstraction and judge when a DSL is worth building.

A quick review of macros

Compressed, from Evolving Clojure Through Macros:

a macro runs at macroexpansion time, between reading and compiling
its arguments arrive as unevaluated forms
its return value is a form, which replaces the call
templates use syntax quote `, unquote ~ and unquote-splicing ~@
macroexpand-1 and macroexpand show what a macro produced
auto-gensym name# prevents a macro’s bindings from capturing the caller’s

If any of that is hazy, go back — this unit builds on all of it and re-derives none of it.

One rule the next section breaks on purpose

The earlier unit taught that introducing an un-gensymed binding is a bug, called variable capture, because it silently changes the meaning of the caller’s code.

Hold onto that. The very next section does it deliberately.

Learning outcomes

recall-the-macro-machinery: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it.

Concepts

anaphoric-macros: sets up the hygiene rule that anaphoric macros deliberately invert

Anaphoric macros

An anaphor is a word referring back to something earlier — “it” in “take the report and file it”. An anaphoric macro provides exactly that: a pronoun for the thing just computed.

The problem

Start with a placeholder computation:

(defn some-computation [x]
  (if (even? x) false (inc x)))

Now use it, and note the duplication:

(if (some-computation 11)
  (* 2 (some-computation 11)))
;=> 24

some-computation runs twice. Naturally you would use let:

(let [computation (some-computation 11)]
  (if computation
    (* 2 computation)))

and Clojure already folds those together with if-let:

(if-let [computation (some-computation 11)]
  (* 2 computation))

But you still had to invent the name computation and write it three times. What you want:

(anaphoric-if (some-computation 11)
  (* 2 it))

Implementing it

(defmacro anaphoric-if [test-form then-form]
  `(if-let [~'it ~test-form]
     ~then-form))

The interesting part is ~'it. Inside a syntax quote, a bare it would be namespace-qualified into user/it and be useless as a binding. ~'it inserts the unqualified symbol — deliberately unhygienic.

(macroexpand-1 '(anaphoric-if (some-computation 11)
                  (* 2 it)))
;=> (clojure.core/if-let [it (some-computation 11)] (* 2 it))

And it works:

(anaphoric-if (some-computation 12)
  (* 2 it))
;=> nil

(anaphoric-if (some-computation 11)
  (* 2 it))
;=> 24

Generalising it

if is not the only form that would benefit. Abstract over the operator:

(defmacro with-it [operator test-form & exprs]
  `(let [~'it ~test-form]
     (~operator ~'it ~@exprs)))

Now it works with any of them:

(with-it if (some-computation 11)
  (* 2 it))
;=> 24

(with-it and (some-computation 11) (> it 10) (* 2 it))
;=> 24

(with-it when (some-computation 11)
  (println "Got it:" it)
  (* 2 it))
Got it: 12
;=> 24

The cost, stated plainly

The binding is invisible at the call site. A reader who does not know the macro cannot tell where it came from — and any it the caller already had is shadowed.

Anaphoric macros are a real technique with a real readability cost. They earn their place only where the pronoun genuinely reads better than a name would.

Learning outcomes

write-an-anaphoric-macro: Write an anaphoric macro that deliberately introduces a binding the caller can use.

Concepts

anaphoric-macros: implements anaphoric-if and generalises it into with-it

Threading in any position

Unit 2’s threading macros insert the value in a fixed position. -> puts it first:

(defn surface-area-cylinder [r h]
  (-> r
      (+ h)
      (* 2 Math/PI r)))

->> puts it last, which suits sequence pipelines:

(defn some-calculation [a-collection]
  (->> (seq a-collection)
       (filter some-pred?)
       (map a-transform)
       (reduce another-function)))

Where fixed positions fail

Add one step whose function does not take the collection last:

(defn another-calculation [a-collection]
  (->> (seq a-collection)
       (filter some-pred?)
       (map a-transform)
       (#(compute-averages-from % another-pred?))))

That #(...) exists only to move the value into the right slot. Clojure’s own answer is as->, which names the intermediate:

(defn another-calculation [a-collection]
  (as-> (seq a-collection) result
    (filter some-pred? result)
    (map a-transform result)
    (compute-averages-from result another-pred?)))

The result of each step binds to result, so you place it wherever it belongs — no anonymous function needed.

thread-it

The anaphoric version uses it instead of a name you supply:

(defn yet-another-calculation [a-collection]
  (thread-it (seq a-collection)
    (filter some-pred? it)
    (map a-transform it)
    (compute-averages-from it another-pred?)))

Its implementation is recursive, and it handles the empty case more gracefully than the built-in:

(->> )
ArityException Wrong number of args (0) passed to: core/->>

(thread-it)
;=> nil

(defmacro thread-it [& [first-expr & rest-expr]]
  (if (empty? rest-expr)
    first-expr
    `(let [~'it ~first-expr]
       (thread-it ~@rest-expr))))

Each step binds it and recurses, so the macro expands into nested let forms.

Learning outcomes

build-a-threading-anaphoric-macro: Build a threading macro that binds each intermediate result.
write-an-anaphoric-macro: Write an anaphoric macro that deliberately introduces a binding the caller can use.

Concepts

anaphoric-macros: builds thread-it, threading a value into any position via it

Shifting computation to compile time

A macro capability with nothing to do with syntax.

Rotation ciphers, without macros

ROT13 replaces each letter with the one 13 places along. Build it generally:

(def ALPHABETS [\a \b \c \d \e \f \g \h \i \j \k \l \m
                \n \o \p \q \r \s \t \u \v \w \x \y \z])

(def NUM-ALPHABETS (count ALPHABETS))
(def INDICES (range 1 (inc NUM-ALPHABETS)))
(def lookup (zipmap INDICES ALPHABETS))

(defn shift [shift-by index]
  (let [shifted (+ (mod shift-by NUM-ALPHABETS) index)]
    (cond
      (<= shifted 0) (+ shifted NUM-ALPHABETS)
      (> shifted NUM-ALPHABETS) (- shifted NUM-ALPHABETS)
      :else shifted)))

(shift 10 13)  ;=> 23
(shift 20 13)  ;=> 7

Build the substitution table:

(defn shifted-tableau [shift-by]
  (->> (map #(shift shift-by %) INDICES)
       (map lookup)
       (zipmap ALPHABETS)))

(shifted-tableau 13)
;=> {\a \n, \b \o, \c \p, \d \q, \e \r, \f \s, \g \t, \h \u, \i \v, ...}

and encrypt with it:

(defn encrypt [shift-by plaintext]
  (let [shifted (shifted-tableau shift-by)]
    (apply str (map shifted plaintext))))

(encrypt 13 "abracadabra")
;=> "noenpnqnoen"

ROT13 is reciprocal, so encrypting twice returns the original:

(encrypt 13 "noenpnqnoen")
;=> "abracadabra"

Decryption is a negative shift, and partial fixes the rotation:

(defn decrypt [shift-by encrypted]
  (encrypt (- shift-by) encrypted))

(def encrypt-with-rot13 (partial encrypt 13))
(def decrypt-with-rot13 (partial decrypt 13))

What is wasteful here

shifted-tableau runs on every call, though it depends only on shift-by — a constant.

memoize would help, but the computation still happens at least once. What we actually want is the table written into the code:

(defn encrypt-with-rot13 [plaintext]
  (apply str (map {\a \n \b \o \c \p} plaintext)))

with the real map complete for all 26 letters. Then there is nothing to compute at runtime at all.

The macro

(defmacro def-rot-encrypter [name shift-by]
  (let [tableau (shifted-tableau shift-by)]
    `(defn ~name [~'message]
       (apply str (map ~tableau ~'message)))))

The macro computes the tableau during expansion and embeds the result:

(macroexpand-1 '(def-rot-encrypter encrypt13 13))
;=> (clojure.core/defn encrypt13 [message]
      (clojure.core/apply clojure.core/str
        (clojure.core/map {\a \n, \b \o, \c \p, \d \q, \e \r, \f \s, \g \t,
                           \h \u, \i \v, \j \w, \k \x, \l \y, \m \z, \n \a,
                           \o \b, \p \c, \q \d, \r \e, \s \f, \t \g, \u \h,
                           \v \i, \w \j, \x \k, \y \l, \z \m} message)))

An inline literal map. And it works:

(def-rot-encrypter encrypt13 13)
(encrypt13 "abracadabra")
;=> "noenpnqnoen"

The reader converts program text into data structures, expanding macros as it goes. `def-rot-encrypter` generates the tableau during that process, and it appears in the final source as an inline lookup table.

the new encrypt13 function at runtime doesn’t do any tableau computation at all. If, for instance, you were to ship this code off to users as a Java library, they wouldn’t even know that shifted-tableau was ever called.

Then a macro to define both directions at once, generating the function names:

(defmacro define-rot-encryption [shift-by]
  `(do
     (def-rot-encrypter ~(symbol (str "encrypt" shift-by)) ~shift-by)
     (def-rot-encrypter ~(symbol (str "decrypt" shift-by)) ~(- shift-by))))

When this is worth doing

When work is expensive, repeated, and determined entirely by constants available at compile time.

All three matter. If the inputs are not known until runtime there is nothing to precompute, and the macro buys nothing but complexity.

And note that no function can do this at any level of cleverness. Functions run at runtime; only a macro executes during compilation.

Learning outcomes

shift-work-to-compile-time: Move computation from runtime to compile time with a macro.

Concepts

compile-time-computation: builds the cipher without macros, then moves the tableau into the expansion

Macros that write macros

A macro returns code. A defmacro is code. So a macro can return a defmacro.

The goal

(declare x y)

(make-synonym b binding)
;=> #'user/b

(b [x 10 y 20] [x y])
;=> [10 20]

b is now an alias for binding.

The template

If you were writing b by hand:

(defmacro b [& stuff]
  `(binding ~@stuff))

So make-synonym must produce exactly that, with b and binding substituted.

The naive attempt

(defmacro make-synonym [new-name old-name]
  `(defmacro ~new-name [& stuff]
     `(~old-name ~@stuff)))

Expanding it shows the trouble:

(macroexpand-1 '(make-synonym b binding))
;=> (clojure.core/defmacro b [& user/stuff]
      (clojure.core/seq (clojure.core/concat
        (clojure.core/list user/old-name) stuff)))

Two problems. stuff became user/stuff, and old-name was left as user/old-name rather than being replaced by binding.

What a nested backquote actually does

Worth checking directly:

(defmacro back-quote-test []
  `(something))
(macroexpand '(back-quote-test))
;=> (user/something)

One backquote produces the form. Two produce code that produces the form, which is why the expansion above is full of concat and list — that is what a syntax quote compiles into.

The working version

(defmacro make-synonym [new-name old-name]
  `(defmacro ~new-name [& ~'stuff]
     `(~'~old-name ~@~'stuff)))

(macroexpand-1 '(make-synonym b binding))
;=> (clojure.core/defmacro b [& stuff]
      (clojure.core/seq (clojure.core/concat
        (clojure.core/list (quote binding)) stuff)))

Compare with the hand-written template — the same thing.

The odd part is ~'~old-name, and the book explains the order:

first, ~old-name is expanded, leaving ~'binding (the value of old-name) for the generated macro. Then the outer backquote is expanded, leaving you with 'binding, which finally becomes (quote binding).

You need this to ensure the value of old-name is not resolved until the generated macro expands.

How to write these

Not by reasoning about quote levels in your head. Macroexpand one level at a time and look, comparing against the hand-written template you are trying to produce.

And use the technique sparingly. It is among the harder things to read in Clojure, and the person modifying it later may be you.

When it is warranted: not for one macro — write that macro. It pays when you have a family of near-identical macros that would otherwise be written by hand, and where the family may grow. Which is exactly what a DSL is.

Learning outcomes

write-a-macro-generating-macro: Write a macro that writes macros.
recall-the-macro-machinery: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it.

Concepts

macro-generating-macros: implements make-synonym and works through the two levels of quoting

Domain-specific languages

The destination of the unit, and of the module’s macro thread.

Two design considerations

Decomposition. Conventional design is top-down: break the problem into smaller pieces until each is small enough to implement. Bottom-up decomposition goes the other way — build a layer of vocabulary that talks about the problem domain, then express the solution in that vocabulary.

Combinability. The pieces of a good DSL compose. Vocabulary you can combine freely covers cases you did not anticipate; vocabulary you cannot only covers what you listed.

The problem: user classification

Segment users of a website by what they did — their session data. First, persistence:

(ns clj-in-act.ch11.session
  (:require redis))

(def redis-key-for :consumer-id)
(def ^:dynamic *session*)

(defn save-session [session]
  (redis/set (redis-key-for session) (pr-str session)))

(defn find-session [consumer-id]
  (read-string (redis/get consumer-id)))

(defmacro in-session [consumer-id & body]
  `(binding [*session* (find-session ~consumer-id)]
     (do ~@body)))

Note in-session: a macro wrapping a body in a binding of the dynamic var *session* — exactly the dynamic-scope technique from unit 3, used to avoid threading the session through every function.

The language we want

(defsegment googling-clojurians
  (and
    (> (count $search-terms) 0)
    (matches? $url-referrer "google")))

(defsegment loyal-safari
  (and
    (empty? $url-referrer)
    (= :safari $user-agent)))

Read those as a domain expert would. They say what a segment is, not how to compute it. The $-prefixed names are session attributes, and nothing in the rule mentions Redis, sessions or lookups.

How it is built

(defmacro defsegment [segment-name & body])

The body is walked, and every $attribute symbol is rewritten into a lookup against *session*. clojure.walk/postwalk does the traversal:

(defn transform-lookups [dollar-attribute]
  ...)

Notice what had to be a macro, and what did not

Most of this DSL is functions and data. transform-lookups is an ordinary function on symbols. The session persistence is ordinary functions.

Only two things are macros: defsegment, because it defines a name and must receive the rule unevaluated to rewrite it; and in-session, because it establishes a binding around a body.

A DSL built mostly from macros is usually a DSL built wrong. This one is a good example of the balance.

Metalinguistic abstraction

The idea underneath, and the one Lisp has always been associated with: when a problem is awkward in your language, build a better language for it.

Most languages do not offer this option, so most programmers never consider it. Clojure does — which makes the discipline of not doing it part of the skill.

Learning outcomes

design-a-dsl: Design a domain-specific language by decomposing a problem bottom-up.
build-the-classification-dsl: Build a working DSL through the user-classification example.
explain-metalinguistic-abstraction: Explain metalinguistic abstraction and judge when a DSL is worth building.

Concepts

metalinguistic-abstraction: covers bottom-up decomposition and builds the user-classification DSL

What you can now build

Every technique in this unit increases the distance between what your code says and what a reader already knows.

an anaphoric macro introduces a binding that appears from nowhere
compile-time computation means the code that runs is not the code you wrote
a macro-generating macro is hard to read even when correct
a DSL must be learned before any program written in it can be read

Against that, the case is real. A good DSL makes the problem visible instead of the plumbing, and lets people who understand the domain read and check the rules.

The test worth applying. Would a competent Clojure programmer, new to this codebase, be able to read it? If your DSL makes that easier because the domain vocabulary is clearer than the plumbing would be, build it. If it makes it harder because they must learn your language first, write the functions.

Learning outcomes

recall-the-macro-machinery: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it.
write-an-anaphoric-macro: Write an anaphoric macro that deliberately introduces a binding the caller can use.
build-a-threading-anaphoric-macro: Build a threading macro that binds each intermediate result.
shift-work-to-compile-time: Move computation from runtime to compile time with a macro.
write-a-macro-generating-macro: Write a macro that writes macros.
design-a-dsl: Design a domain-specific language by decomposing a problem bottom-up.
build-the-classification-dsl: Build a working DSL through the user-classification example.
explain-metalinguistic-abstraction: Explain metalinguistic abstraction and judge when a DSL is worth building.

Concepts

metalinguistic-abstraction: collects the case for and against building a language
macro-generating-macros: collects the technique that makes DSL families practical

Conclusion

Anaphoric macros invert the hygiene rule on purpose.

~'it inserts an unqualified symbol so the caller can use it. The earlier unit called this capture and told you to prevent it; here it is the feature — at the cost of a binding that appears from nowhere.
thread-it threads a value into any position, not only the first or last.

-> and ->> fix the position, forcing an #() whenever a function does not take its subject there. Binding each intermediate result to it removes the constraint.
Only a macro can move work to compile time.

The cipher’s tableau depends solely on a constant, so def-rot-encrypter computes it during expansion and emits it as an inline literal. Ship the library and nobody knows shifted-tableau was ever called.
A macro can write a macro, and the difficulty is entirely in the quoting.

~'~old-name unquotes, quotes, and unquotes again so the value is not resolved until the generated macro expands. Write these by macroexpanding one level at a time, not by reasoning in your head.
A good DSL is mostly functions and data.

In the classification DSL only defsegment and in-session are macros — one defines a name and rewrites an unevaluated rule, the other establishes a binding. Everything else is ordinary code.
Metalinguistic abstraction is a real option and a real cost.

When a problem is awkward, you can build a better language for it. Most languages do not offer that, which is why the discipline of not doing it is part of the skill.

Where the module ends

This is the last unit. Across the module you have gone from reading your first parenthesis to building languages: the three pillars, the working vocabulary, the building blocks underneath, concurrency through immutability, macros, closures pushed until they became objects, and finally languages built on languages.

The unifying fact — the one that made the last three units possible — has been true since unit 1: your program is data, and you can compute with it.

← More Macros and DSLs

--- title: Lecture notes — More Macros and DSLs keywords: - ver. 1.0.0 date: '2026-08-20' --- [← More Macros and DSLs](index.qmd){.hg-back} ## Where we are In **More on Functional Programming** you pushed closures until they became objects, and then decided not to use them — because knowing you *can* build something is what makes not building it a decision. This unit does the same thing with macros, at a larger scale. **Evolving Clojure Through Macros** added constructs one at a time — `unless`, `randomly`, `assert-true`. Here we go from adding constructs to building **languages**. It is the last unit of the module, and it ends where Lisp has always pointed: at the idea that when a problem is awkward in your language, you can build a better one for it. ## What you will be able to do 1. `recall-the-macro-machinery` — Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it. 2. `write-an-anaphoric-macro` — Write an anaphoric macro that deliberately introduces a binding the caller can use. 3. `build-a-threading-anaphoric-macro` — Build a threading macro that binds each intermediate result. 4. `shift-work-to-compile-time` — Move computation from runtime to compile time with a macro. 5. `write-a-macro-generating-macro` — Write a macro that writes macros. 6. `design-a-dsl` — Design a domain-specific language by decomposing a problem bottom-up. 7. `build-the-classification-dsl` — Build a working DSL through the user-classification example. 8. `explain-metalinguistic-abstraction` — Explain metalinguistic abstraction and judge when a DSL is worth building. ## What we will cover - **Anaphoric macros** — deliberately introducing a binding named `it`. - **Compile-time computation** — moving work out of runtime entirely. - **Macro-generating macros** — macros whose expansion is a `defmacro`. - **Metalinguistic abstraction** — building a language for the problem, then solving it there. ## From constructs to languages Four steps, each larger than the last: 1. **Anaphoric macros** — inverting the hygiene rule you were taught 2. **Compile-time computation** — which no function can do 3. **Macro-generating macros** — for whole families at once 4. **Domain-specific languages** — building a small language and solving the problem in it ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "recall-the-macro-machinery" >}}: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it. - {{< badge outcome "explain-metalinguistic-abstraction" >}}: Explain metalinguistic abstraction and judge when a DSL is worth building. ::: ## A quick review of macros Compressed, from **Evolving Clojure Through Macros**: - a macro runs at **macroexpansion time**, between reading and compiling - its arguments arrive as **unevaluated forms** - its return value is a **form**, which replaces the call - templates use syntax quote `` ` ``, unquote `~` and unquote-splicing `~@` - `macroexpand-1` and `macroexpand` show what a macro produced - auto-gensym `name#` prevents a macro's bindings from capturing the caller's If any of that is hazy, go back — this unit builds on all of it and re-derives none of it. ::: {.callout-important title="One rule the next section breaks on purpose"} The earlier unit taught that introducing an un-gensymed binding is a **bug**, called variable capture, because it silently changes the meaning of the caller's code. Hold onto that. The very next section does it deliberately. ::: ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "recall-the-macro-machinery" >}}: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "anaphoric-macros" >}}: sets up the hygiene rule that anaphoric macros deliberately invert ::: ## Anaphoric macros An *anaphor* is a word referring back to something earlier — "it" in "take the report and file **it**". An **anaphoric macro** provides exactly that: a pronoun for the thing just computed. ### The problem Start with a placeholder computation: ```clojure (defn some-computation [x] (if (even? x) false (inc x))) ``` Now use it, and note the duplication: ```clojure (if (some-computation 11) (* 2 (some-computation 11))) ;=> 24 ``` `some-computation` runs **twice**. Naturally you would use `let`: ```clojure (let [computation (some-computation 11)] (if computation (* 2 computation))) ``` and Clojure already folds those together with `if-let`: ```clojure (if-let [computation (some-computation 11)] (* 2 computation)) ``` But you still had to invent the name `computation` and write it three times. What you want: ```clojure (anaphoric-if (some-computation 11) (* 2 it)) ``` ### Implementing it ```clojure (defmacro anaphoric-if [test-form then-form] `(if-let [~'it ~test-form] ~then-form)) ``` The interesting part is `~'it`. Inside a syntax quote, a bare `it` would be namespace-qualified into `user/it` and be useless as a binding. `~'it` inserts the **unqualified symbol** — deliberately unhygienic. ```clojure (macroexpand-1 '(anaphoric-if (some-computation 11) (* 2 it))) ;=> (clojure.core/if-let [it (some-computation 11)] (* 2 it)) ``` And it works: ```clojure (anaphoric-if (some-computation 12) (* 2 it)) ;=> nil (anaphoric-if (some-computation 11) (* 2 it)) ;=> 24 ``` ### Generalising it `if` is not the only form that would benefit. Abstract over the operator: ```clojure (defmacro with-it [operator test-form & exprs] `(let [~'it ~test-form] (~operator ~'it ~@exprs))) ``` Now `it` works with any of them: ```clojure (with-it if (some-computation 11) (* 2 it)) ;=> 24 (with-it and (some-computation 11) (> it 10) (* 2 it)) ;=> 24 (with-it when (some-computation 11) (println "Got it:" it) (* 2 it)) Got it: 12 ;=> 24 ``` ::: {.callout-important title="The cost, stated plainly"} The binding is **invisible at the call site**. A reader who does not know the macro cannot tell where `it` came from — and any `it` the caller already had is shadowed. Anaphoric macros are a real technique with a real readability cost. They earn their place only where the pronoun genuinely reads better than a name would. ::: ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "write-an-anaphoric-macro" >}}: Write an anaphoric macro that deliberately introduces a binding the caller can use. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "anaphoric-macros" >}}: implements anaphoric-if and generalises it into with-it ::: ## Threading in any position Unit 2's threading macros insert the value in a **fixed** position. `->` puts it first: ```clojure (defn surface-area-cylinder [r h] (-> r (+ h) (* 2 Math/PI r))) ``` `->>` puts it last, which suits sequence pipelines: ```clojure (defn some-calculation [a-collection] (->> (seq a-collection) (filter some-pred?) (map a-transform) (reduce another-function))) ``` ### Where fixed positions fail Add one step whose function does not take the collection last: ```clojure (defn another-calculation [a-collection] (->> (seq a-collection) (filter some-pred?) (map a-transform) (#(compute-averages-from % another-pred?)))) ``` That `#(...)` exists only to move the value into the right slot. Clojure's own answer is `as->`, which names the intermediate: ```clojure (defn another-calculation [a-collection] (as-> (seq a-collection) result (filter some-pred? result) (map a-transform result) (compute-averages-from result another-pred?))) ``` The result of each step binds to `result`, so you place it wherever it belongs — no anonymous function needed. ### thread-it The anaphoric version uses `it` instead of a name you supply: ```clojure (defn yet-another-calculation [a-collection] (thread-it (seq a-collection) (filter some-pred? it) (map a-transform it) (compute-averages-from it another-pred?))) ``` Its implementation is recursive, and it handles the empty case more gracefully than the built-in: ```clojure (->> ) ArityException Wrong number of args (0) passed to: core/->> (thread-it) ;=> nil ``` ```clojure (defmacro thread-it [& [first-expr & rest-expr]] (if (empty? rest-expr) first-expr `(let [~'it ~first-expr] (thread-it ~@rest-expr)))) ``` Each step binds `it` and recurses, so the macro expands into nested `let` forms. ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "build-a-threading-anaphoric-macro" >}}: Build a threading macro that binds each intermediate result. - {{< badge outcome "write-an-anaphoric-macro" >}}: Write an anaphoric macro that deliberately introduces a binding the caller can use. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "anaphoric-macros" >}}: builds thread-it, threading a value into any position via `it` ::: ## Shifting computation to compile time A macro capability with nothing to do with syntax. ### Rotation ciphers, without macros ROT13 replaces each letter with the one 13 places along. Build it generally: ```clojure (def ALPHABETS [\a \b \c \d \e \f \g \h \i \j \k \l \m \n \o \p \q \r \s \t \u \v \w \x \y \z]) (def NUM-ALPHABETS (count ALPHABETS)) (def INDICES (range 1 (inc NUM-ALPHABETS))) (def lookup (zipmap INDICES ALPHABETS)) ``` ```clojure (defn shift [shift-by index] (let [shifted (+ (mod shift-by NUM-ALPHABETS) index)] (cond (<= shifted 0) (+ shifted NUM-ALPHABETS) (> shifted NUM-ALPHABETS) (- shifted NUM-ALPHABETS) :else shifted))) (shift 10 13) ;=> 23 (shift 20 13) ;=> 7 ``` Build the substitution table: ```clojure (defn shifted-tableau [shift-by] (->> (map #(shift shift-by %) INDICES) (map lookup) (zipmap ALPHABETS))) (shifted-tableau 13) ;=> {\a \n, \b \o, \c \p, \d \q, \e \r, \f \s, \g \t, \h \u, \i \v, ...} ``` and encrypt with it: ```clojure (defn encrypt [shift-by plaintext] (let [shifted (shifted-tableau shift-by)] (apply str (map shifted plaintext)))) (encrypt 13 "abracadabra") ;=> "noenpnqnoen" ``` ROT13 is reciprocal, so encrypting twice returns the original: ```clojure (encrypt 13 "noenpnqnoen") ;=> "abracadabra" ``` Decryption is a negative shift, and `partial` fixes the rotation: ```clojure (defn decrypt [shift-by encrypted] (encrypt (- shift-by) encrypted)) (def encrypt-with-rot13 (partial encrypt 13)) (def decrypt-with-rot13 (partial decrypt 13)) ``` ### What is wasteful here `shifted-tableau` runs **on every call**, though it depends only on `shift-by` — a constant. `memoize` would help, but the computation still happens at least once. What we actually want is the table written into the code: ```clojure (defn encrypt-with-rot13 [plaintext] (apply str (map {\a \n \b \o \c \p} plaintext))) ``` with the real map complete for all 26 letters. Then there is nothing to compute at runtime at all. ### The macro ```clojure (defmacro def-rot-encrypter [name shift-by] (let [tableau (shifted-tableau shift-by)] `(defn ~name [~'message] (apply str (map ~tableau ~'message))))) ``` The macro computes the tableau **during expansion** and embeds the result: ```clojure (macroexpand-1 '(def-rot-encrypter encrypt13 13)) ;=> (clojure.core/defn encrypt13 [message] (clojure.core/apply clojure.core/str (clojure.core/map {\a \n, \b \o, \c \p, \d \q, \e \r, \f \s, \g \t, \h \u, \i \v, \j \w, \k \x, \l \y, \m \z, \n \a, \o \b, \p \c, \q \d, \r \e, \s \f, \t \g, \u \h, \v \i, \w \j, \x \k, \y \l, \z \m} message))) ``` An inline literal map. And it works: ```clojure (def-rot-encrypter encrypt13 13) (encrypt13 "abracadabra") ;=> "noenpnqnoen" ``` ![The reader converts program text into data structures, expanding macros as it goes. `def-rot-encrypter` generates the tableau during that process, and it appears in the final source as an inline lookup table.](11_more_macros_and_dsls/img-0.jpeg) > the new `encrypt13` function at runtime doesn't do any tableau computation at all. If, for instance, you were to ship this code off to users as a Java library, they wouldn't even know that `shifted-tableau` was ever called. Then a macro to define both directions at once, generating the function **names**: ```clojure (defmacro define-rot-encryption [shift-by] `(do (def-rot-encrypter ~(symbol (str "encrypt" shift-by)) ~shift-by) (def-rot-encrypter ~(symbol (str "decrypt" shift-by)) ~(- shift-by)))) ``` ::: {.callout-note title="When this is worth doing"} When work is **expensive**, **repeated**, and determined entirely by **constants available at compile time**. All three matter. If the inputs are not known until runtime there is nothing to precompute, and the macro buys nothing but complexity. And note that no function can do this at any level of cleverness. Functions run at runtime; only a macro executes during compilation. ::: ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "shift-work-to-compile-time" >}}: Move computation from runtime to compile time with a macro. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "compile-time-computation" >}}: builds the cipher without macros, then moves the tableau into the expansion ::: ## Macros that write macros A macro returns code. A `defmacro` **is** code. So a macro can return a `defmacro`. ### The goal ```clojure (declare x y) (make-synonym b binding) ;=> #'user/b (b [x 10 y 20] [x y]) ;=> [10 20] ``` `b` is now an alias for `binding`. ### The template If you were writing `b` by hand: ```clojure (defmacro b [& stuff] `(binding ~@stuff)) ``` So `make-synonym` must produce exactly that, with `b` and `binding` substituted. ### The naive attempt ```clojure (defmacro make-synonym [new-name old-name] `(defmacro ~new-name [& stuff] `(~old-name ~@stuff))) ``` Expanding it shows the trouble: ```clojure (macroexpand-1 '(make-synonym b binding)) ;=> (clojure.core/defmacro b [& user/stuff] (clojure.core/seq (clojure.core/concat (clojure.core/list user/old-name) stuff))) ``` Two problems. `stuff` became `user/stuff`, and `old-name` was left as `user/old-name` rather than being replaced by `binding`. ::: {.callout-tip title="What a nested backquote actually does"} Worth checking directly: ```clojure (defmacro back-quote-test [] `(something)) (macroexpand '(back-quote-test)) ;=> (user/something) ``` One backquote produces the form. Two produce **code that produces the form**, which is why the expansion above is full of `concat` and `list` — that is what a syntax quote compiles into. ::: ### The working version ```clojure (defmacro make-synonym [new-name old-name] `(defmacro ~new-name [& ~'stuff] `(~'~old-name ~@~'stuff))) ``` ```clojure (macroexpand-1 '(make-synonym b binding)) ;=> (clojure.core/defmacro b [& stuff] (clojure.core/seq (clojure.core/concat (clojure.core/list (quote binding)) stuff))) ``` Compare with the hand-written template — the same thing. The odd part is `~'~old-name`, and the book explains the order: > first, `~old-name` is expanded, leaving `~'binding` (the value of `old-name`) for the generated macro. Then the outer backquote is expanded, leaving you with `'binding`, which finally becomes `(quote binding)`. You need this to ensure the value of `old-name` **is not resolved until the generated macro expands**. ::: {.callout-important title="How to write these"} Not by reasoning about quote levels in your head. **Macroexpand one level at a time and look**, comparing against the hand-written template you are trying to produce. And use the technique sparingly. It is among the harder things to read in Clojure, and the person modifying it later may be you. ::: **When it is warranted:** not for one macro — write that macro. It pays when you have a **family** of near-identical macros that would otherwise be written by hand, and where the family may grow. Which is exactly what a DSL is. ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "write-a-macro-generating-macro" >}}: Write a macro that writes macros. - {{< badge outcome "recall-the-macro-machinery" >}}: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "macro-generating-macros" >}}: implements make-synonym and works through the two levels of quoting ::: ## Domain-specific languages The destination of the unit, and of the module's macro thread. ### Two design considerations **Decomposition.** Conventional design is **top-down**: break the problem into smaller pieces until each is small enough to implement. **Bottom-up** decomposition goes the other way — build a layer of vocabulary that talks about the *problem domain*, then express the solution in that vocabulary. **Combinability.** The pieces of a good DSL compose. Vocabulary you can combine freely covers cases you did not anticipate; vocabulary you cannot only covers what you listed. ### The problem: user classification Segment users of a website by what they did — their session data. First, persistence: ```clojure (ns clj-in-act.ch11.session (:require redis)) (def redis-key-for :consumer-id) (def ^:dynamic *session*) (defn save-session [session] (redis/set (redis-key-for session) (pr-str session))) (defn find-session [consumer-id] (read-string (redis/get consumer-id))) (defmacro in-session [consumer-id & body] `(binding [*session* (find-session ~consumer-id)] (do ~@body))) ``` Note `in-session`: a macro wrapping a body in a `binding` of the dynamic var `*session*` — exactly the dynamic-scope technique from unit 3, used to avoid threading the session through every function. ### The language we want ```clojure (defsegment googling-clojurians (and (> (count $search-terms) 0) (matches? $url-referrer "google"))) ``` ```clojure (defsegment loyal-safari (and (empty? $url-referrer) (= :safari $user-agent))) ``` Read those as a domain expert would. They say what a segment *is*, not how to compute it. The `$`-prefixed names are session attributes, and nothing in the rule mentions Redis, sessions or lookups. ### How it is built ```clojure (defmacro defsegment [segment-name & body]) ``` The body is walked, and every `$attribute` symbol is rewritten into a lookup against `*session*`. `clojure.walk/postwalk` does the traversal: ```clojure (defn transform-lookups [dollar-attribute] ...) ``` ::: {.callout-note title="Notice what had to be a macro, and what did not"} Most of this DSL is **functions and data**. `transform-lookups` is an ordinary function on symbols. The session persistence is ordinary functions. Only two things are macros: `defsegment`, because it **defines a name** and must receive the rule **unevaluated** to rewrite it; and `in-session`, because it establishes a **binding** around a body. A DSL built mostly from macros is usually a DSL built wrong. This one is a good example of the balance. ::: ### Metalinguistic abstraction The idea underneath, and the one Lisp has always been associated with: when a problem is awkward in your language, **build a better language for it**. Most languages do not offer this option, so most programmers never consider it. Clojure does — which makes the discipline of *not* doing it part of the skill. ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "design-a-dsl" >}}: Design a domain-specific language by decomposing a problem bottom-up. - {{< badge outcome "build-the-classification-dsl" >}}: Build a working DSL through the user-classification example. - {{< badge outcome "explain-metalinguistic-abstraction" >}}: Explain metalinguistic abstraction and judge when a DSL is worth building. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "metalinguistic-abstraction" >}}: covers bottom-up decomposition and builds the user-classification DSL ::: ## What you can now build Every technique in this unit increases the distance between what your code says and what a reader already knows. - an **anaphoric macro** introduces a binding that appears from nowhere - **compile-time computation** means the code that runs is not the code you wrote - a **macro-generating macro** is hard to read even when correct - a **DSL** must be learned before any program written in it can be read Against that, the case is real. A good DSL makes the *problem* visible instead of the plumbing, and lets people who understand the domain read and check the rules. **The test worth applying.** Would a competent Clojure programmer, new to this codebase, be able to read it? If your DSL makes that easier because the domain vocabulary is clearer than the plumbing would be, build it. If it makes it harder because they must learn your language first, write the functions. ::: {.hypergen-refs .outcomes} <h1>Learning outcomes</h1> - {{< badge outcome "recall-the-macro-machinery" >}}: Recall the macro machinery from the earlier unit and be ready to use it without re-deriving it. - {{< badge outcome "write-an-anaphoric-macro" >}}: Write an anaphoric macro that deliberately introduces a binding the caller can use. - {{< badge outcome "build-a-threading-anaphoric-macro" >}}: Build a threading macro that binds each intermediate result. - {{< badge outcome "shift-work-to-compile-time" >}}: Move computation from runtime to compile time with a macro. - {{< badge outcome "write-a-macro-generating-macro" >}}: Write a macro that writes macros. - {{< badge outcome "design-a-dsl" >}}: Design a domain-specific language by decomposing a problem bottom-up. - {{< badge outcome "build-the-classification-dsl" >}}: Build a working DSL through the user-classification example. - {{< badge outcome "explain-metalinguistic-abstraction" >}}: Explain metalinguistic abstraction and judge when a DSL is worth building. ::: ::: {.hypergen-refs .concepts} <h1>Concepts</h1> - {{< badge concept "metalinguistic-abstraction" >}}: collects the case for and against building a language - {{< badge concept "macro-generating-macros" >}}: collects the technique that makes DSL families practical ::: ## Conclusion - Anaphoric macros invert the hygiene rule on purpose. > `~'it` inserts an unqualified symbol so the caller can use it. The earlier unit called this capture and told you to prevent it; here it is the feature — at the cost of a binding that appears from nowhere. - `thread-it` threads a value into any position, not only the first or last. > `->` and `->>` fix the position, forcing an `#()` whenever a function does not take its subject there. Binding each intermediate result to `it` removes the constraint. - Only a macro can move work to compile time. > The cipher's tableau depends solely on a constant, so `def-rot-encrypter` computes it during expansion and emits it as an inline literal. Ship the library and nobody knows `shifted-tableau` was ever called. - A macro can write a macro, and the difficulty is entirely in the quoting. > `~'~old-name` unquotes, quotes, and unquotes again so the value is not resolved until the generated macro expands. Write these by macroexpanding one level at a time, not by reasoning in your head. - A good DSL is mostly functions and data. > In the classification DSL only `defsegment` and `in-session` are macros — one defines a name and rewrites an unevaluated rule, the other establishes a binding. Everything else is ordinary code. - Metalinguistic abstraction is a real option and a real cost. > When a problem is awkward, you can build a better language for it. Most languages do not offer that, which is why the discipline of not doing it is part of the skill. ### Where the module ends This is the last unit. Across the module you have gone from reading your first parenthesis to building languages: the three pillars, the working vocabulary, the building blocks underneath, concurrency through immutability, macros, closures pushed until they became objects, and finally languages built on languages. The unifying fact — the one that made the last three units possible — has been true since unit 1: **your program is data, and you can compute with it**. ::: {.hg-nav} [← More Macros and DSLs](index.qmd){.hg-prev} :::