State and the Concurrent World

Clojure

2026-08-20 10:15

Where we are

A promise from unit 1

Introducing Clojure claimed Clojure separates values from identities,

that immutability makes shared mutable state largely moot,

and named vars, atoms, refs and agents.

Then it stopped.

This unit pays it off

1. The problem — and why locking is not the fix
2. The reframing — identity versus value
3. Clojure’s way — persistent structures, managed references
4. The four types
5. The unified model — and how to choose
6. Futures and promises

Watch for this

Almost nothing here is a concurrency primitive.

No mutexes. No critical sections.

The work is done by making values immutable —

and defining the one moment an identity takes a new one.

What goes wrong with shared state

State is not the problem

The real world is full of perceived changes: people change, plans change, the weather changes, and the balance in a bank account changes.

The problem is sharing it between threads and updating it.

The anomalies

Lost updates — two threads read 10, both write 11. Answer should be 12.

Dirty reads — reading data mid-update.

Unrepeatable reads — never able to read the same data twice.

Phantom reads — reading data that has been deleted.

The traditional answer

Locks. Only one thread runs a protected section at a time.

Reasonable — until more than one piece of data needs a coordinated change.

Three disadvantages

• Less multithreaded than before. Others wait while one thread holds the lock.
• Excessive. A reader must lock too, blocking other readers.
• You must remember. Lock the right things, in the right order.

And that last one is the worst

No compile-time warning. No runtime warning.

Just a program that behaves unexpectedly.

The knowledge of what to lock cannot be expressed in the program.

Everyone in the software industry knows how well documentation works.

New problems locking creates

Issue	Description
Deadlock	Threads each wait for locks the other holds.
Starvation	A thread never gets the resources to finish.
Livelock	Both keep moving, neither progresses. Two people in a hallway.
Race condition	Interleaving produces the wrong result — rarely, and unrepeatably.

Identities and values

The flaw in most OO languages

these languages conflate the idea of what Rich Hickey calls identity with that of state.

Favourite movies

As a child: Disney and Pixar.

As a grownup: Tim Burton and Robert Zemeckis.

favorite-movies changed over time.

Or did it?

Two different sets

There are two sets. Neither ever changed.

What changed is which set the entity refers to.

So there are two concepts

• an identity — someone’s favourite movies
• the sequence of values it assumes over time

Which gives a definition:

State is the value of an identity at a particular point in time.

The separation, drawn

The identity never changes; it refers to different values over time.

You already accept this for numbers

x = 101

Nobody expects this to work:

x.setUnitsDigit(3)
x.setTensDigit(2)

x = 101 + 22

x now points at 123 — a completely new value.

Clojure’s move

Extend that from numbers and strings to everything.

Why it dissolves the problem

If a value cannot change:

• a reader can never see a partial update
• no lock is needed to read, ever
• two threads can hold the same value with no interaction

All that is left is the instant an identity swaps one value for another.

Managed references

First, the performance objection

Copying on every update grows linearly with size.

Unusable in production.

The requirements

An immutable structure must:

• leave the old version usable when it mutates
• match the performance of the mutable version

Persistent data structures

A persistent data structure is one that preserves the previous version of itself when it’s modified.

All of Clojure’s core structures are persistent.

They share structure instead of copying.

Four managed references

Type	Useful for
ref	Shared, synchronous, coordinated changes
agent	Shared, asynchronous, independent changes
atom	Shared, synchronous, independent changes
var	Isolated changes — thread-local

Three axes: coordinated or independent · synchronous or asynchronous · shared or thread-local

Refs and STM

Creating and reading

(def all-users (ref {}))

(deref all-users)  ;=> {}
@all-users         ;=> {}

all-users
;=> #<Ref@227e9896: {}>

Reading needs no transaction and never blocks.

Mutating needs a transaction

(ref-set all-users {})
IllegalStateException No transaction running

(dosync
  (ref-set all-users {}))
;=> {}

alter

(defn add-new-user [login budget-amount]
  (dosync
    (let [current-number (count @all-users)
          user (new-user (inc current-number) login budget-amount)]
      (alter all-users assoc login user))))

(add-new-user "amit" 1000000)
;=> {"amit" {:id 2, :login "amit", :monthly-budget 1000000, :total-expenses 0}}

And commute for commutative operations — more concurrency, weaker ordering.

What STM is

Database transactions, applied to shared memory.

Lock-free. And optimistic, where locking is pessimistic.

How a transaction runs

Any number of threads may begin the transaction.

Changes are isolated — only the changing thread sees them.

The first to finish commits. Others abort and retry.

ACI, not ACID

• Atomic — all changes visible at one instant, or none
• Consistent — validators can reject; retries on conflict
• Isolated — in-transaction values are private

No durability — this is volatile memory, not a database.

MVCC

Each thread gets a snapshot when its transaction starts.

readers never block writers (or other readers) … writers never block readers either

The rule you must internalise

A transaction can be retried.

So the code inside a dosync must be free of side effects.

This is not small print — it is the main practical rule for refs, and it is why the agents section exists.

Agents

Send a function, do not set a value

(def total-cpu-time (agent 0))

(send total-cpu-time + 700)
(deref total-cpu-time)
;=> 700

+ is applied to the agent’s current value and 700, later, on another thread.

The send call returns immediately.

send or send-off

send — fixed pool, for CPU-bound actions

send-off — expanding pool, for actions that block

A blocking action sent with send occupies a thread others need.

Waiting

(await & the-agents)
(await-for timeout-in-millis & the-agents)

Mostly a testing tool.

Failure is the substantive part

(def bad-agent (agent 10))

(send bad-agent / 0)
;=> #<Agent@125b9ec1 FAILED: 10>

(deref bad-agent)
;=> 10

(send bad-agent / 2)
ArithmeticException Divide by zero

Further sends fail too, until cleared.

Inspect and clear

(agent-error bad-agent)
;=> #<ArithmeticException java.lang.ArithmeticException: Divide by zero>

(clear-agent-errors bad-agent)

An agent that has silently stopped accepting work is a real failure mode.

Side effects in transactions

A dosync may retry, so it must be pure.

But real programs must send an email when a transaction commits.

The answer

Sends made inside a transaction are held until it commits —

and discarded if it retries or aborts.

So an agent is the correct place for the side effect.

Exactly once, only on success.

Atoms and vars

Atoms

Synchronous, unlike agents.

Independent, unlike refs.

(def total-rows (atom 0))
(swap! total-rows + 100)

Two ways to change one

(reset! atom new-value)
(swap! the-atom the-function & more-args)
(compare-and-set! the-atom old-value new-value)

swap! is a retry loop

Read the current value · compute the new one · install it only if unchanged.

If it changed — discard and retry.

So the function passed to swap! must be free of side effects.

Notice

That is the same rule as the STM’s, by a different mechanism.

It follows from optimistic concurrency generally.

Atoms give no coordination

Two atoms changed one after another are two atomic changes.

With a moment in between where one has moved and the other has not.

If that matters, you needed refs.

Vars: thread-local

(def ^:dynamic *mysql-host*)

(defn db-query [db]
  (binding [*mysql-host* db]
    (count *mysql-host*)))

(def mysql-hosts ["test-mysql" "dev-mysql" "staging-mysql"])

(pmap db-query mysql-hosts)
;=> (10 9 13)

Each thread rebinds for itself. Nothing to coordinate.

One interface, four semantics

Creating

(def a-ref   (ref 0))
(def an-agent (agent 0))
(def an-atom  (atom 0))

Reading

(deref a-ref)     ; or @a-ref
(deref an-agent)  ; or @an-agent
(deref an-atom)   ; or @an-atom

Identical for all four.

Reading is never coordinated, never blocks, cannot fail.

Mutation — where they differ

Refs	Agents	Atoms
(ref-set ref v)	(send agent f & args)	(reset! atom v)
(alter ref f & args)	(send-off agent f & args)	(swap! atom f & args)
(commute ref f & args)		(compare-and-set! atom old new)

Same shape: a function of the old value, installed under defined rules.

Watching

(def adi (atom 0))

(defn on-change [the-key the-ref old-value new-value]
  (println "Hey, seeing change from" old-value "to" new-value))

(add-watch adi :adi-watcher on-change)

(swap! adi inc)
Hey, seeing change from 0 to 1

(remove-watch adi :adi-watcher)

The same for all four.

The design

Four genuinely different semantics behind one interface.

The variation is confined to the one place it must be —

the moment of change.

Choosing a reference type

Vars

Only useful for isolating changes.

Not coordinating. Not sharing.

Cannot be written to by multiple parts of your code.

Atoms

The vast majority of the time you’ll be using atoms because you don’t need anything more.

Two drawbacks: no coordination, and no side effects in swap!.

Refs are atoms with coordination

Split state into several refs and update them atomically in dosync.

Same tradeoff as one big lock versus several small ones.

Changes still must be side-effect free.

Agents

The only type that tolerates side effects.

The cost: error states to check and clear, and asynchronous completion.

The combination worth remembering

A mostly pure mutation with refs —

plus a little side effect with an agent,

which runs only if the transaction succeeds.

A test worth applying

Changing two atoms and worrying about the gap between them?

You have discovered your change was coordinated after all.

Futures and promises

Neither is for state

unlike reference types they can only ever have one value

What distinguishes them: the value may not yet be known.

Something slow

(defn long-calculation [num1 num2]
  (Thread/sleep 5000)
  (* num1 num2))

(defn long-run []
  (let [x (long-calculation 11 13)
        y (long-calculation 13 17)
        z (long-calculation 17 19)]
    (* x y z)))

(time (long-run))
;=> 10207769

Fifteen seconds.

With futures

(future & body)

(time (fast-run))
;=> 10207769

About five seconds on four cores.

The longest, not the sum.

Promises

(def p (promise))

(def value (deref p))

Blocks until filled.

(deliver promise value)

Fills it. Once.

The difference

A future computes its own value — you decide what work happens.

A promise is filled by someone else — you decide when to wait.

A future is a computation. A promise is a rendezvous.

Summary

The six things to carry away

• The problem is not state, not mutation, but shared mutation.
• Locking works and moves the burden of correctness onto you, permanently.
• An identity is a stable name for a succession of immutable values.
• Persistent structures make immutability affordable.
• Four types, two questions: coordinated? synchronous?
• Optimistic concurrency means your function may run more than once.

The idea is smaller than the machinery

No mutexes. No critical sections.

Making values immutable removes almost all of the problem.

What remains is the single moment an identity takes a new value.

Where next

Evolving Clojure Through Macros changes subject entirely.

Back to a thread running since unit 1:

your program is a list, and the compiler reads it as data.