Elixir and the Four Horsemen of the Typeocalypse

Introduction

I like Elixir, a lot. Frankly, if you're writing any sort of web service I don't see why you would use anything else. I even built this blog with Elixir, complete with its own bespoke Markdown parser, also written in Elixir. Why? Because it was fun! Elixir's developer experience is very good, far better than anything else I've used for the web. Every time I set up a new Phoenix project it's been a joy to use, and every time I have to drop back to JavaScript or Python or whatever else I really miss that polished experience.

See, the thing about the Elixir ecosystem is that it's well put-together. I'm not talking about big, flashy things like immutability or the preemptive runtime. I'm talking about little things. The syntax and its resulting AST are incredibly consistent, which means that even when you're using a DSL (e.g. Ecto.Query), the shape of the code is still legible. The module namespaces are implemented sanely, separated from the filesystem - something that doesn't seem like such a big deal until you have to write your first Python import hack. The core libraries, test tooling, docs, and package management are all excellent.

In the JS world, you can't so much as utter the word npm under your breath without being inundated with dozens of warnings from outdated transient dependencies, each of which has several "vulnerabilities", many of them "severe" - a word which here means "vulnerable to DoS". In the batteries included Python world, you don't even get a functioning package manager out of the box. You have to create a "virtual environment". It doesn't even have a lockfile! None of this is a problem in Elixir; it all works exactly how you would expect.

This is possible because Elixir has a fairly tight-knit community. It's popular enough to be relevant, but not much more than that. It's common to see the creator of the language in the commit logs for various libraries, fixing little things. There's a culture in which developers actually care about the experience people have using the language. When something is broken, it gets fixed; I don't just mean bugs, I mean the experience itself.

Which, er, brings me to my point: Elixir's type situation. It's bad. Compared to the rest of Elixir, it's really bad. But there's some nuance to this: like I said, Elixir often nails the little things, and the problems with types in Elixir are a compounding of many little things. I think these things can be fixed, and I think they will be fixed, but in the meantime it's worth talking about them.

The Four Horsemen

A lot of people, who have never written Elixir before, seem to think it has no type system at all. This is, of course, not true: Elixir is dynamically typed, sure, but it still has types. This has been rehashed a thousand times already, and I have nothing to add here, but in case this is your first time there are generally two axes:

• Static / Dynamic: Whether types are checked at compile-time or run-time, respectively

• Strong / Weak: Whether types are checked or not checked at run-time, respectively

This is, naturally, a simplification. Strong and weak typing are confusing terms because there is a broad spectrum of possible type safety. But practically speaking, despite being dynamic, Elixir is a very safe language that strongly enforces its types at run-time. On the other hand, despite having static types, C is a very unsafe language in which you can do many terrible things at run-time.

And that's usually where these sorts of discussions end up. Someone argues that a lack of static types will lead to bugs, someone else counters that Elixir is still safe because types are checked at run-time and, like, you guys do have tests, right? I find these discussions very dissatisfying because they're missing the point. The primary value of a static type system isn't safety: it's iteration speed.

I like it when I start to write out the name of a function and a little box appears that tells me what it expects. This is, I think, 99% of the value of static types. Strangely, the only place I've ever seen this sentiment echoed is this humorous Grug Developer page. The other 1%, by the way, is catching type errors as you write out the code rather than 2 seconds later when you actually run it. I don't mean to trivialize this: those 2 seconds are actually very valuable. It's well known that improvements in iteration speed provide nonlinear improvements in productivity, and as a result the gains end up being quite substantial. I'm a fan.

But the reality is that you can get nearly all of the value I just described by grafting a simple type annotation system onto your language. In Python, these are called type hints, and they're a part of the language spec. In JavaScript, they originally took the form of type annotations inside JSDoc comments.

Of course, once you have all these types floating around for documentation reasons, you may as well also run a gradual type checker on them. And so people do! In Python, you use mypy (or pyright, or a couple others). In JavaScript, you use TypeScript, which also provides its own type syntax because type hints were never added to JS proper (a tragedy).

In Elixir, the type annotations are called typespecs, and the type checker is called Dialyzer. And typespecs are almost good! Almost. The problem is that typespecs and Dialyzer aren't really part of Elixir. The word "grafted" is quite apt. They're just useful enough that people use them, but not quite powerful enough to be used exclusively.

As a result, you have to deal with run-time types a lot as well, and the two type systems don't work very well together. Making matters worse, it's very difficult to properly integrate with the Dialyzer types, so, every time somebody needs types in their library, they integrate with the run-time type system instead. Except the run-time type system isn't expressive enough, so they then graft their own type checking on top of that, again, and invent yet another type system.

This has become a bit of a problem; a problem which I hereby dub the Typeocalypse. And we will spend the rest of article exploring the eponymous Four Horsemen of the Typeocalypse: run-time types, Dialyzer typespecs, and two ad-hoc type systems from Elixir's two most popular "killer app" libraries: Ecto and Phoenix.

Pattern Matching

See, Elixir code isn't quite structured the way your usual, run-of-the-mill, C-derived, object-oriented code is structured. Elixir is a functional language; everything is immutable. And nearly all control flow, save for the occasional if statement, runs through this weird construct called pattern matching.

If you've only ever worked in C-style languages, pattern matching is weird. It violates the most fundamental, sacred principle that has been seared into your mind since you wrote your first line of code. Namely, that when you put something on the left of an equals sign (like x = 3), then the value on the right is assigned to the value on the left. With pattern matching, this is no longer the case; instead, the values on each side of the equals sign are reconciled and then assignments are made, but only if they match up. If they don't match up, the pattern match fails, which will either result in a control flow decision or an exception (a crash).

A number of languages (including Python) have added support for pattern matching in recent years, inspired by the functional style. But make no mistake: in these languages, pattern matching is an afterthought - syntactic sugar. In Elixir, pattern matching is everything.

def lookup_user(user_id) do
  # This function returns
  # {:ok, %User{}} or {:error, "User not found!"}
  if Database.user_exists_with_id(user_id) do
    user = Database.get_user_with_id(user_id)
    {:ok, user}
  else
    {:error, "User not found!"}
  end
end

def get_username!(user_id) do
  # This line will crash if the lookup_user function returns
  # {:error, "User not found!"}
  # because it doesn't match.
  {:ok, %User{username: username}} = lookup_user(user_id)
  username  # This is a return statement
end

def get_username(user_id) do
  # This time, we just return nil if we receive the error
  case lookup_user(user_id) do
    {:ok, %User{username: username}} ->
      username
    {:error, _error_msg} ->
      nil
  end
end

When you write Elixir, virtually all control flow looks like this. Pattern matching is used for assignment, case statements, function statements, and even many of your test assertions. And when something doesn't match up, for example if lookup_user/1 returned something other than a User{} struct, the program would immediately crash.

As a result, Elixir control flow is relentlessly typed. It's not like Python where you get an object or a tuple back and you don't know what's in it unless you bother to check. In Elixir, those type checks are a natural consequence of how the code is structured. Countless developer hours have been lost to debugging because JavaScript decided to coerce an int into a string, or worse, 0 into false. TypeScript will not save you from this. Elixir will.

But while pattern matching may produce safe code, it does little to help with documentation or autocomplete. Those are the realm of static types. Pattern match statements are often buried within functions, and not easily surfaced. Some can be included in function signatures, sure, but those often don't match up with what the function really expects, especially when dealing with collections. As an example:

def count_unique_names(users) when is_list(users) do
  users
  |> Enum.map(fn %User{} = user -> user.username end)
  |> Enum.uniq()
  |> Enum.count()
end

This function has a guard in its signature. Guards are an extension of pattern matching that allow you to perform additional checks. In this case, is_list/1 ensures that users is a list. The anonymous function passed to Enum.map/2 also includes a struct pattern match in its signature, which effectively type checks that each entry in users is a %User{} struct.

The problem is that the full type information, namely that users is a list of %User{} structs, is not included in the signature; it only states users is a list. Which means that extra type information won't show up in autocomplete, and it won't show up in your docs. If you want autocomplete and docs (I certainly do!), you have to give in and start using typespecs.

Typespecs

Typespecs are Elixir's type annotation syntax, in a similar vein to Python type hints. They look like this:

@spec count_unique_names([%User{}]) :: non_neg_integer
def count_unique_names(users) when is_list(users) do
  users
  |> Enum.map(fn %User{} = user -> user.username end)
  |> Enum.uniq()
  |> Enum.count()
end

Typespecs are placed above function names with the @spec syntax. Here, %User{} types a User struct, so [%User{}] types a list of User structs. The :: separates the function signature from the return type, in this case non_neg_integer. Looking at this example, the problems start to make themselves obvious: typespecs and run-time types don't match up! There are now two different type systems in our function. That can't be good, can it?

Another thing that might jump out at you here is that the types are declared above the function, separate from the arguments. A common gut reaction is that this is a bad thing; I felt the same way, initially. Why should I have to write out the function signature twice when every other language only makes me write it once?

Once you have some experience with the language, it becomes clear that this is not a big deal. The function name is easily autocompleted, so it's not a burden to type out. And besides, it kinda has to be this way, because in Elixir functions are also pattern matched.

@spec greet(:normal | :cowboy) :: String.t
def greet(:normal), do: "Hello there!"
def greet(:cowboy), do: "Howdy!"

This function has more than one body, depending on which argument you pass in. However, it can still only have one typespec. Functions like this are very common in Elixir, so there's really no way around it: typespecs have to be separate. Those colon things (:normal and :cowboy) are atoms, by the way. They're something between enums and strings, but not really either. They're used pretty much everywhere.

Typespecs are really great, and I personally like to type pretty much every function. I generally only make exceptions for module-private functions where the signature is obvious at a glance. I also like to use Dialyzer, which I've noticed has a bit of a reputation in the Elixir community for being... bad, I guess? Strangely, this hasn't been my experience at all; it works just fine for me. Perhaps because I only use it in my editor (nvim, via the elixir-ls language server). I don't see the point of running it on the command line or in CI or anything like that - that's why I have tests, after all. I just want the quick popups!

The real problem with typespecs is that they're a bit of an afterthought. Along with Dialyzer, they pre-date Elixir (Dialyzer is from Erlang), but they're not actually a part of the runtime. They're really just type hints, stripped and thrown away as soon as the code is actually compiled. This, combined with the fact that they don't actually match up with the run-time type system, creates trouble downstream for library developers: there's no type system to hook into, so you have to build your own. Which brings us to...

Ecto Schemas

Ecto is Elixir's database toolkit. It's like an ORM (Object-Relational Mapper), but not really; Elixir is a functional language, and has no objects, so there's nothing to map. Instead, Ecto uses schemas, which are essentially just Elixir structs with a fancy macro syntax that are easily wired up to your database. You might be able to see where this is going already: schemas have fields, and fields have, you know... types.

defmodule User do
  use Ecto.Schema

  schema do
    field :username, :string
    field :age, :integer
  end
end

That's an Ecto schema, and those atoms at the end of each field declaration are, in fact, types. But they're not pattern matching types, and they're not typespec types. They're Ecto types. sigh

Like I said, Ecto schemas are just structs. They're autogenerated by a macro, and they come with some fancy database tooling and something called changesets for form validation. But fundamentally, underneath, they're just structs.

defmodule User do
  defstruct [:username, :age]
end

That's a struct. Structs, in turn, are really just maps (a map looks like %{some_key: "some value"}) with some fancy compile-time checking on top. Wait, compile-time checking? Like static types? Well, almost! The structs themselves are types, but their fields don't have types. It works like this:

# This is valid
alice = %User{username: "Alice", age: 20}

# This is a compile error (:occupation is an unknown key)
bob = %User{username: "Bob", age: 21, occupation: "Cryptographer"}

You can even make keys mandatory with the @enforce_keys attribute.

defmodule User do
  @enforce_keys [:username, :age]
  defstruct [:username, :age]
  # Or, because @enforce_keys is a module attribute,
  # you can avoid repeating the keys with:
  # defstruct @enforce_keys
end

Now if you try to declare a struct without the required keys, you get a compile-time error!

# This is a compile error (missing key :age)
alice = %User{username: "Alice"}

This is really great! Catching these errors at compile-time speeds up iteration and improves the developer experience, and in my experience it really does help, especially when renaming struct keys. But the struct fields still aren't typed. If you want to type them, you're going to have to reach for typespecs again.

With typespecs, you can declare a type alias using @type.

@type greeting :: :normal | :cowboy

It is customary to use the alias t for a module's type. For example, String.t refers to a string. So, if we want to type our struct, we can declare a type alias as such:

defmodule User do
  # Usually you would use `%__MODULE__{` instead of `%User{` here
  # to avoid having to repeat the module name in multiple places
  @type t :: %User{
    username: String.t,
    age: non_neg_integer,
  }

  @enforce_keys [:username, :age]
  defstruct @enforce_keys
end

Now we can get type checking on struct fields via Dialyzer! Unfortunately, we had to write everything out twice, and it's kind-of ugly. And unlike with functions and their multiple bodies, this repetition is completely unnecessary, and it's much more annoying to write out as you have to include the field names in the type declaration.

All of this schema stuff is built into Elixir, and has nothing to do with Ecto. But like I said, Ecto generates structs for you via the schema macro, and you provide Ecto with type information about each field. So you would think it would just generate types for you, right? Wrong. That's "out of scope". So now if you want to type everything, your schemas will look like this:

defmodule User do
  use Ecto.Schema
  import Ecto.Changeset

  @type t :: %__MODULE__{
    username: String.t,
    age: non_neg_integer,
  }

  schema do
    field :username, :string
    field :age, :integer
  end

  @spec changeset(%User{}, map) :: %Ecto.Changeset{}
  def changeset(%User{} = user, params) when is_map(params) do
    user
    |> cast(params, [:username, :age])
    |> validate_required([:username, :age])
    |> validate_number(:age, greater_than_or_equal_to: 0)
  end
end

That hurts. And the types aren't even the same! Add in the changeset with a bit of pattern matching and there are three different type systems in this little module, each of which is slightly different.

Now, at this point, you might be thinking I'm being a bit unfair. After all, Ecto is responsible for interfacing with databases, and databases have their own types. So of course Ecto defines its own types; the entirely library basically exists to do type conversion!

But what bothers me is not that Ecto defines its own types. A library should be able to create its own types! That's an expected feature of practically any language. The thing is, Ecto doesn't just have its own types; it has its own type system! And this type system does not properly integrate with the other two type systems that you already have to deal with when writing Elixir code.

Ecto's responsibilities as a type converter are masking the underlying issue, which is that Elixir lacks a properly extensible type system that libraries can build on and integrate with.

Indeed, while using Ecto (which is, to be clear, an excellent library), I never really recognized the underlying Typeocalypse as a problem. I found it pretty annoying that it doesn't generate typespecs, but I figured the authors probably had a good reason and left it at that.

It was only after Phoenix 1.7 came out, with its declarative component assigns, that I actually started to notice something was wrong.

Phoenix Declarative Assigns

Now, technically, declarative assigns were a feature of LiveView 0.18, not Phoenix 1.7. This is because the Phoenix.Component module which handles components is actually a part of LiveView (for historical reasons). However, both releases came out at roughly the same time (though 1.7 had a very long alpha period), and Phoenix 1.7 took big steps toward integrating components into "normal" Phoenix applications, so I really think of declarative assigns as part of the Phoenix release.

The components look like this:

def normal_greeting(assigns) do
  ~H"""
  <span>Hello, <%= @name %>!</span>
  """
end

def cowboy_greeting(assigns) do
  ~H"""
  <span>Howdy, <%= @name %>!</span>
  """
end

def greeting(assigns) do
  # `assigns` is just a map, e.g.
  # %{greeting_type: :cowboy, name: "Alice"}
  # (it contains a couple of special values as well, not shown)
  ~H"""
  <div class="greeting">
    <%= case @greeting_type do %>
      <% :normal -> %>
        <.normal_greeting name={@name} />
      <% :cowboy -> %>
        <.cowboy_greeting name={@name} />
    <% end %>
  </div>
  """
end

If you're familiar with React, they're kind-of like function components; the difference being that Elixir, unlike JS, is actually a functional language, and so they're actually pure (ok, not actually). You can use them with LiveView or just with a normal, static HTML page. When you're using LiveView, there is an event mechanism for updating the values stored in a root assigns map which holds the state of the LiveView.

That ~H-plus-multi-line-string thing is HEEx, a variant of EEx (Embedded Elixir) which parses HTML. The "Embedded" in EEX refers to the fact that the Elixir language is actually embedded into the strings; it's not a watered-down templating language. This is a very good thing. If we've learned anything from 10 years of JSX, it's been that having access to a real programming language in your templates is critical for freedom and sanity. As you can see, there's no problem with embedding a fully-functional case statement into our component's HTML.

The assigns value passed into each component is just a simple map containing the values needed to render that component. The values in the map can then be accessed inside the HEEx via the @value syntax. This works because the ~H sigil is actually a macro, and it compiles the template into Elixir code which in turn has access to the variables in scope. If you're using your components in a LiveView, the assigns also implement change-tracking so they can re-render when values change, like React.

As I said before, I like to type as many functions as I can. So can we type these components? Well, technically, yes. For a start, assigns is technically just a map, so you can technically just pattern match on it. There is a syntax to pattern match on map values, similar to the struct syntax you saw earlier in this article.

def greeting(%{greeting_type: greeting_type, name: name} = assigns)
    when is_atom(greeting_type) and is_binary(name) do
  # ... same as before
end

Ouch, the function signature got so long I had to split it onto a second line. Not a great start. But at least we have run-time validation of our types! By the way, is_binary/1 is checking if name is a string. In Elixir, strings are actually just binary data. The typespec type String.t is also really just an alias for binary. But String.t still communicates that a string is expected, so you should still use it in your type declarations.

What about typespecs? We could do that too!

@spec greeting(%{greeting_type: :normal | :cowboy, name: String.t}) :: term
def greeting(%{greeting_type: greeting_type, name: name} = assigns)
    when is_atom(greeting_type) and is_binary(name) do
  # ... same as before
end

I've set the return type to term (which means any) because, frankly, I have no idea what type these components return. Okay, technically they return a Phoenix.LiveView.Rendered struct, but that's an implementation detail; you will never see this struct during normal usage. It doesn't actually matter because nobody does this. Pattern matching on assigns is used rarely for components with multiple bodies, but I don't think I've ever seen anyone declare typespecs for them. I mean, just look at that function signature. It's practically illegible.

This situation is further complicated by slots, which allow you to compose components by passing in named children. They look like this:

def modal(assigns) do
  ~H"""
  <div class="modal">
    <div class="header"><%= render_slot(@header) %></div>
    <div class="body"><%= render_slot(@inner_block) %></div>
    <div class="footer"><%= render_slot(@footer) %></div>
  </div>
  """
end

def greeting_modal(assigns) do
  ~H"""
  <.modal>
    <:header>Greeting Modal</:header>

    Hello, <%= @name %>!

    <:footer>
      <button>Close</button>
    </:footer>
  </.modal>
  """
end

Slots are a somewhat recent addition to Phoenix, and they're a very important one: slots enable reusable, component-defined composition, allowing developers to more effectively build out design systems and third-party component libraries. They're a big improvement for building more complex Phoenix apps, especially with LiveView. But the assigns pattern has a problem which slots have further exposed: they're hard to document. Remember how it went when we tried to add types to a component?

And now, with these fancy slots, people are starting to build out component libraries. Third-party component libraries! With lots of assigns and slots! So documentation becomes, you know... important.

Enter declarative assigns:

slot :header
slot :inner_block, required: true
slot :footer, required: true

def modal(assigns) do
  ~H"""
  <div class="modal">
    <div class="header"><%= render_slot(@header) %></div>
    <div class="body"><%= render_slot(@inner_block) %></div>
    <div class="footer"><%= render_slot(@footer) %></div>
  </div>
  """
end

attr :greeting_type, :atom, default: :normal
attr :name, :string, required: true

def greeting_modal(assigns) do
  ~H"""
  <.modal>
    <:header>Greeting Modal</:header>

    <%= case @greeting_type do %>
      <% :normal -> %>
        Hello, <%= @name %>!
      <% :cowboy -> %>
        Howdy, <%= @name %>!
    <% end %>

    <:footer>
      <button>Close</button>
    </:footer>
  </.modal>
  """
end

Now we have these slot and attr declarations (they're macros) that allow us to document required attributes, set defaults, and... wait, are those types? Yes, yes they are! And they're not pattern matching types, and they're not typespec types, and they're definitely not Ecto types. What we've just discovered is, in fact, a fourth type system. sigh

The problem is that, as has been our experience with all the other type systems in Elixir, this new type system doesn't quite do everything we need. Sure, it injects documentation into your docstring, along with type information, which is great! But the types are, once again, half-baked; as you can see in the example, @name is supposed to be :normal or :cowboy, but the only type available is atom. The typespec type system allows us to solve this (with a union type, :normal | :cowboy), and the pattern matching system solves it inside the function (via the case statement, which will crash if we provide anything else). But declarative assigns don't. And it's the same problem as before for collections as well. There is no way to declare that you want a list of User structs, or anything like that.

On the bright side, the declarative assigns can do compile-time type checking! But only on literals. It doesn't work like Dialyzer, which is an actual type checker that tries to infer the type of every value.

# This is type checked :)
<.greeting_modal name="Alice" />

# This is not type checked :(
<.greeting_modal name={String.reverse("Alice")} />

The reason I spent so much time detailing Phoenix components is to make it clear, even to someone who's never written Elixir code before, that there are reasons these decisions were made. Something like declarative assigns was needed, and they do make things better. It was entirely reasonable to add them to Phoenix.

And the reason I needed to say that was so that I can say this: declarative assigns are not good. They're okay, sure. But they're not good. And that hurts because most things in Elixir, and in Phoenix, are good!

The Typeocalypse

The reason declarative assigns are not good is the same reason Ecto types, typespec types, and pattern matching types are not good. None of them do everything that's required of a type system! They all take on some portion of what's needed, be it documentation, or static typing, or library-specific needs like typing your database columns. But the fragmentation is brutal; as I've tried to show, these systems just do not work with each other. There is no proper bridge between compile-time and run-time types, and instead you're just expected to specify both. There is no type system for libraries to hook into, so they have to keep inventing their own. Phoenix and Ecto are not niche libraries: they're the most important libraries in the ecosystem. They form the killer app of Elixir! And they're both fighting the language.

There is light at the end of this tunnel. Elixir will be getting a new type system. It will not be an extension of typespecs, or a set of extra pattern matching features. It will be a completely new type system, from the ground up. And it's very important that this be the new Elixir type system, and not the fifth Elixir type system.

The type checker must be strict enough that it can prove functions safe at compile-time, so that sanity pattern match checks can be omitted. And it must provide enough extensibility that libraries can properly integrate with it, even if they have run-time features.

I am very optimistic that these things will happen. The latest blog post about the new type system has an entire section dedicated to the integration of run-time and static types. The solutions presented would go a long way towards solving the friction between type systems that plagues Elixir code today.

And once Elixir has a proper type system to call its own, integrated into the compiler, I have no doubt that the extensibility issues will go away. Elixir's compiler APIs and macros are very powerful, which is why, for example, HEEx templates can compile directly into Elixir code. The biggest problem with Dialyzer is that it's not a part of Elixir at all, and the new system should completely solve that problem.

This new type system is likely not going to be finished for a while, but that's okay. Elixir is already great, and nothing is perfect. The only thing that matters is that things get better. I can wait.