Family Ties part 4: Scoping Out the Scene

This is the fourth of a series of articles on what I’ve learned about Erlang (and Elixir) from writing Erl2ex, an Erlang-to-Elixir transpiler. This week we take a look at variable scoping, comparing Elixir’s caret operator with Erlang’s single assignment, and we’ll explore a few surprises lurking in how variables are scoped in the presence of different language constructs.

Is Elixir data really immutable?

Immutability is one of the most important features of many functional languages, including Erlang and Elixir. Preventing changes to existing data simplifies concurrency management, enables a variety of performance optimizations, and aligns your code with well-understood architectural patterns.

But wait… one thing we may have learned early on was that, while Erlang enforces single-assignment—any given variable name can be bound only once—Elixir allows variables to be reassigned.

% This is not allowed in Erlang.
A = foo.
A = bar.

# However, this IS allowed in Elixir.
a = :foo
a = :bar

Does this mean that Elixir actually does allow mutation? After all, we’re modifying the value of variable “a” above, right?

As we look at variable binding and scoping, we should start by emphasizing an important distinction. Elixir data is just as immutable as Erlang data. In the example, the values :foo and :bar remain unchanged, immutable, in memory. If you are working with one of those values, it cannot be changed out from under you. All that is changing is which value the variable “a” points to.

In this article, we’ll see that variable binding and scoping behavior in Elixir and Erlang are sometimes different, and can occasionally be confusing. However, it is important to remember that both languages retain a commitment to data immutability that underlies much of their shared concurrency and reliability stories.

Scoping variables

Most modern languages employ some form of lexical scoping. This means that the hierarchy in a program’s structure dictates the scope of a variable. A variable defined in a block remains visible within that block, and typically within sub-blocks, but cannot be accessed outside the block (unless it is exported somehow). Here is an example in Elixir:

# Elixir code.
def scope_example() do
  # Define a variable "list".
  list = [1,2,3]
  fn(elem) ->
    # Variable "list" is accessible here
    Enum.member?(list, elem)
  end)
end
# Variable "list" is not accessible here

If a variable in an inner scope has the same name as one in an outer scope, the inner variable hides the outer one; the inner is said to “shadow” the outer. To see this more clearly, let’s begin with a Ruby method that sums the values in an array. (Yes, you could also use Enumerable.inject, but suppose you wrote it in a more “imperative” style.)

# Ruby method that sums values in an array.
def sum(array)
  partial_sum = 0
  array.each do |elem|
    # Modifies the existing partial_sum
    partial_sum = partial_sum + elem
  end
  partial_sum
end

This code relies on the fact that Ruby allows you to modify the “partial_sum” variable from inside the “do” block, a common pattern in more “imperative” languages. Trying to translate this “directly” into Elixir would not work:

# Elixir function that doesn't work as desired.
def broken_sum(list) do
  partial_sum = 0
  Enum.each(list, fn(elem) ->
    # Creates a new inner partial_sum; does NOT modify the existing var
    partial_sum = partial_sum + elem
  end)
  # Always returns 0
  partial_sum
end

Elixir functions cannot modify variables from an outer scope. Instead, Elixir creates a new variable in the inner scope, shadowing the outer variable. And that makes sense if you think about it: an inner function might eventually be called from, say, another process, and you don’t want it affecting this process’s state. It is a feature that limits side effects. (For similar reasons, Java requires that outer variable accessed by an inner class be declared final. Certain other languages, such as Ruby and Javascript, do allow an inner function to reach out and modify their context, and that opens the door to some subtle bugs when using those languages.)

Of course, you would be correct to criticize such Elixir code as not properly employing a functional style. Idiomatic Elixir would likely use Enum.reduce(). And of course, we’d expect that attempting such code in Erlang would fail completely, because you can’t rebind the variable at all. And that’s the point. Variable scoping and binding behavior are key distinctions in different languages, and often force you to structure your code differently. As we’ll see, the implications can be subtle and surprising.

The caret or the stack

In January, Jose Valim, creator of Elixir, wrote an article on immutability and variable rebinding in Elixir and Erlang. Jose asked the question, is one language more “safe” than the other due to variable rebinding? He shows that, while immutability makes both languages are “safe” in terms of shielding data against clobbering by other processes, both languages are actually vulnerable to bugs that can be introduced if someone changes variable bindings by modifying the code. Elixir variables bind by default, so a value can change unexpectedly if you change the code to insert a new binding. Erlang variables either bind or reference depending on the current state, so behavior of an expression can change unexpectedly if you insert a new binding.

Jose’s article may feel a bit arcane on first glance, but it is well worth understanding because it highlights the variable life cycle and scope that will let us explore some idiosyncracies of both languages.

When an Erlang variable appears on the left side of a match, the behavior depends on the context: if this is the first appearance of the variable in that scope, it is bound at that time, otherwise it is merely referenced. Elixir, on the other hand, forces you to be explicit in choosing whether to bind a variable or reference its earlier value. A variable on the left side of a match is bound, unless it has a caret, in which case it is referenced.

Here’s a simple example in Erlang:

% Erlang function that crashes if the two arguments are not equal.
assert_equal(X, Y) ->
  A = X,
  A = Y.

The two matches in the function do different things: the first binds the variable A, but the second only references the variable (throwing a match error if the value of A is different from the value of Y). If we were to translate this to Elixir, it might look like this. Notice how the semantic difference is now explicit.

# Equivalent Elixir function.
def assert_equal(x, y) do
  a = x
  ^a = y
end

This becomes ever more important as we study the behavior in the presence of nested scopes. For example, as we saw earlier, an Elixir function definition creates an inner scope that shadows outer variables. Is the same true in Erlang? Sort of. Consider this example:

% What does this Erlang function return?
disjoint_scopes() ->
  F = fun() -> A = 1 end,
  F(),
  A = 2,  % A is bound here (and the previous A is no longer in scope)
  A.

The “A” variable in function “F” is in an inner scope that no longer exists at the end of “foo”, so we’d expect that A can then be bound in the outer scope, and the value 2 would be returned. This is indeed the case.

The Elixir equivalent of the above function is:

# The equivalent Elixir function.
def disjoint_scopes() do
  f = fn -> a = 1 end
  f.()
  a = 2  # a is bound here (and the previous a is no longer in scope)
  a
end

However, what if we swapped the order of the two bindings of variable “A”?

% What does this Erlang function return?
overlapping_scopes() ->
  A = 2,                   % This line moved up
  F = fun() -> A = 1 end,  % What does this do now?
  F(),
  A.

Now it throws a match error. Erlang now considers that “inner” A variable to be a reference to the outer A, and it fails because 1 != 2. Perhaps that was unexpected at first glance, but it kind-of makes sense. Previously, when the outer “A” variable was bound second, the scopes of the two variables never overlap. However, now when the outer “A” variable is bound first, it is lexically still in scope inside function “F”. Erlang has to resolve this conflict somehow, and it could have done so in two ways. The inner variable could reference the outer variable. Or it could shadow the outer variable and create a new local variable inside function “F”. Erlang’s strategy is to choose the former.

Elixir, on the other hand, allows you, the programmer, to make the choice by including or omitting the caret. The Elixir equivalent of the above Erlang code is:

# The equivalent Elixir function.
def overlapping_scopes() do
  a = 2
  f = fn -> ^a = 1 end  # Note caret. The reference is explicit.
  f.()
  a
end

However, remember the choice that Erlang had: either refrence the outer variable or create a local variable that shadows the outer variable. You have the same choice in Elixir with the caret operator. If you omit the caret in Elixir, you are not only deciding to bind rather than reference, but you are actually changing the scope of that inner variable. It does not rebind or affect the outer “a” variable, but it creates a local “a” inside the inner function.

# Omitting the caret changes the scope of "a"
def nested_scopes() do
  a = 2
  f = fn -> a = 1 end  # If caret omitted: now a local variable
  f.()
  a        # returns 2, the value of the outer variable
end

So far so good? Well now we get to the surprising part…

Exposing the exports

Scope creation, especially when interacting with control structures, tends to be messy in pretty much any language. Erlang and Elixir are, unfortunately, no exception. We saw above that function scopes can partially “leak” in certain cases. Here’s another example.

Remember from above that an Erlang variable in an “inner” function remains confined to its inner scope as long as that variable wasn’t previously bound.

% This Erlang function returns 2.
disjoint_scopes() ->
  F = fun() -> A = 1 end,
  F(),
  A = 2,  % Binds to A, because the previous A is no longer in scope
  A.

This is not true for some other control structures.

% This Erlang function crashes if you pass in anything other than 1.
only_one(X) ->
  case X of
    1 -> A = ok;
    _ -> A = notOk
  end,
  A = ok,  % Variable A is already bound, so this is a reference.
  A.

In the above case, the “A” variables in the case statement are “exported” to their surrounding scope. Variable “A” has already been bound at the time of the later match, so that match will succeed only if you pass 1 to the function. In other words, its Elixir equivalent is:

# The equivalent Elixir function.
def only_one(x) do
  case(x) do
    1 -> a = :ok
    _ -> a = :notOk
  end
  ^a = :ok  # Explicitly make this a reference
  a
end

Elixir does force you to make it explicit that a match is taking place, so I think it is a bit more clear what is going on than in Erlang. However, the fact remains that variables inside the case statment are leaking out into the surrounding scope in both languages.

Similarly, if you bind the variable A before the case statement, then A cannot be bound inside the case statement, because the outer variable is still in scope:

% This Erlang function crashes if you pass in anything other than 1.
also_only_one(X) ->
  A = ok            % This is a bind
  case X of
    1 -> A = ok;    % This matches against the existing value of A
    _ -> A = notOk  % This also matches against the existing value
  end,
  A.

The equivalent in Elixir:

# The equivalent Elixir function.
def also_only_one(x) do
  a = :ok
  case(x) do
    1 -> ^a = :ok     # Reference the outer "a"
    _ -> ^a = :notOk
  end
  a
end

One more quirk to explore. Recall that if you bind a variable before defining an inner function, that variable is visible within the function, and Erlang cannot bind it again.

% This Erlang function throws a match error
overlapping_scopes() ->
  A = 2,
  F = fun() -> A = 1 end,  % References outer "A"
  F(),                     % Match error here
  A.

However, if the “inner” variable is bound as an argument, Erlang treats it as a new variable, and shadows the outer one.

% This Erlang function returns 2.
shadowing_argument() ->
  A = 2,
  F = fun(A) -> A end,  % Argument "A" is local, shadows outer variable
  F(1),                 % No match error!
  A.

Again, Elixir, through its pin operator, gives you the choice of either behavior: referencing the outer variable or creating a new shadowing variable in the inner scope. Here is the Elixir equivalent of the above:

# This Elixir function returns 2.
def shadowing_argument() do
  a = 2
  f = fn (a) -> a end  # No caret: this "a" is a local variable
  f.(1)
  a        # returns 2, the value of the outer variable
end

If you include the caret, you reference the outer variable:

# This Elixir function throws a match error.
def match_outer_variable() do
  a = 2
  f = fn (^a) -> a end  # Reference outer variable in pattern match
  f.(1)                 # Crashes here: no match for f(1)
  a
end

If you’re thoroughly confused at this point, I don’t blame you. Because of behavior like this, it is generally good practice, in both languages, to avoid reusing variable names, even if they live in different scopes and you expect one to shadow the other. The rules are subtle and can be surprising, and for the sake of clarity of your code, it’s best to avoid corner cases like these that we’ve been exploring.

Where to go from here

Chasing down all the special cases around variable scopes was an enormous headache for Erl2ex. It’s important to be aware that there are some surprises, in both languages.

If you haven’t read Jose’s article, I recommend it. It’s a useful read.

The good news is that Elixir is trying to simplify its scoping rules by deprecating the confusing constructs, especially exporting variables from control structures such as “if” and “case” statements. Elixir 1.2 already emits a warning at compile time for some cases, and Elixir 1.3 will deprecate additional cases. Daniel Perez recently blogged about some of the upcoming changes.

Next time, we’ll look at the exception handling in the two languages and how they relate. Feel free to browse the index of articles in this series, and stay tuned for more on Erlang and Elixir’s family ties.