“for” comprehensions

The for comprehension is a powerful language feature. It is like a “Swiss Army Knife”. It includes many features and options. At its simplest, it may remind you of Enum.map/2.

for num <- [1, 2, 3, 4, 5] do
  num * 2
end
#=> [2, 4, 6, 8, 10]

The above for comprehension returns a list where each element is the return value of the do/end block. However, instead of passing a function like in Enum.map/2, you provide the function body code inside the do/end block.

Basic for anatomy

The basic layout of the for comprehension is this:

for pattern <- enumerable do
  expression
end

The part pattern <- enumerable is called a “generator”.

The fact that this has its own special name tells you that it might do something different. What happens when the pattern doesn’t match?

for num when is_number(num) <- ["abc", 123, "def", 456] do
  num
end
# [123, 456]

Interestingly, when the pattern doesn’t match an entry in the list it gets filtered out!

Filter clause

The for comprehension also includes support for a separate and dedicated “filter” clause. What’s cool about the filter clause is that it can perform operations that are not allowed in a guard clause. Here’s where it goes in the comprehension.

Similar to the pattern match, if the filter clause doesn’t match, the entry is filtered out. Let’s look at a working example to understand it better.

for value when is_binary(value) <- ["abc", 123, "def", 456], String.contains?(value, "e") do
  value
end
#=> ["def"]

You may be asking, “Why can’t I just conditionally add the result with the “expression” portion?” If you try that here, you see that the entry is included in the resulting list, but no explicit value was returned so it includes a nil value. This isn’t the behavior you intended.

for value when is_binary(value) <- ["abc", 123, "def", 456] do
  if String.contains?(value, "e") do
    value
  end
end
#=> [nil, "def"]

The filter clause is applied before the do/end block ever executes. Depending on the pattern match and any filter clauses, the do/end block may never execute.

Comprehensions ignore or discard all elements where the filter expression returns false or nil. All other values pass and are included.

Here’s the order that things happen:

1. Element of enumerable is checked against pattern
2. If pattern matches, filter constraint is applied
3. If filter passes, do/end block is executed

Exercise #1 – Preferred user points

You are writing a game for a client and during game play, the client wants you to give preferential treatment to users that have characteristics important to them. Yes, it’s completely unfair… but it’s their game. The client really likes the letter combination “uc” in a name. They think of the words “…you see…” and just find it humorous. Yeah, weird client, I know!

So during game play, your function Comprehension.award_unfair_points/2 will be called with a list of users. You should first filter out any users that are not active. An additional filter should be applied if their name contains the letters “uc”. So active users with the letter combination should have their points incremented by the given number of points and this privileged set of users should be returned in a list.

The tests for this exercise are here:

mix test test/comprehension_test.exs --only describe:"award_unfair_points/2"

  def award_unfair_points(users, points) do
    for %User{active: true} = user <- users, String.contains?(user.name, "uc") do
      %User{user | points: user.points + points}
    end
  end

Multiple generators

The for comprehension supports using multiple generators. Remember the “generator” part is: n <- [1, 2, 3]. This is what it looks like with two generators. This example create a small multiplication table:

for x <- 1..3, 
    y <- 1..3 do
  "#{x} * #{y} = #{x*y}"
end
#=> [
#=>   "1 * 1 = 1",
#=>   "1 * 2 = 2",
#=>   "1 * 3 = 3",
#=>   "2 * 1 = 2",
#=>   "2 * 2 = 4",
#=>   "2 * 3 = 6",
#=>   "3 * 1 = 3",
#=>   "3 * 2 = 6",
#=>   "3 * 3 = 9"
#=> ]

And yes, you can do more than 2 generators. Feel free to play with that.

Multiple generators create a Cartesian Product or a cross-join. It goes through each x value when y is 1. Then it goes through each x value again when y is now 2 and so on.

Exercise #2 – Build a chess board

In this example you need to build a Chessboard. A Chessboard is an 8×8 grid but each square has a unique name making them clearly addressable.

Using a for comprehension, build a list of the squares. The structure of each square will be a map that looks like this: %{col: "a", row: 1, name: "a1"}. The function to create is Comprehension.build_chessboard/0. It should return a list of 64 square entries. There are unit tests to check that the board is built correctly.

The tests for this exercise are here:

mix test test/comprehension_test.exs --only describe:"build_chessboard/0"

  def build_chessboard() do
    for row <- 1..8,
        col <- ["a", "b", "c", "d", "e", "f", "g", "h"] do
      %{col: col, row: row, name: "#{col}#{row}"}
    end
  end

Special options

The for comprehension has a few additional options that add some extra behavior. Let’s cover those now.

Into and Uniq

All of the examples up to now show the for comprehension returning a list as the result. Using the :into option, it can return any data structure that supports the Collectable protocol. The Enum.into/2 function does the work and you can read more about how it works on the documentation there. Just be aware that the for comprehension will do that for you when you use this option. This is what it looks like:

for n <- [a: 1, b: 2, c: 3], into: %{}, do: n
#=> %{a: 1, b: 2, c: 3}

This converts a keyword list into a map. The into: %{} tells it what to convert it into and provides an initial value.

The uniq: true option will guarantee the results are only added to the collection if they were not returned before.

You can read more about the :into and :uniq options in the documentation.

Reduce

This option works like Enum.reduce/3. When used, the comprehension returns the accumulator, not a list. To use it, we pass in the initial value for the accumulator. Doing a reduce changes the syntax of the comprehension. Because we need to keep receiving the accumulator as it iterates through the enumerable, there is a special form that uses -> to do it. Seeing it in action will help:

for n <- 1..5, reduce: 0 do
  acc ->
    acc + n
end
#=> 15

The option reduce: 0 says we want to reduce and provides the initial value. In this case, we start with a 0. Then the acc -> is how we receive the accumulator being passed in. Our return value will become the new accumulator value.

You can read more about the :reduce option in the documentation.

Exercise #3 – Total team points

Our game has users who have received points. Users are part of a team and we need to be able to total up all the points awarded to the team. Create the function Comprehension.team_points/1 which receives a list of users. The team may have lost users along the way and they became “inactive”. Filter out inactive users. Return the sum of all the active user’s points. Use the for comprehension and the :reduce option to do this.

The tests for this exercise are here:

mix test test/comprehension_test.exs --only describe:"team_points/1"

  def team_points(users) do
    for %User{active: true} = user <- users, reduce: 0 do
      acc ->
        acc + user.points
    end
  end

for comprehension recap

Using a for comprehension can be more elegant than the equivalent version using Enum functions. There are still functions and situations where other solutions work better so this isn’t the ultimate solution.

A few key points to remember:

• Basic usage is like Enum.map/2
• Generators with pattern matching filter out non-matches
• Filter clause can be added for additional filtering that doesn’t work in a guard clause
• Multiple generators can be used to create a Cartesian Product
• Options like :into and :reduce add extra data transformation behavior

The for comprehension is powerful and really is like a Swiss Army Knife. Like a Swiss Army Knife that includes a screwdriver feature, it can operate as a screwdriver but a dedicated or set of varied screwdrivers will work better at times. Still, it includes many features that work well together and it is a valuable addition to your toolbox.

The for comprehension includes the features and abilities of a number of Enum functions. Let’s review that list briefly and how they are used:

• Enum.map/2 – By using a basic generator without any matching
• Enum.filter/2 – Through both the pattern and the “filter” clause
• Enum.into/2 – Through the in into: collectable option
• Enum.uniq/1 – Through the uniq: true option
• Enum.reduce/3 – Through the reduce: initial_acc option and the special -> clause