blog/comprehensions-are-common-python

[utterances-bot]

Comprehensions are Common Python | The When of Python Blog

List comprehensions, and other types of comprehensions such as set comprehensions, are such an important part of Python we should treat them as Common Python - that is, as a feature we should all know so we can basically read each other's work. Even though they are Common Python we should only use this compressed syntax when it is relatively easy to understand. The rule of thumb is that we should change to another approach if our comprehension becomes an _in_comprehension.

https://when-of-python.github.io/blog/comprehensions-are-common-python.html

[nathanjmcdougall]

On "Incomprehensions" and the general rule of thumb:

if a comprehension stops being easy to read, use an alternative approach instead e.g. a for loop or map.

I would say:

1. Map should nearly never be used.
2. Functionalizing the expressions in the for loops should be the first recourse
3. Splitting a single comprehension into multiple comprehensions is okay if you want to store the intermediate steps of a calculation rather than having it all done in one function.
4. For loops could sometimes be used, but usually they're not the best option.

Let's consider a toy example:

outputs  = [func4(func2(func1(value)).method3()).attribute5 for value in values if func7(value.attribute6)]

Most of the time, I think this would be the most readable (Functional form):

def get_output(value):
    func1_value = func1(value)
    func2_value = func2_value (value)
    method3_value = method3_value.method3()
    func4_value = func4(method3_value)
    output = func4_value.attribute5
    return output

def is_okay(value):
    attribute6_value = value.attribute6
    return func7(attribute6_value)

outputs = [get_output(value) for value in values if is_okay(value)]

In some cases, it can be better to do something like this (Comprehension chaining), so that you store the intermediate steps of a calculation:

attribute6_values = [value.attribute6 for value in values]
is_okays = [func7(value) for value in attribute6_values]

func1_values = [func1(value) for value in values]
func2_values = [func2(value) for value in func1_values]
method3_values = [value.method3() for value in func2_values]
func4_values = [func4(value) for value in method3_values]
attribute5_values = [value.attribute5 for value in method3_values]

outputs = [value for value, is_okay in zip(attribute5_values, is_okays) if is_okay]

And, of course, you'd often use a mix of both of these (some comprehension chaining together with some functionalization). The examples above are a little exaggerated; you wouldn't necessarily need/want to create a separate variable for every single function call/attribute access in get_output, nor need to store every single list in the chained example.

Lastly, let's look at the for loop:

outputs = []
for value in values:
    attribute6_values = value.attribute6
    is_okay = func7(attribute6_value)

    if is_okay:
        func1_value = func1(value)
        func2_value = func2_value (value)
        method3_value = method3_value.method3()
        func4_value = func4(method3_value)
        output = func4_value.attribute5

        outputs.append(output)

The reasons I don't like this so much are:

1. It's double indented, which is fairly ugly for something so simple.
2. It has "noise": idiomatic lines for when initializing outputs and appending to it.
3. It's harder to convince yourself that this process is parallelizable (each iteration of the for loop is independent of the previous & subsequent iterations), and likewise harder to take advantage of that, e.g. with concurrent.futures
4. It's harder to test: Generally if we're got a complex expression like func4(func2(func1(value)).method3()).attribute5 it's worth testing. That goes for the chained-comprehension approach too, mind you.

[grantps]

Just read What’s in a List Comprehension? An overview of Python’s alternative to loops and why you should use them. A good article in many ways but I was intrigued by the discussion of nested comprehensions.

With a list comprehension, you can shrink the entire nested for loop down to just one line:

>>> nested_lst = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> [print(number) for inner_lst in nested_lst for number in inner_lst]
1
2
3
4
5
6
7
8
9
[None, None, None, None, None, None, None, None, None]

I know, it looks a bit confusing. It confused me at first too. But if you read the line carefully, it follows the same logic as the nested for loop

I have always found nested comprehensions confusing too and don't consider them worth the bug risk. I would prefer a for loop. Our code is often unavoidably hard to read - which is why we should make it as readable and easy to reason about as possible when we can.

[grantps]

Here is a resource from an obviously very enthusiastic comprehension expert: https://mathspp.gumroad.com/l/comprehending-comprehensions

[grantps]

There is even a cheatsheet - https://mathspp.gumroad.com/l/cheatsheet_list_comps_101 - although my inclination is to avoid anything beyond the reasonably simple and easy-to-read.

[grantps]

And here is a discussion of how to use the ternary operator in the first part of a comprehension (which is often surprisingly useful) e.g.

long_names = [name if len(name) > 5 else '?' for name in names]

If You Find if..else in List Comprehensions Confusing, Read This, Else…