break a list comprehension

[ihatehippies]

In order for my code to be efficient, loops need to be terminated prematurely to prevent unnecessary iterations. I know that list comprehension is fast that the 'for' statement, but I'm having trouble finding a suitable way to break them. Here's what I've come up with

try:
	[x if x < 5 else 1/0 for x in range(5)]
except:
	pass

[0, 1, 2, 3, 4]

I'm just using an exception to break it.

[TrustyTony]

I often find myself writing small list comrehensions in sequence and after puting the normal for by putting them in opposite order in the end of list comprehension.
It is maybe possible to use generator expressions to imitate for loop break cleaner than zero division error. Maybe better to use normal for those parts. Got any example code to compare your exception code in comparision to normal loop.

Here is one stupid experiment of doing return function for linear search list comprehension:

def ret(a):
    raise StopIteration, str(a)

target ='c'
x=['a','b','c','d','e']

try:
    res = [x[i] for i in range(len(x)) if x[i]==target and ret((x[i],i))]
    print 'Not found'

except StopIteration as si:
	print str(si)
print res

""" Output:
('c', 2)

Traceback (most recent call last):
  File "D:/Python Projects/linear_comprehension.py", line 10, in <module>
    print res
NameError: name 'res' is not defined
"""

Notice that the previously generated values in list are lost as the assignment to res does not succeed. So it results in NameError for print res.

So cleaned up:

def ret(a):
    raise StopIteration, str(a)

target ='c'
x=['a','b','c','d','e']

try:
    [x[i] for i in range(len(x)) if x[i]==target and ret((x[i],i))]
    print 'Not found'

except StopIteration as si:
	print str(si)

""" Output:
('c', 2)
"""

[snippsat]

In order for my code to be efficient, loops need to be terminated prematurely to prevent unnecessary iterations

Yes list comprehension are faster than ordinary loop,but dont do to much Premature optimization.

One of the frequently repeated mantras I hear from Python veterans is, "Premature optimization is the root of all evil"

So first write your code in a readable and clear way,before doing to much optimization.
If exception handling with list comprehension is to difficult then dont do it.

Learn to test your code,dont relay on that some code is faster than other code.
More on this later.

To your code.

try:
    print [x if x < 3 else 1/0 for x in range(5)]
except: #Dont ever do this
    pass
#Now this code should break and user dont know anything, no message is coming

#Test in IDLE
>>> print [x if x < 3 else 1/0 for x in range(5)]
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero
>>> 

try:
    print [x if x < 3 else 1/0 for x in range(5)]
except ZeroDivisionError:
    #Always give message to user
    print 'And ZeroDivisionError has happend'

'''-->ZeroDivisionError has happend'''

So write it without list comprehension.

try:
    l= []
    for x in range(5):
        if x < 3:
            l.append(x)
        else:
            1/0
except ZeroDivisionError:
    #Always give message to user
    print 'ZeroDivisionError has happend'
print l
'''-->ZeroDivisionError has happend
[0, 1, 2]
'''

So we test the code when it dont break,to see the time differntence.
For this we use timeit module,now it run the code in 1000000 loops.

import timeit

l = '''
l = []
for x in range(5):
    if x < 5:
        l.append(x)
    else:
        1/0
'''
#------------#
ls = '''
ls = [x if x < 5 else 1/0 for x in range(5)]
'''

#print timeit.Timer(stmt = l).timeit(number=1000000)
'''-->2.28020209272'''

print timeit.Timer(stmt = ls).timeit(number=1000000)
'''-->1.46803546261'''

As ecpectet list comprehension is a litte faster than the ordinary loop,but if code get to complex don't try to wreck your brain to get code in a list comprehension because it`s a little faster.

[vegaseat]

Replace your list comprehension with a generator expression ( simply replace [] with () ), then you can use the ifilter() method of module itertools to limit the range of values ...

# create an iterable of square numbers and return only numbers that
# are above 20 and below 200 using itertools.ifilter(predicate, iterable)

from itertools import ifilter

# use a generator expression
square_gen = ( x**2 for x in range(100) )

result = list(ifilter(lambda x: 20 < x < 200, square_gen))

print(result)

"""result >>>
[25, 36, 49, 64, 81, 100, 121, 144, 169, 196]
"""

[TrustyTony]

start quote:

ls = '''
ls = [x if x < 5 else 1/0 for x in range(5)]
'''

end quote.

This expression does not raise the zero division error, so it is not timing the same thing.

>>> ls = [x if x < 5 else 1/0 for x in range(20)]


Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    ls = [x if x < 5 else 1/0 for x in range(20)]
ZeroDivisionError: integer division or modulo by zero
>>> ls

Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    ls
NameError: name 'ls' is not defined
>>> ls = [x if x < 5 else 1/0 for x in range(5)]
>>> ls
[0, 1, 2, 3, 4]
>>>

[TrustyTony]

Replace your list comprehension with a generator expression ( simply replace [] with () ), then you can use the ifilter() method of module itertools to limit the range of values ...

# create an iterable of square numbers and return only numbers that
# are above 20 and below 200 using itertools.ifilter(predicate, iterable)

from itertools import ifilter

# use a generator expression
square_gen = ( x**2 for x in range(100) )

result = list(ifilter(lambda x: 20 < x < 200, square_gen))

print(result)

"""result >>>
[25, 36, 49, 64, 81, 100, 121, 144, 169, 196]
"""

Unfortunately in Python3, the filter has replace ifilter, so to work in Python 2.6 and 3.1:

# create an iterable of square numbers and return only numbers that
# are above 20 and below 200 using itertools.ifilter(predicate, iterable)

from __future__ import print_function, generators

def check(x):
    print(x, end=', ')
    return 20 < x < 200
     
# use a generator expression
square_gen = ( x**2 for x in range(100) )

result = list(filter(check, square_gen))

print('\nResult:', result)

"""result
0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521, 1600, 1681, 1764, 1849, 1936, 2025, 2116, 2209, 2304, 2401, 2500, 2601, 2704, 2809, 2916, 3025, 3136, 3249, 3364, 3481, 3600, 3721, 3844, 3969, 4096, 4225, 4356, 4489, 4624, 4761, 4900, 5041, 5184, 5329, 5476, 5625, 5776, 5929, 6084, 6241, 6400, 6561, 6724, 6889, 7056, 7225, 7396, 7569, 7744, 7921, 8100, 8281, 8464, 8649, 8836, 9025, 9216, 9409, 9604, 9801, 
Result: [25, 36, 49, 64, 81, 100, 121, 144, 169, 196]
"""

Also by defined debug function we can see that this code does not demonstrate the stopping of list comprehension or generator in first found answer, but goes through all square list.

[snippsat]

This expression does not raise the zero division error, so it is not timing the same thing.

Yes i know that,the timing was to shown speed off list comprehension vs ordinary loop.

[TrustyTony]

I googled this discussion for adding exception for list comprehension.
http://mail.python.org/pipermail/python-dev/2006-April/064407.html

"tomer filiba" <tomerfiliba at gmail.com> wrote:
> "[" <expr> for <expr> in <expr> [if <cond>] [except
> <exception-class-or-tuple>: <action>] "]"

Note that of the continue cases you offer, all of them are merely simple
if condition (though the file example could use a better test than
os.path.isfile).

    [x for x in a if x.startswith("y") except AttributeError: continue]
    [x for x in a if hasattr(x, 'startswith') and x.startswith("y")]

    [1.0 / x for x in y except ZeroDivisionError: continue]
    [1.0 / x for x in y if x != 0]

    [open(filename) for filename in filelist except IOError: continue]
    [open(filename) for filename in filelist if os.path.isfile(filename)]

The break case can be implemented with particular kind of instance
object, though doesn't have the short-circuiting behavior...

class StopWhenFalse:
    def __init__(self):
        self.t = 1
    def __call__(self, t):
        if not t:
            self.t = 0
            return 0
        return self.t

z = StopWhenFalse()

Assuming you create a new instance z of StopWhenFalse before doing the
list comprehensions...

    [x for x in a if z(hasattr(x, 'startswith') and x.startswith("y"))]
    [1.0 / x for x in y if z(x != 0)]
    [open(filename) for filename in filelist if z(os.path.isfile(filename))]


If you couldn't guess; -1, you can get equivalent behavior without
complicating the generator expression/list comprension syntax.

 - Josiah

It offers alternative for break but unfortunately says though doesn't have the short-circuiting behavior...

[snippsat]

Exception handling in list comprehension is difficult/impossible.
Here is Alex Martelli answer to the question.

There is no built-in expression in Python that lets you ignore an exception (or return alternate values &c in case of exceptions), so it's impossible, literally speaking, to "handle exceptions in a list comprehension"

http://stackoverflow.com/questions/1528237/how-can-i-handle-exceptions-in-a-list-comprehension-in-python

[TrustyTony]

Found still this: Is there anything like a list comprehension terminator

Call of example here was wrong with gen instead of while_. Here use case which proves it does not hit ZeroDivision error. Also my under function version instead of lambda.

If somebody wants to test the performance of these, it would be nice to know

from itertools import takewhile

def while_(gen, cond):
    for item in gen:
        if not cond(item):
            return
##        print item # debug
        yield item

def under(x):
    return lambda y :(y<x)

# while_
print([3./(3-x) for x in while_(range(5), lambda x: x<=2)])

# takewhile from itertools
print([3./(3-x) for x in takewhile(lambda x:x<=2, range(5))])

# variation with def (performance penalty?) under using python style < not <=
print([3./(3-x) for x in takewhile(under(3), range(5))])

P.S. would it make sense to have unary < returning same as under function above as dropping one side of comparison seems to be biggest use of lambda in Python? <3 would give: lambda x: x<3

[snippsat]

That`s not exceptions handling in list comprehension,i think Alex Martelli answer is the correct answer to exception handling in list comprehension.

so it's impossible, literally speaking, to "handle exceptions in a list comprehension"

[TrustyTony]

That`s not exceptions handling in list comprehension,i think Alex Martelli answer is the correct answer to exception handling in list comprehension.

But the point of this thread not exception handling, but short circuit break from middle of list comprehension when solution is found. Exception is just mechanism that we must use if we do not use generator.

[snippsat]

But the point of this thread not exception handling, but short circuit break from middle of list comprehension when solution is found. Exception is just mechanism that we must use if we do not use generator.

Yes i know,and for me the correct anwer is to say that exception cannot be used inside a list comprehension.