I'm having a weird problem code is running differently in a module than in line

[Peacer]

a = [[[0] for col in range(4)] for row in range(4)]

I do that code and it runs fine, and then I manipulate that list with code I have later and it runs fine at a reasonable speed.

But if I put this code in a module.

Module ListToFillUp.py

def make(x,y,content):
    list = [[content for col in range(x)] for row in range(y)]
    return list

Module that I'm working in

b = ListToFillUp.make(4, 4, [0])

And then I import the module into the other module and use this line to make the list, it is fine and it is identical to the other list as checked by ==, but it encounters code later on that it just hangs on until I have a memory error.

There could be a problem with the code later on that I didn't post, but the bigger problem is how there seems to be an actual difference between the lists created. But when I compare the two lists with == it comes up as true. And I see the lists printed out and they are the same. I even do "copy.deepcopy(b)" and the problem still exists. So is there anything else I can check to see if they are different or if there is another explanation why I am getting a memory error?

[Peacer]

UPDATE: I was using Python 3.1 before now I am using Python 2.6 and the problem no longer happens. Is there a way I can make it work in Python 3.1 or there is a reason why there's a problem?

UPDATE TO UPDATE: I tested it in Python 2.6 and 2.7, and they both work accordingly. But in Python 3.1 and 3.2 it fails for lists created by the module.

[woooee]

You will have to put trace points in your program to see where it is hanging. I doubt it is here. Also, do not use "list" as a variable name as it is already used to name a function that returns a list, and that could be the problem because references to "list" are destroyed upon exiting the function, but it is impossible to tell from the code posted.

[Peacer]

You will have to put trace points in your program to see where it is hanging. I doubt it is here. Also, do not use "list" as a variable name as it is already used to name a function that returns a list, and that could be the problem because references to "list" are destroyed upon exiting the function, but it is impossible to tell from the code posted.

That isn't the problem I have it as this in my actual program.

def make(x,y,content):
    return [[content for col in range(x)] for row in range(y)]

[Gribouillis]

To me there is no memory error, but there is a more subtle error. The lists [0] in b are all the same

>>> a = [[[0] for col in range(4)] for row in range(4)]
>>> a
[[[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]
>>> def make(x,y,content):
...     list = [[content for col in range(x)] for row in range(y)]
...     return list
... 
>>> b = make(4, 4, [0])
>>> b
[[[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]
>>> b[0][0][0] = 9
>>> b
[[[9], [9], [9], [9]], [[9], [9], [9], [9]], [[9], [9], [9], [9]], [[9], [9], [9], [9]]]
>>> a[0][0][0] = 9
>>> a
[[[9], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]

see the difference ?

[Peacer]

To me there is no memory error, but there is a more subtle error. The lists [0] in b are all the same

>>> a = [[[0] for col in range(4)] for row in range(4)]
>>> a
[[[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]
>>> def make(x,y,content):
...     list = [[content for col in range(x)] for row in range(y)]
...     return list
... 
>>> b = make(4, 4, [0])
>>> b
[[[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]
>>> b[0][0][0] = 9
>>> b
[[[9], [9], [9], [9]], [[9], [9], [9], [9]], [[9], [9], [9], [9]], [[9], [9], [9], [9]]]
>>> a[0][0][0] = 9
>>> a
[[[9], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]

see the difference ?

That's really odd because when I run code

spreadSamplea = ListToFillUp.make(4, 4, [0])
spreadSamplea[0][0][0] = 100
print(spreadSamplea)

And this is what I get.
[[[100], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]

[Gribouillis]

That's really odd because when I run code

spreadSamplea = ListToFillUp.make(4, 4, [0])
spreadSamplea[0][0][0] = 100
print(spreadSamplea)

And this is what I get.
[[[100], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]], [[0], [0], [0], [0]]]

It's impossible. Can you post ListToFillUp.py exactly as it is ?

[Peacer]

You're right. It is impossible. And I do believe it is part of the problem, but I'm not exactly sure why it is reacting differently for the versions of Python.

[Gribouillis]

You're right. It is impossible. And I do believe it is part of the problem, but I'm not exactly sure why it is reacting differently for the versions of Python.

The only way for us to tell is to see the whole code. The solution lies in the code that we don't see.

[Peacer]

You are right, but before I put up the code I'm curious what is the difference between a and b in your post? Why does one react differently?

[Gribouillis]

You are right, but before I put up the code I'm curios what is the difference between a and b in your post? Why does one react differently?

Because in the expression [[0] for col in range(4)] , the list constructor [0] is executed for each value of col, building each time a new list. On the other hand, when you call make(4, 4, [0]), the content argument is only built once.

[Peacer]

Because in the expression [[0] for col in range(4)] , the list constructor [0] is executed for each value of col, building each time a new list. On the other hand, when you call make(4, 4, [0]), the content argument is only built once.

Wow, that's awesome. I have been using that module this whole time and it never created any problems because I would change a[0][0] instead of what's inside of it. It didn't cause any problems until now, but if you didn't point this out I would've surely had a problem later on. Thanks a lot.

[Peacer]

So what would you suggest I do to recreate the make function so that I can send whatever I want in there and expect the content to be executed multiple times. I used deepcopy but it is over 20 times slower than how it was before.

[Gribouillis]

So what would you suggest I do to recreate the make function so that I can send whatever I want in there and expect the content to be executed multiple times. I used deepcopy but it is over 20 times slower than how it was before.

If the content is always a list, you could use

def make(x,y,content):
    result = [[list(content) for col in range(x)] for row in range(y)]
    return result

Or if the content is always a list containing a single item, you could write

def make(x,y,item):
    result = [[[item] for col in range(x)] for row in range(y)]
    return result

You could also pass a function which creates the content

def make(x,y,contentmaker):
    result = [[contentmaker() for col in range(x)] for row in range(y)]
    return result

def f():
    return [0]

b = make(4, 4, f)

It all depends on the values that you expect to give to the content parameters.

[Peacer]

The thing is, it might be a list sometimes, while other times it could be an integer. And I want good speeds for both.

[Gribouillis]

The thing is, it might be a list sometimes, while other times it could be an integer. And I want good speeds for both.

You can use

>>> from functools import partial
>>> def make(x, y, content):
...  f = partial(list, content) if isinstance(content, list) else lambda: content
...  return [[f() for col in range(y)] for row in range(x)]
... 
>>> make(4, 4, 10)
[[10, 10, 10, 10], [10, 10, 10, 10], [10, 10, 10, 10], [10, 10, 10, 10]]
>>> b = make(4, 4, [10])
>>> b[0][0][0] = 99
>>> b
[[[99], [10], [10], [10]], [[10], [10], [10], [10]], [[10], [10], [10], [10]], [[10], [10], [10], [10]]]

In python 2, replace range by xrange for speed. Also it seems more logical to me that x is the number of rows and y the number of columns.

Another solution is

>>> from copy import copy
>>> def make(x, y, content):
...  return [[copy(content) for col in range(y)] for row in range(x)]
... 
>>> make(3, 4, 10)
[[10, 10, 10, 10], [10, 10, 10, 10], [10, 10, 10, 10]]
>>> b = make(3, 4, [10])
>>> b[0][0][0] = 99
>>> b
[[[99], [10], [10], [10]], [[10], [10], [10], [10]], [[10], [10], [10], [10]]]

[Peacer]

In python 2, replace range by xrange for speed. Also it seems more logical to me that x is the number of rows and y the number of columns.

It does, but when you want to access it b[y][x] is a lot more anti-intuitive. The code you gave me has a good speed for lists, with it being a little faster than my code I have made, but it is significantly slower for anything else.

This is my code I just made.

def makeNew(x,y,content):
    contentType = type(content)
    if contentType == type([]):
        return [[list(content) for col in range(x)] for row in range(y)]
    elif contentType == type({}):
        return [[dict(content) for col in range(x)] for row in range(y)]
    else:
        return [[content for col in range(x)] for row in range(y)]

I don't like type checking it seems very unpythonic. So I have it type check for lists and dictionaries, I didn't include tuples because they are immutable. But should I anyways? Is there ever a time where it could be a problem. Other than that does it look like it won't have problems? I am also trying to get faster speeds than this.

Also, copy is a lot slower.

[Gribouillis]

It does, but when you want to access it b[y][x] is a lot more anti-intuitive. The code you gave me has a good speed for lists, with it being a little faster than my code I have made, but it is significantly slower for anything else.

This is my code I just made.

def makeNew(x,y,content):
    contentType = type(content)
    if contentType == type([]):
        return [[list(content) for col in range(x)] for row in range(y)]
    elif contentType == type({}):
        return [[dict(content) for col in range(x)] for row in range(y)]
    else:
        return [[content for col in range(x)] for row in range(y)]

I don't like type checking it seems very unpythonic. So I have it type check for lists and dictionaries, I didn't include tuples because they are immutable. But should I anyways? Is there ever a time where it could be a problem. Other than that does it look like it won't have problems? I am also trying to get faster speeds than this.

Also, copy is a lot slower.

I compared different versions for an integer argument

from __future__ import print_function
import sys
if sys.version_info >= (3,):
    xrange = range

def makeA(x, y, item):
    return [[item for col in xrange(x)] for row in xrange(y)]

def makeB(x, y, item):
    return [[item] * x for row in xrange(y)]

def makeC(x, y, item):
    return [[item] * x] * y

from timeit import timeit

print("python", sys.version_info[:3])
print("results in microseconds")
for f in "makeA makeB makeC".split():
    print(f, timeit("make(4, 5, 10)", "from __main__ import %s as make" % f))
    
""" my results -->
[t300009] speed.py
python (2, 6, 5)
results in microseconds
makeA 4.00190997124
makeB 2.26678800583
makeC 0.693363904953
[t300009] python3 speed.py
python (3, 1, 1)
results in microseconds
makeA 5.51726484299
makeB 2.44721198082
makeC 0.627732992172
"""

makeC is the fastest.

[Peacer]

Oh wow, that greatly increases the speed of my programs thanks so much.

[Gribouillis]

Oh wow, that greatly increases the speed of my programs thanks so much.

It remains to find the fastest implementation when the content argument is a list. We'll see that tomorrow, because it's very late here. Good night.

[Peacer]

Never mind, I made an integer list using makeC and if you change any of the integers it will change the other integers inside that list at the same x position. makeB works for integers, but not for lists. makeA still is the best for lists it seems, so I have an if statement to check whether or not it is a list.

def make2(x,y,content):
    contentType = type(content)
    if contentType == type([]):
        return [[list(content) for col in range(x)] for row in range(y)]
    elif contentType == type({}):
        return [[dict(content) for col in range(x)] for row in range(y)]
    else:
        return [[content] * x for row in range(y)]

[Gribouillis]

Never mind, I made an integer list using makeC and if you change any of the integers it will change the other integers inside that list at the same x position. makeB works for integers, but not for lists. makeA still is the best for lists it seems, so I have an if statement to check whether or not it is a list.

def make2(x,y,content):
    contentType = type(content)
    if contentType == type([]):
        return [[list(content) for col in range(x)] for row in range(y)]
    elif contentType == type({}):
        return [[dict(content) for col in range(x)] for row in range(y)]
    else:
        return [[content] * x for row in range(y)]

Yes, you're right, makeC doesn't work. For lists, the following is faster than makelistA by a few percents (10% in python 3 for item = [10])

def makelistB(x, y, item):
    p = x*y
    L = [list(item) for i in xrange(p)]
    return [ L[i:i+x] for i in xrange(0,p,x) ]