Member Avatar for memory

I am new to python and numpy.

When I wrote: zeros((20000,20000),int), it returned "MemoryError".

Does this mean that this array exceeds python's memory limit?

Is there a way to define such big an array in python?

Thanks.

  • 2 Contributors
  • 2 Replies
  • 121 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by memory

Dani AI

Generated

Short answer: the error means the OS refused the allocation, not a Python-imposed quota. A 20,000×20,000 array is 400,000,000 elements. If each element is 8 bytes (typical on 64-bit builds) that’s about 3.2 GB; if 4 bytes (common for 32-bit ints) it’s about 1.6 GB. Allocator overhead, alignment, and the process address-space limit on 32-bit systems make such allocations fail even when physical RAM is larger. was right to point out it’s the underlying C allocator and the process limits that matter, and that breaking the problem into smaller pieces is often the practical fix.

Practical options to handle very large arrays:

  • Use 64-bit Python + 64-bit numpy on a machine with enough RAM.
  • Reduce element size (use np.int32, np.float32, bool, etc.) when precision allows.
  • Keep the array on disk with memory-mapping: numpy.memmap (disk-backed, avoids one big malloc but needs disk space).
  • Use sparse representations if the array is mostly zeros: scipy.sparse.
  • Use chunked, out-of-core libraries like Dask to operate on portions that fit in memory.

Example patterns:

import numpy as np
# disk-backed array (creates a file on disk, needs enough free space)
fp = np.memmap('big.dat', dtype=np.int32, mode='w+', shape=(20000, 20000))
from scipy import sparse
# build a sparse matrix when only a few entries are nonzero
rows = [0, 10, 1000]
cols = [1, 20, 3000]
data = [1, 2, 3]
A = sparse.csr_matrix((data, (rows, cols)), shape=(20000, 20000))

Troubleshooting tips: check available RAM and swap (free -m, top), confirm 64-bit vs 32-bit Python, and test with smaller shapes before scaling up. Memory-mapped and out-of-core approaches trade speed for feasibility; choose based on access pattern and sparsity. For documentation see numpy.memmap and Dask array docs for out-of-core workflows.

Recommended Answers

All 2 Replies

Member Avatar for Gribouillis

On a 32 bits system, it's more than 1.5 Gb of ram. It could be more than what your system can allocate for your process. I don't think it has much to do with python: the array is allocated by a malloc-like function in numpy. You should probably break your problem into smaller subproblems.

Member Avatar for memory

I see.

Thank you very much!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.