list of list of files

Question

alabandit 0 Newbie Poster

16 Years Ago

I need to monitor a number of folders over a period of time. so i pass the files in each folder to a list of list.

import os

path_to_watch = []
path_to_watch.append("/path1/")
path_to_watch.append("/path2/")
path_to_watch.append("/path3/")

before = [],[]
for item, folder in enumerate(path_to_watch):
    print "i am here", item, " : ", folder
    before[item].append(dict ([(f, None) for f in os.listdir (folder)]))

were path1 to 3 are actual paths on my system

When it reaches the final folder it gives me the error

tuple index out of range

any ideas?

python

2 Contributors
7 Replies
263 Views
8 Hours Discussion Span
Latest Post 16 Years Ago Latest Post by TrustyTony

Dani AI

Generated 9 Months Ago

The error you saw (IndexError: tuple index out of range) happens because your per-folder container was smaller than the number of folders you enumerated. Writing something like before = [],[] makes a 2-tuple containing two lists; if you then enumerate three folders, before[2] does not exist. The safe way is to create one snapshot container per path up front (either a list-of-lists or a dict keyed by path) so indexes/keys always match the paths being watched.

A clearer, more robust pattern is to keep a mapping path -> set-of-names and update each path independently. This avoids fragile index math and makes per-path reporting simple. Example (Python 3 / pathlib style):

from pathlib import Path
import time

paths = [Path('/path1'), Path('/path2'), Path('/path3')]
snapshots = {p: {e.name for e in p.iterdir() if e.is_file()} for p in paths}

while True:
    time.sleep(3)
    for p in paths:
        try:
            current = {e.name for e in p.iterdir() if e.is_file()}
        except OSError:
            continue
        added = current - snapshots[p]
        removed = snapshots[p] - current
        if added:
            print("Added in %s: %s" % (p, ", ".join(added)))
        if removed:
            print("Removed in %s: %s" % (p, ", ".join(removed)))
        snapshots[p] = current

Practical tips: validate each path exists before monitoring; catch OSErrors from iterdir(); ignore temporary/partial filenames (or detect atomic moves); use absolute paths; and throttle scanning to avoid high CPU for large folders. For production or many folders, prefer an event-driven watcher (inotify / FSEvents / Windows API via a library such as watchdog) instead of polling.

This keeps things simpler than juggling nested tuples/lists (the root of the original bug) and follows the same detection idea suggested but in a per-path, less error-prone structure.

All 7 Replies

TrustyTony 888 ex-Moderator

16 Years Ago

Does this help you?

Pretty is my module which I posted you can replace it with normal prints and use your paths of course.

import os
import pretty

path_to_watch = []
path_to_watch.append("/Tony/Tests")
path_to_watch.append("/Python Projects/")
path_to_watch.append("/Python26/Lib/site-packages")

before = [],[]
for item, folder in enumerate(path_to_watch):
##    print "i am here", item, " : ", folder
    exp=[(f, None) for f in os.listdir (folder)]
    print 'Before: ',len(before),
    pretty.ppr(before)
    pretty.ppr(item)
    pretty.ppr(before[item])    
##    print item, before[item], len(before)
    if item: before[item].append(dict (exp))

>>> 
Before:  2 
(
  [], 
  [])
0

[]
Before:  2 
(
  [], 
  [])
1

[]
Before:  2 
(
  [], 
  [{'ppitcs_code': None}])
2

Traceback (most recent call last):
  File "D:\Tony\Tests\watchthese.py", line 16, in <module>
    pretty.ppr(before[item])
IndexError: tuple index out of range
>>>

What are you trying to accomplish with this jungle of dicts, Nones and lists?
What expected final result?

Edited 16 Years Ago by TrustyTony because: n/a

TrustyTony 888 ex-Moderator

16 Years Ago

Why not to do simply:

import os,time
path_to_watch='/Tony'
before = set(f for f in os.listdir(path_to_watch))
while 1:
    time.sleep (3)
    after = set(f for f in os.listdir(path_to_watch))
    added = [f for f in after if not f in before]
    removed = [f for f in before if not f in after]
    if added:
        print "Images Added: ", ", ".join (added)
    if removed:
        print "Images Removed: ", ", ".join (removed)
    before = after

Reply to this topic

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

alabandit 0 Newbie Poster · Answer 1 · 2010-05-07T16:31:45+00:00

Does this help you?

Pretty is my module which I posted you can replace it with normal prints and use your paths of course.

import os
import pretty

path_to_watch = []
path_to_watch.append("/Tony/Tests")
path_to_watch.append("/Python Projects/")
path_to_watch.append("/Python26/Lib/site-packages")

before = [],[]
for item, folder in enumerate(path_to_watch):
##    print "i am here", item, " : ", folder
    exp=[(f, None) for f in os.listdir (folder)]
    print 'Before: ',len(before),
    pretty.ppr(before)
    pretty.ppr(item)
    pretty.ppr(before[item])    
##    print item, before[item], len(before)
    if item: before[item].append(dict (exp))

>>> 
Before:  2 
(
  [], 
  [])
0

[]
Before:  2 
(
  [], 
  [])
1

[]
Before:  2 
(
  [], 
  [{'ppitcs_code': None}])
2

Traceback (most recent call last):
  File "D:\Tony\Tests\watchthese.py", line 16, in <module>
    pretty.ppr(before[item])
IndexError: tuple index out of range
>>>

What are you trying to accomplish with this jungle of dicts, Nones and lists?
What expected final result?

I need to monitor a number of folders for changes, depending on the type of file and folder it will trigger a action.
then second part of the script checks every couple of minutes for changes to the folder structure, i just need lists of all the folders monitored.

the code for a single folder i found in a tutorial online

before = dict ([(f, None) for f in os.listdir (path_to_watch)])
while 1:
  time.sleep (3)
  after = dict ([(f, None) for f in os.listdir (path_to_watch)])
  added = [f for f in after if not f in before]
  removed = [f for f in before if not f in after]
  if added:
	print "Images Added: ", ", ".join (added)
        #main_function(added)

  if removed:
	print "Images Removed: ", ", ".join (removed)
  before = after

alabandit 0 Newbie Poster · Answer 2 · 2010-05-07T17:34:21+00:00

I need to monitor a number of folders for changes, depending on the type of file and folder it will trigger a action.
then second part of the script checks every couple of minutes for changes to the folder structure, i just need lists of all the folders monitored.
the code for a single folder i found in a tutorial online
before = dict ([(f, None) for f in os.listdir (path_to_watch)])
while 1:
  time.sleep (3)
  after = dict ([(f, None) for f in os.listdir (path_to_watch)])
  added = [f for f in after if not f in before]
  removed = [f for f in before if not f in after]
  if added:
	print "Images Added: ", ", ".join (added)
        #main_function(added)

  if removed:
	print "Images Removed: ", ", ".join (removed)
  before = after

simplified the code to...

before = [],[]
path_to_watch.append("/path1/")
path_to_watch.append("/path2/")
path_to_watch.append("/path3/")

for item, folder in enumerate(path_to_watch):
    before[item].append("1")

alabandit 0 Newbie Poster · Answer 3 · 2010-05-07T18:05:53+00:00

after more googling i got it to work by,

import os

path_to_watch = [[0] for i in range(len(path_to_watch))]
path_to_watch.append("/path1/")
path_to_watch.append("/path2/")
path_to_watch.append("/path3/")

before = [],[]
for item, folder in enumerate(path_to_watch):
    print "i am here", item, " : ", folder
    before[item].append(dict ([(f, None) for f in os.listdir (folder)]))

miss understood how to declare the listoflist

thanks every one for your help

alabandit 0 Newbie Poster · Answer 4 · 2010-05-07T18:09:09+00:00

thanks tonyjv

not the problem but certainly a improvement on the code.

TrustyTony 888 ex-Moderator Team Colleague Featured Poster · Answer 5 · 2010-05-07T18:28:01+00:00

For multiple paths just wrap the list comprehension inside other one for paths and mayby register the files with os.path.realpath(f) instead of only f.

Better to put that repeated code in function as it is needed inside and out of while.

list of list of files

Dani AI

Recommended Answers Collapse Answers

All 7 Replies

Recommended Answers