Start New Discussion within our Software Development Community

This snippet composes functions of the form func(infile, outfile) which read an input file and transform its content to write an output file. A function compose((f, g, h,...)) is defined which returns a new such file transformation function which successively applies f, g, h, using in-memory temporary files.

# Title: Compose file transformation functions in python
# Date: 2011 Oct 27 
# Author: Gribouillis for the python forum at www.daniweb.com
# License: public domain
# Use this code freely in your programs.

try:
    from io import StringIO # python 3
except ImportError:
    from cStringIO import StringIO # python 2

def last_marked(iterable):
    """Generate pairs (islast, item) with the iterable's items
    
    The boolean value 'islast' is False unless the item is the last in the sequence.
    """
    it = iter(iterable)
    try:
        prev = next(it)
    except StopIteration:
        raise
    for item in it:
        yield False, prev
        prev = item
    yield True, prev
    
def compose(file_transforms):
    """Compose a sequence of file transformation functions to create a new one.

    Here a file transformation function has the signature
        def func(infile, outfile):
            ...
    where infile is a file opened for reading and outfile is a file opened for
    writing.
    
    compose() takes a sequence of file transformation functions and returns a
    new file transformation function which runs each function one after the other,
    using StringIO (memory files) as temporary files. For example, given 3 such
    functions f, g, h:
        
        >>> F = compose((f, g, h))
        >>> F(open("infile.txt"), open("outfile.txt", "w"))
    """
    def composition(infile, outfile):
        fh = infile
        for islast, func in last_marked(file_transforms):
            temp = outfile if islast else StringIO()
            func(fh, temp)
            fh, temp = temp, None
            if not islast:
                fh.seek(0)
    return composition
The article starter has earned a lot of community kudos, and such articles offer a bounty for quality replies.