Posts by Gribouillis

I don't understand your problem. If you have the list

L = [('A', 1), ('B', 2), ('C', 3), ('D', 4),('A', 5), ('B', 6), ('C', 7), ('D', 8)]

and you want a excel sheet such as

A B C D A B C D
1 2 3 4 5 6 7 8

you can do either

writer.writerows(zip(*L))

or

writer.writerow[(k for k, v in L)]
writer.writerow([v for k, v in L])

and of course, in your case, do first

L = list(flatten_dict(root))

All this will work very well as long as you dont have very large xml files (but in this case, the code can be adapted)

The flatten_dict() function returns an iterable sequence of (key, value) pairs. You can turn this to a list of pairs with list(flatten_dict(root)).

A list of pairs L (or a iterable of pairs) can be transposed with zip(*L).

>>> L = [('A', 1), ('B', 2), ('C', 3), ('D', 4)]
>>> list(zip(*L))
[('A', 'B', 'C', 'D'), (1, 2, 3, 4)]

You don't understand the output of flatten_dict(). You can do this

def main():
    with open('source.xml', 'r', encoding='utf-8') as f: 
        xml_string = f.read() 
    xml_string= xml_string.replace('�', '') #optional to remove ampersands. 
    root = ElementTree.XML(xml_string) 

    writer = csv.writer(open("test_out.csv", 'wt'))
    writer.writerows(zip(*flatten_dict(root)))

if __name__ == "__main__":
        main()

Also main() was not called in your code, due to indentation.

No, fileList is ['python.exe','java.exe','myassign.py','mydoc.doc'] and fileExtension is 'doc'.

If there is still a dot at the end of a column header, it would be better to remove it ('Response.R.' becomes 'Response.R'). For this, use the rstrip('.') method.

By default, the csv module selects the 'excel' dialect.

It is not better to use writerows(). It is only shorter if you only want to call writerow() several times. Shorter code looks better.

The problem is that I dont understand your rule for key generation. If you want to keep only the last word, it is very easy to do

    for key, value in flatten_dict(root):
        key = key.rstrip('.').rsplit('.', 1)[-1]
        print(key,  value)

edit: also, you can start with my generator and change the code the way you want to generate a different key.

It looks easy if you write pseudo-code

def fileExtensionExists(fileList, fileExtension):
    for fileName in fileList:
        get this filename''s extension, call this 'extension' (use parseExtension)
        if extension == fileExtension:
            return True
    # if we reach here, the extension was not met
    # it means that the extension does not exist in the list
    return False

def parseExtension(fileName):
    if there is a dot in filename:
        return everything after the last dot
    else:
        return the empty string

Now you should be able to transform this into python.

Here is a variant which handles lists differently by enumerating the list items and removing their common tag. It works for MonthDayCount in your example, but I'm not sure it will work the way you want for all the lists in your files.

import xml.etree.cElementTree as ElementTree 
from xml.etree.ElementTree import XMLParser

def flatten_list(aList, prefix=''):
    for i, element in enumerate(aList, 1):
        eprefix = "{}{}".format(prefix, i)
        if element:
            # treat like dict 
            if len(element) == 1 or element[0].tag != element[1].tag: 
                yield from flatten_dict(element, eprefix+'.')
            # treat like list 
            elif element[0].tag == element[1].tag: 
                yield from flatten_list(element, eprefix+'.')
        elif element.text: 
            text = element.text.strip() 
            if text: 
                yield eprefix, text


def flatten_dict(parent_element, prefix=''):
    prefix = prefix + parent_element.tag + '.'
    if parent_element.items():
        for k, v in parent_element.items():
            yield prefix + k, v
    for element in parent_element:
        eprefix = prefix + element.tag + '.'
        if element:
            # treat like dict - we assume that if the first two tags 
            # in a series are different, then they are all different. 
            if len(element) == 1 or element[0].tag != element[1].tag: 
                yield from flatten_dict(element, prefix=prefix)
            # treat like list - we assume that if the first two tags 
            # in a series are the same, then the rest are the same. 
            else: 
                # here, we put the list in dictionary; the key is the 
                # tag name the list elements all share in common, and 
                # the value is the list itself
                yield from flatten_list(element, prefix=eprefix)
            # if the tag has attributes, add those to the dict
            if element.items(): …

I transformed the activestate recipe into a generator which flattens the xml structure. Look at the ouput, then try to define how you would transform the keys to get correct column names

import xml.etree.cElementTree as ElementTree 
from xml.etree.ElementTree import XMLParser 
import json 
import csv 
import tokenize 
import token 
try: 
    from collections import OrderedDict 
    import json 
except ImportError: 
    from ordereddict import OrderedDict 
    import simplejson as json 
import itertools 
import six 
import string 
#from csvkit import CSVKitWriter 


def flatten_list(aList, prefix=''): 
    for element in aList:
        if element: 
            # treat like dict 
            if len(element) == 1 or element[0].tag != element[1].tag: 
                yield from flatten_dict(element, prefix)
            # treat like list 
            elif element[0].tag == element[1].tag: 
                yield from flatten_list(element, prefix)
        elif element.text: 
            text = element.text.strip() 
            if text: 
                yield prefix, text


def flatten_dict(parent_element, prefix=''):
    prefix = prefix + parent_element.tag + '.'
    if parent_element.items():
        for k, v in parent_element.items():
            yield prefix + k, v
    for element in parent_element:
        eprefix = prefix + element.tag + '.'
        if element:
            # treat like dict - we assume that if the first two tags 
            # in a series are different, then they are all different. 
            if len(element) == 1 or element[0].tag != element[1].tag: 
                yield from flatten_dict(element, prefix=prefix)
            # treat like list - we assume that if the first two tags 
            # in a series are the same, then the rest are the same. 
            else: 
                # here, we put the list in dictionary; the key is the 
                # tag name the list elements all share in common, and 
                # the value is the list …

Using user input could be anything

That's why you have a variable or function parameter named filename. You must use this variable instead of "python.exe".

One part of the problem is that we don't know which command or (python?) statement fails to read the entire file. Can you describe more precisely where it fails.

I see that you are using an old activestate recipe to parse xml and transform it into a dictionary. It means that the comments in this code don't have anything to do with your specific xml files.

You could traverse the parsed xml tree directly and generate on the fly the items of your final flat dictionary. It would be much more hard-hitting. This would probably mean two or three generators invoking each other recursively through the use of yield from statements.

The key algorithmic points remain the same: what is your actual xml file's structure and what are your rules to create target key, value pairs ?

Hi. Here is a small experiment in the python console

>>> "python.exe".rsplit('.', 1)
['python', 'exe']

It should be easy now !

Yes put the snippet in a file named postprocess.py then write

from postprocess import post_process

Here is an example of what you can do. This function transforms a dictionary by exploding the inner lists if they contain only strings

from postprocess import post_process

@post_process(dict)
def explode_lists(adict):
    for key, value in adict.items():
        if isinstance(value, list):
            if all(isinstance(x, str) for x in value):
                for i, x in enumerate(value, 1):
                    yield ('{}{}'.format(key, i), x)
                continue
        yield key, value

if __name__ == '__main__':
    D = {'D_B': ['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'],
        'F_Int32': ['0','0','0','0'],
        'OTF': '0',
        'PBDS_Double': ['0', '0', '0', '0', '0', '0', '0', '0'],
        'SCS_String': ['1', '2']}
    print(explode_lists(D))


""" my output -->
{'SCS_String2': '2', 'SCS_String1': '1', 'PBDS_Double1': '0', 'PBDS_Double3': '0', 'PBDS_Double2': '0', 'PBDS_Double5': '0', 'PBDS_Double4': '0', 'PBDS_Double7': '0', 'PBDS_Double6': '0', 'PBDS_Double8': '0', 'F_Int321': '0', 'F_Int323': '0', 'F_Int322': '0', 'F_Int324': '0', 'D_B5': '0', 'D_B4': '0', 'D_B7': '0', 'D_B6': '0', 'D_B1': '0', 'D_B3': '0', 'D_B2': '0', 'D_B9': '0', 'D_B8': '0', 'OTF': '0', 'D_B11': '0', 'D_B10': '0'}
"""

In this code, I used a very useful snippet that I wrote long ago in the following file postprocess.py

# postprocess.py

def post_process(*filters):
    """Decorator to post process a function's return value through a
    sequence of filters (functions with a single argument).

    Example:

        @post_process(f1, f2, f3)
        def f(*args, **kwd):
            ...
            return value

        then calling f(...) will actually return f3( f2( f1( f(...)))).

        This can also be used to convert a generator to a function
        returning a sequence type:

        @post_process(dict)
        def my_generator():
            ...
            yield key, value

    """

    def decorate(func):
        from functools import wraps
        @wraps(func)
        def wrapper(*args, **kwd): …

Here is a simple transformation snippet

>>> key = 'spam'
>>> L = ['foo', 'bar', 'baz']
>>> [('{}{}'.format(key, i), value) for i, value in enumerate(L, 1)]
[('spam1', 'foo'), ('spam2', 'bar'), ('spam3', 'baz')]

the key is replicated n number of times for n number of items in its associated list

Every output is possible, only the rules which govern the production of items must be very carefuly defined. If you want DB1, DB2, etc, you must explain by which rule D_B becomes DB1, DB2 etc. In the same way, which rule transforms F_Int32 into F1, F2, etc.

Python can implement any precise rule that you define, but it cannot define the transformation rules for you.

Interestingly, with your function on other files I get the following error

I made the assumption that if the json file contains arrays with more than one item, all these items are strings. It fails in the files that you tried. Only strings can be concatenated. You must define the list of pairs that must be generated in these cases.

One final comment to make: Is it possible to write a function that applies to each key such that if the key is associated with a value, the key is replicated n number of times for n number of items in its associated list.

Again, this must be clarified with a json example and the complete list of pairs key/value that you want to produce with this json data.

Here is the kind of code that you can try

import csv 
import json 
import sys 

def shrink(v):
    while True:
        if isinstance(v, list):
            if len(v) == 1:
                v = v[0]
            elif v:
                assert all(isinstance(x, str) for x in v)
                v = ''.join(v)
            else:
                raise ValueError('Empty list')
        elif isinstance(v, dict) and len(v) == 1:
            v = next(iter(v.values()))
        else:
            return v

def flatten(obj):
    assert isinstance(obj, dict)
    for k, v in obj.items(): 
        v = shrink(v)
        if isinstance(v, dict):
            yield from flatten(v)
        else:
            yield k, v

if __name__ == "__main__": 
    with open("data2.json") as f: 
        data = json.load(f) 

    pairs = list(flatten(data))
    print(pairs)

    writer = csv.writer(sys.stdout) 
    header = writer.writerow([k for k, v in pairs]) 
    row = writer.writerow([v for k, v in pairs])

The idea is to shrink the values with the following rules: if a value is a list with a single element, it is replaced by this element (shrunk). If the value is a list with more than 1 element, it is assumed that all the elements are strings, and they are concatenated and the result replaces the value. If the value is a dict with a single key, it is replaced by the value of this single item.

My output with the modified json file above is

[('DLA', '0'), ('FC', '00000'), ('PC', '0'), ('WC', '0'), ('CN', None), ('Description', None), ('Code', '0'), ('CMC', '0')]
DLA,FC,PC,WC,CN,Description,Code,CMC
0,00000,0,0,,,0,0

If you want to adapt the flatten function to the case where the json code contains arrays, you must first describe which key/value pairs you expect as output from the above json file and why. This question is independent from python. It can be coded only if it can be described precisely.

Ok, it looks easy. The ValueError is thrown by the json parser (function json.load()) because your json file is an invalid json file. It contains a pair key/value outside of an object. If I paste your json code in http://jsonlint.com for example, the parser says that a { or a [ is expected at the beginning. Also there are missing closing delimiters at the end

The json grammar can be seen at http://json.org/

Here is a modified valid json file

{
"PAC": {
            "Account": [{
                "PC": "0",
                "CMC": "0",
                "WC": "0",
                "DLA": "0",
                "CN": null,
                "FC": {
                    "Int32": ["0",
                    "0",
                    "0",
                    "0",
                    "0"]
                },
                "F": {
                    "Description": null,
                    "Code": "0"
                }
}]
}
}

Unfortunately, this is not sufficient because your code is not ready to handle arrays in json data (I mean [...]). For this, you should probably try to call load() without the object hook and change the flatten() generator.

It is because temp is a string (str type) while 85 is an integer. Integer comparison is different from string comparison (string ordering is lexicographic order). You should do

temp = int('32')

to convert to int for example.

This problem does not exist in recent versions of python where comparison of unordered data types is impossible. See the difference between python 2.7

>>> '32' > 85
True

and python 3.4

>>> '32' > 85
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: str() > int()

If you have the choice, learn python with python 3.4.

The description is not clear at all. We need a minimal working example: a small json object, and a short piece of code which raises the described exception (ValueError ...). You can add the expected result.

On my system, I installed the hh command from Click Here (shell history suggest box). I can navigate history with the hh command and remove specific entries with the del key.

Another way is to clear the whole history with commands such as cat /dev/null > ~/.bash_history && history -c && exit, but check that the history does not come back after rebooting.

These commands remain dangerous even after they've been used, because they remain in your shell history, with a risk of being recalled mistakenly, for example a ^r in a bash terminal may easily recall commands. I wouldn't accept to have a rm -rdf * in my shell history.

I would say rm -rf *.

If you input [z * 5 for z in range(2, 10, 2)], it does not interfere.

decorators only work on function call, not on definition

On the contrary, the decorator is called once, immediately after the function's definition

>>> def deco(f):
...  print f.__name__, 'is being defined'
...  return f
... 
>>> @deco
... def foo(x):
...  return x * x
... 
foo is being defined
>>> foo(3)
9
>>> 

Between the raw_input() and your printing x, there was an input(). This is where x was modified.

About the dangers of input(), the point is that when a program writes write a list!, the user does not think that his input can have a malicious effect such as erasing files or starting unwanted programs, etc. There should be some mechanism in your program to prevent this from happening. Python's eval() function is too powerful, or it must be used in conjonction with code restricting the string that can be passed to input().

An example is given in this snippet where an expression is carefully analysed before being given to eval(), thus allowing only the evaluation of mathematical expressions.

When [x * 3 for x in range(2, 10, 2)] is evaluated as a result of the input function, the variable x takes the values 2, 4, 6, 8. When the x is printed, its value is the last one, 8. You can avoid this by using another variable name.

Conclusion: don't use input() in python 2, it is too dangerous.

If I use , in the place of +''+ It gave this

You need to be precise. The output you are showing separates every arguments with a comma. Where does it come from ? Did you print some python variable ? How is this command used ? So please explain what you mean by it gave this. For example in a python list ['a', 'b'] there is a comma, but it is only a representation of the list.It does not mean that 'a' and 'b' are separated by a comma.

If I use next(iter(...)) it is giving many errors

Again, can you post the error message ?

Replace +''+ with ,. Why didn't you use next(iter(...)) as I told you ?

The file is probably not yours, or has wrong permissions. Open a terminal and type ls -l file.php to see the permissions. Post the result here.

touch foo creates file foo, if that's what you mean, but I probably missed something in your question. (?)

dict.iteritems() disappeared from python in python 3.0. Use dict.items() in python 3, which returns a view (in python 2, dict.items() used to return a list).

Instances don't use less memory if the method is defined outside the class definition. Instances don't keep pointers to the method. They only keep a pointer to the class, which in turns keeps a pointer to the method.

If you want, you can add the function to the class after the class definition

def classfunc( inst, arg ):
    pass

class mainclass: pass

mainclass.classfunc = classfunc

but there is no benefit as compared to writing the function definition directly in the class' body.

Also in python 2 only, write class mainclass(object): pass otherwise it is an old-style class which behaves slightly differently from ordinary classes (they are python 1 classes).

EDIT: The correct way to defined memory-thrifty instances in python is to use the __slots__ member which prevents instances from allocating a dictionary.

It seems very easy because the file is a TAB-separated file. Here is my code in python 2

#!/usr/bin/env python
# -*-coding: utf8-*-
'''doc
'''
from __future__ import (absolute_import, division,
                        print_function, unicode_literals)
import codecs

def process_file(filename):
    with codecs.open(filename, encoding='utf8') as ifh:
        for line in ifh:
            row = line.split('\t')
            english, hindi = row[-2:]
            print('English:', english)
            print('Hindi:', hindi)

if __name__ == '__main__':
    process_file('hindmonocorp05.txt')

And the result

No it will be very fast.

I suppose your data is stored in a file. Can you send such a file with say 10 lines of data ?

I think your data is not valid python code. For example to read the hindi part, I had to replace all the \N with \ \N (double backslash) otherwise python would not read the unicode string.

Here is what I get when I print the hindi part:

ताजा\JJ साँसें\N_NN और\CC_CCD चमचमाते\JJ दाँत\N_NN आपके\PR_PRP व्यक्तित्व\N_NN को\PSP निखारते\V_VM हैं\V_VAUX ।\R

does it have something to do with what you want ?

Your question doesn't mean much. The unicode does not have to be converted to the target language. It is already in that language. What you want is probably to display the unicode with the language's human glyphs, which is something else. For this, the print function should work, for example I don't know hindi but

>>> y = u'\u0924\u093e\u091c\u093e'
>>> print(y)
ताजा
>>> 

It would be great if you could attach a small file to a post containing your exact data (for example a csv file in UTF8. You can zip the file to attach it to a post).

Edit: it's too bad, we still have this bug with the unicode characters in the forum!

It seems fixed for me.

I have the same issue in qupzilla 1.8.6.

Yes, you can enter the commas explicitely and parse the input later. For example you can remove every comma before converting to float or int

userdata = input('please enter the amount of the check   ')
userdata = userdata.replace(',', '')
amount = int(userdata)

You can improve this by adding a loop and input validation, for example a valid input matches the regex

r'^\s*\d{1,3}{\,\d{3}}*\s*$'

You need a GUI to do that, for example a tkinter Entry widget plus supporting code to handle each key stroke and mouse event sent to this widget.

Do you already have some code for the calculator ? Are you using a GUI toolkit ?

No, I only typed man getpid in a terminal and at the top of the man page, there was

       #include <sys/types.h>
       #include <unistd.h>

See http://pubs.opengroup.org/onlinepubs/7908799/xsh/unistd.h.html for example. sys/types.h is for pid_t.

You only need to add prints to see what's going on. Here is my version, with pid numbers. Everything seems to run the way we described above

#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

#define SLEEP 1

int main(int argc, char** argv){
int i = 2;
for (i=2; i<=4; i++)
{
printf("Hello %d (pid %d, ppid %d)\n", i, getpid(), getppid());
sleep(SLEEP);
if ((i & 1) && fork() == 0)
{
sleep(SLEEP);
printf("OS %d (pid %d, ppid %d)\n", i, getpid(), getppid());
fork();
sleep(SLEEP);
printf("ready to fork %d (pid %d, ppid %d)\n", i, getpid(), getppid());
if (fork() > 0) {

sleep(SLEEP);
exit(1); }

sleep(SLEEP);
printf("Students %d (pid %d, ppid %d)\n", i, getpid(), getppid());
}
else
printf("World %d (pid %d, ppid %d)\n", i, getpid(), getppid());
}
return 0;
}

/*
 * Example output with comments
 * 
Hello 2 (pid 4870, ppid 4742)           4870 is A
World 2 (pid 4870, ppid 4742)
Hello 3 (pid 4870, ppid 4742)
World 3 (pid 4870, ppid 4742)
Hello 4 (pid 4870, ppid 4742)
World 4 (pid 4870, ppid 4742)
OS 3 (pid 4871, ppid 4870)              4871 is B
12:15 494489] ready to fork 3 (pid 4871, ppid 1804)   B is about to fork D
ready to fork 3 (pid 4872, ppid 4871)   4872 is C, about to fork E
Students 3 (pid 4874, ppid 4872)        4874 is E (parent C)
Students 3 (pid 4873, ppid 4871)        4873 is D (parent B)
Hello 4 (pid 4874, ppid 4872)
Hello 4 (pid 4873, ppid 4871)
World 4 (pid 4873, ppid 1804)
World 4 (pid 4874, ppid 1804)
 */

Edit: notice that B (4871) is adopted by init (1804) after A has died.

No in principle B is created only at loop i=3. Can you post the whole code ?

Here is a python program with almost the same statements

from os import fork, getpid
from sys import exit

def main():
    i = 2
    while i <= 4:
        print('PID', getpid(), ': Hello', i)
        if (i & 1) and (fork() == 0):
            print('PID', getpid(), ': OS', i)
            fork()
            if fork() > 0:
                exit(1)
            print('PID', getpid(), ': Students', i)
        else:
            print('PID', getpid(), ': World', i)
        i += 1
    return 0

if __name__ == '__main__':
    exit(main())

""" my output -->
('PID', 7621, ': Hello', 2)
('PID', 7621, ': World', 2)
('PID', 7621, ': Hello', 3)
('PID', 7621, ': World', 3)
('PID', 7621, ': Hello', 4)
('PID', 7621, ': World', 4)
('PID', 7622, ': OS', 3)
('PID', 7625, ': Students', 3)
('PID', 7625, ': Hello', 4)
('PID', 7625, ': World', 4)
('PID', 7624, ': Students', 3)
('PID', 7624, ': Hello', 4)
('PID', 7624, ': World', 4)
"""

A is 7621, the child B is 7622, the grandchild and grandgrandchild are 7624 and 7625 (which is which ?)

The expression i & 1 is true only if i is odd. This will happen only if i equals 3 in the loop. In that case fork() is executed and creates a child process. Let A be the main process and B the child process created at this moment. The if statement's body is only executed in process B (because that's where fork() returned 0). Then, the fork() at line 14 is executed, spawning a grandchild process say C. Line 15 is executed in both B and C, spawning another grandchild D and a grandgrandchild E. B and C exit at line 16 with status 1 while D and E both print Students and go to next loop with i == 4.

I never did recover the data

Everybody backs up!