Hi, there

With a lot of help in this forum, I strated to use Python to accomplish something. Here I am looking for guide about how to improve my code speed.

This code is to determine the normal direction of a polygon with four points.
Every four points have 6 different orders and so i calculate them to find one with maximum value, which is the order I need. It works and however runs slowly when I use it to handle a lot of data.

path_value=[28817,29283,30676,29284,30677]

def order_nodes(path_value):
#    print path_value
    f9 = open('inf.txt', 'r')
    alllines9 = f9.readlines()
    f9.close()
    x=[0,0,0,0,0,0]   # save all coordinate values of these nodes into a
    y=[0,0,0,0,0,0]
   
    i=1
    #  get all coordinate values of these nodes
    while i < len(path_value):
        for line100 in alllines9:
            line100=line100.strip()
            line1000=line100.split(',')
#            print line1000[0]
            if float(path_value[i]) == float(line1000[0]):
                x[i-1]=line1000[1]
                y[i-1]=line1000[2]
        i=i+1
#        print "i=", i

    # fetch all coordinate values of these nodes
    x[-1]=x[0]
    y[-1]=y[0]

    # determine these order
    total_area=[0,0,0,0,0,0]
    total_order=[0,0,0,0,0,0]
    
    # case 0
    j=0
    area=0
    while j < len(x)-1:
        area= area + (float(x[j])*float(y[j+1])-float(y[j])*float(x[j+1]))
        j=j+1
    total_area[0]=area
    total_order[0]=[1,2,3,4]

    # case 1
    x1=x[:]
    y1=y[:]
  
    x1_temp=x1[2]
    y1_temp=y1[2]

    x1[2]=x1[3]
    y1[2]=y1[3]
    x1[3]=x1_temp
    y1[3]=y1_temp

    total_order[1]=[1,2,4,3]

    j1=0
    area1=0
    while j1 < len(x1)-1:
        area1= area1 + (float(x1[j1])*float(y1[j1+1])-float(y1[j1])*float(x1[j1+1]))
        j1=j1+1
    total_area[1]=area1
    
        # case 2
    x2=x[:]
    y2=y[:]
     
    x2_temp=x2[1]
    y2_temp=y2[1]

    x2[1]=x2[2]
    y2[1]=y2[2]
    x2[2]=x2_temp
    y2[2]=y2_temp

    total_order[2]=[1,3,2,4]
   
    j2=0
    area2=0
    while j2 < len(x2)-1:
        area2= area2 + (float(x2[j2])*float(y2[j2+1])-float(y2[j2])*float(x2[j2+1]))
        j2=j2+1
    total_area[2]=area2

        # case 3
    x3=x2[:]
    y3=y2[:]

     
    x3_temp=x3[2]
    y3_temp=y3[2]

    x3[2]=x3[3]
    y3[2]=y3[3]
    x3[3]=x3_temp
    y3[3]=y3_temp

    total_order[3]=[1,3,4,2]

    j3=0
    area3=0
    while j3 < len(x3)-1:
        area3= area3 + (float(x3[j3])*float(y3[j3+1])-float(y3[j3])*float(x3[j3+1]))
        j3=j3+1
   
    total_area[3]=area3
    
   # case 4
    x4=x[:]
    y4=y[:]
     
    x4_temp1=x4[1]
    x4_temp2=x4[2]
    y4_temp1=y4[1]
    y4_temp2=y4[2]
        
    x4[1]=x4[3]
    x4[2]=x4_temp1
    x4[3]=x4_temp2
 
    y4[1]=y4[3]
    y4[2]=y4_temp1
    y4[3]=y4_temp2

    total_order[4]=[1,4,2,3]
    
    j4=0
    area4=0
    while j < len(x4)-1:
        area4= area4 + (float(x4[j4])*float(y4[j4+1])-float(y4[j4])*float(x4[j4+1]))
        j4=j4+1
    
    total_area[4]=area4

    # case 5
    x5=x4[:]
    y5=y4[:]
 
    x5_temp=x5[2]
    y5_temp=y5[2]

    x5[2]=x5[3]
    y5[2]=y5[3]
    x5[3]=x5_temp
    y5[3]=y5_temp

    total_order[5]=[1,4,3,2]
    j5=0
    area5=0
    while j < len(x5)-1:
        area5= area5 + (float(x5[j5])*float(y5[j5+1])-float(y5[j5])*float(x5[j5+1]))
        j5=j5+1

    total_area[5]=area5
    max_location = total_area.index(max(total_area))
    node_order=total_order[max_location]

    return node_order

node_order=order_slab_nodes(path_value)

Could you please comment my code? Thank you so much.

ning

First of all, I don't know what the code does.
From a merely technical point of view, I say the followings.

I think you should improve the code before improving the speed.
The majority of the code seems to be copy-pasted. All the "cases" can possibly merged into one codeblock or function.
Some wise man said: "The design is finished, when you cannot remove anything, not when you cannot add anything" Or something in this meaning:)

Speeding improvements can be seen:

  • Why so much float cast? Why not read in floats?
  • More use of += and *= syntax is helpful.
  • Len is evaluated for every loop step, but it remains constant.
  • The loop can be solved with an enumeration, too. No need to jump on indexes.

This line:
if float(path_value) == float(line1000[0]):
will give you major headaches, since floats are usually off by very small values and are such not always equal.

Use the function fuzzy_equals for that:

def fuzzy_equals(a, b, delta=0.000001):
    """
    used for comparison of floating point numbers a and b
    """
    return -delta < a-b < delta

Also Python has a nice profiler available to check the efficiency of a give function. Use something like this:

import cProfile

def order_nodes(path_value):
    ...
    ...

cProfile.run("order_nodes()")
...
...

The many calls using function float() will make you suffer!

Also, what is the call "order_slab_nodes(path_value)"?

Last not least check into module psyco. It gives major speed improvements as it compiles to native 386 machine code rather than Python bytecode.

First of all, I don't know what the code does.
From a merely technical point of view, I say the followings.

I think you should improve the code before improving the speed.
The majority of the code seems to be copy-pasted. All the "cases" can possibly merged into one codeblock or function.
Some wise man said: "The design is finished, when you cannot remove anything, not when you cannot add anything" Or something in this meaning:)

Speeding improvements can be seen:

  • Why so much float cast? Why not read in floats?
  • More use of += and *= syntax is helpful.
  • Len is evaluated for every loop step, but it remains constant.
  • The loop can be solved with an enumeration, too. No need to jump on indexes.

Hi, Slate

Thank you so much for your suggestion.

I modified this code for += and Len and am still looking for flaots and enumeration you mentioned. I also appreciate the coding principle you address. Let me carry on in this direction.

Be honest, I just moved in Python two months ago. Any minor guide and suggestion would be very useful for me.

best wishes

ning

This line:
if float(path_value) == float(line1000[0]):
will give you major headaches, since floats are usually off by very small values and are such not always equal.

Use the function fuzzy_equals for that:

def fuzzy_equals(a, b, delta=0.000001):
    """
    used for comparison of floating point numbers a and b
    """
    return -delta < a-b < delta

Also Python has a nice profiler available to check the efficiency of a give function. Use something like this:

import cProfile

def order_nodes(path_value):
    ...
    ...

cProfile.run("order_nodes()")
...
...

The many calls using function float() will make you suffer!

Also, what is the call "order_slab_nodes(path_value)"?

Last not least check into module psyco. It gives major speed improvements as it compiles to native 386 machine code rather than Python bytecode.

Hi, Ene Uran

Thank you for your suggestion.

In order to avoid the problem associated with float equality, I used INT stead of FLOAT in "if float(path_value) == float(line1000[0]):".

I used profile modus to assess this code, which results in


246664 function calls in 3.556 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 :0(close)
        1    0.000    0.000    0.000    0.000 :0(index)
        8    0.000    0.000    0.000    0.000 :0(len)
        1    0.000    0.000    0.000    0.000 :0(max)
        1    0.007    0.007    0.007    0.007 :0(readlines)
        1    0.002    0.002    0.002    0.002 :0(setprofile)
   123324    0.862    0.000    0.862    0.000 :0(split)
   123324    0.813    0.000    0.813    0.000 :0(strip)
        1    0.001    0.001    3.554    3.554 <string>:1(?)
        1    1.871    1.871    3.553    3.553 order_slab.py:5(order_shell_nodes)
        1    0.000    0.000    3.556    3.556 profile:0(order_shell_nodes(path_value))
        0    0.000             0.000          profile:0(profiler)

This result surprises me since strip() and split() are dominant time killers. So I add "break" into the code as

i=1
    #  fetch all coordinate values of these nodes
    while i < len(path_value):
        for line100 in alllines9:
            line100=line100.strip()
            line1000=line100.split(',')
#            print line1000
            if int(path_value[i]) == int(line1000[0]):
                x[i-1]=line1000[1]
                y[i-1]=line1000[2]
                break
        i=i+1

Now pofile modus produces

41159 function calls in 0.606 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        8    0.003    0.000    0.003    0.000 :0(acquire)
        4    0.000    0.000    0.000    0.000 :0(allocate_lock)
        2    0.000    0.000    0.000    0.000 :0(append)
        1    0.000    0.000    0.000    0.000 :0(close)
        2    0.000    0.000    0.000    0.000 :0(dumps)
        3    0.000    0.000    0.000    0.000 :0(get)
       10    0.000    0.000    0.000    0.000 :0(get_ident)
        5    0.000    0.000    0.000    0.000 :0(has_key)
        1    0.000    0.000    0.000    0.000 :0(index)
        4    0.000    0.000    0.000    0.000 :0(isinstance)
       14    0.000    0.000    0.000    0.000 :0(len)
        1    0.000    0.000    0.000    0.000 :0(max)
        2    0.000    0.000    0.000    0.000 :0(pack)
        1    0.007    0.007    0.007    0.007 :0(readlines)
        4    0.000    0.000    0.000    0.000 :0(release)
        2    0.000    0.000    0.000    0.000 :0(select)
        2    0.000    0.000    0.000    0.000 :0(send)
        1    0.002    0.002    0.002    0.002 :0(setprofile)
    20502    0.145    0.000    0.145    0.000 :0(split)
    20502    0.138    0.000    0.138    0.000 :0(strip)
        1    0.001    0.001    0.604    0.604 <string>:1(?)
        2    0.000    0.000    0.000    0.000 <string>:1(fileno)
        1    0.310    0.310    0.603    0.603 order_slab.py:5(order_shell_nodes)
        1    0.000    0.000    0.606    0.606 profile:0(order_shell_nodes(path_value))
        0    0.000             0.000          profile:0(profiler)
       14    0.000    0.000    0.000    0.000 rpc.py:149(debug)
        2    0.000    0.000    0.004    0.002 rpc.py:206(remotecall)
        2    0.000    0.000    0.000    0.000 rpc.py:216(asynccall)
        2    0.000    0.000    0.003    0.002 rpc.py:236(asyncreturn)
        2    0.000    0.000    0.000    0.000 rpc.py:242(decoderesponse)
        2    0.000    0.000    0.003    0.002 rpc.py:277(getresponse)
        2    0.000    0.000    0.000    0.000 rpc.py:285(_proxify)
        2    0.000    0.000    0.003    0.002 rpc.py:293(_getresponse)
        2    0.000    0.000    0.000    0.000 rpc.py:315(newseq)
        2    0.000    0.000    0.000    0.000 rpc.py:319(putmessage)
        3    0.000    0.000    0.000    0.000 rpc.py:545(__getattr__)
        2    0.000    0.000    0.000    0.000 rpc.py:584(__init__)
        2    0.000    0.000    0.004    0.002 rpc.py:589(__call__)
        2    0.000    0.000    0.000    0.000 threading.py:111(release)
        2    0.000    0.000    0.000    0.000 threading.py:126(_acquire_restore)
        2    0.000    0.000    0.000    0.000 threading.py:133(_release_save)
        2    0.000    0.000    0.000    0.000 threading.py:143(_is_owned)
        2    0.000    0.000    0.000    0.000 threading.py:147(Condition)
        2    0.000    0.000    0.000    0.000 threading.py:152(__init__)
        2    0.000    0.000    0.003    0.002 threading.py:195(wait)
        4    0.000    0.000    0.000    0.000 threading.py:39(__init__)
       10    0.000    0.000    0.000    0.000 threading.py:44(_note)
       10    0.000    0.000    0.000    0.000 threading.py:672(currentThread)
        2    0.000    0.000    0.000    0.000 threading.py:76(RLock)
        2    0.000    0.000    0.000    0.000 threading.py:81(__init__)
        2    0.000    0.000    0.000    0.000 threading.py:93(acquire)

I am still thinking about how to improve it further.

I defined several similar code to find a proper one. "order_slab_nodes(path_value)" is one of them. Please just ignore it.

I am looking for module psyco.

best wsihes

ning

Here is an example of module psyco:

# time a function using time.time()
# also uses Psyco JIT i386 compiler to speed up code 
# typical result: 1.6 sec with Psyco, 5.3 sec without
# for Python25 on Windows download psyco-1.6.win32-py25.exe
# from http://psyco.sourceforge.net/

import time

def print_timing(func):
    """
    create a timing decorator function
    for relatively slow functions
    use
    @print_timing 
    just above the function you want to time
    """
    def wrapper(*arg):
            t1 = time.time()
            res = func(*arg)
            t2 = time.time()
            fs = '%s took %0.3f ms'
            print( fs % (func.__name__, (t2-t1)*1000) )
            return res
    return wrapper

@print_timing
def primes(n):
    """ 
    returns a list of prime numbers from 2 to < n 
    using a sieve algorithm
    """
    if n < 2:  
        return []
    elif n == 2: 
        return [2]
    s = range(3, n+2, 2)
    mroot = n ** 0.5
    half = (n + 1)/2
    i = 0
    m = 3
    while m <= mroot:
        if s[i]:
            j = (m * m - 3)/2
            s[j] = 0
            while j < half:
                s[j] = 0
                j += m
        i = i + 1
        m = 2 * i + 3
    return [2]+[x for x in s if x]


if __name__ == '__main__':
    print("without psyco")
    prime_list = primes(10000000)

    print( '-'*30)

    # import Psyco JIT compiler/optimizer if available
    try:
        import psyco
        psyco.full()
        print("with psyco")
    except ImportError:
        print("module psyco not used ...")
    prime_list = primes(10000000)

"""
my output -->
without psyco
primes took 5326.000 ms
------------------------------
with psyco
primes took 1616.000 ms
"""

Hi, guys

Thank you for reading my simple code and in particular Ene Uran's further guide about psyco. However it does not work in my machine due to a unknown reason.

I am still struggling to improve my code, which is used to calculate the area of a planar polygon. Later on I will extedn it into a planar polygon in a space. Unfortunately the speed of my code makes it impossible to handle a realistic problem.

best wishes

ning

Hi, Guys

Just update my progress.

Following slate's words about programming, I checked my code and made three changes

1) use a better way to swap variables by replacing

x2_temp=x2[1]
    y2_temp=y2[1]

    x2[1]=x2[2]
    y2[1]=y2[2]
    x2[2]=x2_temp
    y2[2]=y2_temp

with

x2[1], x2[2] = x2[2], x2[1]
        y2[1], y2[2] = y2[2], y2[1]

2) delete strip() in the loop since it is doing nothing

3) put split() in IF

In other words, after my modification of 2) and 3), I got a new WHILE loop below:

#  fetch all coordinate values of these nodes
    while i < len(path_value):
        for line100 in alllines9:
            if str(path_value[i]) in line100:   
                line100=line100.split(',')
                x[i-1]=line100[1]
                y[i-1]=line100[2]
                break
        i+=1

Eventually, function call-in time of the code improves from 0.606 CPU seconds to 0.055, acoording profile modulus. It is a big step for me.

Now I am looking for a better way to locate a specified line in a text file, which has the same pattern (a label, and its coordinate values) as follows:

25125,17.82052,21.77562,3.4
25126,19.60312,22.79209,3.4
25127,20.29419,21.59513,3.4
25128,20.61147,23.38916,3.4
25129,21.30927,22.18054,3.4
25130,21.17482,25.06984,3.4

.

This text file is simple but has millions of lines.

best wishes

ning

If you can provide the external file loaded in the code, I will look into it.

Hi slate

Thank you so much for your kindness. Let me try again as I enjoy this learning process.

Have a nice day

Ning

You can download module scipy that has all sorts of highly speed optimized functions used in science. Take a short look at the tutorial for scipy at:
http://www.scipy.org/Wiki/Documentation?action=AttachFile&do=get&target=scipy_tutorial.pdf

Scipy needs another module called numpy, The downloads are free from:
http://www.scipy.org/Download

Hi, vegaseat

Thank you for providing the information. I know Scipy, although I never tried it before.

best wishes

ning

Hi, guys

Another update:

I just found error in the code

#  fetch all coordinate values of these nodes
    while i < len(path_value):
        for line100 in alllines9:
            if str(path_value[i]) in line100:   
                line100=line100.split(',')
                x[i-1]=line100[1]
                y[i-1]=line100[2]
                break
        i+=1

Becasue I use

str(path_value[i]) in line100

, it picks up wrong data. For example

if path_value=[786,2430,4931,939,4982], this code picks wrongly up

2430
line100: ['761', '14.24304', '22.07602', '-4\n']

as it matches "2430" in '14.24304'.

If I use RE, the speed of this code increases up to 1.95 CPU s, do you have any way to sort it out? Thank you a lot.

ning

Hi, there

I update my code here as I believe, as a Python beginner, I can get some guide from a lot of experts here.

At present, I add a littl patch into the WHILE loop as

while i < loop_limit:
        for line100 in alllines9:
            if str(path_value[i]) in line100:   
                line100=line100.split(',')
                if int(path_value[i]) == int(line100[0]):
                    x[i-1]=line100[1]
                    y[i-1]=line100[2]
                    break
        i+=1

It works now. However I am not sure how I can improve its speed. I am stduying weave modus of Scipy.

best wishes

ning

This article has been dead for over six months. Start a new discussion instead.