Member Avatar for Swetha_1
Swetha_1 0 Newbie Poster

for example: consider the contents of file1.txt:

1 0 9227 1152 34 2
2 111 7622 1120 34 2
3 68486 710 1024 14 2
6 265065 3389 800 22 2
7 393152 48438 64 132 3
8 412251 46744 64 132 3
9 430593 50866 256 95 4
10 430730 10770 256 95 4
11 433750 12701 256 14 3
12 437926 2794 64 34 2
13 440070 43 32 96 3
14 440102 44 32 96 3
15 440357 43 32 96 3
16 440545 43 32 96 3
17 440599 43 32 96 3
18 440625 43 32 96 3
19 440999 84 32 96 0
20 441574 44 32 96 3
which the contains n jobs with 6 fields(i,e column(0-5))

Now, for example, I take the first 19 jobs as history. Then I need to start reading from the 20th and so on comparing the columns 3,4,5 which matches with the above jobs in the history.For the first recent occurrence i,e 18 440625 43 32 96 3.I need to check the condition if col1 of the 20th job is greater than the (col1 * 1.1) of the first recent occurrence i,e 18th if true it add the col2 to a list and should read the next recent occurrence i,e 17th job and so on until the if condition fails.Then need to add the 20th line to history making the history count of 21 there by updating the history and so on till the end of the file is reached......... Can any one suggest a code in python by which its possible for me to read the 20st line and compare with the above history and continue to 21,22,23------------------ until the end of the file is reached......

  • 1 Contributor
  • 0 Replies
  • 116 Views

Dani AI

Generated

Per 's description: treat the first N rows as a history window, then for each incoming row find prior rows whose columns 3,4,5 match the incoming row; walk those prior matches from newest to oldest and, for each, check whether the incoming row's compare-column is greater than the prior row's value times 1.1. Collect the prior row's add-column into a list until that check fails, then add the incoming row to history and continue. The implementation below is parameterized so you can change which columns are matched, which column is compared, which column is appended, the multiplier, and the initial history size.

def process_file(path, history_count=19, match_indices=(3,4,5),
                 compare_index=1, add_index=2, threshold=1.1, delimiter=None):
    def split(line):
        parts = line.strip().split() if delimiter is None else line.strip().split(delimiter)
        return parts if parts else None

    index_by_key = {}
    results = []

    with open(path, 'r') as fh:
        # load initial history_count non-empty lines
        loaded = 0
        while loaded < history_count:
            line = fh.readline()
            if not line:
                break
            parts = split(line)
            if not parts:
                continue
            key = tuple(parts[i] if i < len(parts) else None for i in match_indices)
            index_by_key.setdefault(key, []).append(parts)
            loaded += 1

        # process remaining lines
        for line in fh:
            parts = split(line)
            if not parts:
                continue
            key = tuple(parts[i] if i < len(parts) else None for i in match_indices)
            matches = []
            try:
                new_val = float(parts[compare_index])
            except (IndexError, ValueError):
                new_val = None

            if new_val is not None and key in index_by_key:
                for prior in reversed(index_by_key[key]):
                    try:
                        prior_val = float(prior[compare_index])
                    except (IndexError, ValueError):
                        break
                    if new_val > prior_val * threshold:
                        matches.append(prior[add_index] if add_index < len(prior) else None)
                    else:
                        break

            results.append(matches)
            index_by_key.setdefault(key, []).append(parts)

    return results

Notes and troubleshooting:

  • All column indices in the function are 0-based. Adjust compare_index and add_index to match your meaning of "col1" and "col2".
  • If the file has a header or non-numeric tokens, skip the header or use the csv module and adapt parsing. The function tries to convert the compare column to float and will skip if conversion fails.
  • For very large files use a sliding fixed-size history (collections.deque) and remove the oldest entry from index_by_key when it falls out of the window to bound memory.
  • The dict index_by_key gives O(k) work per line where k is number of prior matches for that key; this is much faster than scanning the whole history each time.
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.