Member Avatar for farhaana.ahmed.58

i have a text document 'topics.txt':

1~cocoa

2~

3~

4~

5~grain~wheat~corn~barley~oat~sorghum

6~veg-oil~linseed~lin-oil~soy-oil~sun-oil~soybean~oilseed~corn~sunseed~grain~sorghum~wheat

7~

8~

9~earn

10~acq

and so on..
here the numbers correspond to the file names, i have about 20000 files.

    import os
    import re
    import sys
    sys.stdout=open('f1.txt','w')
    from collections import Counter
    from glob import glob

    def removegarbage(text):
        text=re.sub(r'\W+',' ',text)
        text=text.lower()
        return text

    folderpath='d:/individual-articles'
    counter=Counter()


    filepaths = glob(os.path.join(folderpath,'*.txt'))

    num_files = len(filepaths)

    with open('topics.txt','r') as filehandle:
        lines = filehandle.read()
        words = removegarbage(lines).split()
       counter.update(words)


    for word, count in counter.most_common():
        probability=count//num_files
        print('{}  {} {}'.format(word,count,probability))

i need my output to be of the form:
word,count of word in topics.txt,probability, list of files,number of files in list,

so far my program works fine till probability, but how do i get a list of the files belonging to a word?
ex: 'grain' must contain the list(5,6,....)

would counting the line number for each word and storing the line number work?

how do i go about it?
Plz help!

Edited by farhaana.ahmed.58
  • 2 Contributors
  • 1 Reply
  • 259 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by chriswelborn

Dani AI

Generated

Quick summary and a few concrete fixes for the approach already in the thread: the topics file is best parsed line-by-line (keep the leading file id), split on ~ to get that id and the tokens, then build two structures: (a) a per-word occurrence count and (b) a per-word set of file ids. That preserves the exact file numbers (don’t rely on line-number) and avoids duplicate file ids when the same word appears more than once on one line. : calling a wholesale removegarbage() over the whole file before splitting will discard the file-id mapping; parse lines first. ’s idea to iterate lines and map words to file lists is right — the improvements below make it more robust and efficient.

Key points to implement

  • Use defaultdict(set) for file lists (dedupes automatically) and Counter or a defaultdict(int) for raw counts.
  • Compute probability with float division; for Python 2 be explicit (e.g. count / float(num_files)) so you don’t get integer-floor results.
  • Decide which denominator you want: total number of article files (len(glob(...))) or number of unique file ids seen in topics.txt — use whichever matches your intent.
  • At output time convert sets to sorted lists and write CSV (or text) with columns: word, total_count, probability, file_list, num_files_in_list.

Example implementation (concise, original)

from collections import defaultdict, Counter
import csv, os, glob

files_by_word = defaultdict(set)
counts = Counter()

with open('topics.txt') as fh:
    for line in fh:
        parts = line.strip().split('~')
        if not parts or not parts[0].strip(): continue
        fid = int(parts[0].strip())
        for tok in parts[1:]:
            w = tok.strip().lower()
            if not w: continue
            counts[w] += 1
            files_by_word[w].add(fid)

num_files = len(glob.glob(os.path.join('d:/individual-articles','*.txt')))
with open('word_report.csv','w',newline='') as out:
    w = csv.writer(out)
    w.writerow(['word','count','probability','file_list','num_files_in_list'])
    for word, cnt in counts.items():
        flist = sorted(files_by_word[word])
        prob = cnt / float(num_files)
        w.writerow([word, cnt, round(prob,6), ','.join(map(str,flist)), len(flist)])

Troubleshooting and extras: if you need the fraction of files containing the word (not occurrences), use len(files_by_word[word]) / float(num_files). If the topics vocabulary is huge or memory is constrained, persist with sqlite or write intermediate results to disk. Normalize tokens carefully (decide whether to keep hyphens like veg-oil), and always verify num_files against the set of file ids found in topics.txt so your probability denominator is correct.

Member Avatar for chriswelborn

I'm not sure of a real efficient way of doing this. Someone else might. But by iterating over each line in your test data I was able to build a dictionary with each word and a list of file numbers as the values, like {"grain":["5", "6"], "soybean": ["6"], ... }. Running this over that many items may not be the best idea, but its a start. The other thing I thought of was restructuring the file to better suit your needs.

wordfiles = {}
# basically cycling through each line,
# maybe you could integrate it into whatever processing you
# are already doing.
for line in testdata.split('\n'):
    # just bypassing any junk/empty lines
    if line.strip(' ').strip('\t').strip('\n') != "":
        items = line.strip(' ').strip('\t').strip('\n').split('~')

        if len(items) > 0:
            fileno=items[0]
            # grab each word from this line
            # empty entries don't count.
            words = [w for w in items[1:] if w]
            for word in words:
                if wordfiles.has_key(word):
                    # key already created, add this fileno to the list
                    wordfiles[word].append(fileno)
                else:
                    # key doesn't exist, create a list value
                    wordfiles[word] = [fileno]

# Check which files "grain" is in.
print "grain is in " + str(len(wordfiles["grain"])) + " files:"
print "File numbers:\n" + "\n    ".join(wordfiles["grain"])
Edited by chriswelborn because: doh! didn't see it solved.
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.