Member Avatar for doomas10

Hello all,

How are you? Hope well. Just a quick question. I have a file which contains abstracts (small texts) and i am looking for certain keywords and their frequency. These keywords are provided from another file. I was thinking to read the keyword file first and the perform a scan on the abstract file. However, if i find a keyword, i would like to find the word before and after as well. Any thoughts of how i can do that?

example

Keyword_file looks like this:

George
myself

abstract_file looks like this:

hello, my name is george. How are you? Today
i am not feeling very well. I consider myself to be
sick.

so i want to find the words 'george' and 'myself' as well as 'is','how' and 'consider', 'to'.

Any suggestions? :?:

  • 3 Contributors
  • 10 Replies
  • 317 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by doomas10

Dani AI

Generated

A compact, production-ready alternative that fills the gaps in the thread: stream the abstract file as a token stream (no full-file read), keep a small sliding window for previous/current/next words, and collect counts with Counters. This handles punctuation more flexibly than naive split, gives the keyword frequencies requested, and reports the most common words appearing immediately before and after each keyword. It builds on 's tokenization idea and 's findall tip while avoiding per-match file I/O.

from collections import Counter
import re

token_re = re.compile(r"[A-Za-z0-9'-]+")   # simple tokens including apostrophes/hyphens

def words(path):
    with open(path, 'r') as fh:
        for line in fh:
            for t in token_re.findall(line.lower()):
                yield t

keywords = set(open('singleheads.txt').read().lower().split())

before = Counter()
after  = Counter()
occurs = Counter()

it = words('abex.txt')
prev = None
cur  = next(it, None)
nxt  = next(it, None)

while cur is not None:
    if cur in keywords:
        occurs[cur] += 1
        before[prev or '<START>'] += 1
        after[nxt or '<END>'] += 1
    prev, cur, nxt = cur, nxt, next(it, None)

print('keyword counts:', occurs.most_common())
print('common-before:', before.most_common(10))
print('common-after:', after.most_common(10))

Notes and troubleshooting: use a set for keyword lookup (O(1) membership). Tokenization is a trade-off—the simple regex above works for English-like text but may need Unicode-aware patterns or an NLP tokenizer for other languages, contractions, or complex punctuation. For multi-word keywords or many patterns, consider a multi-pattern search (Aho–Corasick) instead of per-token checks. Preserve sentence boundaries if whole-sentence context is required (sentence tokenizers or simple split on [.?!] can help). Also, avoid opening/writing the output file inside the match loop (open once, write results at the end) to reduce system-call overhead.

Recommended Answers

All 10 Replies

Member Avatar for Gribouillis

You can iterate over the words, or build a list, with a regular expression (the re module)

import re

abstract = """hello, my name is george. How are you? Today
i am not feeling very well. I consider myself to be
sick."""

word_pattern = re.compile(r"\w+")

print list(word_pattern.findall(abstract))

""" my output -->
['hello', 'my', 'name', 'is', 'george', 'How', 'are', 'you', 'Today', 'i', 'am', 'not', 'feeling', 'very', 'well', 'I', 'consider', 'myself', 'to', 'be', 'sick']
"""
Edited by Gribouillis because: n/a
Member Avatar for doomas10

i try the list and i think within that area a solution can be found.however, i do not know how to search for the words after and before. for example:

filename='abex.txt' 
wordlist=re.split('\s+', file(filename).read().lower())
print 'words in text:', len(wordlist) 
print wordlist
print ""

filename2='singleheads.txt'
wordlist2=re.split('\s+', file(filename2).read().lower())
print 'words in the single head file:', len(wordlist2)
print wordlist2

    
for word in wordlist2:
    if word in wordlist:
       print word

you see this code prints only the common ones which is super. But i can not understand how can i ask the algorithm to fetch the words before and after the keywords. is there a function that can do that?:?:

Member Avatar for Gribouillis

You can use enumerate. Supposing you have 2 lists, keywords and word_list,

keyword_set = set(keywords) # better use a set
for i, w in enumerate(word_list):
    if w in keyword_set:
        word_before = word_list[i-1] if i > 0 else ''
        word_after = word_list[i+1] if i+1 < len(word_list) else ''
        print("%s <%s> %s" % (word_before, w, word_after))
Edited by Gribouillis because: n/a
Member Avatar for doomas10

thanks! it worked awesome! but here is another problem. you see with this code

filename='abex.txt' 
wordlist=re.split('\s+', file(filename).read().lower()) #ta kanei lowercase
filename2='singleheads.txt'
wordlist2=re.split('\s+', file(filename2).read().lower())

punctuation=re.compile(r'[.?!,":;]')   #remove the punctuation
for word in wordlist:
    word=punctuation.sub("",word) 

keyword_set = set(wordlist2)
for i,w in enumerate(wordlist): #it gives to the list items numbers
    if w in keyword_set:
        before_word = wordlist[i-1] if i > 0 else ''
        after_word = wordlist[i+1] if i+1 < len(wordlist) else ''
        print "%s <%s> %s" % (before_word,w,after_word)

it brings nice results! but words that may have with them a .?!" are excluded since the keyword list has "clean" words. i tried to fix it with

punctuation=re.compile(r'[.?!,":;]')   #remove the punctuation
for word in wordlist:
    word=punctuation.sub("",word)

with three lines and it seems to work fine. but i can not make a connection between the new wordlist(the clean one now) and set. Keep hitting my head on the wall but i can not find a way. Something says it is going to be very very simple!:sweat:

Member Avatar for Gribouillis

Forget about punctuation: build the words list as I did in my first post above, using the regex r'\w+'.

Member Avatar for doomas10

i did and it works fine! thank you very much for your help :-):cool: for the record this is my final algorithm:

import re
terms='singleheads.txt'
wordlist=re.split('\s+', file(terms).read().lower())

abstract=open('abex.txt','r')
abstract2=abstract.read().lower() 
abstract3=str(file2)

word_pattern = re.compile(r"\w+")
doom=list(word_pattern.findall(abstract3))
print doom
print ""

keyword_set = set(wordlist)
for i,w in enumerate(doom): #it gives to the list items numbers
    if w in keyword_set:
        before_word = doom[i-1] if i > 0 else ''
        after_word = doom[i+1] if i+1 < len(doom) else ''
        print "%s <%s> %s" % (before_word,w,after_word)
        sephiroth=open('staib.txt','a')
        sephiroth.write(str(before_word)+ " "+ "<" + str(w) + ">" + str(after_word) + "\n")
        sephiroth.close()

thanks for the help!

Member Avatar for Gribouillis

Nice, but you should not open 'staib.txt' for each iteration. Open it before the loop starts and close it after the loop. Opening a file is an expensive system call.

Member Avatar for doomas10

ok :-) thanks for the tip :-))

Member Avatar for snippsat

print list(word_pattern.findall(abstract))

Just a tips.
re.findall is returning a list,so there is not necessary to use list().

import re

text = '''\
hello, my name is george. How are you? Today
i am not feeling very well. I consider myself to be
sick.
'''

word_pattern = re.findall(r'\w+', text)
print word_pattern

""" Out-->
['hello', 'my', 'name', 'is', 'george', 'How', 'are', 'you', 'Today', 'i', 'am', 'not', 'feeling', 'very', 'well', 'I', 'consider', 'myself', 'to', 'be', 'sick']
"""
Edited by snippsat because: n/a
commented: indeed ! +4
Member Avatar for doomas10

thanks snippsat :-)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.