Member Avatar for koveras vehcna

Hello everyone, I am currently working on text processing with Python and I want to parse a .txt file into its sentences as a whole. I tried to create some regular expressions but I failed to do so. I only managed to come up with a regex that splits each paragraph into its sentences. How can I split the text into sentences ? -Also, if a sentence is long, does not end in the same paragraph, it should be printed as a whole too- Thanks.

  • 3 Contributors
  • 2 Replies
  • 1K Views
  • 14 Hours Discussion Span
  • Latest Post Latest Post by TrustyTony

Dani AI

Generated

Short answer: treat the file as one text for sentence splitting (dont rely on paragraph boundaries), normalize "soft" line breaks, then use a real sentence-boundary detector instead of blind regex. Splitting on a dot (as suggested) breaks on abbreviations, decimals and ellipses; s itertools trick is clever but effectively reimplements SBD and will still be brittle for real text.

A simple, practical pipeline you can apply right away:

  1. read the whole file (correct encoding),
  2. collapse single line breaks into spaces so wrapped lines become one sentence,
  3. run a sentence tokenizer (not a single regex). Example idea:
import re
from nltk.tokenize import sent_tokenize

text = open('input.txt', encoding='utf-8').read()
# replace single newlines (soft wraps) with a space; keep blank lines if you want real paragraph breaks
text = re.sub(r'(?<!\n)\n(?!\n)', ' ', text)
sentences = sent_tokenize(text)
for s in sentences:
    print(s)

For robust results use a library: NLTK's Punkt-based sent_tokenize handles common abbreviations and can be trained on domain text; it requires the punkt model. (nltk.org)

If you need higher accuracy in messy or domain-specific corpora, try spaCy (dependency-parser or its Sentencizer component) or train/customize your tokenizer and abbreviation lists. Also watch for hyphenation at line ends and manual markup — normalize those before tokenizing and test interactively on representative text. (spacy.io)

Recommended Answers

All 2 Replies

Member Avatar for woooee

Split on the period. Note that the following doesn't make sense/requires an example.

Also, if a sentence is long, does not end in the same paragraph, it should be printed as a whole too

Edited by woooee because: n/a
Member Avatar for TrustyTony

Here is example, but English messy way of "Quoting sentence." should be fixed to make this work:

import itertools as it
endsentence = ".?!"
filein = 'd:/test/advsh12.txt'
sentences = it.groupby(open(filein).read(),
                       lambda x: any(x.endswith(punct)
                                     for punct in endsentence))
for number,(truth, sentence) in  enumerate(sentences):
    if truth:
        print number//2+1,':',previous+''.join(sentence).replace('\n',' ')
    previous = ''.join(sentence)
    if number>=2*100: break ## 100 first sentences

We use itertools groupby to separate the sentence,punctuation,sentence,punctuation pairs and join them in pairs when reading punctuation (truth is True). We check only end of words not to stop at 1.23 for example.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.