Member Avatar for Felulah

Hi everybody,

I'm new at Python and this forum has been a great help so far, so I hope someone is willing to help me on the following problem.
It can count the number of sentences and words in a text, but after I have split up a text into sentences, I do not know how to count the words in each sentence separately so I can calculate the standard deviation.

Grts

  • 3 Contributors
  • 3 Replies
  • 2K Views
  • 6 Days Discussion Span
  • Latest Post Latest Post by Felulah

Dani AI

Generated

If your goal is per-sentence word counts (for a mean and standard deviation) and then the SD of type-token ratio (TTR), a small tokenizer is enough. Building on ’s idea to use a regex and ’s point about sentence boundaries, decide on simple rules first: treat ., ?, ! as sentence terminators and count words as sequences of letters, optionally with an apostrophe (it’s, don’t). That keeps the math stable and the code short.

Here is a self-contained example. It returns per-sentence word counts, their mean and sample SD, and the mean and sample SD of TTR computed sentence-by-sentence. Use statistics.pstdev instead if you want population SD.

import re
import statistics as stats

SENT_SPLIT = re.compile(r'(?<=[.!?])\s+')
WORD_RE = re.compile(r"[A-Za-z]+(?:'[A-Za-z]+)?")

def split_sentences(text):
    return [s for s in SENT_SPLIT.split(text.strip()) if s]

def words(sentence):
    return WORD_RE.findall(sentence)

def per_sentence_word_counts(text):
    return [len(words(s)) for s in split_sentences(text)]

def ttr_per_sentence(text):
    ttrs = []
    for s in split_sentences(text):
        ws = [w.lower() for w in words(s)]
        if ws:
            ttrs.append(len(set(ws)) / len(ws))
    return ttrs

def summary(text):
    counts = per_sentence_word_counts(text)
    ttrs = ttr_per_sentence(text)

    wc_mean = stats.mean(counts) if counts else 0.0
    wc_sd = stats.stdev(counts) if len(counts) > 1 else 0.0

    ttr_mean = stats.mean(ttrs) if ttrs else 0.0
    ttr_sd = stats.stdev(ttrs) if len(ttrs) > 1 else 0.0

    return {"counts": counts, "wc_mean": wc_mean, "wc_sd": wc_sd,
            "ttr_mean": ttr_mean, "ttr_sd": ttr_sd}

Notes:

  • Normalize to lowercase before computing TTR so "The" and "the" match.
  • Hyphens and numbers: if you want to count them as words, extend WORD_RE. For example, add - or \d thoughtfully.
  • Abbreviations (e.g., "Dr.") can split naively; if that matters, swap in an NLP sentence tokenizer later.

Recommended Answers

All 3 Replies

Member Avatar for Gribouillis

You can use a regex for words

import re
word_re = re.compile(r"[A-Za-z]+")

def count_words(sentence):
    return word_re.subn('', sentence)[1]

print(count_words("Give me bacon and eggs, said the other man."))
Member Avatar for vegaseat

To extract the sentences of a text into a list, you have to establish some rules. A simple rule may be that all sentences end with one of these characters '.' or '?' or '!'

Now you can extract the text's sentences, for a typical example see:
http://www.daniweb.com/forums/showthread.php?p=1175950#post1175950

Member Avatar for Felulah

Thanks for your help! I have been able to calculate the average word and sentence length and their standard deviation, but now I am stuck on calculating the sd of the type-token ratio :-s

Grts

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.