Member Avatar for dragonflyheli

Hello everyone,
I'm looking for some help with these simple tasks. I actually need this just for linguistic analysis, so I'm sorry for asking probably dumb questions. :)

There is a simple code that uses grep to find lines that contain a certain word in one file.

linecount=`grep "someword" $1/*file.txt | wc -l`
echo $linecount
wordcount=`grep "someword" $1/file.txt | cut -f2- | wc -w`
echo $wordcount
echo 'avg words per line:'
echo "scale=2; $wordcount / $linecount" | bc

What would be the simplest way to:
- find the maximum line length (in words)? wc -L should be probably used somehow?
- count the vocabulary size (simply number of different tokens) for all the found lines? I could only apply uniq -c to lines, not words

I really appreciate any help. many thanks in advance!

  • 2 Contributors
  • 2 Replies
  • 133 Views
  • 10 Hours Discussion Span
  • Latest Post Latest Post by dragonflyheli

Dani AI

Generated

Clarification for and response to : since the markup tokens uniquely identify one natural-language line, count matched lines (not all occurrences inside a line). For max line length in words use an awk scan that matches the pattern, tracks NF (number of fields) and computes max, total and average in one pass — this avoids separate grep + wc steps and handles multiple files in the shell glob ($1/*file.txt).

awk '/someword/ {
  if (NF > max) max = NF
  total += NF
  lines++
}
END {
  if (lines) printf "max_words=%d\nlines=%d\ntotal_words=%d\navg=%.2f\n", max, lines, total, total/lines
  else print "no matches"
}' "$1"/*file.txt

To count vocabulary size (unique tokens) across the matched lines, normalize and strip punctuation while collecting tokens into an associative array in awk, then count array keys. This stays in one pass and avoids sort -u for large sets:

awk '/someword/ {
  for (i=1;i<=NF;i++) {
    w = $i
    gsub(/^[^[:alnum:]]+|[^[:alnum:]]+$/, "", w)
    if (w != "") uniq[tolower(w)] = 1
  }
}
END {
  c = 0
  for (w in uniq) c++
  print c
}' "$1"/*file.txt

Notes and troubleshooting: wc -L measures characters, not words. If lines start with an identifier you need to skip (you used cut earlier), set -F'\t' and process fields from $2 onward or strip the leading field inside awk. For robust Unicode tokenization (diacritics, scripts, word joins) use a Unicode-aware tool (Perl/Python tokenizers). awk details (tolower, associative arrays) are documented in the gawk manual.

Recommended Answers

All 2 Replies

Member Avatar for Salem

If you have a line
"their therapist is over there"

and you're searching "the", what would you want
0 - nothing matches the actual word "the"
1 - the line contains "the" somewhere
3 - there are three places where "the" appears

Member Avatar for dragonflyheli

Actually the words being searched belong to technical markup, each identifying unambiguously one line in natural language, which needs to be parsed; so in this case generally possible ambiguities in search can be ignored.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.