Member Avatar for farawaydsky

Hi,

I have a requirement where I have to identify duplicates from a file based on the first 6 chars (It is fixed width file of 12 chars length) and whenever a duplicate row is found, its original and duplicate row's last 2 chars should be updated to all 0's if they are not same. (I mean last 2 digits of original and duplicate row should be same, if not then default to 00 else keep them as is)

I can use multiple loops and get the results but I would need something which will be faster

here is the sample input and output

input:
1251233Y1234
1221249N8821
1231116Y9945
1231113Y2123
1231109Y3212
1231123N1214
1231126N1214

output should be:
1251233Y1234
1221249N8821
1231116Y9900
1231113Y2100
1231109N3212
1231123N1214
1231126N1214 (Since last 2 digits are same nothing changed)

Any help in achieving the above result using either awk/sed will be greatly appreciated.

Thanks,
Faraway

  • 2 Contributors
  • 2 Replies
  • 131 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by L7Sqr

Dani AI

Generated

A concise, memory-efficient awk approach that preserves the original line order and only zeroes the last two characters for groups (same first 6 chars) that contain differing last-two values. This keeps the logic in one awk invocation (read/store/check, then emit), avoiding nested loops over the file. was right that higher-level languages can be easy, but awk does this cleanly with substr and associative arrays (see GNU awk string functions and arrays for details: String functions and Arrays).

Algorithm (brief): read each line, extract key = first 6 chars and tail = last 2 chars; store the whole line and key indexed by record number; remember the first tail seen per key and mark a flag if a different tail is found; in the END block print stored lines, replacing the last two characters with "00" for keys that were flagged. This preserves order and runs in linear time; memory usage equals the file size because lines are stored.

Example awk implementation (save as a one-liner or script):

awk '
{
  n = length($0)
  k = substr($0,1,6)
  t = (n >= 12 ? substr($0,11,2) : (n >= 2 ? substr($0,n-1,2) : ""))
  rec[NR] = $0
  key[NR] = k
  if (!(k in first)) first[k] = t
  else if (first[k] != t) diff[k] = 1
}
END {
  for (i = 1; i <= NR; i++) {
    line = rec[i]
    if (diff[key[i]]) {
      prefix = (length(line) >= 10 ? substr(line,1,10) : line)
      print prefix "00"
    } else
      print line
  }
}
' infile > outfile

Troubleshooting notes: handle CRLF input by running tr -d "\r" if needed; if records can be shorter than 12 chars the script uses safe fallbacks; for very large files that cannot fit in memory, use a sort+stream or a two-pass solution to avoid holding all lines in RAM (or use a streaming language like Perl/Python as suggested).

Recommended Answers

All 2 Replies

Member Avatar for L7Sqr

Are you required to use awk/sed? This would be much easier if you could employ something like perl/ruby/python.
In general, the loop body would look something like:

if first_line
    previous_line = current_line
    continue
end if

saved_current_line = current_line

if previous_line[0..6] == current_line[0..6]
    previous_line[-2..-1] = "00"
    current_line[-2..-1] = "00"
end if

previous_line = saved_current_line

output previous_line
output current_line

Of course, this completely ignores the case when you have an odd number of sequential lines with a matching prefix - you'd have to add logic in to handle that.

Member Avatar for L7Sqr

Whoops. The last part of that should output before reassigning to previous_line as well as checking for the last line instead of blindly printing current_line at each loop iteration.

So something like:

output previous_line

previous_line = saved_current_line

if last_line
    output current_line
end if
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.