this time I have only one textfile and I want to remove all duplicate lines so that each and every single line in it is unique unto itself.

with open("textfile") as w:
 for line in w:
  W = line
  with open("textfile") as d:
   c = 0
   for line in d:
    D = line
    if W.strip() == D.strip(): c = c + 1
    if c == >1:
     #how can I delete D?
     #and will after deleting it, does this mean that I can make sure that my W won't skip it?

Edited 3 Years Ago by nouth

my oldtext.txt
"Which language do you like?
I like python programming.
Which language do you like?
I like python programming.
I like python programming."

oldFile = open("oldtext.txt")
newFile = open("newtext.txt", "w")
text = []
for line in oldFile:
    if line not in text:
        text.append(line)
        newFile.write(line)
newFile.close()

my newtext.txt
"Which language do you like?
I like python programming."

>>> ================================ RESTART ================================
>>> 
Which language do you like?

I like python programming.

If the order of lines does not matter, you can use

  with open('filelein.txt') as infile,  open('fileout.txt', 'w') as outfile:
           outfile.write(''.join(set(infile))) 

Edited 3 Years Ago by pyTony

I want to do it in such a way as to so that I can understand exactly what is happening.

with open(file) as f:
    for line in f:
        if i < 1:
            i = i + 1
        else:
            #I want to remove line from f

how can I delete line from f within for line in f?

okay, I got distracted with this

oldFile = open("oldtext.txt")
newFile = open("newtext.txt", "w")
text = []
for line in oldFile:
    if line not in text:
        text.append(line)
        newFile.write(line)
newFile.close()

I cannot understand what text = [] does and how it can work with if line not in text:
but can I change it to this?

if line not in text:
    text.append(line)
    oldFile.write(line)

or maybe I read something somewhere from about saving it to temporary files, so maybe newFile can be temporary and then it can be used with to reoverwrite the oldFile? no not like append

with open("textfile") as w:
    for line in w:
        W = line
        with open("textfile") as d:
            c = 0
            for line in d:
                D = line
                if W.strip() == D.strip(): c = c + 1
                if c == >1:
                    #I want to delete line in d
#but this is definitely the way that I want to do it, notice I am still opening the same "textfile" twice and then making a new variable for it's line

Edited 3 Years Ago by nouth

so maybe newFile can be temporary and then it can be used with to reoverwrite the oldFile?

Yes this is normal way,you iterate over contend and do what you need,then save contend in a new file.
If you need newtext to be oldtext you rename the files.
os.rename(oldtext','newtext') or shutil.move('oldtext','newtext')

Python has fileinput that can do inplace edit of orginal file.
It's ok,but the method over with rename is just as good.
If order doesn't matter,soultion by pyTony is good.

'''lines.txt-->
line 1
line 2
line 2
line 3
'''

import fileinput
import sys

def remove_dup(fname):
    text = []
    for line in fileinput.input(fname, inplace=True):
        if line not in text:
            text.append(line)
            sys.stdout.write(line)

fname = 'lines.txt'
remove_dup(fname)

'''lines.txt-->
line 1
line 2
line 3
'''

Edited 3 Years Ago by snippsat

I have learnt what text = [] does now
but I still have some trouble understanding this and how it works outfile.write(''.join(set(infile))) from pyTony

I haven't used import yet but can understand that for line in fileinput.input(fname, inplace=True): and sys.stdout.write(line) have something to do with import fileinput and import sys

but my next question I will ask will have to be about this

"Write to outfile joined lines of set of items (lines) in file infile"
Set of values has only one instance of same item.

Comments
thanks :)
This question has already been answered. Start a new discussion instead.