In line editing without creating a new file

[boshu]

Hi all,

Once again I would like to seek your help.
I got one file (test.hrh) with a number of flags. Some falgs are defined (#define), some not (#undef).

I got another file which a flag list which should be added to the file if not there, and if exists then need to check the status and change if needed.

#test.hrh file content:

#ifndef __BLDSERVICESSOFTWARE_HRH
#define __BLDSERVICESSOFTWARE_HRH

/** #FF_TOPCONTACTSWIDGET_COMPONENT */
#undef FF_TOPCONTACTSWIDGET_COMPONENT                      

/** #FF_VISUAL_RADIO_COMPONENT */
#undef FF_VISUAL_RADIO_COMPONENT                           

/** #FF_WEB_SEARCH_COMPONENT  */
#undef FF_WEB_SEARCH_COMPONENT                             

/** #__MOBILESEARCH_CLASSIC_IDLE */
#define __MOBILESEARCH_CLASSIC_IDLE                         

#endif  // __BLDSERVICESSOFTWARE_HRH

List of flags:
===============
1. FF_HERE_AND_NOW_COMPONENT 
2. FF_WEB_SEARCH_COMPONENT

Note:
1 should be added to the file as #define FF_HERE_AND_NOW_COMPONENT
2 already exists but as #undef, should be changed to #define FF_WEB_SEARCH_COMPONENT

I implemented it as:
Create a new file (test.new) and parsed all the test.hrh contents there. While parsin I checked whether the flag exists, if not then added the whole string like "#define FLAG". If it was defined and everthing is right then doing nothing. Else change the status from "undef FLAG" to "#define FLAG".

Finally deleted the "test.hrh" file and renamed "test.new" to tet.hrh.

Can it be done other way? Without creating a new file?

[d5e5]

Read about edit-in-place. I don't have time right now to customise but here is an old example I've used before for editing a file, saving a backup, and the edited file automatically replaces the original.

#!/usr/bin/perl
use strict;
use warnings;

my $filename = '/home/david/Programming/Perl/data/var.txt';

#So you can use diamond operator and inplace editing
push @ARGV, $filename;

#Enable inplace editing. Copy of input file saved as 'var.txt.bk'
$^I = ".bk"; 

my $replaced = 0;

while(<>){
    chomp;
    if ($replaced == 0
        and s/the line to be replaced/this is the new line/){
        $replaced++;
    }
    print "$_\n";
}

[boshu]

Hi David
Thanks for the guidance and hint.
I could not make it work. I must have missed something in the article.
I tried another way which is:

use warnings;
use strict;


my @files = "test.hrh";
my @cf = ("FF_HERE_AND_NOW_COMPONENT", "FF_HERE_AND_NOW_ADD_IT");
my $replaced = 0;

foreach my $inFile ( @files ) {
    chomp($inFile);

    print "PROCESSING: $inFile \n";

    open( OLD , "<$inFile")       or die "cant open ${inFile}: $! \n";
    open( HAND , ">${inFile}.new") or die "cant open ${inFile}.new \n";

    
    foreach my $cf (@cf)
   	{
   		print "$cf\n";
    		while(<OLD>) {
   			chomp($cf);
    		
    		if ((/^#.*\s+(\w+)/) && ($replaced == 0))
    		{
    			my $flag = $1;
    			#print "$flag\n";
    			if ($flag eq $cf)
    			{
    				print HAND "#define $cf\n";
    				$replaced++;
    			}
    			else
    			{
    				print HAND "#define $flag\n";
    			}
    				
				}
    		else
    		{
    			print HAND $_;
    		}

    		}
  	}
    close(HAND);
    close(OLD);
    rename("${inFile}.new",${inFile}) or die "cant rename ${inFile}.new \n";

}

But this only changes the #undef part for existing flag, cannot add the new one.

I shall try again.
Boshu

[erezschatz]

Hi all,

Create a new file (test.new) and parsed all the test.hrh contents there. While parsin I checked whether the flag exists, if not then added the whole string like "#define FLAG". If it was defined and everthing is right then doing nothing. Else change the status from "undef FLAG" to "#define FLAG".

Finally deleted the "test.hrh" file and renamed "test.new" to tet.hrh.

Can it be done other way? Without creating a new file?

Create an array, and save each line you parse as an array item. then write the array to the file:

print $FILEHANDLE join "\n" @array;

[boshu]

Getting back to David,
Can multiple strings be replaced using the inline edit option?
Lets says the flags are store in an array?

Any help?

[d5e5]

Getting back to David,
Can multiple strings be replaced using the inline edit option?
Lets says the flags are store in an array?

Any help?

Yes. Inplace (AKA inline) editing of a file opens whatever filename it finds in @ARGV, lets you read it any way you want (one line at a time, all lines at once into an array, or slurping entire file into a string -- it doesn't matter which) and lets you print the contents back to the same file after you make any changes that you want.

The trick with editing a file inplace is to push the filename into the @ARGV array and then read from the empty <> operator and just print the edited contents without specifying a filehandle. In effect inplace editing just renames or moves the original file to a backup file and creates a new file with the original file name which you print data to by print statements that do NOT specify a filehandle.

[boshu]

Is that for editing only?
How about adding strings, lines if required?

[d5e5]

Is that for editing only?
How about adding strings, lines if required?

Print whatever you want to the file. As I said, all inplace edit does is rename the original file and let you read it, then whatever you print goes into a new file that has the original file name. That's all. If you do an inplace edit of a file and don't print anything, the result is an empty file. Or you can edit a file containing 5 lines, replace those lines (by printing 5 different lines), or delete them (by not printing them) or add 100 new lines (by printing 100 lines of whatever you want).

[boshu]

Thanks Dave.
This could be very handy and would solve my problem in few lines I guess, - ONLY if I could use it properly.

[boshu]

Ok.
One more question?
How can I loop through the flags (in an array) while inside the Diamond operator. That would print the lines many times?

Could you please give an example?

[d5e5]

Take for example the test.hrh file you posted at the beginning of this thread. The following tries to read it all into an array (and for some reason closes it, although I didn't want to close it), then loops through the array of flags to check and modifies the array, then prints the array to test.hrh. Because it is an inplace edit, it creates a backup file named test.hrh.bk.

#!/usr/bin/perl
use strict;
use warnings;

my %found; #Keep track of flags found in file;

#The following is a list of flags to add or change
my @check = qw(FF_HERE_AND_NOW_COMPONENT FF_WEB_SEARCH_COMPONENT);

my $filename = 'test.hrh';
#So you can use diamond operator and inplace editing
push @ARGV, $filename;
#Enable inplace editing. Copy of input file saved as 'test.hrh.bk'
$^I = ".bk";

my @file = <>; #Read all records into array
chomp(@file);

foreach my $chk(@check){
    foreach my $line(@file){
        if ($line =~ s/^#(define|undef)(\s$chk)/#define $2/){
            $found{$chk}++;
        }
    }
}

push @file, "\n\n#Here's the tricky part.\n#Let's try to add the flags not found\n";
foreach my $chk(@check){
    push @file, "#define $chk" if not exists $found{$chk};
}

#I didn't know that reading all records of the file would close it,
#but it did... so I have to open it for output.
open ARGVOUT, '>', $filename or die "$!";#Open empty file of same name for output
print ARGVOUT join "\n", @file;
close ARGVOUT;

[boshu]

Most appreciated David.
The logic looks simple and practical.
Works for me just like I wanted.

One more thing. The file I am trying to edit inline has an ending line and all added lines must be added before that flag (#endif // __BLDSERVICESSOFTWARE_HRH).

How can I get around with that?

Right now its adding after, like:

#endif // __BLDSERVICESSOFTWARE_HRH

#Here's the tricky part.
#Let's try to add the flags not found

Thanks again and hope to learn more from YOU guys.

[d5e5]

Some of the programmers at StackOverflow have told me that when doing an inplace edit, reading all the lines into an array all at once is bad, because the file gets closed before I get a chance to print to it. Please try the following, hopefully it works better:

#!/usr/bin/perl
use strict;
use warnings;

my %found; #Keep track of flags found in file;

my $filename = 'test.hrh';
#So you can use diamond operator and inplace editing
push @ARGV, $filename;
#Enable inplace editing. Copy of input file saved as 'test.hrh.bk'
$^I = ".bk";

my $end_flag = '#endif  // __BLDSERVICESSOFTWARE_HRH';
my $end_flag_regex = qr($end_flag);

#Read file into array line by line until line matches end flag
my @file = ();
while (<>) {
    chomp;
    
    if (m/$end_flag_regex/) {
        # edit and print lines
        my @processed_file = edit_lines(@file);
        push @processed_file, $_;
        # last chance to print before ARGVOUT gets reset
        print join "\n", @processed_file;
        @file = ();
    }
    else
    {
        push @file, $_;
    }
}

sub edit_lines{
    my @f = @_;

    #The following is a list of flags to add or change
    my @check = qw(FF_HERE_AND_NOW_COMPONENT FF_WEB_SEARCH_COMPONENT);
    
    foreach my $chk(@check){
        foreach my $line(@f){
            if ($line =~ s/^#(define|undef)(\s$chk)/#define $2/){
                $found{$chk}++;
            }
        }
    }
    
    push @f, "#Add the flags not found\n";
    foreach my $chk(@check){
        push @f, "#define $chk" if not exists $found{$chk};
    }
    return @f;
}

[boshu]

Thanks David.
I shall try this solution shortly.
I have encountered one problem when trying the previous version.

I changed the following line

#my $filename = 'test.hrh';
my $filename = $ARGV[0];

And then called the script with "test.hrh" as a parameter like C:\test\inline_replace.pl test.hrh

It give the following error:
C:\test>inline_replace.pl test.hrh
Can't do inplace edit on test.hrh: File exists at C:\test\inline_replace.pl line
17, <> line 146.

C:\test>

Do you happen to know why its giving me this error?

-Boshu

[k_manimuthu]

## Code block 1 
my $filename = 'test.hrh';
#So you can use diamond operator and inplace editing
push @ARGV, $filename;

Here $fliename add into defalut array (@ARGV).

inline_replace.pl test.hrh

Here the file name directly assigned to @ARGV. you want to pass the file name in command line option, you no need to keep the 'code block 1' in your program.

[boshu]

But the file name is coming from another place as an argument as $ARGV[0];

[d5e5]

But the file name is coming from another place as an argument as $ARGV[0];

Muthu is correct. If @ARGV contains at least one element (no matter where it comes from) then trying to copy the value of $ARGV[0] and pushing it back into @ARGV will mess up what Perl is trying to do. If you want to run the script with the filename argument already in @ARGV then modify it as follows:

#!/usr/bin/perl
use strict;
use warnings;

my %found; #Keep track of flags found in file;

if (@ARGV == 0){
    my $filename = 'test.hrh';
    #So you can use diamond operator and inplace editing
    push @ARGV, $filename;
    warn "No file name argument so I assume you want to edit $filename.\n"
}

#Enable inplace editing. Copy of input file saved as 'test.hrh.bk'
$^I = ".bk";
#...
#...
#...

[boshu]

I changed the code as you suggested but the problem persists,
Here is the changed script:

#!/usr/bin/perl
use strict;
use warnings;

my %found; #Keep track of flags found in file;

my $filename = $ARGV[0];

if (@ARGV == 0){
    my $filename = 'test.hrh';
    #So you can use diamond operator and inplace editing
    push @ARGV, $filename;
    warn "No file name argument so I assume you want to edit $filename.\n"
}


#The following is a list of flags to add or change
my @check = qw(FF_HERE_AND_NOW_addit FF_WEB_SEARCH_COMPONENT);

#So you can use diamond operator and inplace editing
push @ARGV, $filename;
#Enable inplace editing. Copy of input file saved as 'test.hrh.bk'
$^I = ".bk";

my @file = <>; #Read all records into array

chomp(@file);


foreach my $chk(@check){
    foreach my $line(@file){
        if ($line =~ s/^#(define|undef)(\s+$chk)/#define $2/){
            $found{$chk}++;
        }
    }
}

#push @file, "\n\n#Here's the tricky part.\n#Let's try to add the flags not found\n";
foreach my $chk(@check){
    push @file, "#define $chk" if not exists $found{$chk};
}

#I didn't know that reading all records of the file would close it,
#but it did... so I have to open it for output.
open ARGVOUT, '>', $filename or die "$!";#Open empty file of same name for output
print ARGVOUT join "\n", @file;
close ARGVOUT;

and when I run without any argument:
H:\test>d5d5.pl
No file name argument so I assume you want to edit test.hrh.
Use of uninitialized value in <HANDLE> at H:\test\d5d5.pl line 26, <> line 2.
Can't open : No such file or directory at H:\test\d5d5.pl line 26, <> line 2.
Use of uninitialized value in open at H:\test\d5d5.pl line 46, <> line 2.
No such file or directory at H:\test\d5d5.pl line 46, <> line 2.

H:\test>

With the filename (test.hrh) in argument:
H:\test>d5d5.pl test.hrh
Can't do inplace edit on test.hrh: File exists at H:\test\d5d5.pl line 26, <> line 2.

H:\test>d5d5.pl

[d5e5]

Sorry, I should have sent you the entire script. My point was that copying a file name from $ARGV[0] and pushing it back into @ARGV is wrong. The result is you have two elements in @ARGV for the same file name so the inplace edit gets all messed up when it tries to rename the same file and reopen the same file twice. Don't do that unless you are NOT passing the file name on the command line as an argument. Pushing a file name into @ARGV is just a trick to use inplace editing without passing an argument on the command line. Here is the script that works for me:

#!/usr/bin/perl
use strict;
use warnings;

my %found; #Keep track of flags found in file;

if (@ARGV == 0){
    my $filename = 'test.hrh';
    #So you can use diamond operator and inplace editing
    push @ARGV, $filename;
    warn "No file name argument so I assume you want to edit $filename.\n"
}

#Enable inplace editing. Copy of input file saved as 'test.hrh.bk'
$^I = ".bk";

my $end_flag = '#endif  // __BLDSERVICESSOFTWARE_HRH';
my $end_flag_regex = qr($end_flag);

#Read file into array line by line until line matches end flag
my @file = ();
while (<>) {
    chomp;
    
    if (m/$end_flag_regex/) {
        # edit and print lines
        my @processed_file = edit_lines(@file);
        push @processed_file, $_;
        # last chance to print before ARGVOUT gets reset
        print join "\n", @processed_file;
        @file = ();
    }
    else
    {
        push @file, $_;
    }
}

sub edit_lines{
    my @f = @_;

    #The following is a list of flags to add or change
    my @check = qw(FF_HERE_AND_NOW_COMPONENT FF_WEB_SEARCH_COMPONENT);
    
    foreach my $chk(@check){
        foreach my $line(@f){
            if ($line =~ s/^#(define|undef)(\s$chk)/#define $2/){
                $found{$chk}++;
            }
        }
    }
    
    push @f, "#Add the flags not found\n";
    foreach my $chk(@check){
        push @f, "#define $chk" if not exists $found{$chk};
    }
    return @f;
}

[boshu]

Thanks David

This script now works perfectly.
However, I noticed that there is no way I could use this by calling from an Ant target, with arguments.

For example, I want to call the script from within an Ant target:

<target name="update-flag">
	<!--update flags-->
   <exec executable="perl">
	<!-- Script that updates the flags -->
	<arg value="${common.tools}/update_carbon_flags.pl"/>
	<!-- Name of the file which should be updated -->
	<arg value="${build.drive/variant.confml"/>
	<!-- Directory wherefrom the flags should be parsed -->
	<arg value="${ccm.project.wa_path}"/>
    </exec>
</target>

But as you mentioned, the script will take whatever arguments it is fed with and process those as file arguments - I guess it cannot be used in the above scenario?

[d5e5]

Thanks David

This script now works perfectly.
However, I noticed that there is no way I could use this by calling from an Ant target, with arguments.

For example, I want to call the script from within an Ant target:

<target name="update-flag">
	<!--update flags-->
   <exec executable="perl">
	<!-- Script that updates the flags -->
	<arg value="${common.tools}/update_carbon_flags.pl"/>
	<!-- Name of the file which should be updated -->
	<arg value="${build.drive/variant.confml"/>
	<!-- Directory wherefrom the flags should be parsed -->
	<arg value="${ccm.project.wa_path}"/>
    </exec>
</target>

But as you mentioned, the script will take whatever arguments it is fed with and process those as file arguments - I guess it cannot be used in the above scenario?

I don't have Ant or Java and have never used them for development so I don't know whether inplace editing will work when called from an Ant target. I guess it depends on what the arguments look like when Ant adds them. How about running a simple test script like the following to see if Ant can call a Perl script and to see the format and values of the arguments that Ant passes. Try calling the following script from your Ant target:

#!/usr/bin/perl
use strict;
use warnings;
use 5.010;

# I add !!!! so we can see if any spaces, etc.
say "Ant called $0 with the following arguments:";
say join "!!!!\n", @ARGV;

[boshu]

Thanks Dave,

I shall call this one through Ant call.
In fact I do run lots of other Perl scripts as Ant call and it works.

I'll get back to you on the result.

Thanks.
Boshu

[boshu]

First I created the following target to run the script from Ant

<target name="run-perl-script">
		<exec executable="perl">
		<arg value="${common.tools}/call_perl_script.pl"/>
		</exec>
		</target>

Then called the target:

E:\Build_E\running\idos\ssdo\branches\mcl-zipsci>hlm -Dbuild.drive=X: -Dbuild.number=1 -Dbuild.system=sbs-ec run-perl-script

Output on screen:

Buildfile: build.xml
     
run-perl-script:
     [exec] Perl v5.10.0 required--this is only v5.6.1, stopped at E:/BUILD_E/running/idos/ssdo/common_tools/call_perl_script.pl line 4.
     [exec] BEGIN failed--compilation aborted at E:/BUILD_E/running/idos/ssdo/common_tools/call_perl_script.pl line 4.
     [exec] Result: 255

My system runs on Perl 5.6.1

[d5e5]

First I created the following target to run the script from Ant

<target name="run-perl-script">
		<exec executable="perl">
		<arg value="${common.tools}/call_perl_script.pl"/>
		</exec>
		</target>

Then called the target:

E:\Build_E\running\idos\ssdo\branches\mcl-zipsci>hlm -Dbuild.drive=X: -Dbuild.number=1 -Dbuild.system=sbs-ec run-perl-script

Output on screen:

Buildfile: build.xml
     
run-perl-script:
     [exec] Perl v5.10.0 required--this is only v5.6.1, stopped at E:/BUILD_E/running/idos/ssdo/common_tools/call_perl_script.pl line 4.
     [exec] BEGIN failed--compilation aborted at E:/BUILD_E/running/idos/ssdo/common_tools/call_perl_script.pl line 4.
     [exec] Result: 255

My system runs on Perl 5.6.1

Please try running the following modified test script from an Ant target:

#!/usr/bin/perl
use strict;
use warnings;
use 5.006_001;

use Cwd;
my $cwd = getcwd();
# I add !!!! so we can see if any spaces, etc.
print "\nCurrent working directory is: $cwd\n";
print "Ant called $0 with the following arguments:\n";
print join "!!!!\n", @ARGV, "\n";

I'm wondering if the directory in which Ant runs the script is different than the one from which you started Perl when testing the script from the command line?

[boshu]

Hi Dave

They are not the same.

[d5e5]

Hi Dave

They are not the same.

It seems to me that the argument containing the file name you want to edit will need to include the correct absolute path. I see in the original Ant target that you posted you had a line like this: <arg value="${build.drive/variant.confml"/> Is that correct? Try adding that to the Ant target that runs the simple test script in my previous post so it can print out the value of the argument(s) and see if the script receives the correct path and file name for the file name that the real script would edit.

[boshu]

Hi David

All these variables ${common.tools},${build.drive},${ccm.project.wa_path} are predefined variables which are loaded before this particular script is called.

Here the Perl script is expected to accept 2 arguments, one is the file to be edited and the other one is a directory name wherefrom it gets all the flags.

Is there any way that I can have two arguments in the Script, as follows:

$filename = $ARGV[0];
$dir = $ARGV[1];

The use of $ARGV[1]; in the script will be following:

#!/usr/bin/perl
use File::Copy;

my $arg1 = $ARGV[0];
my $dir =  $ARGV[1];

opendir (DIR, "$dir/") or die "Couldn't open directory, $!";
my @FILES = grep(/.txt/,readdir(DIR));
closedir (DIR);

# read uda_flags
foreach my $file (@FILES) {
		open(INFO,"$dir\\$file");
    while (<INFO>){
    	if ($_ =~ /^Carbon_flags\:(.*)/i){  
				push (@carbon_flags, "$1");    		 
			}                                
		}
  }
close INFO;

Please note: this @carbon_flags is equivalent to "qw(FF_HERE_AND_NOW_COMPONENT FF_WEB_SEARCH_COMPONENT);" what we have been using in the example earlier.

So, in practice the script is supposed to do 2 things:
1. Collect Carbon flags from the text files underneath a particulr folder (which is $ARGV[1]).
2. Edit and apend a file (which is passed as $ARGV[0]) against the Carbon flags

-Boshu

[d5e5]

Save and remove the $ARGV[1] argument from the @ARGV array (using the pop function) before enabling inplace editing. Then inplace editing can proceed because the only argument remaining in @ARGV is the name of the file you want to edit.

#!/usr/bin/perl
use strict;
use warnings;

my %found; #Keep track of flags found in file;

if (@ARGV != 2) { 
      die "Usage: $0 filename dir\n"; 
}

my $carbon_flags_dir = pop; #Remove $ARGV[1] from @ARGV and assign to variable
print "The folder containing text files of carbon flags is: $carbon_flags_dir\n";
#
#Do your opendir, read text files, push flags into your @carbon_flags here
#before enabling inplace editing
#

#Enable inplace editing. Copy of input file saved as 'test.hrh.bk'
$^I = ".bk";

my $end_flag = '#endif  // __BLDSERVICESSOFTWARE_HRH';
my $end_flag_regex = qr($end_flag);

#Read file into array line by line until line matches end flag
my @file = ();
while (<>) {
    chomp;
    
    if (m/$end_flag_regex/) {
        # edit and print lines
        my @processed_file = edit_lines(@file);
        push @processed_file, $_;
        # last chance to print before ARGVOUT gets reset
        print join "\n", @processed_file;
        @file = ();
    }
    else
    {
        push @file, $_;
    }
}

sub edit_lines{
    my @f = @_;

    #The following is a list of flags to add or change
    my @check = qw(FF_HERE_AND_NOW_COMPONENT FF_WEB_SEARCH_COMPONENT);
    
    foreach my $chk(@check){
        foreach my $line(@f){
            if ($line =~ s/^#(define|undef)(\s$chk)/#define $2/){
                $found{$chk}++;
            }
        }
    }
    
    push @f, "#Add the flags not found\n";
    foreach my $chk(@check){
        push @f, "#define $chk" if not exists $found{$chk};
    }
    return @f;
}

[k_manimuthu]

<target name="run-perl-script">
	<exec executable="perl">
	<arg value="${common.tools}/call_perl_script.pl"/>
	</exec>
</target>

I don't know the Ant script. As your previous reply I understand the 'call_perl_script.pl' will be executed. Better you have passed the file and directory for 'arg' tag. You will be modify the below line and the try again your previous reply code.

<arg value="${common.tools}/call_perl_script.pl FILE_NAME DIR_NAME"/>

So FILE_NAME treated as '$ARGV[0]' DIR_NAME treated as '$ARGV[1]'.