EXCEL CSV FILES Handling

[jorgeflorencio]

...

Hello everyone i have some csv files that are the result of the inspection machine...

these CSV files have the following layout

,SRFF File: D:\SPI Master Program List\200-34-02\200-34-02-B-01-R.SRF
,Panel Name: Panel Description
,Units: Microns

,Date,Time,PanelId,Board,Location,Part,Package,HeightAvgResult,HeightAvg,HeightAvgUpFail,HeightAvgLowFail,HeightAvgTarget,HeightRangeResult,HeightRange,HeightRangeMax,AreaAvgResult,AreaAvg,AreaAvgUpFail,AreaAvgLowFail,AreaAvgTarget,AreaRangeResult,AreaRange,AreaRangeMax,VolumeAvgResult,VolumeAvg,VolumeAvgUpFail,VolumeAvgLowFail,VolumeAvgTarget,VolumeRangeResult,VolumeRange,VolumeRangeMax,XOffset,YOffset,Rotation,Scaling,HAVFailedFeatureResult,HAVFailedFeatures,HAVFailedFeatureMax,RegFailedFeatureResult,RegFailedFeatures,RegFailedFeatureMax,BridgeFailedFeatureResult,BridgeFailedFeatures,BridgeFailedFeatureMax

,07/19/2008,00:48:55,#280,Module 1 ,IC2,TSOP8,SOP8_1,F,138.539642,238.000000,84.000000,140.000000,P,9.226400,140.000000,P,427747.781250,729933.140625,243311.046875,486622.090000,P,26717.400391,486622.093750,P,60986780.000000,122628772.800000,34063548.000000,68127092.600000,P,3087930.000000,68127096.000000,-22.886625,-23.317499,-0.039985,-0.131595,P,0,1,P,0,1,P,0,1
,07/19/2008,00:48:55,#280,Module 1 ,C38,0402,0402_1_3,P,131.077194,238.000000,84.000000,140.000000,P,12.636800,140.000000,P,265426.750000,470565.468750,156855.156250,313710.320000,P,5929.500000,313710.312500,P,36161180.000000,79054999.200000,21959722.000000,43919444.800000,P,2340260.000000,43919444.000000,-26.636499,-28.285000,-0.201710,-0.635564,P,0,1,P,0,1,P,0,1
,07/19/2008,00:48:55,#280,Module 1 ,C66,0402,0402_1_3,P,146.934555,238.000000,84.000000,140.000000,P,4.062300,140.000000,P,266860.500000,470565.468750,156855.156250,313710.320000,P,15511.000000,313710.312500,P,40048568.000000,79054999.200000,21959722.000000,43919444.800000,P,235260.000000,43919444.000000,-32.004501,-21.340000,-0.163571,-1.147347,P,0,1,P,0,1,P,0,1

so i need some help how to create a structure in order to maintain the values related to each column

Date,Time,PanelId,Board,Location,Part,Package,HeightAvgResult,HeightAvg,HeightAvgUpFail,HeightAvgLowFail,HeightAvgTarget,HeightRangeResult,HeightRange,HeightRangeMax,AreaAvgResult,AreaAvg,AreaAvgUpFail,AreaAvgLowFail,AreaAvgTarget,AreaRangeResult,AreaRange,AreaRangeMax,VolumeAvgResult,VolumeAvg,VolumeAvgUpFail,VolumeAvgLowFail,VolumeAvgTarget,VolumeRangeResult,VolumeRange,VolumeRangeMax,XOffset,YOffset,Rotation,Scaling,HAVFailedFeatureResult,HAVFailedFeatures,HAVFailedFeatureMax,RegFailedFeatureResult,RegFailedFeatures,RegFailedFeatureMax,BridgeFailedFeatureResult,BridgeFailedFeatures,BridgeFailedFeatureMax

in order then to export the data that have Fails - F - and reject the ones that have Passed - P - with the respective column header...

please give me some tips
[/TEX]

[Alex Edwards]

If you need a line to be read from a particular value (like a date), you could do one of many things--

-(Hard) create a regular-expression to match dates (in MM/DD/YYYY format) and separate lines once a new date has been analyzed.
-(Easy) create a method that takes a String argument and determines if it is a date by checking for the appropriate characters in a date-format String literal.

--I'd store each line in an ArrayList<String>.

From there I'd probably tokenize each comma separated value and for the particular line then store it in a <String, ArrayList<String> > Map.

Go to my profile, then code snippets and look at "Search Engine" to get an idea of why you'd do this.

You can easily change the separation from a space to a comma in my code snippet.

From there you can use "p" or "f" keys to locate all Modules that passed or failed.

Since the ones of interest are ones that failed, simply retrieve the ArrayList<String> associated with the key "f", and write the information to a file or do what you want with it.

Hopefully this post was helpful.

Edit: If you want to do this an easier way it may help to invest time in a Swing Application and implement a JTable with the columns and rows of the information then have some kind of iterator to retrieve particular rows of information of interest based on a value of a column/row (basically use a double array or an ArrayList of ArrayList<String> values to store the information relative to the JTable and use actionListeners to update the ArrayLists when an update to the JTable has been made.)

[jorgeflorencio]

If you need a line to be read from a particular value (like a date), you could do one of many things--

-(Hard) create a regular-expression to match dates (in MM/DD/YYYY format) and separate lines once a new date has been analyzed.
-(Easy) create a method that takes a String argument and determines if it is a date by checking for the appropriate characters in a date-format String literal.

--I'd store each line in an ArrayList<String>.

From there I'd probably tokenize each comma separated value and for the particular line then store it in a <String, ArrayList<String> > Map.

Go to my profile, then code snippets and look at "Search Engine" to get an idea of why you'd do this.

You can easily change the separation from a space to a comma in my code snippet.

From there you can use "p" or "f" keys to locate all Modules that passed or failed.

Since the ones of interest are ones that failed, simply retrieve the ArrayList<String> associated with the key "f", and write the information to a file or do what you want with it.

Hopefully this post was helpful.

Edit: If you want to do this an easier way it may help to invest time in a Swing Application and implement a JTable with the columns and rows of the information then have some kind of iterator to retrieve particular rows of information of interest based on a value of a column/row (basically use a double array or an ArrayList of ArrayList<String> values to store the information relative to the JTable and use actionListeners to update the ArrayLists when an update to the JTable has been made.)

Thank you for the tips...
I'm planning to do this... however i'm rusty again...what i have done so far was to count the number of F =fails that each csv file had... and assign to the name of the file and colect data regarding the date of creation of the file... i'm going to have a look on your sample.. i hope that u don't mind... by the way,,,, can you give a small sample how to

" Since the ones of interest are ones that failed, simply retrieve the ArrayList<String> associated with the key "f", and write the information to a file or do what you want with it."

its not so easy to me... i understand if you wont reply to this request... i'm learning!!

thanks by the tips

[Alex Edwards]

Ok I'd like to apologize - I didn't realize your values extended so far!

What exactly deems a unit as a failure? I see many "passes" values in each row. What is the condition for a unit to be a "fail?"

[jorgeflorencio]

You don't need to apologize ... i was not so explicit...

In fact what i need to colect is... lets say that i have a faillure... this failure is based in the cross analysis of the values provided... lets say that i have a faillure that is located in
"HeightAvgResult,HeightAvg,HeightAvgUpFail,HeightAvgLowFail"

F,238.539642,238.000000,84.000000,140.000000

what i need is only the value that is in front of the F.. in this case 238.539642 and what kind of faillure that is related to .. in this case... HeightAvgResult...

So the info will appear:
Module 1 ,IC2,TSOP8,SOP8_1,Height,F,138.539642...

i hope that you can understand what i mean.. thanks once again

[Alex Edwards]

There are a few ways we can do this.

In my opinion, the best way to do this is create a class that accepts these kind of values and stores them in appropriate data types (the types listed in each column in the first row (Date, time... etc), so it would be wise to use Strings).

For now we'll have 3 classes (maybe more) and have them hold all of the data we need.

When you read the lines from the file you'll store them in the classes the same way I mentioned before.

The classes toString method will be overridden to display all of the information for that particular line of data.

Now this may or may not complicate things but you can write methods that return if a particular column is a fail or not.

Now you can just use an iterator (or array of these classes) where each has different information and when a fail comes up upon invocation of the classes's method and it returns true you can display the toString method of the class that has the information you need.

This is probably the simplest way to do it. I can provide an example but it'll take time to sift through the different value per column.

[jorgeflorencio]

Thank u for the tips.. i will try to find examples in the Web in order to guid me... because i'm not so good..

the reason i'v choose only the fails was that sometimes i have 1500 files with i don't how many lines... so the data that i need is basically the faillures,,, F, so this will allow me to make my output smaller and provide the correct data regarding the failures...

So... the idea will be if the line has a faillures i will collect the data regarding that line split the line and place the needed data in "Classes" when you mean classe you are refering to a txt file ??
regarding the array... i'm not so good with controling arrays... i suck...i know... but slowly i can get there if well guided.. i want to learn....

[Alex Edwards]

I've provided a class that should help you with this project. Examine it closely--

public class InspectionResults{

	public static final byte  HEIGHT_AVG_RESULT = 0,
							  HEIGHT_RANGE_RESULT = 1,
							  AREA_AVG_RESULT = 2,
							  AREA_RANGE_RESULT = 3,
							  VOLUME_AVG_RESULT = 4,
							  VOLUME_RANGE_RESULT = 5,
							  HAV_FAILED_FEATURE_RESULT = 6,
							  REG_FAILED_FEATURE_RESULT = 7,
							  BRIDGE_FAILED_FEATURE_RESULT = 8;
	private String retrievedData[];
	private boolean failed[];

	/**
	 * Constructs this InspectionResult with the data stored in the args.
	 * This class expects 44 values within the range of the args.
	 */
	public InspectionResults(String... args){
		retrievedData = args;
		boolean temp[] ={
			((retrievedData[7].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[12].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[15].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[20].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[23].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[28].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[35].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[38].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[41].equalsIgnoreCase("F")) ? true: false)
		};
		failed = temp;
	}

	/**
	 * Returns true if the given value has failed, returns false otherwise.
	 * It's preferred to use the constants defined within this class to get the
	 * desired information, and not regular ints.
	 */
	public boolean hasFailed(byte result) throws Exception{
		if(result >= 0 && result < failed.length)
			return failed[result];
		else{
			throw new Exception("Attempt to access invalid result type! Use the Result Constants to avoid this error!");
		}
	}

	/**
	 * Returns the value next to the specified result.
	 */
	public String getAdjacentValue(byte result) throws Exception{
		if(result >= 0 && result < retrievedData.length - 1)
			return retrievedData[result + 1];
		else throw new Exception("Error! Attempt to access column with either no adjacent value or outside of data-range!");
	}

	/**
	 * Simply returns a String representing the data for each value in this class.
	 */
	@Override
	public String toString(){
		String temp = "";
		for(String element : retrievedData){
			if(element.toString() != retrievedData[retrievedData.length - 1])
				temp += element + ", ";
			else temp += element;
		}
		return temp;
	}
}

If you can manage to retrieve each line of information from the file and store it in a new object of this class you should be able to eliminate a world of trouble.

I took the time to look through which values had a P or F next to it. The Constructor of this class is expecting an Array of Strings, so you will need to use a tokenizer for each line you examine to tokenize a line of value separated by commas.

If you want to turn a line of information into an Array you can use the
String.split(",") command (where String is replaced by the actual value that contains the String-value of the line of information you retrieve), to turn your line into an array of Strings. From there you can create a new object of this class that will handle the comparisons for you (checking if a particular value is P or F) and also enables you to retrieve the Adjacent value (which actually needs to be edited - more constants are needed to get the appropriate adjacent values OR you could just enter the numbers of the column to get the adjacent values which may be annoying but I forgot to implement that).

Hopefully this helps you out.

[jorgeflorencio]

Uau.. you are fast.. i will give a look and tryout with my program... i have evrything done regarding access the folders read/access the csv files, count the number of ocurrences... store the data in the txt file and also send the data to a DB.... this is what is missing i will try and tomorrow i will give you the feedback.... thank you once again... C u

[jorgeflorencio]

regarding
"so you will need to use a tokenizer for each line you examine to tokenize a line of value separated by commas."

I have that as well...

[Alex Edwards]

regarding
"so you will need to use a tokenizer for each line you examine to tokenize a line of value separated by commas."

I have that as well...

Then all you need to do is store the tokenized values in an array and use that array to create an object of the class listed above (remember, it's expecting an array of Strings of length 44 (from indices 0-43)).

[Alex Edwards]

I used this as a reference, it might make the code clearer in case it isn't already--

,SRFF File: D:\SPI Master Program List\200-34-02\200-34-02-B-01-R.SRF
,Panel Name: Panel Description
,Units: Microns

,Date,
Time,
PanelId,
Board,
Location,
Part,
Package,
HeightAvgResult, // 7
HeightAvg,
HeightAvgUpFail,
HeightAvgLowFail,
HeightAvgTarget,
HeightRangeResult, // 12
HeightRange,
HeightRangeMax,
AreaAvgResult, // 15
AreaAvg,
AreaAvgUpFail,
AreaAvgLowFail,
AreaAvgTarget,
AreaRangeResult, // 20
AreaRange,
AreaRangeMax,
VolumeAvgResult, // 23
VolumeAvg,
VolumeAvgUpFail,
VolumeAvgLowFail,
VolumeAvgTarget,
VolumeRangeResult, // 28
VolumeRange,
VolumeRangeMax,
XOffset,
YOffset,
Rotation,
Scaling,
HAVFailedFeatureResult, // 35
HAVFailedFeatures,
HAVFailedFeatureMax,
RegFailedFeatureResult, // 38
RegFailedFeatures,
RegFailedFeatureMax,
BridgeFailedFeatureResult, // 41
BridgeFailedFeatures,
BridgeFailedFeatureMax



,07/19/2008,
00:48:55,
#280,
Module 1 ,
IC2,
TSOP8,
SOP8_1,
F, // 7
138.539642,
238.000000,
84.000000,
140.000000,
P, // 12
9.226400,
140.000000,
P, // 15
427747.781250,
729933.140625,
243311.046875,
486622.090000,
P, // 20
26717.400391,
486622.093750,
P, // 23
60986780.000000,
122628772.800000,
34063548.000000,
68127092.600000,
P, // 28
3087930.000000,
68127096.000000,
-22.886625,
-23.317499,
-0.039985,
-0.131595,
P, // 35
0,
1,
P, // 38
0,
1,
P, // 41
0,
1

,07/19/2008,00:48:55,#280,Module 1 ,C38,0402,0402_1_3,P,131.077194,238.000000,84.000000,140.000000,P,12.636800,140.000000,P,265426.750000,470565.468750,156855.156250,313710.320000,P,5929.500000,313710.312500,P,36161180.000000,79054999.200000,21959722.000000,43919444.800000,P,2340260.000000,43919444.000000,-26.636499,-28.285000,-0.201710,-0.635564,P,0,1,P,0,1,P,0,1
,07/19/2008,00:48:55,#280,Module 1 ,C66,0402,0402_1_3,P,146.934555,238.000000,84.000000,140.000000,P,4.062300,140.000000,P,266860.500000,470565.468750,156855.156250,313710.320000,P,15511.000000,313710.312500,P,40048568.000000,79054999.200000,21959722.000000,43919444.800000,P,235260.000000,43919444.000000,-32.004501,-21.340000,-0.163571,-1.147347,P,0,1,P,0,1,P,0,1

[jorgeflorencio]

what i have similar to split is...

String st = record;

                String []splitSt = st.split(",");

                System.out.println("string : " + st );
                System.out.println("length :" + splitSt.length );
                int i;

                //Connecting to the database
                //String URL =
                //"jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=E:\\public\\Log\\Results\\spimon.mdb;}",
                String HOST = "jdbc:odbc:spimon",
                DRIVER = "sun.jdbc.odbc.JdbcOdbcDriver";
                Class.forName(DRIVER)/*.newInstance()*/;
                myConn = DriverManager.getConnection(HOST,"","");
                //Connection myConn = DriverManager.getConnection(URL,"","");

                //using the database 
                Statement stat = myConn.createStatement();
                PreparedStatement updaterecord;
                String query="INSERT INTO logspi01 VALUES(?,?,?,?,?,?,?)"; //tem que ter o mesmo numero de campos da DB
                updaterecord = myConn.prepareStatement(query);


                for (i = 0; i < splitSt.length; i++ )
                {
                    //updaterecord.setString ((i+1), splitSt[i]); /*or*/

                    updaterecord.setString((1),splitSt[0]); //machine
                    updaterecord.setString((2),splitSt[1]); //package
                    updaterecord.setString((3)," ");        //lotnumber
                    updaterecord.setString((4),splitSt[2]); //Fail Location
                    updaterecord.setString((5),splitSt[3]); //Fail Qty
                    updaterecord.setString((6),splitSt[4]); //Date
                    updaterecord.setString((7),splitSt[5]); //Time
                }
                updaterecord.executeUpdate();
                //int x = updaterecord.executeUpdate();
                stat.close();
                myConn.close();
                //myConn = null;
                //System.out.println("  Inserting Data " + x + " row(s)");

[Alex Edwards]

Here's and updated version of the class I gave you - should allow you to use the constants to retrieve data relative to pass-fail fields more leniently.

public class InspectionResults{

	public static final byte  HEIGHT_AVG_RESULT = 7,
							  HEIGHT_RANGE_RESULT = 12,
							  AREA_AVG_RESULT = 15,
							  AREA_RANGE_RESULT = 20,
							  VOLUME_AVG_RESULT = 23,
							  VOLUME_RANGE_RESULT = 28,
							  HAV_FAILED_FEATURE_RESULT = 35,
							  REG_FAILED_FEATURE_RESULT = 38,
							  BRIDGE_FAILED_FEATURE_RESULT = 41;
	private String retrievedData[];
	private boolean failed[];

	/**
	 * Constructs this InspectionResult with the data stored in the args.
	 * This class expects 44 values within the range of the args.
	 */
	public InspectionResults(String... args){
		retrievedData = args;
		boolean temp[] ={
			((retrievedData[7].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[12].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[15].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[20].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[23].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[28].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[35].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[38].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[41].equalsIgnoreCase("F")) ? true: false)
		};
		failed = temp;
	}

	/**
	 * Returns true if the given value has failed, returns false otherwise.
	 * It's preferred to use the constants defined within this class to get the
	 * desired information, and not regular ints.
	 */
	public boolean hasFailed(byte result) throws Exception{
		switch(result){
			case HEIGHT_AVG_RESULT:
				return failed[0];
			case HEIGHT_RANGE_RESULT:
				return failed[1];
			case AREA_AVG_RESULT:
				return failed[2];
			case AREA_RANGE_RESULT:
				return failed[3];
			case VOLUME_AVG_RESULT:
				return failed[4];
			case VOLUME_RANGE_RESULT:
				return failed[5];
			case HAV_FAILED_FEATURE_RESULT:
				return failed[6];
			case REG_FAILED_FEATURE_RESULT:
				return failed[7];
			case BRIDGE_FAILED_FEATURE_RESULT:
				return failed[8];
			default :
				throw new Exception("Attempt to access invalid result type! Use the Result Constants to avoid this error!");
		}
	}

	/**
	 * Returns the value next to the specified result.
	 */
	public String getAdjacentValue(byte result) throws Exception{
		if(result >= 0 && result < retrievedData.length - 1)
			return retrievedData[result + 1];
		else throw new Exception("Error! Attempt to access column with either no adjacent value or outside of data-range!");
	}

	/**
	 * Simply returns a String representing the date for each value in this class.
	 */
	@Override
	public String toString(){
		String temp = "";
		for(String element : retrievedData){
			if(element.toString() != retrievedData[retrievedData.length - 1])
				temp += element + ", ";
			else temp += element;
		}
		return temp;
	}
}

[jorgeflorencio]

:(

I was reading your code... i'm really lousy...

thats why i need to know how should i use your code sorry....:( My code is

private class Start extends Task<Void, Void> {
	@Override protected Void doInBackground() throws InterruptedException {
			                         
			try {
							
				////Redirecting outputFrame = new Redirecting(false, false, null,500, 300, JFrame.HIDE_ON_CLOSE);
					
				/*... Initialize array of indentations */
				INDENTS[0] = INDENT_STR;
				for (int i = 1; i < MAX_DEPTH; i++) {
					INDENTS[i] = INDENTS[i-1] + INDENT_STR;
				}

				/*... Initialize read files.... */ 
				File Folder = new File("E:\\public\\Log\\Log");
				File FolderLD = new File("E:\\public\\Log\\LotData");
				File HostFile = new File("E:\\public\\Log\\SPIlog.txt");
				File HostFileLD = new File("E:\\public\\Log\\SPIlotData.txt");
				File LotInput = new File ("z:\\DataLot\\");
				
				File file = null;       
				File[] files = null;       
				FileReader fr = null;       
				BufferedReader br = null;
				FileOutputStream fos = null; 
				String line = " ";
				
				/* backup the file */
				setMessage("backup the SPI Log file");
				//setProgress(k, 0, 2);
				try {
					copyFile(new File("E:\\public\\Log\\SPIlog.txt"),new File("E:\\public\\Log\\Backups\\log.txt"));
				} catch (IOException e1) {
					e1.printStackTrace();
				}
				System.out.println("Backup File SPIlog.txt..........................Done");
				
				File LInput = new File ("z:\\SE300 Outputc");
				
				File HostFileLD01 = new File("E:\\public\\Log\\LotData\\LD01.log");
				File HostFileLD02 = new File("E:\\public\\Log\\LotData\\LD02.log");
				File HostFileLD03 = new File("E:\\public\\Log\\LotData\\LD03.log");
				File HostFileLD04 = new File("E:\\public\\Log\\LotData\\LD04.log");
				File HostFileLD05 = new File("E:\\public\\Log\\LotData\\LD05.log");
				File HostFileLD06 = new File("E:\\public\\Log\\LotData\\LD06.log");
				File HostFileLD07 = new File("E:\\public\\Log\\LotData\\LD07.log");
				File HostFileLD08 = new File("E:\\public\\Log\\LotData\\LD08.log");
				File HostFileLD09 = new File("E:\\public\\Log\\LotData\\LD09.log");
				File HostFileLD010 = new File("E:\\public\\Log\\LotData\\LD10.log");
				File HostFileLD011 = new File("E:\\public\\Log\\LotData\\LD11.log");
				File HostFileLD012 = new File("E:\\public\\Log\\LotData\\LD12.log");
				File HostFileLD013 = new File("E:\\public\\Log\\LotData\\LD13.log");
				File HostFileLD014 = new File("E:\\public\\Log\\LotData\\LD14.log");
				File HostFileLD015 = new File("E:\\public\\Log\\LotData\\LD15.log");
				File HostFileLD016 = new File("E:\\public\\Log\\LotData\\LD16.log");
				File HostFileLD017 = new File("E:\\public\\Log\\LotData\\LD17.log");
				File HostFileLD018 = new File("E:\\public\\Log\\LotData\\LD18.log");
				File HostFileLD019 = new File("E:\\public\\Log\\LotData\\LD19.log");
				File HostFileLD020 = new File("E:\\public\\Log\\LotData\\LD20.log");
				File HostFileLD021 = new File("E:\\public\\Log\\LotData\\LD21.log");
				File HostFileLD022 = new File("E:\\public\\Log\\LotData\\LD22.log");
				File HostFileLD023 = new File("E:\\public\\Log\\LotData\\LD23.log");
				
				File HostFile01 = new File("E:\\public\\Log\\Log\\SPI01.log");
				File HostFile02 = new File("E:\\public\\Log\\Log\\SPI02.log");
				File HostFile03 = new File("E:\\public\\Log\\Log\\SPI03.log");
				File HostFile04 = new File("E:\\public\\Log\\Log\\SPI04.log");
				File HostFile05 = new File("E:\\public\\Log\\Log\\SPI05.log");
				File HostFile06 = new File("E:\\public\\Log\\Log\\SPI06.log");
				File HostFile07 = new File("E:\\public\\Log\\Log\\SPI07.log");
				File HostFile08 = new File("E:\\public\\Log\\Log\\SPI08.log");
				File HostFile09 = new File("E:\\public\\Log\\Log\\SPI09.log");
				File HostFile10 = new File("E:\\public\\Log\\Log\\SPI10.log");
				File HostFile11 = new File("E:\\public\\Log\\Log\\SPI11.log");
				File HostFile12 = new File("E:\\public\\Log\\Log\\SPI12.log");
				File HostFile13= new File("E:\\public\\Log\\Log\\SPI13.log");
				File HostFile14 = new File("E:\\public\\Log\\Log\\SPI14.log");
				File HostFile15 = new File("E:\\public\\Log\\Log\\SPI15.log");
				File HostFile16 = new File("E:\\public\\Log\\Log\\SPI16.log");
				File HostFile17 = new File("E:\\public\\Log\\Log\\SPI17.log");
				File HostFile18= new File("E:\\public\\Log\\Log\\SPI18.log");
				File HostFile19 = new File("E:\\public\\Log\\Log\\SPI19.log");
				File HostFile20 = new File("E:\\public\\Log\\Log\\SPI20.log");
				File HostFile21 = new File("E:\\public\\Log\\Log\\SPI21.log");
				File HostFile22 = new File("E:\\public\\Log\\Log\\SPI22.log");
				File HostFile23 = new File("E:\\public\\Log\\Log\\SPI23.log");
			
				
				/* Creates the Object Log folder*/
				
				File fY = new File("E:\\public\\Log\\");
				/* if it doesn't exists*/
				if(!fY.exists())
					System.out.println("Log Folder was created");
					setMessage("Log Folder was created");
					//setProgress(k, 3, 4);
				fY.mkdir();
				/* if exists*/
				if(fY.exists())
					//System.out.println("Log Folder................................... Done;");
					setMessage("Log Folder................................... Done;");
					//setProgress(k, 5, 6);
				File fYY = new File("E:\\public\\Log\\Log");
				/* if it doesn't exists*/
				if(!fYY.exists())
					//System.out.println("Log SubFolder was created");
					setMessage("Log SubFolder was created");
				//setProgress(k, 7, 8);
				fYY.mkdir();
				/* if exists*/
				if(fYY.exists())
					//System.out.println("Log SubFolder................................... Done;");
					setMessage("Log SubFolder................................... Done;");
				//setProgress(k, 9, 10);
				
				File fxY = new File("E:\\public\\Log\\LotData");
				/* if it doesn't exists*/
				if(!fxY.exists()){
					//System.out.println("LotData SubFolder was created");
					setMessage("LotData SubFolder was created");
					//setProgress(k, 10, 11);
					fxY.mkdir();}
				/* if exists*/
				if(fxY.exists())
					//System.out.println("LotData SubFolder............................... Done;");
					setMessage("LotData SubFolder............................... Done;");
				
				
				/* ---------------------SPI01----------------------- */
				/*Creates the Object file SPI1.log */
				//outputFrame.aTextArea.setText("");
				File f1 = new File("E:\\public\\Log\\Log\\SPI01.log"); 
				if(f1.exists())
					//System.out.println("Log File SPI01..................................... Done;");
					setMessage("Log File SPI01..................................... Done;");
				/* if it doesn't exists */
				if(!f1.exists())
					//System.out.println("The Log file does not exist");
					setMessage("The Log file does not exist");
				/* New file ... exception popup */
				try {
					if(f1.createNewFile())
						//System.out.println("Log file created.................................. Done;");
						setMessage("Log file created.................................. Done;");
				}
				catch (IOException e) {
					//System.out.println("I/O Error");
					setMessage("I/O Error");
				}
				/*Creates the Object file LD01.log */
				File ff1 = new File("E:\\public\\Log\\LotData\\LD01.log"); 
				if(ff1.exists())
					//System.out.println("LotData SPI01..................................... done;");
					setMessage("LotData SPI01..................................... done;");
				/* if it doesn't exists */
				if(!ff1.exists()){
					setMessage("The LotData file does not exist");
					//System.out.println("The LotData file does not exist");
					/* New file ... exception popup */
					try {
						if(ff1.createNewFile())
							//System.out.println("LotData file created.................................. done;");
							setMessage("LotData file created.................................. done;");
					}
					catch (IOException e) {
						//System.out.println("I/O Error");
						setMessage("I/O Error");
					}
				}		

				/* mapping */
				/*   Creates a temporary Network Drive before starting the cycle */
				String mapSPI01 = new String("net use z: \\\\172.21.34.1\\d$ qmy001 /user:172.21.34.1\\administrator /Persistent:No");            
				Runtime rt = Runtime.getRuntime();
				try     {
					Process con = rt.exec(mapSPI01);
					con.waitFor();
					//System.out.println("Drive mapping Spi 01......................... Done");
					setMessage("Drive mapping Spi 01......................... Done");
					}
				catch (IOException e)   {
					System.out.println(e);
					}
				catch (InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
					
					/*Copy LotData contents from d:\DataLot\LotData.csv to the Respective LD.log*/
					try {       
						if (LotInput.isDirectory()) { 
							//fos = new FileOutputStream(HostFileLD01);   
							files = LotInput.listFiles();
							xw = new FileWriter(HostFileLD01);
							
							for (int i = 0; i < files.length; i++) {       
								file = files[i];       
			
								fr = new FileReader(file);       
								br = new BufferedReader(fr); 
								x=1;
			
								while ((line = br.readLine()) != null) {
									//System.out.println(line);
									xw.write(x+","+line + System.getProperty("line.separator")) ;//change line
									xw.flush();
								}
							} 
							
						}  
						       
					}
					catch (FileNotFoundException fnex) {       
						fnex.printStackTrace();       
					} 
					catch (IOException ioex) {       
						ioex.printStackTrace();       
					}
					/*end of copy*/
					
					setMessage("Collecting Data from SPI 01");
					/* Scanning */
					File root = new File("z:\\SE300 Output");
					//File root = new File("K:\\spi15\\SE 300 Output\\CSV");
					if (root != null && root.isDirectory()) {
						try{
							fw = new FileWriter("E:\\public\\Log\\Log\\SPI01.log");
							x=1;
							listRecursively(root, 0);
							fw.close();
							}
					catch(NullPointerException np){
						np.printStackTrace();
					}
					catch(IOException exc){
						exc.printStackTrace();
					}
				}
				else
				{
					//System.out.println("Not a directory: " + root);
					setMessage("Not a directory: " + root);}

				/* Deletes temporary Network Drive before finishing the cycle */
					try
			        {            
			            String[] cmd = new String[3];
			                  
			            cmd[0] = "cmd.exe" ;
			            cmd[1] = "/C" ;
			            cmd[2] = "net use z: /delete /y";
			       
			      
			            Runtime rtoff = Runtime.getRuntime();
			            //System.out.println("Executing " + cmd[0] + " " + cmd[1] 
			            //                  + " " + cmd[2]);
			            setMessage("Executing " + cmd[0] + " " + cmd[1] 
			              			                               + " " + cmd[2]);
			            Process proc = rtoff.exec(cmd);
			           
			            // any error message?
			            StreamGobbler1 errorGobbler = new 
			                StreamGobbler1(proc.getErrorStream(), "ERROR");            
			            
			            // any output?
			            StreamGobbler1 outputGobbler = new 
			                StreamGobbler1(proc.getInputStream(), "OUTPUT");
			                
			            // kick them off
			            errorGobbler.start();
			            outputGobbler.start();
			                                    
			            // any error???
			            int exitVal = proc.waitFor();
			            setMessage("ExitValue: " + exitVal);
			        } catch (Throwable t)
			          {
			            t.printStackTrace();
			          }
//				String mapSPIDEL = new String("net use z: /delete /y");            
//				Runtime rtoff = Runtime.getRuntime();
//				try     {
//					Process con = rtoff.exec(mapSPIDEL);
//					con.waitFor();
//					System.out.println("Network Drive deleted......................... Done");
//				}
//				catch (IOException e)   {
//					System.out.println(e);
//				}

and then goes to the second machine and so on and so on.. i'm using a recursive method to read and access all the csv files...

[jorgeflorencio]

/* .........................Recursive list............................... */
	public  void listRecursively(File fdir, int depth) throws IOException {
		
		/*Transform milliseconds time to gregorian time */
		long datefiles = fdir.lastModified();
		SimpleDateFormat Date = new SimpleDateFormat (" dd/MM/yyyy , HH:mm:ss aaa");
		Date nDate = new Date(datefiles);
		
		String F = ",F,";
		int count = 0;
		
		/*Line counter*/
		try
		{
			RandomAccessFile File = new RandomAccessFile(fdir,"r");
			long lastline=File.length();
			File.close();
			FileReader fileRead = new FileReader(fdir);
			BufferedReader bufferReader = new BufferedReader(fileRead);
			Scanner scan = new Scanner(fdir);
			while(scan.hasNextLine())
			{
				if(scan.nextLine().contains(F))
					count++;
			}
			
			fileRead.close();
			bufferReader.close();
			
//			/*Start Line counter*/
//			LineNumberReader lineRead = new LineNumberReader(fileRead);
//			lineRead.skip(lastline);
//			int countline = lineRead.getLineNumber()-6; //number of default lines = 6
//			fileRead.close();
//			lineRead.close();
//			/*End Line counter*/
			
		/* Output1 */
			if (fdir.getPath().endsWith(".csv") /*&& fdir.lastModified() > HostFile.lastModified()*/)
				System.out.println(INDENTS[depth] + x +", "+fdir.getName() +", "+ count +","+Date.format(nDate));
		/* Output2 */
			if (fdir.getPath().endsWith(".csv") /*&& fdir.lastModified() > HostFile.lastModified()*/)
				fw.write( INDENTS[depth] + x +", "+ fdir.getName() +", "+ count +","+ Date.format(nDate)+ System.getProperty("line.separator"));
			fw.flush();
		//if (fdir.getPath().endsWith(".csv") && (fdir.length()/512 )>= 1 && fdir.length()/512 <= 3)	
		}
		catch(IOException e){
		} 
		if (fdir.isDirectory() && !fdir.isHidden() && depth < MAX_DEPTH) {
			for (File f : fdir.listFiles()){  // Go over each file/subdirectory.
				listRecursively(f, depth+1);
				}}}

[jorgeflorencio]

Hi Alex!

First i would like to say. Thank you very much!

Secondly... i'm lost i don't know how to use your code despite i understand the concept behind! I need to reed more in order to apply your knowledge...

I was stopped for 1,5 years...

Regards,

jorge

[Alex Edwards]

RandomAccessFile File = new RandomAccessFile(fdir,"r");
			long lastline=File.length();
			File.close();
			FileReader fileRead = new FileReader(fdir);
			BufferedReader bufferReader = new BufferedReader(fileRead);
			Scanner scan = new Scanner(fdir);
			while(scan.hasNextLine())
			{
				if(scan.nextLine().contains(F))
					count++;
			}
			
			fileRead.close();
			bufferReader.close();

Ok your problem is that you need to read a big line of text and yoou're using a Scanner instead of a Buffered Reader. Buffered Readers, as the name implies, are made for long lines of text and can be size for scanning lines of the appropriate length.

After scanning a line, you may want to do something like parse the first comma (since they are comma separated values and I believe the first value you encounter would be considered a " " or "" due to the comma that exists before the data).

Create a new object of my class immediately after doing so. It would look something like this--

BufferedReader br = new BufferedReader(fileRead/*, int line size*/ );

// set length of line you will be reading via BufferedReader constructor

while(br.ready()){
     StringBuilder sb = new StringBuilder(br.readLine());
     
     // truncate or replace or remove unnecessary values like first comma and the last

     InspectionResults ir = new InspectionResults( sb.toString(). split(","));

// perform some action if a particular fail is acknowledged via my class
//store information if needed
}

By the way, this smells like a sub-module or a part of a company project >_>

[jorgeflorencio]

Hi again...

I'm completely lost... despite the fact i don't know how to use your code... my head is almost getting nuts... i've found that some records have 32 instead of 45.. therefore i'm using the ... If (linesize =45)...i'm applying your code.. else... i'm using your codebase but adjusted for the record in question

else

[Alex Edwards]

Here's a test I ran using this file--

INSPECT.txt

,07/19/2008,00:48:55,#280,Module 1 ,IC2,TSOP8,SOP8_1,F,138.539642,238.000000,84.000000,140.000000,P,9.226400,140.000000,P,427747.781250,729933.140625,243311.046875,486622.090000,P,26717.400391,486622.093750,P,60986780.000000,122628772.800000,34063548.000000,68127092.600000,P,3087930.000000,68127096.000000,-22.886625,-23.317499,-0.039985,-0.131595,P,0,1,P,0,1,P,0,1
,07/19/2008,00:48:55,#280,Module 1 ,C38,0402,0402_1_3,P,131.077194,238.000000,84.000000,140.000000,P,12.636800,140.000000,P,265426.750000,470565.468750,156855.156250,313710.320000,P,5929.500000,313710.312500,P,36161180.000000,79054999.200000,21959722.000000,43919444.800000,P,2340260.000000,43919444.000000,-26.636499,-28.285000,-0.201710,-0.635564,P,0,1,P,0,1,P,0,1
,07/19/2008,00:48:55,#280,Module 1 ,C66,0402,0402_1_3,P,146.934555,238.000000,84.000000,140.000000,P,4.062300,140.000000,P,266860.500000,470565.468750,156855.156250,313710.320000,P,15511.000000,313710.312500,P,40048568.000000,79054999.200000,21959722.000000,43919444.800000,P,235260.000000,43919444.000000,-32.004501,-21.340000,-0.163571,-1.147347,P,0,1,P,0,1,P,0,1

import java.io.*;
import java.util.Arrays;

public class InspectionResults{

	public static final byte  HEIGHT_AVG_RESULT = 7,
							  HEIGHT_RANGE_RESULT = 12,
							  AREA_AVG_RESULT = 15,
							  AREA_RANGE_RESULT = 20,
							  VOLUME_AVG_RESULT = 23,
							  VOLUME_RANGE_RESULT = 28,
							  HAV_FAILED_FEATURE_RESULT = 35,
							  REG_FAILED_FEATURE_RESULT = 38,
							  BRIDGE_FAILED_FEATURE_RESULT = 41;
	private String retrievedData[];
	private boolean failed[];

	/**
	 * Constructs this InspectionResult with the data stored in the args.
	 * This class expects 44 values within the range of the args.
	 */
	public InspectionResults(String... args){
		retrievedData = args;
		boolean temp[] ={
			((retrievedData[7].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[12].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[15].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[20].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[23].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[28].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[35].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[38].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[41].equalsIgnoreCase("F")) ? true: false)
		};
		failed = temp;
	}

	public static void main(String... args){

		FileReader fr = null;
		BufferedReader br = null;
		try{
			fr = new FileReader(new File("INSPECT.txt"));
			br = new BufferedReader(fr);
		}catch(Exception e){e.printStackTrace();}


		String dwArray[][] ={ {""}, {""}, {""} };

		for(int i = 0; i < dwArray.length; i++){
			String temp[] = null;

			try{ temp = br.readLine().split(",");}catch(Exception f){f.printStackTrace(); System.exit(1);};
			temp = Arrays.<String>copyOfRange(temp, 1, temp.length);
			dwArray[i] = temp;
		}


		InspectionResults ir[] =
		{
			new InspectionResults(dwArray[0]),
			new InspectionResults(dwArray[1]),
			new InspectionResults(dwArray[2])
		};

		System.out.println(ir[0]); // as an example
		spacer(3);

		try{
			System.out.println(ir[0].hasFailed(InspectionResults.HEIGHT_AVG_RESULT));
			System.out.println(ir[0].getAdjacentValue(InspectionResults.HEIGHT_AVG_RESULT));
		}catch(Exception e){
			System.out.println(e);
		}

		try{
			fr.close();
			br.close();
		}catch(Exception e){
		}
	}

	private static void spacer(int lines){
		for(int i = 0; i < lines; i++)
			System.out.println();
	}

	/**
	 * Returns true if the given value has failed, returns false otherwise.
	 * It's preferred to use the constants defined within this class to get the
	 * desired information, and not regular ints.
	 */
	public boolean hasFailed(byte result) throws Exception{
		switch(result){
			case HEIGHT_AVG_RESULT:
				return failed[0];
			case HEIGHT_RANGE_RESULT:
				return failed[1];
			case AREA_AVG_RESULT:
				return failed[2];
			case AREA_RANGE_RESULT:
				return failed[3];
			case VOLUME_AVG_RESULT:
				return failed[4];
			case VOLUME_RANGE_RESULT:
				return failed[5];
			case HAV_FAILED_FEATURE_RESULT:
				return failed[6];
			case REG_FAILED_FEATURE_RESULT:
				return failed[7];
			case BRIDGE_FAILED_FEATURE_RESULT:
				return failed[8];
			default :
				throw new Exception("Attempt to access invalid result type! Use the Result Constants to avoid this error!");
		}
	}

	/**
	 * Returns the value next to the specified result.
	 */
	public String getAdjacentValue(byte result) throws Exception{
		if(result >= 0 && result < retrievedData.length - 1)
			return retrievedData[result + 1];
		else throw new Exception("Error! Attempt to access column with either no adjacent value or outside of data-range!");
	}

	/**
	 * Simply returns a String representing the data for each value in this class.
	 */
	@Override
	public String toString(){
		String temp = "";
		for(String element : retrievedData){
			if(element.toString() != retrievedData[retrievedData.length - 1])
				temp += element + ", ";
			else temp += element;
		}
		return temp;
	}
}

[Alex Edwards]

How often do your lines vary in information? You could go about this a few ways

-Create classes to handle those cases
-Adapt the current class to simply handle all cases.

I'd go for number 2 to be safe, but in order to do that more information about what's varying will be needed.

[jorgeflorencio]

HI..
The information will vary only with 44 and 32... so i'm adapting the code for 32 as well...

I'm going to try first.. and then will tell Further more.. THHHAANNNKKK YOOUUU SOO MUCH...

[Alex Edwards]

HI..
The information will vary only with 44 and 32... so i'm adapting the code for 32 as well...

I'm going to try first.. and then will tell Further more.. THHHAANNNKKK YOOUUU SOO MUCH...

For the final values declared in global scope of the class, you may additionally (or inversely) want to declare an array or enum that holds those values so you can simply iterate through them when processing your information.

I hope everything turns out well for you.

-Alex

[jorgeflorencio]

Thank you for your help...

Eclipse is giving me an error on this line ¨

temp = Arrays.<String>copyOfRange(temp, 1, temp.length);

in copyOfRange...

what library are you using?

[Alex Edwards]

Notice the import statements --

import java.io.*;
import java.util.Arrays;

/*from Arrays*/ static <T> T[] Arrays.copyOfRange( T[] array, int from, int size )

[jorgeflorencio]

package spimonitoring;

import java.io.*;
import java.util.Arrays;
import java.lang.reflect.Array;

public class InspectionResults2{

	public static final byte  HEIGHT_AVG_RESULT = 7,
							  HEIGHT_RANGE_RESULT = 12,
							  AREA_AVG_RESULT = 15,
							  AREA_RANGE_RESULT = 20,
							  VOLUME_AVG_RESULT = 23,
							  VOLUME_RANGE_RESULT = 28,
							  HAV_FAILED_FEATURE_RESULT = 35,
							  REG_FAILED_FEATURE_RESULT = 38,
							  BRIDGE_FAILED_FEATURE_RESULT = 41;
	
	private String retrievedData[];
	private boolean failed[];

	/**
	 * Constructs this InspectionResult with the data stored in the args.
	 * This class expects 44 values within the range of the args.
	 */
	public InspectionResults2(String... args){
		retrievedData = args;
		boolean temp[] ={
			((retrievedData[7].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[12].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[15].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[20].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[23].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[28].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[35].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[38].equalsIgnoreCase("F")) ? true: false),
			((retrievedData[41].equalsIgnoreCase("F")) ? true: false)
		};
		failed = temp;
	}

	public static void main(String... args){

		FileReader fr = null;
		BufferedReader br = null;
		try{
			fr = new FileReader(new File("INSPECT.txt"));
			br = new BufferedReader(fr);
		}catch(Exception e){e.printStackTrace();}


		String dwArray[][] ={ {""}, {""}, {""} };

		for(int i = 0; i < dwArray.length; i++){
			String temp[] = null;

			try{ temp = br.readLine().split(",");
			}
			catch(Exception f){f.printStackTrace(); 
			System.exit(1);
			};
			temp = Arrays.<String>copyOfRange(temp, 1, temp.length);
			//static <T> T[] Arrays.copyOfRange( T[] array, int from, int size )
			dwArray[i] = temp;
		}


		InspectionResults2 ir[] =
		{
			new InspectionResults2(dwArray[0]),
			new InspectionResults2(dwArray[1]),
			new InspectionResults2(dwArray[2])
		};

		System.out.println(ir[0]); // as an example
		spacer(3);

		try{
			System.out.println(ir[0].hasFailed(InspectionResults2.HEIGHT_AVG_RESULT));
			System.out.println(ir[0].getAdjacentValue(InspectionResults2.HEIGHT_AVG_RESULT));
		}catch(Exception e){
			System.out.println(e);
		}

		try{
			fr.close();
			br.close();
		}catch(Exception e){
		}
	}

	private static void spacer(int lines){
		for(int i = 0; i < lines; i++)
			System.out.println();
	}
	
	
	
	 
	/**
	 * Returns true if the given value has failed, returns false otherwise.
	 * It's preferred to use the constants defined within this class to get the
	 * desired information, and not regular ints.
	 */
	public boolean hasFailed(byte result) throws Exception{
		switch(result){
			case HEIGHT_AVG_RESULT:
				return failed[0];
			case HEIGHT_RANGE_RESULT:
				return failed[1];
			case AREA_AVG_RESULT:
				return failed[2];
			case AREA_RANGE_RESULT:
				return failed[3];
			case VOLUME_AVG_RESULT:
				return failed[4];
			case VOLUME_RANGE_RESULT:
				return failed[5];
			case HAV_FAILED_FEATURE_RESULT:
				return failed[6];
			case REG_FAILED_FEATURE_RESULT:
				return failed[7];
			case BRIDGE_FAILED_FEATURE_RESULT:
				return failed[8];
			default :
				throw new Exception("Attempt to access invalid result type! Use the Result Constants to avoid this error!");
		}
	}

	/**
	 * Returns the value next to the specified result.
	 */
	public String getAdjacentValue(byte result) throws Exception{
		if(result >= 0 && result < retrievedData.length - 1)
			return retrievedData[result + 1];
		else throw new Exception("Error! Attempt to access column with either no adjacent value or outside of data-range!");
	}

	/**
	 * Simply returns a String representing the data for each value in this class.
	 */
	@Override
	public String toString(){
		String temp = "";
		for(String element : retrievedData){
			if(element.toString() != retrievedData[retrievedData.length - 1])
				temp += element + ", ";
			else temp += element;
		}
		return temp;
	}

Something is wrong is giving me an error...

[jorgeflorencio]

temp = Arrays.<String>copyOfRange(temp, 1, temp.length);

[Alex Edwards]

It may (or may not) have something to do with the Standard Edition of Java you are currently using.

What JDK are you currently using?

[Alex Edwards]

Notice the difference in these two links, for example.

SE 1.4.2 -- http://java.sun.com/j2se/1.4.2/docs/api/java/util/Arrays.html
SE 6.0 -- http://java.sun.com/javase/6/docs/api/java/util/Arrays.html

[jorgeflorencio]

1.5... in the factory...

6 in my laptop,,,, but i'm not using it now :(