0

oho...ok...
I have 2 test files
1. 348270 records. 30.4MB
2. 10 records. 864bytes

When I test the smaller file, it works great...but when I tried the larger file, it doesn't even move on to say "The file was uploaded."

Code:

if(in_array($ext, $allowedext))				
					{					
						$filename = $_FILES['file']['tmp_name'];
						//move_uploaded_file($filename,"upload/".$_FILES['file']['name']);											
						//$fh = fopen($_FILES['file']['tmp_name'], 'r');					
						$handle = fopen($filename, "r"); 			
						
						while (($data = fgetcsv($handle, 100000, ',', '"')) !== FALSE)					
						{
							//echo ("data<br />");
							//echo $data;
							
							//echo ("<br />data_array<br />");
							//$data_array = explode(",",$data); 
							//print_r ($data_array);
							
							//echo ("<br />");
							$value1 = $data[0];
							$value2 = $data[1];
							$value3 = $data[2];
							$value4 = $data[3];
							$value5 = $data[4];
							$value6 = $data[5];
							$value7 = $data[6]; 
							$value8 = $data[7]; 
							$value9 = $data[8];
							$value10 = $data[9]; 
							$value11 = $data[10]; 
							$value12 = $data[11]; 
											
							$query = "INSERT INTO testtsttbills VALUES ('$value1','$value2','$value3','$value4','$value5','$value6','$value7','$value8','$value9','$value10','$value11','$value12')";						
							$qry = mysql_query($query,$conn);					
						}						
						//$contents = fread($handle, filesize($filename));					
						fclose($handle);							
						//echo $contents;					
						echo "<div align='center'><font color='red'>The file was uploaded.</font></div>";				
					}
0

I put the

set_time_limit(0)

to the top of my script.

These are the settings I have in my php.ini file:

max_execution_time = 60
max_input_time = 60
memory_limit = 128M

post_max_size = 80M

;upload_tmp_dir = "c:/uploads/tmp" (do I need to remove the semicolon to the beginning of this line in the .ini file?)
upload_max_filesize = 80M

I tried upload the smal file, and it works...I then tried the larger file, and in about 5-10 seconds the page is refreshed...and nothing is uploaded...

0

max_execution_time = 60 means the script can take a maximum of 60 seconds to execute (and since you are trying to upload a file which is around 30mb, it will take more than umm..(10 mins ?) long time to process!) . So, you can either set it to 0 or use set_time_limit(0) in your script. If you are working on file uploads, then you have to remove the semicolon before upload_tmp_dir.

0

well I increased the amount of time from 60 to 1800...that should give way for any file that may be more than 30MB...I also removed the semicolon...
to be clear....can I use

EITHER set_time_limit(0) in the script OR max_execution_time = 1800

or can I use

set_time_limit(0) in the script AND max_execution_time = 1800

:)

0

No matter what you have for max_execution_time in php.ini, But if set_time_limit is 0 (in your script), it uses set_time_limit(0) because its executed during the runtime!

0

:?: hmmm.. keep printing $data.. Ie., print_r($data); and see if it prints any data for the large file.. also check how many records does it print..

0

nav33n,
I have tried uploading files with different amount of records...I started of with 10 records, 1000 records, 5000 records...and they were uploaded (seperately), I tried 10000 records, 100000 records, and up...and they obviously didn't work...
I noticed that while uploading the file with the 5000 records, the pc sorta froze...does this consume alot of resources for such a small amount of data to be uploaded?

this is the print_r

Array ( [0] => 127000769106 [1] => 630-3228 [2] => 1/11/2007 [3] => 7:58:46 [4] => DY [5] => TRINIDAD [6] => 18683555181 [7] => 4:00:00 [8] => 0 [9] => 3.2 [10] => 3.2 [11] => P0 )

Array ( [0] => 127000769106 [1] => 630-3228 [2] => 1/11/2007 [3] => 8:05:44 [4] => DY [5] => TRINIDAD [6] => 18687631516 [7] => 8:00:00 [8] => 0 [9] => 6.4 [10] => 6.4 [11] => P0 )
.
.
.

while (($data = fgetcsv($handle, 100000, ',', '"')) !== FALSE)					
					{
						print_r($data);
						echo("<br />"); 
						$value1 = $data[0];
						$value2 = $data[1];
						$value3 = $data[2];
						$value4 = $data[3];
						$value5 = $data[4];
						$value6 = $data[5];
						$value7 = $data[6]; 
						$value8 = $data[7]; 
						$value9 = $data[8];
						$value10 = $data[9]; 
						$value11 = $data[10]; 
						$value12 = $data[11]; 
						echo("<br />"); 
										
						$query = "INSERT INTO testtsttbills (account,service,date,time,timebandtype,destination,callednumber,mins,airtimecharge,tollcharge,amt,privatenumber) VALUES ('$value1','$value2','$value3','$value4','$value5','$value6','$value7','$value8','$value9','$value10','$value11','$value12')";						
						$qry = mysql_query($query,$conn);					
					}
0

Yeah.. it does.. Try opening a 30 mb file in notepad.. You can take a nap by the time it opens the file.. Umm.. I don't know how to process a large file.. Lets see if someone else has an idea :(

0

okie dokie....
well, I would put it upload now, and when I return to work on Friday, I would check it...(tomorrow is a Public Holiday in my country...so I won't be doing any work on it...that should give it enough time to fully run)
nav33n, thanks so much for your help thus far...:):):)

0

WellI got bad news, good news, and bad news...
Bad News: It wasn't uploading the full file, so I just tried 100000 records...:(
Good News: It uploaded the file with 100000 records but...:)
Bad News: In the database, the data wasn't stored, but the rows were created...:(

0

:confused: beats me...:icon_confused:
let me repost the code...perhaps there's we missed...

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Test Upload Bill</title>
<style type="text/css">
<!--
.style28 {
	font-family: Verdana, Arial, Helvetica, sans-serif;
	font-size: 18px;
	font-weight: bold;
	color: #006600;
}
.style29 {
	font-family: Verdana, Arial, Helvetica, sans-serif;
	font-size: 14px;
	font-weight: bold;
}
-->
</style>


<style type="text/css">
<!--
.style27 {
	font-family: Verdana, Arial, Helvetica, sans-serif;
	font-size: 10px;
	font-weight: bold;
}
body {
	background-color: #CCFFCC;
}
.style30 {
	color: yellow;
	font-size: 14px;
	font-family: Verdana, Arial, Helvetica, sans-serif;
}
.style31 {
	color: #CCFFCC;
	font-weight: bold;
}
.style32 {color: yellow; font-weight: bold; }
.style33 {font-size: 14px; font-family: Verdana, Arial, Helvetica, sans-serif;}
-->
</style>
</head>

<body onload="goforit()">
<table width="200" border="0" align="center">
  <tr>
    <th colspan="6" scope="col"><img src="images/apBanner2.png" width="962" height="225" /></th>
  </tr>
  <tr>
    <td width="155"><img src="images/btnT.png" width="150" height="33" /></td>
    <td width="155"><img src="images/btnA.png" width="150" height="33" /></td>
    <td width="155"><img src="images/btnI.png" width="150" height="33" /></td>
    <td width="155"><img src="images/btnS.png" width="150" height="33" /></td>
    <td width="327" colspan="2">
	<div align="center">
	  <script>
		/*
		Live Date Script- 
		© Dynamic Drive (www.dynamicdrive.com)
		For full source code, installation instructions, 100's more DHTML scripts, and Terms Of Use,
		visit http://www.dynamicdrive.com
		*/
		
		var dayarray=new Array("Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday")
		var montharray=new Array("January","February","March","April","May","June","July","August","September","October","November","December")
			
		function getthedate()
		{
			var mydate=new Date()
			var year=mydate.getYear()
			                if (year < 1000)
			year+=1900
			var day=mydate.getDay()
			var month=mydate.getMonth()
			var daym=mydate.getDate()
			if (daym<10)
				daym="0"+daym
			var hours=mydate.getHours()
			var minutes=mydate.getMinutes()
			var seconds=mydate.getSeconds()
			var dn="AM"
			if (hours>=12)
				dn="PM"
			if (hours>12)
			{
				hours=hours-12
			}
			if (hours==0)
				hours=12
			if (minutes<=9)
				minutes="0"+minutes
			if (seconds<=9)
				seconds="0"+seconds
			//change font size here
			var cdate="<small><font color='006600' face='Verdana'><b>"+dayarray[day]+" "+montharray[month]+" "+daym+", "+year+"    "+hours+":"+minutes+":"+seconds+" "+dn
				+"</b></font></small>"
			if (document.all)
			document.all.clock.innerHTML=cdate
			else if (document.getElementById)
			document.getElementById("clock").innerHTML=cdate
			else
			document.write(cdate)
		}
		if (!document.all&&!document.getElementById)
			getthedate()
		function goforit()
		{
			if (document.all||document.getElementById)
				setInterval("getthedate()",1000)
		}			
		</script>
    <span id="clock"></span>        
	</div>		</td>
 </tr>
  <tr>
    <td colspan="6">&nbsp;</td>
  </tr>
    <tr><td colspan="6">&nbsp;</td></tr>  <tr>
    <td colspan="6">
  	<form action="<? echo $_SERVER['REQUEST_URI']; ?>" enctype="multipart/form-data" method="post">
	<table>					
		  <tr>
			  <td>
			      <p align="left"><span class="style4 style25 style28">Bill Uploader</span></p>
			      <p align="left"><span class="style21 style29">The Bill Uploader works in 2 steps.</span></p>					 
			  </td>
		  </tr>
		  <tr><td><p>&nbsp;</p></td></tr>
		  <tr>
		  	<td>
			      <table width="950" border="0" align="center" cellpadding="1" cellspacing="0" >
				<tr>
				  <td><p>&nbsp;</p></td>
				  <td colspan="4" bgcolor="#006600">
				  	<p align="left" class="style32">
				  	<span class="style21  style33">Step 1: Browse for file</span></p>						  
				  </td>
				  <td><p>&nbsp;</p></td>
				</tr>
				<tr>
				  <td><p>&nbsp;</p></td>
				  <td bgcolor="#006600"><p>&nbsp;</p></td>
				  <td width="125" bgcolor="#006600" scope="row"><div align="left" class="style13 style29 style31">Browse</div>
				  </td>
				  <td width="356" bgcolor="#006600">
				  	<div align="left">
						<strong>
							<input type="file" name="file" />
						</strong>		</div>						  
				  </td>
				  <td bgcolor="#006600"><p>&nbsp;</p></td>
				  <td><p>&nbsp;</p></td>
				</tr>
				<tr width="950">
				  <td><p>&nbsp;</p></td>
				  <td colspan="4" bgcolor="#006600">
				  	<p align="left"><strong>
					  <span class="style21 style30">Step 2: Upload the file selected</span></strong></p>						  						  						  </td>
				  <td><p>&nbsp;</p></td>
				</tr>
				<tr>
				  <td><p>&nbsp;</p></td>
				  <td bgcolor="#006600"><p>&nbsp;</p></td>
				  <td bgcolor="#006600"><p>&nbsp;</p></td>
				  <td bgcolor="#006600">
					  <div align="left">
						<strong>
							<input type="submit" value="Upload" name="btnSubmit" />
						</strong>	  </div>  		
				  </td>
				  <td bgcolor="#006600"><p>&nbsp;</p></td>
				  <td><p>&nbsp;</p></td>
				</tr>
			  </table>  				    
			</td>
		</tr>
	</table>
</form>	
</td>
 </tr>
  <tr>
  	<td colspan="6" >
		
	  <div align="left" class="style29">
		<? 					
			include ("./connect.php");			
			global $conn;			
			ini_set("display_errors", "1");		
			set_time_limit(0);
				
			if(count($_FILES) > 0)			
			{					
				$ext = "";				
				$ext = substr(trim($_FILES["file"]["name"]), -4);				
				$allowedext = array(".txt", ".csv", ".sql"); 				
				
				if(in_array($ext, $allowedext))				
				{					
					$filename = $_FILES['file']['tmp_name'];				
					$handle = fopen($filename, "r"); 			
					
					while (($data = fgetcsv($handle, 100000, ',', '"')) !== FALSE)					
					{
						print_r($data);
						echo("<br />"); 
						$value1 = $data[0];
						$value2 = $data[1];
						$value3 = $data[2];
						$value4 = $data[3];
						$value5 = $data[4];
						$value6 = $data[5];
						$value7 = $data[6]; 
						$value8 = $data[7]; 
						$value9 = $data[8];
						$value10 = $data[9]; 
						$value11 = $data[10]; 
						$value12 = $data[11]; 
						echo("<br />"); 
										
						$query = "INSERT INTO testtsttbills (account,service,date,time,timebandtype,destination,callednumber,mins,airtimecharge,tollcharge,amt,privatenumber) VALUES ('$value1','$value2','$value3','$value4','$value5','$value6','$value7','$value8','$value9','$value10','$value11','$value12')";						
						$qry = mysql_query($query,$conn);					
					}				
					fclose($handle);				
					echo "<div align='center'><font color='red'>The file was uploaded.</font></div>";				
				}				
				else				
				{					
					echo "<div align='center'><font color='red'>You are trying to upload an invalid file format. Please try again.</font></div>";		
				}			
			}		
		?>		
    </div></td>
  </tr>
  <tr><td colspan="6">&nbsp;</td></tr>
  <tr>
    <td colspan="6" bgcolor="#339900">&nbsp;</td>
  </tr>
  <tr>
    <td colspan="6"><div align="center"><span class="style27">&copy;ITT Department.</span></div></td>
  </tr>
</table>
</body>
</html>
0

I tested your code with a small csv file and it works. :-/

The only thing (*I think*) that can go wrong is datatype of the columns not matching with the provided values.

0

But it works with the small csv I have....
I just deleted most of the records from full csv file, and renamed it to 'smallcsv'...so it's the same datatypes throughout...*sigh*...did u try with a rather large file? about 300000 records...

0

No! I don't have such a large file!

Okay! Since I was jobless, I created a file of 40+ mb (and around 8 lac dummy records) and tried to upload it.. (ofcourse, I had to increase upload_max_filesize = 100M and post_max_size = 100M ) to 100mb.

Its working fine.. :) The script is still running,

Showing rows 0 - 29 (~12,8351 total, Query took 0.0006 sec)

and its still inserting the records..

Edit 2:

Showing rows 0 - 29 (~74,7061 total, Query took 0.0007 sec)

and still inserting..
No problem with your script. Now, I am confused. :S

0

:-O huh?!?:confused: :icon_confused: :icon_sad:
I don't know why it doesn't work here...:@ let me scrutinize the code once more...I'll let you know what's going on...

0

Here is my table:

Field Type Null Key Default Extra
account varchar(20) NO
service varchar(20) YES
date varchar(20) NO
time varchar(20) NO
timebandtype varchar(20) YES
destination varchar(20) NO
callednumber varchar(20) NO
mins varchar(20) NO
airtimecharge varchar(20) NO
tollcharge varchar(20) YES
amt varchar(20) NO
privatenumber varchar(20) NO

Is there a setting in the MySQL Administrator to set the file upload size?

1

OK, so the script works. I think it has something to do with the syntax within the file, let's debug.

replace line 225:

$qry = mysql_query($query,$conn);

with

$qry = mysql_query($query,$conn) or die("<b>A fatal MySQL error occured</b>.\n<br />Query: " . $query . "<br />\nError: (" . mysql_errno($conn) . ") " . mysql_error($conn));

then run again and post your output.

Votes + Comments
Great Job.....Thanks :)
0

Hmm. The only way that I know to be sure is to look at the sql, and that's a lot of sql. Another option would be to break the file into parts and upload them one at a time to help to isolate the problem.

0

I really don't think it has anything to do with the file size at all. I know I'm not an admin but I've imported Gig dump files into bran new mysql databases before and never heard about any type of file size limitation before.

0

Hey....Good Morning,
It is 8:16am here when I started to upload the file of 30.4MB...now the time is 8:20 am and it seems to be uploading still...
R0bb0b, I did put in the die statement at the end of my query, and this was my test, but it seems to working....I also did not upload the csv file, but the txt file...so I will keep you all posted...8:24am and still uploading...I hope it works now...

It is 8:33am and it worked (only problem is that instead of loading the total 348270 records, it loaded 348269 records then gave the error message...)...I have attached it...

Attachments errMsg.PNG 10.44 KB
0

Ok, So I checked my file, and it really contains only 348269 records...therefore it works...

1. I uploaded a txt file and not a csv file
2. I replaced

$qry = mysql_query($query,$conn);

with

$qry = mysql_query($query,$conn) or die("<b>A fatal MySQL error occured</b>.\n<br />Query: " . $query . "<br />\nError: (" . mysql_errno($conn) . ") " . mysql_error($conn));

those were the only 2 things that I did differently today...

0

No, the upload went ahead perfectly....all the data was uploaded, and filled out in the table...so I am seeing the 300000+ records in the table...

I don't understand why it didn't work before though.....

0

I am testing it with the csv file now...

Well, It uploaded the csv file perfectly as well....hmmm...I wonder if it's because of the die statement....I will re-run the upload without the die statement...

I had commented off the print_r in the code...could that have affected the upload?

This question has already been answered. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.