| | |
how to import a csv file to sql server 2005 database table using C#
Please support our C# advertiser: Intel Parallel Studio Home
![]() |
•
•
Join Date: Nov 2006
Posts: 436
Reputation:
Solved Threads: 72
There are a number of ways to do this.
Using directing insert statement is one, another is using a stored procedure, calling as ssis package is another.
The easiest is probably the insert statement.
Simple get you SqlConnection, then use an SqlCommand.
use the string.Format method to parse the data into the command.
string Sql = string.Format("insert into foo(field1,field2) values('{0}',{1})",csv_value1,csv_value2);
SqlConnection conn = new SqlConnection(conectionstring)
conn.Open()
... Loop through all lines parsing the file into vars or an array
SqlCommand cmd = new SqlCommand(sql,conn)
cmd.ExecuteNonQuery();
... loop back
... finally close the connection, done
// Jerry
Using directing insert statement is one, another is using a stored procedure, calling as ssis package is another.
The easiest is probably the insert statement.
Simple get you SqlConnection, then use an SqlCommand.
use the string.Format method to parse the data into the command.
string Sql = string.Format("insert into foo(field1,field2) values('{0}',{1})",csv_value1,csv_value2);
SqlConnection conn = new SqlConnection(conectionstring)
conn.Open()
... Loop through all lines parsing the file into vars or an array
SqlCommand cmd = new SqlCommand(sql,conn)
cmd.ExecuteNonQuery();
... loop back
... finally close the connection, done
// Jerry
•
•
Join Date: Mar 2008
Posts: 13
Reputation:
Solved Threads: 0
hi jerry i have read your code but i did not understand it how to implement it can you send my couple of code please i have tried it in this manner:
on doing this i am getting an error of "Input array is longer than the number of columns in this table." so could you please help me.....
C# Syntax (Toggle Plain Text)
string filepath = @"C:\Documents and Settings\jitendra\Desktop\RIC_SELL_080209.csv"; int i = 0; StreamReader sr = new StreamReader(filepath); string line = sr.ReadLine(); string[] value = line.Split(','); DataTable dt = new DataTable(); DataRow row ; foreach (string dc in value) { dt.Columns.Add(new DataColumn()); } do { row = dt.NewRow(); row.ItemArray = line.Split(); dt.Rows.Add(row); i = i + 1; line = sr.ReadLine(); } while (line != string.Empty); SqlBulkCopy bc = new SqlBulkCopy(con.ToString(), SqlBulkCopyOptions.TableLock); bc.DestinationTableName = "mytable"; bc.BatchSize = dt.Rows.Count; con.Open(); bc.WriteToServer(dt); bc.Close(); con.Close();
on doing this i am getting an error of "Input array is longer than the number of columns in this table." so could you please help me.....
Last edited by jitupatil_2007; Apr 10th, 2008 at 3:04 am.
•
•
Join Date: Nov 2006
Posts: 436
Reputation:
Solved Threads: 72
Here is a version of your code that was slightly modified, and works on a table with a few thousand rows.
The problem with the ""Input array is longer than the number of columns in this table." error is that the file has a column with an embedded comma. This causes the split to return additional columns.
This is a very typical problem with using CSV files. There is no way of knowing where the embedded comma is part of data, or used as a delimiter.
The only thing you can do is to inspect the line for commas, and replace them with a special character, then split, then find the special character and change it back into a comma.
// Jerry
PS If solved, please mark this topic as solved
The problem with the ""Input array is longer than the number of columns in this table." error is that the file has a column with an embedded comma. This causes the split to return additional columns.
This is a very typical problem with using CSV files. There is no way of knowing where the embedded comma is part of data, or used as a delimiter.
The only thing you can do is to inspect the line for commas, and replace them with a special character, then split, then find the special character and change it back into a comma.
C# Syntax (Toggle Plain Text)
private void button3_Click(object sender, EventArgs e) { SqlConnection con = new SqlConnection(@"Data Source=SHAWHP\SQLEXPRESS;Initial Catalog=FOO;Persist Security Info=True;User ID=sa"); string filepath = "C:\\params.csv"; StreamReader sr = new StreamReader(filepath); string line = sr.ReadLine(); string[] value = line.Split(','); DataTable dt = new DataTable(); DataRow row; foreach (string dc in value) { dt.Columns.Add(new DataColumn(dc)); } while ( !sr.EndOfStream ) { value = sr.ReadLine().Split(','); if(value.Length == dt.Columns.Count) { row = dt.NewRow(); row.ItemArray = value; dt.Rows.Add(row); } } SqlBulkCopy bc = new SqlBulkCopy(con.ConnectionString, SqlBulkCopyOptions.TableLock); bc.DestinationTableName = "tblparam_test"; bc.BatchSize = dt.Rows.Count; con.Open(); bc.WriteToServer(dt); bc.Close(); con.Close(); }
// Jerry
PS If solved, please mark this topic as solved
•
•
Join Date: Nov 2006
Posts: 436
Reputation:
Solved Threads: 72
"The only thing you can do is to inspect the line for commas, and replace them with a special character, then split, then find the special character and change it back into a comma."
After some thought, this statement probably won't work. If your CSV has the text columns wrapped in double quotes, then you can probably scrub the data to account for extra embedded commas but, if not... well you are kind-a screwed.
// Jerry
After some thought, this statement probably won't work. If your CSV has the text columns wrapped in double quotes, then you can probably scrub the data to account for extra embedded commas but, if not... well you are kind-a screwed.
// Jerry
•
•
Join Date: Nov 2006
Posts: 436
Reputation:
Solved Threads: 72
Hopefully you adjusted the connection string to use your own server, and not mine ?
Did you walk it through the debugger and see what is happening ?
If you still can not get it to work, send me your project and csv file, and I will see what is wrong with it.
My email is shawhj@meadowcrk.com
// Jerry
Did you walk it through the debugger and see what is happening ?
If you still can not get it to work, send me your project and csv file, and I will see what is wrong with it.
My email is shawhj@meadowcrk.com
// Jerry
•
•
Join Date: Mar 2008
Posts: 13
Reputation:
Solved Threads: 0
he jerry your email address is not working i tried to send my project but its not working by the way i have created new code which is working fine and its up and running it can import about 20000 records from a file to a table the only problem occurs is that when i try to import a file having records having 500000 records i get this kind of error message
System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.WriteSni()
at System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte flushMode)
at System.Data.SqlClient.TdsParser.WriteByteArray(Byte[] b, Int32 len, Int32 offsetBuffer, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.WriteEncodingChar(String s, Int32 numChars, Int32 offset, Encoding encoding, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.WriteValue(Object value, MetaType type, Int32 actualLength, Int32 encodingByteSize, Int32 offset, TdsParserStateObject
see the code of mine goes here
now can you please help me... thanking you in advance..
System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.WriteSni()
at System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte flushMode)
at System.Data.SqlClient.TdsParser.WriteByteArray(Byte[] b, Int32 len, Int32 offsetBuffer, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.WriteEncodingChar(String s, Int32 numChars, Int32 offset, Encoding encoding, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.WriteValue(Object value, MetaType type, Int32 actualLength, Int32 encodingByteSize, Int32 offset, TdsParserStateObject
see the code of mine goes here
C# Syntax (Toggle Plain Text)
static void Main(string[] args) { string connectionstring = @"Data Source=ACC-222\SQLEXPRESS;Initial Catalog=mydata;Integrated Security=True"; SqlConnection con = new SqlConnection(connectionstring); con.Open(); SqlCommand mycount = new SqlCommand("select count(*) from dbo.mytable",con); long countstart = Convert.ToInt32(mycount.ExecuteScalar()); Console.WriteLine("stating row count={0}", mycount); DbConnectionStringBuilder csb = new DbConnectionStringBuilder(); csb["Provider"] = "Microsoft.Jet.OLEDB.4.0"; csb["Data Source"] = @"C:\Documents and Settings\jitendra\Desktop"; csb["Extended Properties"] = "Text;HDR=NO"; string cs = csb.ConnectionString; OleDbConnection txtconnection = new OleDbConnection(cs); txtconnection.Open(); OleDbCommand mycommand = new OleDbCommand("Select * from RIC_customer_20080318.csv", txtconnection); OleDbDataReader dr = mycommand.ExecuteReader(CommandBehavior.SequentialAccess); SqlConnection seccon = new SqlConnection(connectionstring); seccon.Open(); SqlBulkCopy bc = new SqlBulkCopy(seccon); bc.DestinationTableName = "dbo.mytable"; try { bc.WriteToServer(dr); } catch (SqlException se) { StreamWriter sr = new StreamWriter(@"C:\error.txt"); Console.WriteLine(se.ToString()); sr.WriteLine(se.ToString()); Console.ReadLine(); } dr.Close(); long countend = Convert.ToInt32(mycount.ExecuteScalar()); Console.WriteLine("Ending row count = {0}", countend); Console.WriteLine("{0} rows were added.", countend - countstart); }
now can you please help me... thanking you in advance..
•
•
Join Date: Nov 2006
Posts: 436
Reputation:
Solved Threads: 72
Curious as to why you are using the Jet driver ? (slow, unreliable)
Next, you should break the transactions up into chunks of < 10,000 lines per chunk.
You might also want to place an Application.DoEvents between chunks.
The Garbage collection subsystem any many other subsystems need a few cycles every now and then to do their job. IOW, make your loop play nice with the OS and Network.
// Jerry
Next, you should break the transactions up into chunks of < 10,000 lines per chunk.
You might also want to place an Application.DoEvents between chunks.
The Garbage collection subsystem any many other subsystems need a few cycles every now and then to do their job. IOW, make your loop play nice with the OS and Network.
// Jerry
![]() |
Other Threads in the C# Forum
- Previous Thread: Display Data on Datagrid
- Next Thread: Problem in checkboxes set
| Thread Tools | Search this Thread |
.net access activedirectory ado.net algorithm array barchart bitmap box broadcast c# check checkbox client combobox contorl control conversion csharp custom database datagrid datagridview dataset datetime degrees deployment development disabled displayingopenforms draganddrop drawing editing editor encryption enum event excel file form format forms ftp function gdi+ httpwebrequest i18n image imageprocessing index index-error input install java label list listbox mandelbrot math mathematics mouseclick mysql operator oracle path photoshop picturebox pixelinversion post prime programming radians regex remoting richtextbox rows server setup sleep socket sql statistics stream string table text textbox thread time timer update user usercontrol validation visualstudio webbrowser windows winforms wpf xml





