RSS Forums RSS
Please support our MS SQL advertiser: Programming Forums
Views: 11156 | Replies: 15 | Solved | Thread Tools  Display Modes
Reply
Join Date: Sep 2006
Location: San Diego, CA USA
Posts: 100
Reputation: M_K_Higa is an unknown quantity at this point 
Rep Power: 3
Solved Threads: 2
M_K_Higa's Avatar
M_K_Higa M_K_Higa is offline Offline
Junior Poster

Question Get data out of excel file stored as an image

  #1  
Sep 22nd, 2006
Greetings,

I've posted this question in other news groups (msdn and a few others) without any response. I hope I don't get into any trouble for posting this question here, but I reach out for your help.

I have an excel file stored in a data base table as an image data type. I would like to somehow read the data from the excel file and extract its data into a data base table using t-sql.

I've used sql statements such as the following but this requires the excel file to reside as a physical file on the sql servers file system.

insert into SpreadsheetDataDump (f1,f2,f3,f4,f5,f6)  
Select f1,f2,f3,f4,f5,f6 
FROM OPENDATASOURCE
('Microsoft.Jet.OLEDB.4.0','Data Source="D:\spreadsheets\test.xls";Extended Properties="Excel 8.0;IMEX=1;HDR=NO"')
...WorksheetName$ 

In my case, I do not want the excel file to reside on the file system because I require many different users to submit excel files. I just cannot afford to expose the file system to so many users, so I've created a web interface for the users to upload the excel file into the database.

As I mentioned, the excel file gets inserted into a table as an image data type. Now I want to select this image data using t-sql and somehow extract the excel spreadsheet data. If I can't do this, can I at least parse the image data into a physical file somewhere on the SQL server? I don't mind parsing it to disk at this point because by now the process is handled by SQL and the end user is out of picture.

Can this be done? Or do I have to parse out the image data as an excel file somewhere on the sql servers file system?

If I have to parse the image data to the file system, can this be done in t-sql? i.e. does t-sql have the ability to stream image data to the file system either natively or by extended stored procedures? (I want to avoid writing a dll wrapper to do this.)

I'm using SQL Server 2000 on Windows 2000 server.

Thanks for any help you can provide.

-Mike
AddThis Social Bookmark Button
Reply With Quote  
Join Date: Sep 2006
Location: San Diego, CA USA
Posts: 100
Reputation: M_K_Higa is an unknown quantity at this point 
Rep Power: 3
Solved Threads: 2
M_K_Higa's Avatar
M_K_Higa M_K_Higa is offline Offline
Junior Poster

Re: Get data out of excel file stored as an image

  #2  
Feb 7th, 2007
Ok folks, I didn't get any replies but I see a few views -- 811 at this point. I completed this task a while ago, but forgot about this thread.

If anyone is interested in how I coded this task, please reply to this thread. If I see anyone interested, I'll post my code. Hopefully the folks who viewed this thread are seeking a solution to the same issues I was facing.
-Mike
Reply With Quote  
Join Date: Mar 2007
Posts: 1
Reputation: juncmayl is an unknown quantity at this point 
Rep Power: 0
Solved Threads: 1
juncmayl juncmayl is offline Offline
Newbie Poster

Re: Get data out of excel file stored as an image

  #3  
Mar 19th, 2007
Originally Posted by M_K_Higa View Post
Ok folks, I didn't get any replies but I see a few views -- 811 at this point. I completed this task a while ago, but forgot about this thread.

If anyone is interested in how I coded this task, please reply to this thread. If I see anyone interested, I'll post my code. Hopefully the folks who viewed this thread are seeking a solution to the same issues I was facing.


I'm definitely interested in seeing how you accomplished this; I'm in this position myself now and would rather not have to reinvent the wheel if i didn't have to...

thanks!
Reply With Quote  
Join Date: Sep 2006
Location: San Diego, CA USA
Posts: 100
Reputation: M_K_Higa is an unknown quantity at this point 
Rep Power: 3
Solved Threads: 2
M_K_Higa's Avatar
M_K_Higa M_K_Higa is offline Offline
Junior Poster

Re: Get data out of excel file stored as an image

  #4  
Mar 19th, 2007
Ok. Hang on; it's going to take a while for me to gather all of the information.
-Mike
Reply With Quote  
Join Date: Sep 2006
Location: San Diego, CA USA
Posts: 100
Reputation: M_K_Higa is an unknown quantity at this point 
Rep Power: 3
Solved Threads: 2
M_K_Higa's Avatar
M_K_Higa M_K_Higa is offline Offline
Junior Poster

Re: Get data out of excel file stored as an image

  #5  
Mar 19th, 2007
First of all, let me recap the situation. I need an interface that allows users to upload excel data into a sql database.

1) I need a way to get excel data into my sql database.

2) I use the following sql statement to get the excel data into my sql database…

  1. INSERT INTO SpreadsheetDataDump (f1,f2,f3,f4,f5,f6)
  2. SELECT f1,f2,f3,f4,f5,f6
  3. FROM OPENDATASOURCE
  4. ('Microsoft.Jet.OLEDB.4.0','Data Source="D:\spreadsheets\test.xls";Extended Properties="Excel 8.0;IMEX=1;HDR=NO"')
  5. ...WorksheetName$

Keep in mind, in this scenario, the excel file must exist on the sql server.

3) Users need a way to upload the excel file onto the sql server.

4) I cannot grant users direct access to the sql server’s file system.

Task: How does the user get the excel file onto the sql server file system without granting access to the sql server’s file system?

In essence, there are three steps in this process. 1. Get excel file into sql table. 2. Put excel file (stored as Image) onto disc. 3. Insert excel data into sql database.

Step-1) I use a web (asp.net) interface to upload the excel file into a sql table as an Image data type. I’m going to defer you to http://aspnet.4guysfromrolla.com/articles/120606-1.aspx, which explains the process of getting a binary file uploaded into a sql table. You’ll get a lot more stuff out of this article, but we’re concerned with the part about getting the file uploaded into the database. I thought it was a neat article, so I'm giving you the link instead of my code.

I use a similar table structure in my database. This is the table structure I use in my code:

  1. CREATE TABLE BinaryDataFiles(
  2. [PkID] [int] IDENTITY(1,1) NOT NULL,
  3. [FileName] [nvarchar](255) ,
  4. [FileSize] [real],
  5. [FileData] [image],
  6. [UploadDate] [datetime] DEFAULT (getdate()),
  7. [ContentType] [nvarchar](30)
  8. )

Step-2) Here’s the part that actually gets the excel file onto the sql server’s file system. I use a console application to dump the excel file (stored as an Image) onto disc. This console application is called from a sql stored procedure.

Here’s the code that streams the image data to file. It compiles to a file called StreamImageData.exe.

  1. using System;
  2. using System.Collections.Generic;
  3. using System.Text;
  4. using System.Data;
  5. using System.Data.SqlClient;
  6. using System.Diagnostics;
  7. using System.IO;
  8. using System.Configuration;
  9.  
  10. namespace StreamImageData {
  11. class Program {
  12. static int Main(string[] args) {
  13. try {
  14. string ConStr = "Data Source=theServer;Initial Catalog=theDatabase;Persist Security Info=True;User ID=theUser;Password=thePassword";
  15. SqlConnection dbConn = new SqlConnection(ConStr);
  16. string sql_str = "select FileSize, FileName, FileData, ContentType from BinaryDataFiles where (PkID = @PkID)";
  17. SqlCommand cmdGetFile = new SqlCommand(sql_str, dbConn);
  18. SqlParameter paramPkID = new SqlParameter("@PkID", SqlDbType.Int);
  19. cmdGetFile.Parameters.Add(paramPkID);
  20. paramPkID.Direction = ParameterDirection.Input;
  21. paramPkID.Value = args[0].ToString(); // The primary key of the blob.
  22. string fileName = args[1].ToString(); // The output file name.
  23. dbConn.Open();
  24. SqlDataReader dr;
  25. dr = cmdGetFile.ExecuteReader();
  26. if(dr.Read()) {
  27. // Create random data to write to the file.
  28. byte[] dataArray = (byte[])dr["FileData"];
  29. using(FileStream fileStream = new FileStream(fileName, FileMode.Create)) {
  30. // Write the data to the file, byte by byte.
  31. for(int i = 0; i < dataArray.Length; i++) {
  32. fileStream.WriteByte(dataArray[i]);
  33. }
  34. }
  35. }
  36. return 0;
  37. } catch(Exception ex) {
  38. Debug.Write(ex.ToString());
  39. return -1;
  40. }
  41. }
  42. }
  43. }

This application expects two arguments – the primary key for the record which stores the image data and an output file name.

I use a stored procedure to execute StreamImageData.exe. The stored procedure goes something like this...

  1. USE [YourDatabase]
  2. GO
  3. CREATE PROCEDURE [dbo].[BinaryDataFiles_StreamImage2File](
  4. @FileName AS varchar(256)
  5. ,@ID AS varchar(64)
  6. ,@Debug AS bit = 0 --defaults to 0 (off)
  7. )
  8. AS
  9. SET nocount ON
  10.  
  11. DECLARE @TheResult int
  12. declare @command_string AS nvarchar(2000)
  13. SET @command_string = '"d:\apps\bin\StreamImageData.exe" '
  14. + @ID + ' ' + @FileName
  15.  
  16. EXEC @TheResult = master.dbo.xp_cmdshell @command_string
  17. print @TheResult
  18.  
  19. declare @ItExists AS int
  20. exec master..xp_fileexist @FileName, @ItExists out
  21. IF @ItExists = 0 -- It failed; the file doesn't exist.
  22. begin
  23. -- Dump some kind of fail message
  24. end

Step-3) Now that the excel file is parsed onto the sql server’s file system, you can insert the excel data into your database table.
Once again, I use some code that looks like the following to get the excel data into the sql table.

  1. INSERT INTO SpreadsheetDataDump (f1,f2,f3,f4,f5,f6)
  2. SELECT f1,f2,f3,f4,f5,f6
  3. FROM OPENDATASOURCE
  4. ('Microsoft.Jet.OLEDB.4.0','Data Source="D:\spreadsheets\test.xls";Extended Properties="Excel 8.0;IMEX=1;HDR=NO"')
  5. ...WorksheetName$

And this is what the SpreadsheetDataDump table’s structure looks like…

  1. CREATE TABLE BinaryDataFiles(
  2. [f1] [nvarchar](255) ,
  3. [f2] [nvarchar](255) ,
  4. [f3] [nvarchar](255) ,
  5. [f4] [nvarchar](255) ,
  6. [f5] [nvarchar](255) ,
  7. [f6] [nvarchar](255)
  8. )

These are the fundamental components of getting this task done. I have all three of these steps launch automatically, so to the user, it seems like the spreadsheet data automatically populated into the database. In the end, I also delete the image data out of the table and delete the excel file from the file system. Once the sql data is put into the table, my application had no reason to keep the file or image data.

There's a lot more that I've done in my production code. I didn't post all of it because I don't know what applies to your situation, so instead of cluttering this up with unnecessary code, I choose to provide the basics. I hope this helps - if not, please let me know your specifics and I will answer them as best as I can.

Also note that in this case, the spreadsheet data is being thrown into varchar fields. You may need to tweak your data if you want it in a different data type other that character data.
-Mike
Reply With Quote  
Join Date: Nov 2007
Posts: 6
Reputation: tatini is an unknown quantity at this point 
Rep Power: 0
Solved Threads: 1
tatini tatini is offline Offline
Newbie Poster

Re: Get data out of excel file stored as an image

  #6  
Nov 2nd, 2007
Is there any way we can specify the file name and location in Datasource as a variable?

Like can we replace the following with a variable?

'Data Source="D:\spreadsheets\test.xls";Extended Properties="Excel 8.0;IMEX=1;HDR=NO"'
Reply With Quote  
Join Date: Sep 2006
Location: San Diego, CA USA
Posts: 100
Reputation: M_K_Higa is an unknown quantity at this point 
Rep Power: 3
Solved Threads: 2
M_K_Higa's Avatar
M_K_Higa M_K_Higa is offline Offline
Junior Poster

Re: Get data out of excel file stored as an image

  #7  
Nov 2nd, 2007
Yes -- you can build the statement and then use the execute command to execute the statement.

I do something similar to the following in my actual code...

  1. declare @text_01 AS varchar(1000)
  2. declare @text_02 AS varchar(500)
  3. declare @spreadsheet AS varchar(250)
  4. declare @worksheet AS varchar(250)
  5. declare @sql_statement AS varchar(2000)
  6.  
  7. SET @spreadsheet = 'c:\path\test.xls'
  8. SET @worksheet = 'Sheet1'
  9.  
  10. SET @text_01 = 'INSERT INTO SpreadsheetDataDump (f1,f2,f3,f4,f5,f6) SELECT f1,f2,f3,f4,f5,f6 '
  11.  
  12. SET @text_02 = ' FROM OPENDATASOURCE(''Microsoft.Jet.OLEDB.4.0'',''Data Source="'
  13. + @spreadsheet + '";Extended Properties="Excel 8.0;IMEX=1;HDR=NO"'')...'
  14. + @worksheet + '$'
  15.  
  16.  
  17. SET @sql_statement = @text_01 + @text_02
  18.  
  19. exec (@sql_statement )

In my actual code, the @spreadsheet and @worksheet variables are passed in as parameters. I've declared them in this sample code for clarity.

Hope this helps.
-Mike
Reply With Quote  
Join Date: Nov 2007
Posts: 6
Reputation: tatini is an unknown quantity at this point 
Rep Power: 0
Solved Threads: 1
tatini tatini is offline Offline
Newbie Poster

Re: Get data out of excel file stored as an image

  #8  
Nov 2nd, 2007
thanks a lot.
Started on the same path before posting that question. Acheieved the result in a similar way.
Reply With Quote  
Join Date: Nov 2007
Posts: 6
Reputation: tatini is an unknown quantity at this point 
Rep Power: 0
Solved Threads: 1
tatini tatini is offline Offline
Newbie Poster

Re: Get data out of excel file stored as an image

  #9  
Nov 2nd, 2007
All is done and working but when i import the data from the excel to sql table, i see a change in the data.
the excel sheet has a column with numbers stored as text. they have a 0 as the first digit. this is gone once the sheet is imported into the column with data type varchar.

is there a way out of this?
Reply With Quote  
Join Date: Sep 2006
Location: San Diego, CA USA
Posts: 100
Reputation: M_K_Higa is an unknown quantity at this point 
Rep Power: 3
Solved Threads: 2
M_K_Higa's Avatar
M_K_Higa M_K_Higa is offline Offline
Junior Poster

Re: Get data out of excel file stored as an image

  #10  
Nov 2nd, 2007
Yes -- this is a hurdle that we have to face. You have to wirte a second routine that will cast your data into correct data types.

I tweak my data by getting the table containing the spreadsheet data and inserting it into a second table. The second table is defined with correct data types, so I have to cast anything that is not a character data type.
-Mike
Reply With Quote  
Reply

Only community members can participate in forum threads. You must register or log in to contribute.



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 

Thread Tools Display Modes
Forums | Blogs | Tutorials | Code Snippets | Whitepapers | RSS Feeds | Advertising
All times are GMT -4. The time now is 3:25 am.
Newsletter Archive - Sitemap - Privacy Statement - Acceptable Use Policy - Contact Us
Forum system based on vBulletin Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
©2003 - 2008 DaniWeb® LLC