Best compression

Thread Solved

Join Date: Sep 2007
Posts: 1,493
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 136
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Nearly a Posting Virtuoso

Best compression

 
0
  #1
Sep 17th, 2009
Hi and I am looking for a compression mechonism that can compress text (just letters and numbers) into binary or all weird symbols. Does anybody know of any good php/mysql codes as I can't get any to work.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - MacGyver Fan
Reply With Quote Quick reply to this message  
Join Date: Sep 2007
Posts: 1,493
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 136
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Nearly a Posting Virtuoso

Re: Best compression

 
0
  #2
Sep 18th, 2009
I have tried running the following mysql queries through the phpmyadmin command console (mysql_query() code) but I seem to be unable to retrieve the value. Here is my code.
INSERT INTO `table` SET `column`=COMPRESS("abcd")
SELECT * FROM `table` WHERE UNCOMPRESS(`column`)="abcd"
Also tried
INSERT INTO `table` SET `column`=COMPRESS("abcd")
SELECT * FROM `table` WHERE `column`=COMPRESS("abcd")
Please advice on how I should use the select query.
Last edited by cwarn23; Sep 18th, 2009 at 2:24 am.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - MacGyver Fan
Reply With Quote Quick reply to this message  
Join Date: Sep 2007
Posts: 1,493
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 136
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Nearly a Posting Virtuoso

Re: Best compression

 
0
  #3
Sep 18th, 2009
I think I may have found something with the bzip2/bz2 library because the gz library didn't compress very well. I will try storing the bzcompress() result in a binary column. Well let you know how my result goes.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - MacGyver Fan
Reply With Quote Quick reply to this message  
Join Date: Sep 2005
Posts: 1,081
Reputation: digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice 
Solved Threads: 66
Moderator
digital-ether's Avatar
digital-ether digital-ether is offline Offline
Veteran Poster

Re: Best compression

 
0
  #4
Sep 18th, 2009
Originally Posted by cwarn23 View Post
I have tried running the following mysql queries through the phpmyadmin command console (mysql_query() code) but I seem to be unable to retrieve the value. Here is my code.
INSERT INTO `table` SET `column`=COMPRESS("abcd")
SELECT * FROM `table` WHERE UNCOMPRESS(`column`)="abcd"
Also tried
INSERT INTO `table` SET `column`=COMPRESS("abcd")
SELECT * FROM `table` WHERE `column`=COMPRESS("abcd")
Please advice on how I should use the select query.
I tried this and it worked:

  1. SELECT * FROM `table` WHERE `compressed` = compress('test')

The only thing I can assume is the column containing the compressed data has to be a blob or varbinary type.
www.fijiwebdesign.com - web design and development and fun
Cpanel Email - Let users Register email accounts on your website upon registration
Ajax Chat - Fully browser based chat!
Reply With Quote Quick reply to this message  
Join Date: Sep 2007
Posts: 1,493
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 136
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Nearly a Posting Virtuoso

Re: Best compression

 
0
  #5
Sep 18th, 2009
I just tried varbinary like before and again varbinary didn't work however I tried blob which I have never heard of before and it worked! Also the bz library works but won't decompress. So using the method you have described how can I decompress the result?
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - MacGyver Fan
Reply With Quote Quick reply to this message  
Join Date: Sep 2005
Posts: 1,081
Reputation: digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice 
Solved Threads: 66
Moderator
digital-ether's Avatar
digital-ether digital-ether is offline Offline
Veteran Poster

Re: Best compression

 
1
  #6
Sep 18th, 2009
Originally Posted by cwarn23 View Post
I just tried varbinary like before and again varbinary didn't work however I tried blob which I have never heard of before and it worked! Also the bz library works but won't decompress. So using the method you have described how can I decompress the result?
Which bz library? Do you mean zlib? Is this in mysql or PHP?

You can just decompress the result in the SQL query:

  1. SELECT uncompress(test) as test FROM `table` WHERE `compressed` = compress('test')

This should give you the column test in decompressed form.

Note:

I wouldn't use:

  1. SELECT * FROM `table` WHERE UNCOMPRESS(`column`)="abcd"

The reason is that you'll run uncompress() on each row in the database table.
You will also be unable to use any index made on that column.

If you use:
  1. `column`= COMPRESS("abcd")
Then you only have to compress the input "abcd" once. You will also be able to use indexes made on the `column` column since you're comparing directly.
www.fijiwebdesign.com - web design and development and fun
Cpanel Email - Let users Register email accounts on your website upon registration
Ajax Chat - Fully browser based chat!
Reply With Quote Quick reply to this message  
Join Date: Sep 2007
Posts: 1,493
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 136
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Nearly a Posting Virtuoso

Re: Best compression

 
0
  #7
Sep 18th, 2009
SELECT uncompress(test) as test FROM `table` WHERE `compressed` = compress('test')
Wow, if that works that will be brillient. I will try it out.

And the confusion you may have had before about the bz library also known as bzip2 or bz2 depend on what version (I found out the hard way) that was another option I was exploring untill I found out it couldn't compress crc32 hashes. So as I said, if that above code works, that will be the solution to all my endless reverse hashing storage problems.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - MacGyver Fan
Reply With Quote Quick reply to this message  
Join Date: Sep 2005
Posts: 1,081
Reputation: digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice 
Solved Threads: 66
Moderator
digital-ether's Avatar
digital-ether digital-ether is offline Offline
Veteran Poster

Re: Best compression

 
0
  #8
Sep 18th, 2009
www.fijiwebdesign.com - web design and development and fun
Cpanel Email - Let users Register email accounts on your website upon registration
Ajax Chat - Fully browser based chat!
Reply With Quote Quick reply to this message  
Join Date: Sep 2007
Posts: 1,493
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 136
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Nearly a Posting Virtuoso

Re: Best compression

 
0
  #9
Sep 18th, 2009
It works!
Even though it requires twice as much space on the mysql cache it works perfectly. Thanks for that very usefull code that I could not find anywhere else.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - MacGyver Fan
Reply With Quote Quick reply to this message  
Join Date: Sep 2005
Posts: 1,081
Reputation: digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice digital-ether is just really nice 
Solved Threads: 66
Moderator
digital-ether's Avatar
digital-ether digital-ether is offline Offline
Veteran Poster

Re: Best compression

 
0
  #10
Sep 18th, 2009
Originally Posted by cwarn23 View Post
It works!
Even though it requires twice as much space on the mysql cache it works perfectly. Thanks for that very usefull code that I could not find anywhere else.
I wouldn't worry about the cache. But if you want, you can uncompress the result on the PHP side.

Assuming you're using the default compression, zlib, then you need to cut off the first 4 bytes before you uncompress the data.

The first 4 bytes is reserved for the length of the uncompressed string.

see: http://dev.mysql.com/doc/refman/5.0/...ction_compress

Here if a full example using PDO:

  1. <?php
  2. /* Connect to an ODBC database using driver invocation */
  3. $dsn = 'mysql:dbname=test;host=localhost';
  4. $user = 'root';
  5. $password = '';
  6.  
  7. try {
  8. $dbh = new PDO($dsn, $user, $password);
  9. } catch (PDOException $e) {
  10. echo 'Connection failed: ' . $e->getMessage();
  11. }
  12.  
  13. $sql = 'SELECT binary(compressed) AS compressed FROM `compressed` WHERE compress(:test) = compressed ';
  14. $sth = $dbh->prepare($sql, array(PDO::ATTR_CURSOR => PDO::CURSOR_FWDONLY));
  15. $sth->execute(array(':test' => 'test'));
  16. $row = $sth->fetch();
  17.  
  18. // compressed result, with first 4 bytes being the length
  19. $result = $row['compressed'];
  20.  
  21. // the length of uncompressed string
  22. $len = substr($result, 0, 4);
  23. // the compressed string
  24. $compressed = substr($result, 4, strlen($result)-4);
  25.  
  26. // uncompress the string
  27. echo gzuncompress($compressed, intval($len)); // test
  28.  
  29. ?>

Edit:

Here is the structure of the table I used in the example:

  1. +------------+------+------+-----+---------+-------+
  2. | Field | Type | Null | Key | Default | Extra |
  3. +------------+------+------+-----+---------+-------+
  4. | value | text | NO | | NULL | |
  5. | compressed | blob | NO | | NULL | |
  6. +------------+------+------+-----+---------+-------+

And the dump:

  1. --
  2. -- Database: `test`
  3. --
  4.  
  5. -- --------------------------------------------------------
  6.  
  7. --
  8. -- Table structure for table `compressed`
  9. --
  10.  
  11. CREATE TABLE IF NOT EXISTS `compressed` (
  12. `value` text COLLATE utf8_unicode_ci NOT NULL,
  13. `compressed` blob NOT NULL
  14. ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
  15.  
  16. --
  17. -- Dumping data for table `compressed`
  18. --
  19.  
  20. INSERT INTO `compressed` (`value`, `compressed`) VALUES
  21. ('test', 0x04000000789c2b492d2e0100045d01c1);
Last edited by digital-ether; Sep 18th, 2009 at 1:47 pm.
www.fijiwebdesign.com - web design and development and fun
Cpanel Email - Let users Register email accounts on your website upon registration
Ajax Chat - Fully browser based chat!
Reply With Quote Quick reply to this message  
Reply


This thread has been marked solved.
Perhaps start a new thread instead?
Message:


Thread Tools Search this Thread



About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC