unixdev.net


Switch to SpeakEasy.net DSL

The Modular Manual Browser

Home Page
Manual: (Debian-5.0)
Page:
Section:
Apropos / Subsearch:
optional field

IO::Compress::Base::FAUsermContributed Perl DocumeIO::Compress::Base::FAQ(3pm)



NAME
       IO::Compress::Base::FAQ -- Frequently Asked Questions about
       IO::Compress::Base

DESCRIPTION
       Common questions answered.

       Compatibility with Unix compress/uncompress.

       This module is not compatible with Unix "compress".

       If you have the "uncompress" program available, you can use this to
       read compressed files

           open F, "uncompress -c $filename |";
           while (<F>)
           {
               ...

       Alternatively, if you have the "gunzip" program available, you can use
       this to read compressed files

           open F, "gunzip -c $filename |";
           while (<F>)
           {
               ...

       and this to write compress files, if you have the "compress" program
       available

           open F, "| compress -c $filename ";
           print F "data";
           ...
           close F ;

       Accessing .tar.Z files

       See previous FAQ item.

       If the "Archive::Tar" module is installed and either the "uncompress"
       or "gunzip" programs are available, you can use one of these
       workarounds to read ".tar.Z" files.

       Firstly with "uncompress"

           use strict;
           use warnings;
           use Archive::Tar;

           open F, "uncompress -c $filename |";
           my $tar = Archive::Tar->new(*F);
           ...

       and this with "gunzip"

           use strict;
           use warnings;
           use Archive::Tar;

           open F, "gunzip -c $filename |";
           my $tar = Archive::Tar->new(*F);
           ...

       Similarly, if the "compress" program is available, you can use this to
       write a ".tar.Z" file

           use strict;
           use warnings;
           use Archive::Tar;
           use IO::File;

           my $fh = new IO::File "| compress -c >$filename";
           my $tar = Archive::Tar->new();
           ...
           $tar->write($fh);
           $fh->close ;

       Accessing Zip Files

       This module does not support reading/writing zip files.

       Support for reading/writing zip files is included with the
       "IO::Compress::Zip" and "IO::Uncompress::Unzip" modules.

       The primary focus of the "IO::Compress::Zip" and
       "IO::Uncompress::Unzip" modules is to provide an "IO::File" compatible
       streaming read/write interface to zip files/buffers. They are not fully
       flegged archivers. If you are looking for an archiver check out the
       "Archive::Zip" module. You can find it on CPAN at

           http://www.cpan.org/modules/by-module/Archive/Archive-Zip-*.tar.gz

       Compressed files and Net::FTP

       The "Net::FTP" module provides two low-level methods called "stor" and
       "retr" that both return filehandles. These filehandles can used with
       the "IO::Compress/Uncompress" modules to compress or uncompress files
       read from or written to an FTP Server on the fly, without having to
       create a temporary file.

       Firstly, here is code that uses "retr" to uncompressed a file as it is
       read from the FTP Server.

           use Net::FTP;
           use IO::Uncompress::Bunzip2 qw(:all);

           my $ftp = new Net::FTP ...

           my $retr_fh = $ftp->retr($compressed_filename);
           bunzip2 $retr_fh => $outFilename, AutoClose => 1
               or die "Cannot uncompress '$compressed_file': $Bunzip2Error\n";

       and this to compress a file as it is written to the FTP Server

           use Net::FTP;
           use IO::Compress::Bzip2 qw(:all);

           my $stor_fh = $ftp->stor($filename);
           bzip2 "filename" => $stor_fh, AutoClose => 1
               or die "Cannot compress '$filename': $Bzip2Error\n";

       How do I recompress using a different compression?

       This is easier that you might expect if you realise that all the
       "IO::Compress::*" objects are derived from "IO::File" and that all the
       "IO::Uncompress::*" modules can read from an "IO::File" filehandle.

       So, for example, say you have a file compressed with gzip that you want
       to recompress with bzip2. Here is all that is needed to carry out the
       recompression.

           use IO::Uncompress::Gunzip ':all';
           use IO::Compress::Bzip2 ':all';

           my $gzipFile = "somefile.gz";
           my $bzipFile = "somefile.bz2";

           my $gunzip = new IO::Uncompress::Gunzip $gzipFile
               or die "Cannot gunzip $gzipFile: $GunzipError\n" ;

           bzip2 $gunzip => $bzipFile
               or die "Cannot bzip2 to $bzipFile: $Bzip2Error\n" ;

       Note, there is a limitation of this technique. Some compression file
       formats store extra information along with the compressed data payload.
       For example, gzip can optionally store the original filename and Zip
       stores a lot of information about the original file. If the original
       compressed file contains any of this extra information, it will not be
       transferred to the new compressed file usign the technique above.

       Using "InputLength" to uncompress data embedded in a larger
       file/buffer.

       A fairly common use-case is where compressed data is embedded in a
       larger file/buffer and you want to read both.

       As an example consider the structure of a zip file. This is a well-
       defined file format that mixes both compressed and uncompressed
       sections of data in a single file.

       For the purposes of this discussion you can think of a zip file as
       sequence of compressed data streams, each of which is prefixed by an
       uncompressed local header. The local header contains information about
       the compressed data stream, including the name of the compressed file
       and, in particular, the length of the compressed data stream.

       To illustrate how to use "InputLength" here is a script that walks a
       zip file and prints out how many lines are in each compressed file (if
       you intend write code to walking through a zip file for real see
       "Walking through a zip file" in IO::Uncompress::Unzip )

           use strict;
           use warnings;

           use IO::File;
           use IO::Uncompress::RawInflate qw(:all);

           use constant ZIP_LOCAL_HDR_SIG  => 0x04034b50;
           use constant ZIP_LOCAL_HDR_LENGTH => 30;

           my $file = $ARGV[0] ;

           my $fh = new IO::File "<$file"
                       or die "Cannot open '$file': $!\n";

           while (1)
           {
               my $sig;
               my $buffer;

               my $x ;
               ($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH
                   or die "Truncated file: $!\n";

               my $signature = unpack ("V", substr($buffer, 0, 4));

               last unless $signature == ZIP_LOCAL_HDR_SIG;

               # Read Local Header
               my $gpFlag             = unpack ("v", substr($buffer, 6, 2));
               my $compressedMethod   = unpack ("v", substr($buffer, 8, 2));
               my $compressedLength   = unpack ("V", substr($buffer, 18, 4));
               my $uncompressedLength = unpack ("V", substr($buffer, 22, 4));
               my $filename_length    = unpack ("v", substr($buffer, 26, 2));
               my $extra_length       = unpack ("v", substr($buffer, 28, 2));

               my $filename ;
               $fh->read($filename, $filename_length) == $filename_length
                   or die "Truncated file\n";

               $fh->read($buffer, $extra_length) == $extra_length
                   or die "Truncated file\n";

               if ($compressedMethod != 8 && $compressedMethod != 0)
               {
                   warn "Skipping file '$filename' - not deflated $compressedMethod\n";
                   $fh->read($buffer, $compressedLength) == $compressedLength
                       or die "Truncated file\n";
                   next;
               }

               if ($compressedMethod == 0 && $gpFlag & 8 == 8)
               {
                   die "Streamed Stored not supported for '$filename'\n";
               }

               next if $compressedLength == 0;

               # Done reading the Local Header

               my $inf = new IO::Uncompress::RawInflate $fh,
                                   Transparent => 1,
                                   InputLength => $compressedLength
                 or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

               my $line_count = 0;

               while (<$inf>)
               {
                   ++ $line_count;
               }

               print "$filename: $line_count\n";
           }

       The majority of the code above is concerned with reading the zip local
       header data. The code that I want to focus on is at the bottom.

           while (1) {

               # read local zip header data
               # get $filename
               # get $compressedLength

               my $inf = new IO::Uncompress::RawInflate $fh,
                                   Transparent => 1,
                                   InputLength => $compressedLength
                 or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

               my $line_count = 0;

               while (<$inf>)
               {
                   ++ $line_count;
               }

               print "$filename: $line_count\n";
           }

       The call to "IO::Uncompress::RawInflate" creates a new filehandle $inf
       that can be used to read from the parent filehandle $fh, uncompressing
       it as it goes. The use of the "InputLength" option will guarantee that
       at most $compressedLength bytes of compressed data will be read from
       the $fh filehandle (The only exception is for an error case like a
       truncated file or a corrupt data stream).

       This means that once RawInflate is finished $fh will be left at the
       byte directly after the compressed data stream.

       Now consider what the code looks like without "InputLength"

           while (1) {

               # read local zip header data
               # get $filename
               # get $compressedLength

               # read all the compressed data into $data
               read($fh, $data, $compressedLength);

               my $inf = new IO::Uncompress::RawInflate \$data,
                                   Transparent => 1,
                 or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

               my $line_count = 0;

               while (<$inf>)
               {
                   ++ $line_count;
               }

               print "$filename: $line_count\n";
           }

       The difference here is the addition of the temporary variable $data.
       This is used to store a copy of the compressed data while it is being
       uncompressed.

       If you know that $compressedLength isn't that big then using temporary
       storage won't be a problem. But if $compressedLength is very large or
       you are writing an application that other people will use, and so have
       no idea how big $compressedLength will be, it could be an issue.

       Using "InputLength" avoids the use of temporary storage and means the
       application can cope with large compressed data streams.

       One final point -- obviously "InputLength" can only be used whenever
       you know the length of the compressed data beforehand, like here with a
       zip file.

SEE ALSO
       Compress::Zlib, IO::Compress::Gzip, IO::Uncompress::Gunzip,
       IO::Compress::Deflate, IO::Uncompress::Inflate,
       IO::Compress::RawDeflate, IO::Uncompress::RawInflate,
       IO::Compress::Bzip2, IO::Uncompress::Bunzip2, IO::Compress::Lzop,
       IO::Uncompress::UnLzop, IO::Compress::Lzf, IO::Uncompress::UnLzf,
       IO::Uncompress::AnyInflate, IO::Uncompress::AnyUncompress

       Compress::Zlib::FAQ

       File::GlobMapper, Archive::Zip, Archive::Tar, IO::Zlib

AUTHOR
       This module was written by Paul Marquess, pmqs@cpan.org.

MODIFICATION HISTORY
       See the Changes file.

COPYRIGHT AND LICENSE
       Copyright (c) 2005-2008 Paul Marquess. All rights reserved.

       This program is free software; you can redistribute it and/or modify it
       under the same terms as Perl itself.



perl v5.10.0                      2008-07-15      IO::Compress::Base::FAQ(3pm)