ZIP (file format)

From Wikipedia, the free encyclopedia

This article needs additional citations for verification. Please help improve this article by adding reliable references. Unsourced material may be challenged and removed. (August 2007)

ZIP

A zip archive icon from the Torchlight icon set.
Filename extension	.zip
Internet media type	application/zip
Uniform Type Identifier	com.pkware.zip-archive
Magic number	PK\003\004 or PK\005\006 (empty archive) (unless a bootstrap script is present)
Developed by	Phil Katz
Type of format	Data compression

The ZIP file format is a data compression and archival format. A ZIP file contains one or more files that have been compressed, to reduce their file size, or stored as-is. A number of compression algorithms are permitted in zip files but as of 2008 only DEFLATE is widely used and supported.

The format was originally evolved by Phil Katz for PKZIP from the previous ARC compression format by Thom Henderson. However, many software utilities other than PKZIP itself are now available to create, modify, or open (unzip, decompress) ZIP files, notably WinZip, BOMBArchiveHelper, KGB Archiver, PicoZip, Info-ZIP, WinRAR, IZArc, 7-Zip, ALZip, TUGZip, PeaZip, Universal Extractor and Zip Genius. Microsoft has included built-in ZIP support (under the name "compressed folders") in later versions of its Windows operating system. Apple has included built-in ZIP support in Mac OS X 10.3 and later via the BOMArchiveHelper utility.The zip, zipcloak, zipnote, zipsplit tools are used widely in unix-like systems.

ZIP files generally use the file extensions ".zip" or ".ZIP" and the MIME media type application/zip. Some software uses the ZIP file format as a wrapper for a large number of small items in a specific structure. Generally when this is done a different file extension is used. Examples of this usage are Java JAR files, Python .egg files, id Software .pk3/.pk4 files, package files for StepMania and Winamp/Windows Media Player skins, XPInstall, as well as OpenDocument and Office Open XML office formats. Both OpenDocument and Office Open XML formats use the JAR file format internally, so files can be easily uncompressed and compressed using tools for ZIP files. Google Earth makes use of KMZ files, which are just KML files in ZIP format. Mozilla Firefox Add-ons are zip files with extension "xpi". Nokia's mobile phone themes are zipped with extension "nth". Sony Ericsson mobile phone themes are zipped with extension "thm".

Contents 1 History 1.1 Early history 1.2 Beyond the command line 1.3 Version history 1.3.1 4.5 1.3.2 5.2 1.3.3 6.1.0 1.3.4 6.2 1.3.5 6.3 2 Technical information 2.1 The format in detail 2.2 Implementing a ZIP application 3 Strong encryption controversy 4 References 5 See also 6 External links

[edit] History

[edit] Early history

The neutrality of this section is disputed. Please see the discussion on the talk page. (January 2008) Please do not remove this message until the dispute is resolved.

The ZIP file format was originally created by Phil Katz, founder of PKWARE.

During the mid-1980s, System Enhancement Associates, a small company run by Thom Henderson, created a file archiving format called ARC, and a corresponding archiver (also called ARC) that could compress and decompress files into this format.^[1] This program was released as shareware for a number of platforms, with the source code included. The file format quickly became a de facto standard.^[1] Phil Katz released a file compatible software package on the IBM Intel DOS platform, known as PKXARC. It used hand-optimized 8088 assembly language, it was considerably faster than SEA's original cross platform implementation in C.

The competition from Katz did not please SEA, who sued Katz for trademark and copyright infringement, as it alleged that Katz had plagiarized sections of the code. Katz lost the lawsuit and was forced to pay $62,500 to SEA to cover their legal fees. It was found during the court case^{[citation needed]} that Katz had used SEA's ARC source code for the majority of the application but had only made code optimizations to increase speed. Primarily he changed the word length used by the algorithm from 12 bits to 13 bits resulting in a higher compression for typical binary files. As a result of the lawsuit, Katz changed the names of his utilities to PKPAK and PKUNPAK.

Katz then went on to create his own file format, which is known worldwide now as the ZIP format (commonly called a "ZIP file"). The ZIP format was more resistant to data loss than the ARC format because of redundant catalog storage; it also was more flexible than ARC, providing room for additional optional compression algorithms and future expansion. Along with the new format, PKZIP included at least one compression algorithm more efficient than any supported by ARC. Once PKZIP was released, many users abandoned ARC because of its slower speed and less effective compression, and because SEA alienated many by seeming to suddenly assert proprietary legal rights over the ARC file format after it had become widely used among the on-line community (similar in this respect to the later GIF patents controversy).

Katz publicly released technical documentation on the ZIP file format making it an open format, along with the first version of his PKZIP archiver, in January 1989. Originally only bundled with registered versions of PKZIP, the APPNOTE.TXT documentation file, titled .ZIP File Format Specification, was later available in PKWARE site.

The name zip (meaning speed) was suggested by Katz's friend Robert Mahoney. They wanted to imply that their product would be faster than ARC and other compression formats of the time.

[edit] Beyond the command line

In the mid 1990s, as more new computers included graphical user interfaces, more users were not comfortable with the command-line operation of PKZIP. Seeing an opportunity, shareware authors began pitching compression and archival programs with graphical user interfaces. Many of these used the ZIP format. WinZip was among the most popular. PKWARE also offered a graphical version of PKZIP. These programs were easier to learn than the older command-line equivalents, but users still had to learn a specialized tool with its own interface for file archival and compression.

In the late 1990s, various file manager software started integrating support for the ZIP format into their user interface. Even earlier, Norton Commander and its clones like Volkov Commander in DOS had started that trend, and that remains the norm for the "Commander-like" or Orthodox file managers like Midnight Commander for Linux and UNIX-like systems and Total Commander (previously Windows Commander) for Windows. The KDE file manager (kfm) supported the ZIP format very early; ZIP support was also first added to Windows Explorer with the Plus! enhancement package in Windows 98 and later included in Windows Me and Windows XP; ZIP format support is also built in the Mac OS Finder (as of Mac OS X, via the BOMArchiveHelper utility), the Nautilus file manager used by GNOME and the Konqueror file manager of newer versions of KDE. By 2002, all major desktop environments included ZIP file support in their file managers: a ZIP file is typically presented as a directory or folder, so that files are copied into and out of it in the same manner as any other folder and the compression is handled in a way largely transparent to the user. This has eliminated the need to learn a specialized tool and interface for file archival and compression.

[edit] Version history

The .ZIP File Format Specification has its own version number, which do not necessarily correspond to the PKZIP version numbers, especially with PKZIP 6 or later. At various times, PKWARE adds preliminary features that allows PKZIP products to extract archives using advanced features, but PKZIP products that create such archives won't be available until next major release.

[edit] 4.5

Documented 64-bit ZIP format.

[edit] 5.2

Documented Strong Encryption Specification.

[edit] 6.1.0

Documented certificate storage.

[edit] 6.2

• 6.2.0: Documented Central Directory Encryption.

[edit] 6.3

• 6.3.0: Documented Unicode (UTF-8) filename storage. Expanded list of supported hash, compression, encryption algorithms.
• 6.3.1: Corrected standard hash values for SHA-256/384/512.
• 6.3.2: Documented compression method 97 (WavPack).

[edit] Technical information

ZIP is a fairly simple archive format that compresses every file separately. Compressing files separately allows for individual files to be retrieved without reading through other data; in theory, it may allow better compression by using different algorithms for different files. However a caveat to this is that archives containing a large number of small files end up significantly larger than if they were compressed as a single file (the classic example of the latter is the common tar.gz archive which consists of a TAR archive compressed using gzip).

The specification for ZIP indicates that files can be stored either uncompressed or using a variety of compression algorithms. However, in practice, ZIP is almost always used with Katz's DEFLATE algorithm, except when files being added are already compressed or are resistant to compression.

ZIP supports a simple password-based symmetric encryption system which is known to be seriously flawed. In particular it is vulnerable to known-plaintext attacks which are in some cases made worse by poor implementations of random number generators.^[2] It also supports spreading archives across multiple removable disks (generally floppy disks, but it could also be used with other removable media).

New features including new compression and encryption (e.g. AES) methods have been documented to .ZIP File Format Specification since version 5.2. WinZip developed AES based standard is used also by 7-Zip but some vendors use other formats.^[3] PKWARE SecureZIP also supports DC2, DC4, DES, 3DES encryption methods, Digital Certificate (X.509)-based encryption and authentication, archive header encryption.^[4]

The original ZIP format had a number of limits (uncompressed size of a file, compressed size of a file and total size of the archive) at 4GB. In version 4.5 of the specification, PKWARE introduced the "ZIP64" format extensions to get around these limitations.

The FAT filesystem of DOS only has a timestamp resolution of two seconds; ZIP file records mimic this. As a result, the built-in timestamp resolution of files in a ZIP archive is only two seconds, though extra fields can be used to store more accurate timestamps.

Since September 2007, the ZIP specification (APPNOTE.TXT) contains a provision to store file names using UTF-8, finally adding Unicode compatibility to ZIP.

The Info-ZIP implementations of the ZIP format adds support for Unix filesystem features, such as user and group IDs, file permissions, and support for symbolic links. The Apache Ant implementation is aware of them to the extent that it can create files with predefined Unix permissions.

The Info-ZIP Windows tools also support NTFS filesystem permissions, and will make an attempt to translate from NTFS permissions to Unix permissions or vice-versa when extracting files. This can result in potentially unintended combinations, e.g. .exe files being created on NTFS volumes with executable permission denied.

[edit] The format in detail

The ZIP file contents are files and directories which are stored in arbitrary order. The location of a file is indicated in the so called central directory which is located at the end of the ZIP file. The files and directories are represented by file entries.

Besides the file data each file entry is introduced by a local header with information about the file such as the comment, file size and file name. The central directory consists of file headers holding the relative offset of the local headers for each file.

Due to the arbitrary order and the fact that the order of the file entries and the corresponding header references in the central directory may be different, the format is non-sequential.

[edit] Implementing a ZIP application

There are some useful development libraries which are available as open source contributions such as the GNU gzip project and Info-ZIP. For Java, the Java SE (Standard Edition) contains the package "java.util.zip"; Zip64File specifically supports large files (larger than 4GB) and treats ZIP files using random access, and the Apache Ant tool contains a more complete implementation released under the Apache Software License. There is also a no-cost .NET Library called DotNetZip available under a public license. The Microsoft NET 3.5 runtime library includes a class System.IO.Packaging.Package^[5] that supports the ZIP format.

According to the PKWARE Inc. ZIP file format specification^[4] supports large files (more than 65000 entries and entries larger than 4GB). File encryption, and the possibility to span a ZIP file over multiple files are supported as well.

However, not all of these features are implemented by the known development libraries.

[edit] Strong encryption controversy

When WinZip 9.0 public beta was released in 2003, WinZip introduced its own AES-256 compression, using a different file format, along with the documentation for the new specification. ^[6] The encryption standards themselves were not proprietary, but PKWARE had not updated APPNOTE.TXT to include Strong Encryption Specification (SES) since 2001, which had been used by PKZIP versions 5.0 and 6.0. WinZip technical consultant Kevin Kearney and StuffIt product manager Mathew Covington accused PKWARE for withholding SES, but PKZIP chief technology officer Jim Peterson claimed that Certificate-based encryption was still incomplete. However, the latest publicly available APPNOTE.TXT at the time was version 4.5 (available on PKWARE's FTP site), which not only omitted SES, but also omitted Deflate64, DCL Implode, BZip2 compression methods used by .ZIP files created by contemporary PKZIP products.

To overcome this shortcoming, contemporary products such as StuffIt and PentaZip 'implemented' strong ZIP encryption by encrypting ZIP archives into a different file format.^[7]

In another controversial move, PKWare applied for a patent in 2003-07-16 describing a method for combining .ZIP and strong encryption to create a secure .ZIP file.^[8]

In the end, PKWARE and WinZip agreed to support each other's products. On 2004-01-21, PKWARE announced the support of WinZip-based AES compression format.^[9] In later version of WinZip beta, it is able to support SES-based ZIP files.^[10] PKWARE eventually released version 5.2 of .ZIP File Format Specification to public, which documented SES.

[edit] References

[edit] See also

• Random Access
• PKZIP
• List of archive formats
• LZW compression method
• Comparison of file archivers

[edit] External links

• Judgment in favor of SEA in SEA v. PKWARE and Phil Katz
• Technical specifications of the PKZIP file formats from info-ZIP
• Current file format specification from PKWARE (including many recent features that are not widely supported)
• 18 Years of ZIP format: Happy Birthday at The Data Compression News Blog
• Comparison of the performances of various methods of data compression (french)
• .NET Zip library - reads and writes zip archives.
• ZIP2 file format specification

v • d • e Archive formats (comparison by type) Archiving only ar · cpio · shar · tar · LBR Compression only bzip2 · gzip · LZMA · lzop · rzip · SQ · compress · LZW Archiving and compression 7z · ACE · ARC · ALZip · ARJ · Cabinet · cpt · DAR · dd · DGCA · .dmg · GCA · kgb · LHA · LZX · PAQ · RAR · qda · sit · SQX · zoo · ZIP Software packaging and distribution deb · pkg · RPM · MSI · JAR (WAR · RAR (Java) · EAR)