Proposal: FAT+ Version: Draft release (revision 2) Changes: rev 1: initial release rev 2: clarified FAT+ is possible on FAT16, use GiB not GB, clarified issue on FAT systems preventing >=4GiB files this spec addresses Primary Authors (based on discussions from): Udo Kuhnt, Luchezar Georgiev, and Jeremy Davis Purpose: To allow file sizes to exceed 4GiB on FAT[1] file systems while maintaining maximum compatibility. Scope: This document is intended to document the physical changes to a FAT file system on disk to support large files (files with a size greater or equal to 4GiB), and compatibility issues for using such extensions with an operating system that is not aware of them. It does not discuss means to access [read/write/modify] these files except briefly and in regards to such access in comparison to operating systems that understand FAT+ versus ones with no knowledge of FAT+. Background: The FAT file system itself poses limits to a file's size by several means, - the file allocation table's bit width (12,16, or 32[28 used] bits corresponding to FAT12, FAT16, and FAT32 respectively) - the size of a cluster, 512 bytes, 1024 bytes, 2048 bytes, ... - the size of the partition, hence actual count of clusters - the size in bytes of the file as stored in a directory entry However, the method used to chain entries means that a file can be of unlimited length provided another free cluster is available (the bit width of cluster index limits the maximum possible free clusters for a particular FAT (FAT12, FAT16, FAT32) and the size of the physical disk limits the maximum possible free clusters on the given partition). DIR ENTRY->Cluster 1->Cluster 2->...->Cluster N->END can increase to DIR ENTRY->Cluster 1->Cluster 2->...->Cluster N->Cluster N+1->END where a maximum file size is approximately max clusters * bytes in a cluster; with max clusters equal to minimal of [clusters in the partition, 2^FAT's bit width] The first three are limits imposed by the nature of the FAT file system as formatted and the physical characteristics of the drive/partition it is placed upon. These do not by themselves prevent the use of 4GiB files (see previous paragraph, these three together determine the maximum free clusters available to store a file, hence maximum size of a file that can be stored on the drive). The fourth restriction (file size stored in directory entry), however, does prevent any OS file system driver that respects file size boundaries [as opposed to cluster chain and possible terminating character such as Ctrl-Z for file termination] from creating and/or modifying a file greater than 4GiB - 1 byte (0xFFFFFFFF bytes) in size. It is this limit that this document tries to overcome in a compatible manner. This means the FAT+ extension may be used on either FAT16 or FAT32 file systems. (Note that usage on FAT16 requires a partition with a cluster size of 128KB or larger. Any OS that supports FAT16 partitions with a cluster size of 128KB should also support FAT16+.) The basic mechanism is compatible with FAT12, but as the maximum count of clusters is too small to exceed current limits, offers no gain other than shared code path for FAT operations. [1] For purposes of this document, please see the hardware white paper from Microsoft titled "Microsoft Extensible Firmware Initiative FAT32 File System Specification, FAT: General Overview of On-Disk Format" version 1.03 for the official FAT file system specification that this proposal extends. Implementation on disk: Given a file that is exactly 4GiB-1 (2^32 - 1 bytes) or less, the file should be stored as a normal entry [no changes from Microsoft's FAT specification, as the whole file size will fit in the SFN directory entry (32bits, max value 0xFFFFFFFF)]. Once a file meets or exceeds 4GiB, the file is physically stored no differently; that is, a new cluster is allocated as needed and the FAT chain updated as normal (last entry points to new cluster, new cluster points to end of chain marker). However, the file size now exceeds the 32 bits available in the SFN directory entry, so additional bits are required. The FAT+ proposal uses 6 additional bits from the byte marked reserved for NT. NT Reserved Byte 0x0C: (DIR_NTRes in the Microsoft FAT file system specification and labeled as "Reserved for use by Windows NT. Set value to 0 when a file is created and never modify or look at it after that.") NT uses this byte in special cases to indicate that a portion of the filename should be displayed in lowercase. Specifically, the third and fourth bits when set indicate that the filename portion and/or extension are in normal 8.3 format except all lowercase. "... they are bit flags (or masks), so if (DIR_NTRes & 0x08) then filename portion is all lowercase and if (DIR_NTRes & 0x10) then extension portion is all lowercase. Note though that the whole filename must be a valid 8.3 name and mixed case within filename or extension can not be used (if it is then a LFN entry is created instead and these bits aren't used [set to 0]) ..."[2] [2] this is an extract from a conversation based on a comment in some public domain source code Windows 9x does not use these bits and will instead create a LFN entry to store the filename case; this leads to a known minor compatibility issue where Windows 9x will display the name in all capitals but NT uses lowercase. This means bits 0-2 and 5-7 are reserved, so these 6 bits are used by a FAT+ file system to store the upper 6 bits of a file's size when the size becomes or exceeds 4GiB, giving 38 bits instead of just 32 to store the size. This allows for a file up to 274877906944 bytes (256GiB). On a normal file (< 4GiB) these size bits should always be 0, so the size can effectively always be treated as a 38bit value. We do not use bits 3-4, as there is no way to discern if the entry corresponds to a large file (>= 4GiB in size) or an NT filename in 8.3 format with lowercase letters. The mere existence of an LFN entry accompanying this SFN entry does not indicate these bits are in use or not as the file could have been created on NT and subsequently renamed on Windows 9x, so the bits left set and an LFN entry created. Compatibility with non-FAT+ aware operating systems: There are several factors to consider, ability to read data in a large file, ability to modify (write) data in a large file, ability to manipulate meta data (file name, file size, file times, ...) about a large file, and ability to access other files or otherwise manipulate directory entries [create, delete, read, modify other files/directories and LFN data]. By limiting the extension to these 6 bits in the directory entry, all normal directory operations (rename/move/delete/etc) and to the limit described below read operations should work without problems and as expected if the FAT specification is fully followed. The OS must leave the 6 reserved bits as-is ("... never modify ..."), a non-conforming OS may clear them even at times other than creation. There should be no compatibility issues with long filenames, as LFN entries should be treated exactly as before (either the OS supports them and uses/updates them, or the OS does not support them and they are lost/invalidated). Additionally, all normal meta data manipulations or other directory modifications should work fine as all data except the NTReserved byte (now upper byte of file size) in the SFN entry are exactly as indicated and used by the FAT specification and conforming file system drivers (please be aware a non-conforming OS may clear the upper bits of the extended file size on any operation that writes the SFN entry to disk including renames or moves resulting in an incorrect file size for the large file; see below for details on this issue). Even if a file is longer than 4 GiB, a non-FAT+ aware OS can still access the *entire* file, if and only if it does only either sequential read or sequential write operations, without any "seek" (position file pointer) operations. Care still needs to be taken to ensure no operation would cause the OS to update the file size field without further testing to ensure it is done in a compatible method [the file remains 4GiB or larger so that the upper 6 bits are preserved]. This means all files, even large ones, can be read by any OS that understands the FAT file system. An unexpected error may still occur if the read would cross or exceed the 4GiB boundary, as the OS may not be prepared for such data; this should only occur in the case that the file system driver allocates enough memory for the whole file as indicated by the non-extended file size field (so does not reserve enough room for the whole file) and then simply reads data in following the cluster chain without regard to how many bytes are actually read in (should not occur in practice and an indication of other potential issues with such an OS if one can read past indicated file size). Reads by non-FAT+ aware OSes will probably be limited to the file size modulo 4GiB (i.e. size indicated by lower 32 bits of 38 bit file size) and not 4GiB. [To overcome this and allow at least 4GiB to be read would require storing the value 0xFFFFFFFF in the normal file size field, but this leaves the issue of where to store these 32 bits then.] A FAT+ aware operating system may allow current APIs to access the full file and reporting its file size as the maximum supported by such APIs, however, this does not effect the on disk structures and so beyond the scope of this document. WARNING: Writing to a large file from a non-FAT+ aware OS could corrupt the file and cause data loss. This is unavoidable. The main issue is the cluster chain could be severed (possibly fixable if no other lost clusters exist, where a chain of one or more lost clusters are the severed clusters and simply need to be re-attached at the end of the file). Writing to a >= 4GiB file should not be done when using a non-FAT+ aware OS unless you wish to damage the file. A particular not FAT+ aware operating system may be capable of safely updating data within the 1st 4GiB (or when using sequential writes even further) of the file, but will most likely set the file size wrong (fix as mentioned below may still be possible). Similar to reading, a FAT+ aware OS may allow writing using existing APIs, possibly limiting writes to within current 4GiB limits. The biggest issue (other than a portion of the file inaccessible) for non-FAT+ aware operating systems is corruption of the file size. Any OS file system driver that does not recognize FAT+ file systems (does not follow the final form of this proposal) and does not follow the MS FAT specification closely may alter [clear] the six bits used for the upper portion of the file size. In practice this should only occur when the file is modified, specifically any action that causes the OS to write out the SFN directory entry may change these bits. For example, current versions of NT (NT4,2000,XP,2003) clear these 6 bits when renaming or moving the file. Note: there is no data loss with this, as by following the cluster chain all data can be reclaimed (the file size reset to a value >= to the original value with all data still there, but with additional junk at the end unless the file was exactly a multiple of the cluster size). Assuming the lower 32 bits of the file size are still valid the correct 38-bit file size can be easily restored should the upper 6 bits become zeroed. To repair the full file size one need only determine how many clusters are in use (follow the cluster chain), multiply this value by size of a cluster, then subtract the unused space in the file's last cluster. The unused space is easily calculated by taking the lower 32 bits mod the cluster size, and subtracting this value from the cluster size. It is expected a chkdsk type program actually perform this fix due to the potential time needed to verify and traverse the cluster chain, especially if many files or an unknown number of files in a volume need repairing. Please note that access (read/write/modify) to all files <= 4GiB is unchanged, so any DOS, Windows, or other operating system that could read/write/modify the files should continue to do so without any additional issues. FAT+ aware but not supported OS: At a minimal a FAT+ aware operating system must preserve the 6 reserved bits now used for extended file size and the OS utilities [chkdsk, etc] must not truncate or terminate abnormally when the cluster chain exceeds the length possible if the file size was limited to 4GiB. The operating system may limit file access [reads and writes] to within the first 4GiB of the file, but when reasonable should allow full access if no explicit file positioning [seek] logic (other than setting to beginning or end of file) is used and the access is done via sequential means (i.e. sequentially reading a file from beginning to end). A FAT+ capable operating system must provide a means to seek within full 38 bits of file and random read/writes; additionally it must maintain the 6 additional size bits and its utilities should support the extended size including where possible a tool to correct the file size when upper six bits are lost. Note: the actual API to do this is not discussed as it is outside the scope of this document. Explored but rejected considerations: The upper two attribute bits can not be used, as although most DOSes do not use them, they have been and may still be in use so would be ambiguous to use them for a different purpose. Using an LFN entry to store data about the file, such as true file size. The problem is most DOS systems are not LFN aware, so any rename, etc would loose the LFN entry and hence the file's size. LFN aware systems suffer from the problem of how to keep them from replacing the data when a user does something such as a rename, and how to keep such an OS from displaying garbage corresponding to the data stored. Using a special format for the SFN filename or other marker values in the SFN directory entry can not be done as this would still require an LFN entry or something similar to store the additional data (true file size, real name, etc), so suffers the same problems [DOS can easily loose the LFN entry, or overwrite the values] as using an LFN entry to store extra data. FAT version update: For maximum safety, on FAT32 volumes the file system version may be set to 0.1 (still a major version of 0, but a minor version of 1 [instead of 0]). Any FAT+ aware operating system should recognize this version as only indicating that large files may be used (upper 6 bits may be stored in the NTReserved byte) and non-FAT+ aware operating systems should fail to mount this volume. It should be noted that a FAT+ aware OS must not assume a FAT version of 0.0 to mean no 4GiB+ files exist as the setting of the version to 0.1 on disk is optional. If needed, a hex editor or other tool can reset the version to 0.0 to allow non-FAT+ aware operating sytems to mount the volume, with all the potential compatibility problems mentioned above. FAT16 volumes do not support a version number, however, the use of large files is only possible on FAT16 partition formated with cluster sizes of 128KB or larger. Since such large cluster sizes are only used by OSes that also support FAT16+, any OS that later adds support for such large clusters must also at least be made FAT+ aware as this indicates large files may exist (note that it gives no diffinite determination, only that such files can be safely be created). Other ideas: If at least one bit is available to indicate a large file, the file size may stored in the standard file size field as a modulo of the cluster size (possibly using ones complement notation) and then the operating system on first access would follow the cluster chain to calculate the true size. This true size should then be cached so future access to the file need not require the possible time consuming cluster chain traversal. Miscellaneous Terms: DOS - Disk Operating System, in this document specifically MS-DOS or compatible operating systems for IBM compatible Personal Computers. FAT - File Allocation Table, technically a table at the beginning of the file system used to store which clusters are allocated to a particular file, an array based linked-list; more commonly used to indicate the usage of a particular file system defined by Microsoft and used by DOS and other operating systems, that this specification extends FAT+ - an extension to the FAT file system to allow usage of large (>=GiB) files SFN - Short File Name, in regards to the FAT file system, the standard directory entry stores only a short 8.3 file name along with other important details such as starting cluster # and file size LFN - Long File Name, in regards to the FAT file system, additional specially crafted directory entries are linked to an SFN entry to store the case preserving file name that may exceed the 8.3 convention and is stored as 16 bit Unicode characters GiB - a binary (not decimal) gigabyte in SI notation == 1024*1024*1024 bytes All trademarks, service marks, etc. are property of their respective owners and use in this document in no way indicate any particular approval or knowledge of this specification. End.