NTFS Sparse Files (NTFS5 only)
A sparse file has an attribute that causes the I/O subsystem to allocate only meaningful (nonzero) data. Nonzero data is allocated on disk, and non-meaningful data (large strings of data composed of zeros) is not. When a sparse file is read, allocated data is returned as it was stored; non-allocated data is returned, by default, as zeros.
NTFS deallocates sparse data streams and only maintains other data as allocated. When a program accesses a sparse file, the file system yields allocated data as actual data and deallocated data as zeros.
NTFS includes full sparse file support for both compressed and uncompressed files. NTFS handles read operations on sparse files by returning allocated data and sparse data. It is possible to read a sparse file as allocated data and a range of data without retrieving the entire data set, although NTFS returns the entire data set by default.
With the sparse file attribute set, the file system can deallocate data from anywhere in the file and, when an application calls, yield the zero data by range instead of storing and returning the actual data. File system application programming interfaces (APIs) allow for the file to be copied or backed as actual bits and sparse stream ranges. The net result is efficient file system storage and access. Next figure shows how data is stored with and without the sparse file attribute set.
Windows 2000 Data Storage
If you copy or move a sparse file to a FAT or a non-NTFS volume, the file is built to its originally specified size. If the required space is not available, the operation does not complete.
About
LSoft Technologies Inc. is a privately owned North American software company. Our goal is to create world’s leading data recovery, security and backup solutions by providing rock solid performance, innovation, and unparalleled customer service.
Sparse file
According to Wikipedia, in computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently when blocks allocated to a file are mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual «empty» space which makes up the block, using less disk space. The full block size is written to disk as the actual size only when the block contains «real» (non-empty) data.
When reading sparse files, the file system transparently converts metadata representing empty blocks into «real» blocks filled with zero bytes at runtime. The application is unaware of this conversion.
Most modern file systems support sparse files, including most Unix variants and NTFS, but notably not Apple’s HFS+. Sparse files are commonly used for disk images (not to be confused with sparse images), database snapshots, log files and in scientific applications.
The advantage of sparse files is that storage is only allocated when actually needed: disk space is saved, and large files can be created even if there is insufficient free space on the file system.
Disadvantages are that sparse files may become fragmented; file system free space reports may be misleading; filling up file systems containing sparse files can have unexpected effects; and copying a sparse file with a program that does not explicitly support them may copy the entire file, including the empty blocks which are not on explicitly stored on the disk, which wastes the benefits of the sparse property of a file.
Creating sparse files
The truncate utility can create sparse files. This command creates a 512 MiB sparse file:
$ truncate -s 512M file.img
The dd utility can also be used, for example:
$ dd if=/dev/zero of=file.img bs=1 count=0 seek=512M
Sparse files have different apparent file sizes (the maximum size to which they may expand) and actual file sizes (how much space is allocated for data on disk). To check a file’s apparent size, just run:
$ du -h --apparent-size file.img
512M file.img
and, to check the actual size of a file on disk:
$ du -h file.img
0 file.img
As you can see, although the apparent size of the file is 512 MiB, its «actual» size is really zero—that’s because due to the nature and beauty of sparse files, it will «expand» arbitrarily to minimize the space required to store its contents.
Making existing files sparse
The fallocate utility can make existing files sparse on supported file systems:
$ fallocate -d copy.img $ du -h copy.img 0 copy.img
Making existing files non-sparse
The following command creates a non-sparse copy of a (sparse) file:
$ cp file.img copy.img --sparse=never $ du -h copy.img 512M copy.img
Creating a filesystem in a sparse file
This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.
Reason: Sparse files do not have to contain a file system, the purpose should be explained. (Discuss in Talk:Sparse file)
Now that we have created a sparse file, it is time to format it with a filesystem; for example ReiserFS:
$ mkfs.reiserfs -f -q file.img
We can now check its size to see how a filesystem has affected it:
$ du -h --apparent-size file.img
512M file.img
$ du -h file.img
33M file.img
As you may have expected, formatting it with a filesystem has increased its actual size, but left its apparent size the same. Now we can create a directory which we will use to mount our file:
# mount --mkdir -o loop file.img mountpoint
Tada! We now have both a file and a folder into which we may store almost 512 MiB worth of information!
Mounting a file at boot
To mount a sparse image automatically at boot, add an entry to your fstab:
/path/to/file.img /path/to/mountpoint reiserfs loop,defaults 0 0
Note: Be sure to include the loop option, otherwise it will not mount.
Detecting sparse files
Since sparse files occupy less blocks than the apparent file size would require, they can be detected by comparing the two sizes. This is not a bulletproof method if the filesystem uses compression, extended attributes take up the difference in space, file is internally fragmented, has indirect blocks, and similar. Still, the standard way to check is:
$ ls -ls sparse-file.bin
If a file size is greater than the allocated size in the first column a file is sparse. The same can be achieved with du by comparing:
$ du sparse-file.bin $ du --apparent-size sparse-file.bin
A step further is to print sparsiness value with find:
$ find sparse-file.bin -printf '%S\t%p\n'
A sparse file has a sparsiness value of less than one whereas normal files have exactly one or just slightly above. The above command can be easily extended to list sparse files in a desired path:
$ find path/ -type f -printf '%S\t%p\n' | gawk '$1 < 1.0 ' | cut -f '2-'
Copying a sparse file
Copying with cp
Normally, cp is good at detecting whether a file is sparse, so it suffices to run:
$ cp file.img new_file.img
Then new_file.img will be sparse. However, cp does have a --sparse=when option. This is especially useful if a sparse file has somehow become non sparse (i.e. the empty blocks have been written out to disk in full). Disk space can be recovered by:
$ cp --sparse=always new_file.img recovered_file.img
Archiving with tar
This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.
Reason: Informal writing, see Help:Style#Language register. (Discuss in Talk:Sparse file)
One day, you may decide to back up your well-loved sparse file, and choose the tar utility for that very purpose; however, you soon realize you have a problem:
$ du -h file.img
33M file.img
$ tar -cf file.tar file.img
$ du -h file.tar
513M file.tar
Apparently, even though the current size of the sparse file is only 33 MB, archiving it with tar created an archive of the ENTIRE SIZE OF THE FILE! Luckily for you, though, tar has a `--sparse' (`-S') flag, that when used in conjunction with the `--create' (`-c') operation, tests all files for sparseness while archiving. If tar finds a file to be sparse, it uses a sparse representation of the file in the archive. This is useful when archiving files, such as dbm files, likely to contain many nulls, and dramatically decreases the amount of space needed to store such an archive.
$ tar -Scf file.tar file.img
$ du -h file.tar
12K file.tar
Resizing a sparse file
This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.
Reason: Dependence on #Creating a filesystem in a sparse file is not apparent, not every sparse file contains a file system. Also too informal writing, see Help:Style#Language register. (Discuss in Talk:Sparse file)
Before we resize a sparse file, let us populate it with a couple small files for testing purposes:
$ for f in ; do touch folder/file$; done
$ ls folder/ file1 file2 file3 file4 file5
Now, let us add some content to one of the files:
$ echo "This is a test to see if it works. " >> folder/file1
$ cat folder/file1 This is a test to see if it works.
Growing a file
Should you ever need to grow a file, you may do the following:
# umount folder # dd if=/dev/zero of=file.img bs=1 count=0 seek=1G 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.2978e-05 s, 0.0 kB/s
This will increase its size to 1 Gb, and leave its information intact. Next, we need to increase the size of its filesystem:
# resize_reiserfs file.img resize_reiserfs 3.6.21 (2009 www.namesys.com) ReiserFS report: blocksize 4096 block count 262144 (131072) free blocks 253925 (122857) bitmap block count 8 (4) Syncing..done resize_reiserfs: Resizing finished successfully.
# mount -o loop file.img folder
Checking its size gives us:
# du -h --apparent-size file.img 1.0G file.img # du -h file.img 33M file.img
. and to check for consistency:
# df -h folder Filesystem Size Used Avail Use% Mounted on /tmp/file.img 1.0G 33M 992M 4% /tmp/folder
# ls folder file1 file2 file3 file4 file5 # cat folder/file1 This is a test to see if it works.
Tools
- sparse-fio — dd-like program to work with files that are sparsely filled with non-zero data
- sparseutils — utilities to work with sparsely-populated files, provides mksparse.py and sparsemap.py , can be installed with pip
Sources
- wikipedia:Sparse_file
- https://web.archive.org/web/20121026035748/http://www.apl.jhu.edu/Misc/Unix-info/tar/tar_85.html
Retrieved from "https://wiki.archlinux.org/index.php?title=Sparse_file&oldid=752459"
Why can't I create sparse files on my NTFS partition under Linux?
On an EXT4 partition both of the created files were sparse ( du -h shows that they use 0 bytes on disk). But on the NTFS partition each file actually uses 100 KB of disk space, according to du -h and also according to ls -lsk .
So: why can't I create sparse files on the NTFS partition? And what can I do to copy the large file as sparse file to the NTFS partition?
The system where this happens:
- Debian 11 (Bullseye) x86-64
- kernel package 5.10.92-1
- ntfs-3g package 1:2017.3.23AR.3-4+deb11u1
According to mount , the NTFS partition is mounted like this:
/dev/sdg1 on /mnt/loop2 type fuseblk (rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
My understanding is that the partition is mounted using the ntfs-3g driver - is that correct? And according to man ntfs-3g that driver supports sparse files. I also did not find any option to enable or disable that feature.
I did not create the NTFS partition myself, but used the existing partition that was on the drive when I bought it.
Разреженные файлы в NTFS
В NTFS есть поддержка разреженных файлов (sparse files). Это такие файлы, которые занимают меньше дискового пространства, чем их собственный размер. Данная технология не имеет отношения к встроенной в NTFS поддержке компрессии файлов, так как экономия места на диске в sparse-файлах основана на другом принципе. Никакого сжатия данных не осуществляется. Вместо этого, в файле высвобождаются области, занятые одними лишь нулями (0x00). Приложение, читающее разреженный файл, дойдя до области с нулями, прочитает нули, но реального чтения с диска не произойдёт.
Таким образом можно создавать файлы гигантского размера, состоящие из нулей, но на диске они могут занимать всего лишь несколько килобайт. Реальное дисковое пространство выделяется тогда, когда вместо 0x00 записываются какие-то другие данные. Разреженность поможет сэкономить дисковое пространство только в таких файлах, в которых есть действительно большие пустые области.
Демонстрировать работу с разреженными файлами я буду с помощью системной утилиты fsutil в командной строке.
Создадим с помощью утилиты пустой файл большого размера:
fsutil file createnew test.nul 10000000000
Присвоим файлу атрибут «sparse»:
fsutil sparse setflag test.nul
Сам по себе атрибут ещё не приводит к экономии дискового пространства. Нужно ещё разметить внутри файла область, которая будет освобождена. У нас весь файл пустой, так что область можно задать в размер файла.
fsutil sparse setrange test.nul 0 10000000000
Готово. Смотрим результат.
Как проделывать эти операции в своих программах с помощью API функций, можете посмотреть в исходном коде небольшой утилиты, которую я написал в 2006-м. Она тоже консольная, и тоже как fsutil умеет присваивать атрибут sparse и задавать диапазон освобождаемой области. Кроме того, моя программка может сама искать в файле пустые области больше некоторого заданного размера и освобождать их. Отпадает необходимость самому вычислять смещения.
- Подробно о разреженных файлах
- Программа для создания разреженных (sparse) файлов в NTFS