# How it works This document describes how HatchZip extracts an archive while giving its space back, and why each step happens in the order it does. Read it before changing anything in `engine.rs`, `journal.rs` or `archive/storage.rs` under `crates/core/src`. ## The space problem A normal extractor reads the archive and writes the output, then the user deletes the archive. At the moment just before that deletion, the disk holds both: the full archive and the full output. For a 180 GB archive that unpacks to 200 GB, that moment needs 380 GB. HatchZip deletes the archive gradually, in step with writing the output, so the disk never has to hold much more than the output alone. ## Why from the end Files can be shortened from the end on every operating system and filesystem (`ftruncate`, `SetEndOfFile`). Removing bytes from the start or the middle of a file needs hole punching (`fallocate` on Linux, `FSCTL_SET_ZERO_DATA` on Windows, `F_PUNCHHOLE` on macOS), which FAT32 and exFAT do not support and which behaves differently on each system. So HatchZip works from the end. Each format describes its archive as a list of **units**: byte ranges that decode on their own, and that can be truncated away once every file in them is safely on disk. | Format | Unit | Layout | |---|---|---| | ZIP | One entry (local header plus data) | Entries in order, then the central directory, the index, at the very end. A self-extracting stub may come first. | | 7z | One solid block, or one file in a non-solid archive | A 32-byte signature header, the packed blocks in order, then the header (the index) at the end. | | tar | One entry, including any long-name or pax records before it | Header and data records in order, then zero blocks. There is no index. | | `.tar.gz`, `.tar.bz2`, `.tar.xz` | The whole archive, extracted from the front instead; see [Compressed tar](#compressed-tar) | One compressed stream holding a tar | | RAR | One file, from its first header; a solid archive is one unit, extracted only while it is kept; see [RAR](#rar) | Headers and data in order, in one file or over volumes that are complete RAR files, a file split over several of them. There is no index. | Units are extracted in reverse order of their position in the file. After each batch the archive is truncated to where the lowest unit of the batch begins. A solid 7z block can only be decoded from its beginning, so it is freed only once all of it is extracted. That makes the largest unit the main cost of in-place extraction: the run needs free space for the output plus roughly the largest unit. 7-Zip's default settings can put a whole archive into one block, which leaves nothing to gain; the space check reports this before anything is touched. For one large file the same cost is avoided where the filesystem can punch holes: see [Freeing space inside a large file](#freeing-space-inside-a-large-file). Directories and empty files in 7z have no data at all, only header entries. They form one extra unit that starts where the header does, so it is done first and truncating it frees only the header, which is already saved. ## Split archives A split archive (`game.7z.001`, `game.7z.002`, ...) as 7-Zip writes it is one archive cut into byte ranges. `Storage` (`crates/core/src/archive/storage.rs`) joins the volumes into one logical file. Truncating it to a length deletes every volume that lies wholly past that length and shortens the one it falls in. Volume boundaries need no special handling: a unit may span two volumes, and the volume past it is deleted once the unit is committed. Any volume's name finds the whole set, and the journal records every volume with its original length so a resume can check that what is left is an untouched prefix. ## The saved tail ZIP and 7z keep their index at the end, which is the first thing truncation removes. Before anything is truncated, HatchZip copies everything after the last unit's data (the "tail") into `.hatchzip-tail` and records its CRC-32 in the journal. For tar the tail is just the closing zero blocks. From then on the archive is parsed through `storage::Reader`, which serves the live volumes below the tail offset and the saved tail above it, so the format library sees the archive exactly as it was. Regions already truncated away read as zeros. The zip crate scans backwards over them while looking for its end record, and zeros never match a signature; for tar, the zeros read like the end of the archive, so parsing what is left yields exactly the units still to do. ## The journal `.hatchzip` is a small JSON file holding: | Field | Meaning | |---|---| | `format` | `zip`, `7z`, `tar`, `tar.gz`, `tar.bz2`, `tar.xz` or `rar`, so a resume never has to guess from a half-eaten file | | `parts` | Every volume with its original length | | `tail_start`, `tail_crc` | Where the saved tail belongs, and its checksum | | `output` | The output folder, so a resume can't be pointed elsewhere | | `total_size` | Total output size, for progress | | `committed_len` | The archive length after the latest commit. Every unit starting at or past it is done. | | `links` | Symlinks and hard links to create at the very end | | `resume` | While a large file is streamed: its unit, how many bytes of it are flushed, the bit of the archive decoding restarts at, and the CRC-32 so far. For a compressed tar, also `stream`, the decoder's and the tar parser's state, tagged `gzip`, `bzip2` or `xz`. See below. | | `done` | Present and true once the final commit is journaled: every unit is extracted, and only the archive's removal and the links are left. A compressed stream's start may already be punched away by then, so a resume finishes without reading it. | | `keep` | Present and true when the extraction keeps the archive; see [Keeping the archive](#keeping-the-archive) | | `case_folders` | Present when entries' names differ only in case: the folders where they meet. The output folder can move to a filesystem that ignores case, and a tar no longer lists the entries it has extracted, so a resume and a restore ask about these again before anything is deleted or written, rather than trusting the answer from the start, and `check` does while there are entries left to plan. `case_beyond_ascii` is set when some of those names go outside ASCII, for a run that goes on where the system normalizes names. A journal with them is at version 7, or 8 with `folders`, which older builds refuse | | `replace` | Present when the extraction replaces existing files, with `moved` once they are all moved aside; see [Replacing existing files](#replacing-existing-files) | | `folders` | Present when the archive has folder entries with permissions or a modified time: how many are saved in `.hatchzip-folders`, and its CRC-32. A resume checks the file before it changes anything. A look at the archive and a restore never read it, and a run with every entry out finishes without it, warning how many folders keep their defaults | | `folders_left` | Present and true once the links are made and the replaced files settled: a resume only sets the folders and cleans up | The journal is at version 3 since `resume` was added. Version 2 journals load as they are; older builds refuse version 3, which matters, because they would delete the half-written file whose archive bytes are already gone. `resume.stream` came later without a new version: `tar.gz`, `tar.bz2` and `tar.xz` journals have it, and builds that don't know those formats refuse the journal before reading it. `done` was added the same way; a build that ignores it truncates and reopens the archive as it always did. The gzip fields are written even at their defaults, because 0.1.8 reads this structure and has no defaults for them; only `kind`, `block`, `level` and `combined_crc` stay out at theirs. A journal with `keep` is version 4, unless a later version takes over: builds from before `keep` refuse it, since they would resume it in place and truncate an archive the user asked to keep. Every other journal is still written as version 3, so an older build can finish it. A journal with `replace` is version 5 unless a later version takes over, kept archive or not: older builds would delete the person's files at the paths of entries left to do, and a restore would not put the replaced files back. A journal with `folders` is version 8, whatever else it records, `case_folders` included: older builds would finish without setting the folders. There is no separate "done" counter: units start at distinct offsets, so `committed_len` alone says which are finished. The journal is always replaced atomically: written to `.hatchzip.tmp`, flushed, renamed over the old one, and the directory flushed. The saved tail is written the same way through `.hatchzip-tail.tmp`. Whatever is at either temporary name first is removed, which removes a link and never what it points to, and the file is then created only where nothing exists, without following a link. ### Whose journal it is The journal and the saved tail are plain files next to the archive, so whoever supplied the archive could have supplied them too. A resume deletes what it finds at the paths of the entries left, so a journal it trusted wrongly could make it delete the person's own files there, even in the default folder. So every extraction also records its archive and output folder in the person's profile, outside the archive's folder (`crates/core/src/chosen.rs`): `%LOCALAPPDATA%\HatchZip`, `~/Library/Application Support/HatchZip` or `$XDG_STATE_HOME/hatchzip`. A resume or restore uses a journal only when that record names the same archive and folder, or when the person names the folder again with `-o`, for example on another computer. Anything else is refused before a file is changed. The record goes when the extraction completes or is restored. Releases before 0.1.6 recorded only folders chosen with `-o` or in the app, so an extraction they started in the default folder resumes with `-o`. Whatever a journal says, it can only name the archive's own volumes, none of them a link, and links inside the output folder. ## One batch Units are processed in batches. Each batch goes through these steps, in this order: 1. **Choose the batch.** Starting from the next uncommitted unit, take units while their extracted size fits in the free space that exists right now, minus `--keep-free`. A batch holds at most 512 units and 2 GB. 2. **Extract in parallel.** Each unit is decoded by a worker thread, and every file in it is written to `.hatchzip-part`, which is created only where nothing exists and never through a link; anything at that name stops the run instead (see [Rejected before starting](#rejected-before-starting)). The decoders verify CRCs as each stream ends, and HatchZip checks the written size against the size the archive declares. Each finished part file is renamed to its real name. Links are not created yet, only recorded. 3. **Flush.** The batch is forced to the physical disk: one `syncfs` on Linux, an fsync of every file and directory elsewhere (`F_FULLFSYNC` on macOS). 4. **Commit.** The journal is updated with the new `committed_len` and the batch's links. 5. **Truncate.** The archive is shortened to `committed_len`, the start of the batch's lowest unit. The freed space becomes available for the next batch. At the end, links are created, then folders get their permissions and modified times, then the remaining archive (at most a self-extractor stub or the 7z signature header) is deleted, then the side files. Every file and link made inside a folder changes its modified time, and a folder without write permission takes no new files, so folders are set last, deepest first, without following a link. A ZIP or 7z folder entry's header can be truncated long before the end, so before anything is extracted the folder entries are written to `.hatchzip-folders`, the same way as the saved tail. Only folders the extraction creates are listed: one already in the output is left as it was, like an existing file. Once the links are made, the journal records `folders_left`, so a resume never makes a link again inside a folder that may no longer take new entries. A resume can also find folders already set without read or search permission, so it first gives the owner full access back, shallowest first, then sets them all again, deepest first. ## Freeing space inside a large file A unit only frees its archive bytes when it is committed, so one 30 GB file needs 30 GB free before any of its bytes can go. Where the archive's filesystem can punch holes (free bytes in the middle of a file without moving the rest), a unit holding one file of at least 512 MiB, stored or deflated in a ZIP or any regular file in a tar, is **streamed** instead, alone, when its output doesn't fit whole in the space free at that moment: 1. Its file is written to `.hatchzip-part`. Every 256 MiB of output, at the next point where decoding can restart: 2. **Flush** the part file and the folders above it. 3. **Journal the resume point**: output length, the bit of the archive decoding restarts at, and the CRC-32 so far. 4. **Punch** the archive from the start of the file's data up to the byte holding that bit. The archive's length doesn't change, and the freed bytes read as zeros. When the file is done it is renamed and committed through the ordinary steps 3 to 5, in the same journal write that clears `resume`. The local header or tar header in front of the data is never punched, so the archive still parses on resume. Stored data can restart at any byte. A deflate stream can restart at any block boundary, the technique of zlib's `examples/zran.c`, using the C API of zlib-rs (`inflatePrime`, `inflateSetDictionary`, `Z_BLOCK`). A boundary may fall inside a byte, so the point is recorded in bits, and the decoder also needs the 32 KiB of output before the point, which back-references reach into. That window is not stored in the journal: it is the last 32 KiB of the part file, flushed before the point was journaled. Hole punching is detected by punching a 1 MiB scratch file next to the archive and measuring that its allocation dropped, since a call can succeed without freeing anything. Without it (FAT32, exFAT, HFS+) every unit is extracted whole, as before. 7z solid blocks are not streamed yet. | Interrupted | State on disk | What a resume does | |---|---|---| | Before the first point is journaled | A part file, nothing punched | Deletes it and starts the file over | | After the flush, before the journal write | The part file holds output past the previous point, which is still the one in force | Cuts the part file back to the previous point and decodes from there; everything it needs is still in the archive | | After the journal write, before or after the punch | The point is durable, the bytes before it may be freed | Cuts the part file to the point and decodes from it, never reading before it | | After the rename, before the commit | The file has its real name, the journal still holds the last point | Renames it back and continues from that point | The rule: **the archive is never punched past the last durable resume point**, and a point is only durable once the output it vouches for is flushed. A half-streamed file is never deleted as uncommitted output. `restore` refuses an archive whose journal holds a resume point, since an entry with holes in it can't be put back. The test suite kills the process at each of the three steps of a point at several points, for a stored ZIP entry, a deflated ZIP entry and a tar entry, and around the final commit (`crates/cli/tests/streaming.rs`); a unit test resumes a deflate stream from every block boundary at levels 6 and 9 (`crates/core/src/archive/resumable.rs`). A deflate block can be as long as its encoder likes. zlib-ng and zlib-rs write level 1 as one block, which has no point to resume from but its start; such a file frees nothing until it ends, and if the space runs out first the run stops and says how much to free. zlib-rs at levels 6 and 9 wrote a block boundary about every 300 KB of output on the unit test's data, and the ZIP entries written by the zip crate and by Python's `zipfile` in the tests above had boundaries throughout; other encoders are not measured. ## Compressed tar A `.tar.gz`, `.tar.bz2` or `.tar.xz` holds the whole tar in one compressed stream, and extraction walks the compressed stream from the front. So these are not split into units extracted from the end: the whole archive is one unit, extracted front to back, and the compressed bytes behind the decoder are freed by punching holes, at resume points as in the previous section. The archive keeps its length until the end, when the unit commits through the ordinary steps 3 to 5, truncating it to nothing, and the empty file is deleted. All three formats share one tar parser (`targz::TarStream`, driven through the `Feed` trait) and differ only in their decoder: `targz::Walker` over a gzip stream, `tarbz2::Walker` over bzip2 blocks, and `tarxz::Walker` over xz blocks. Before anything is touched, HatchZip decodes the whole stream once. For gzip (`targz::open`), that lists the entries for the checks in [Rejected before starting](#rejected-before-starting), verifies every gzip member's CRC-32 and length against its trailer, and measures what the space check needs. For xz (`tarxz::open`), the xz container is parsed from its end first (stream footer, index, then every block header the walk reaches), since that is the only place the block extents are written; each block's CRC-32 or CRC-64 is then verified as the walk decodes it. SHA-256 checks, unknown filters, concatenated streams and dictionaries over 1 GiB are refused before anything is read for extraction. A damaged archive is refused here, unchanged. The extraction then decodes it a second time. This pass reads the whole archive, so once it has run for half a second the command line shows a progress bar and the GUI the share read ("Reading archive… 37%"). Measured on an Intel Core i7-14700HX with the archive in the page cache: 4.8 s for 488.6 MB holding 1 GiB of word-like text, about 225 MB of output a second; 0.8 s for 1.07 GB holding 1 GiB of random data; 2.1 s for 23.4 MB holding 20 GiB of zeros. The GUI opens an archive once to list it and again to extract it, so there the pass runs twice before the extraction starts. A resumed run makes the same pass from its resume point, since the bytes before it may be gone, so damage found on resume also stops the run before it changes anything. A resume point can be any deflate block boundary, the start of any gzip member, a CRC-checked bzip2 block boundary or byte-aligned stream start, or the start of any xz block, except inside an entry's header records. Every 256 MiB of output, at the next such point: 1. **Flush** the file being written, every file finished since the last point, and their folders: one `syncfs` on Linux; elsewhere each file is flushed as it is finished, and the folders at the point. 2. **Journal the point** in `resume`: the bit decoding restarts at, and in `resume.stream` the decoder's state, tagged with `kind`: `gzip` (whether that is a member start, the CRC-32 and length of the member's output so far, and the member's last 32 KiB of output, the deflate window), `bzip2` (stream level, combined CRC so far, and whether this is a stream header), or `xz` (which block the point precedes). The offset in the tar and where in the tar the point falls are stored for every codec: between entries, inside a file's data (`resume.written` says how much of it is written), inside padding or data that extracts to nothing, or past the end marker. Links found since the last point go into the same write. The gzip window spans tar headers and several files, so unlike a single ZIP entry's it can't be read back from one output file; it is stored in the journal, as 64 KiB of hex. Bzip2 and xz blocks borrow nothing from before them, so their points need no window at all. 3. **Punch** the archive from where this run last punched up to the byte holding that bit. A run's first punch starts at 0, in case the last run stopped between steps 2 and 3. The next punch starts one overlap before that byte, which is one filesystem block on Linux, 64 KiB on macOS, and the 16-cluster unit NTFS frees sparse ranges in on Windows, so the block a filesystem that rounds inwards left behind at the end of the previous punch is covered too; it still never reaches the byte itself. A bzip2 point can fall inside a byte, which stays whole. An xz point is always a block header, at a whole byte, and the punch stops before it: the header is what a resume parses, so it must stay whole. | Interrupted | State on disk | What a resume does | |---|---|---| | Before the first point is journaled | Some output, nothing punched | Deletes that output, all of it still in the archive, and starts over | | After the flush, before the journal write | Output past the point in force, which is the previous one | Deletes the files that come after that point, cuts the file the point is inside back to its recorded length, and decodes from the point | | After the journal write, before or after the punch | The new point is durable | The same, from the new point, never reading before it | | Around the final commit | As for every batch | As for every batch | Bzip2 has no index, and its block signatures can start at any bit. The decoder reads compressed bytes in large chunks and looks up each 2-byte pair at an even offset in a table of the pairs either signature contains at any alignment; every signature contains such a pair, so only positions next to a match are tested bit by bit. It retains each block's raw bytes; only at a candidate does it shift whole words to byte alignment and wrap them in a synthetic one-block stream. The candidate is accepted only if that stream decodes and verifies the block CRC. A false signature inside compressed data, including one immediately after a real boundary, is ignored when decoding fails. Every combined stream CRC is checked too, including on resume. Concatenated streams, levels 1 through 9, `.tbz2` and `.tbz` aliases, and split `.tar.bz2.001` sets work. Corruption, truncation, nonzero stream padding, trailing data and a bare bzip2 file without a tar are refused before extraction. A resumed decoder starts reading at the byte containing its recorded bit and never reads the freed prefix. An xz stream is a sequence of independent blocks, so the xz walk works block by block. Each block's filter chain is built from its own header (LZMA2 at the end, wrapped by the BCJ or delta filters the header names before it) over a source limited to that block's own compressed bytes, so a decoder cannot read into the next block. When the block's declared uncompressed size is reached, its CRC-32 or CRC-64 is read from the offset the index records and compared with the bytes just decoded; a mismatch stops the run as damage. The check bytes are read through a second reader, which is why the scan reads a few bytes per block beyond the compressed data. A resumed run parses the container again from its end, but the blocks an earlier run punched away have no headers left. Their extents come from the index alone and they are marked as unvalidated; only the blocks from the resume point on are read whole and checked. The resume point must name a whole block whose header lies exactly at the archive byte the journal records, and the walk refuses a journal that disagrees. A single-threaded xz (`xz -T1`) puts the whole tar in one block, so nothing can be freed while it decodes. Such an archive is refused for in-place extraction with a message naming `--keep-archive`, which extracts it and leaves the archive whole. A stream made with `--check=none` carries no block check, so damage inside a block could not be told from the original bytes before the block's archive bytes are freed; it is refused in place for the same reason and also extracts with `--keep-archive`. xz's default multithreaded mode (one block per three dictionaries of output) gives points far more often than the 256 MiB step. Without hole punching (FAT32, exFAT) the stream is extracted the same way and resume points are still journaled, so a crash still resumes from the last one, but nothing is freed until the end: the space check asks for room for every file next to the whole archive, and `check` says why. When the output is on another disk, nothing is punched either, since it would not help there. `restore` refuses a compressed tar whose journal holds a resume point: some of the stream may be released, and a compressed tar can't be rebuilt from what is left. Before the first point nothing was released, so restore removes the partial output and the journal and leaves the original archive. The space check counts, for the stream, the highest the files written so far ever run ahead of the compressed bytes consumed, over every point the scan met; plus one step; plus the most file output the scan saw between two consecutive points, since a step can only end at a point; plus a read buffer and a margin at each end of the punched range; never more than all the files. Links are made at the end, once the archive is gone. The scan meets exactly the points the extraction will, since both decode from the same place with the same buffer sizes. Measured on an 820 MiB tmpfs holding a 385.9 MB `.tar.gz` made by GNU tar 1.35 and gzip 1.13 (`-6`) from 501.1 MB in 2,016 entries (seven 50 MB random files, six 25 MB text files, 2,000 small JSON files), with 473.9 MB free: GNU `tar -xzf` ran out of space. `hatchzip check` asked for 400.2 MB plus the 5 MB kept free; during the extraction the lowest free space, sampled every 10 ms, was 111.8 MB, a peak of 362.1 MB. It finished in 1.0 s with 351.6 MB free, as the check said, and every file's SHA-256 matched. With punching disabled (`HATCHZIP_NO_PUNCH`, debug build) the check asks for 508.2 MB and refuses on that disk; on a 1000 MiB tmpfs that run peaked at 508.2 MB. Measured on an Apple M4 with the archive in the page cache and the release build: a 15.7 MB `.tar.xz` made from 133.9 MB of word-like text in 14 entries by `tar -cf - | xz -T4 -6 --block-size=8MiB` (xz 5.8.4; one stream, 16 blocks, CRC-64). `check` asked for 133.9 MB; the extraction took 1.48 s, about 90 MB of output a second, and every file matched. Killed 0.7 s into a run, the resumed run finished with the same tree. A one-block archive from `xz -T1` was refused in place, naming `--keep-archive`, and extracted with it, leaving the archive whole. Measured on an Intel Core i7-13620H (Windows 11, plugged in, power mode Best performance) with a Micron 2500 NVMe volume formatted as NTFS, using the MSVC release build and the archive in the page cache: a 578.9 MB `.tar.bz2` made with Git Bash's bzip2 1.0.8 (`/mingw64/bin/bzip2 -9`) from a 1074.3 MB tar holding 1 GiB in 1,008 files (random data, text and 1,000 small JSON files). Five rounds ran every command back to back, with fresh archive copies written and flushed before each round; medians, with the range in brackets: `bzip2 -dc` with its output discarded 29.0 s (28.8 to 29.7), Windows `tar -xf` (bsdtar 3.8.9 with libbzip2 1.0.8) 31.8 s (31.7 to 32.2), `hatchzip check` 35.5 s (34.5 to 39.9) and `hatchzip extract --yes` 77.1 s (71.9 to 78.7). The Rust bzip2 decoder HatchZip uses (`libbz2-rs-sys` 0.2.5) took 32.0 s on its own in three separate rounds, so the check adds about 10% to it for finding and verifying block boundaries. The extraction decodes the archive twice, once to check it before anything is touched and once to extract; writing the files is a small part of it, since the same tar as `.tar.gz` extracts in 3.0 s and Windows tar unpacks it uncompressed in 0.9 s. 7-Zip 26.01, which decodes bzip2 on several cores, took 18.4 s to unpack the `.tar.bz2` to a `.tar` and that to files. HatchZip used `--keep-free 0` and a fresh archive copy for every extraction, and every extracted file in every round matched its source SHA-256. On the same machine, on a 1.66 GB NTFS virtual disk holding the same 578.9 MB `.tar.bz2`, with 1,063.8 MB free: Windows `tar -xf` ran out of space. `hatchzip check --keep-free 5MB` asked for 767.3 MB plus the 5 MB kept free; during the extraction the lowest free space, sampled every 10 ms, was 548.4 MB. It finished in 72.3 s with 568.8 MB free, as the check said, and every file's SHA-256 matched. `crates/cli/tests/tarbz2.rs` checks real bzip2 archives, both false-signature fixtures, corruption and combined CRC failures, split sets, keep-archive and restore, and crashes at every durable-point step. Unit tests in `tarbz2.rs` resume at every reported block boundary and stream start with a reader that refuses any read before the durable point. pax global headers, which `git archive` writes at the start of every tarball, are skipped, in plain tar as well. `crates/cli/tests/targz.rs` kills the process at each of the three steps of a point at several points, repeatedly within one run, around the final commit, in multi-member and split archives and in archives from GNU tar, gzip and pigz, and checks after each crash that nothing past the journaled point was touched and, where `HATCHZIP_EXPECT_PUNCH` is set, that the space before it was given back. Unit tests in `resumable.rs` resume a multi-member gzip stream from every point it reports and decode it with input buffers down to one byte. `crates/cli/tests/tarxz.rs` does the same for xz: 1 MiB blocks from `lzma-rust2`'s `XzWriter`, every crash step, repeated crashes inside one entry (a resume point inside the big file, at a block boundary), a damaged block or container, damage found on resume, the space plan with and without punching, keep-archive and restore, SHA-256 checks, concatenated streams and a single-block archive. ## RAR RAR support is the `rar` cargo feature, on in the release builds and off by default; see [the format notes](formats.md#dependency-licenses). A build without it refuses a RAR archive before anything is touched. A RAR archive that is not solid stores its files one after another, each header followed by the file's data, like tar. A volume set (`game.part1.rar`, `game.part2.rar`, ... or `game.rar`, `game.r00`, ...) is not one archive cut into byte ranges, as a `.7z.001` set is: each volume is a complete RAR file with its own headers, and a file that doesn't fit is split over consecutive volumes, its header repeated in each piece. `Storage` joins the volumes as it does any split set, and every file is a unit that starts at its first header. So the archive is truncated behind each committed batch exactly as tar is, and a volume is deleted once the truncation point falls before it. A split file belongs to the volume it starts in, so the volumes holding its later pieces stay until all of it is committed. What is left is always a prefix of the set whose headers are untouched, and the last volume left simply ends after a file, without its end of archive header, which UnRAR reads like any other. `archive/rar.rs` reads the headers of every volume itself (RAR 5 and RAR 4, with their checksums) before anything is touched, as tar does. Nothing needs to be saved first: there is no central index and every volume carries its own headers, so the saved tail is empty. A resumed run reads the headers of what is left, up to where the last commit cut it, and finds exactly the units still to do. Before the first run starts, UnRAR lists the set too and every file's name is compared with the one read here, since extraction finds each file by the name UnRAR reports; a name UnRAR reads differently is refused then, not halfway through. Names are read in full, past the 1023 characters of UnRAR's fixed buffer, and a `\` in a RAR 5 name, where it is part of the name rather than a separator, is written as `_` on every system, as WinRAR does on Windows. UnRAR is driven through its C interface and does the decoding only. It finds a file only by reading every header before it, so a batch of RAR units is decoded on one thread, lowest file first, by one archive handle that walks forward from the volume the lowest file starts in (the format says so with `Format::serial`). A helper thread runs UnRAR and pushes each file's data through a bounded channel to the reader the engine's sinks take. Only one archive handle is open at a time, since UnRAR keeps state in globals and is not documented as safe to use from several threads at once, and the handle is closed before the batch commits, so nothing holds a volume open while it is truncated or deleted. A stored file that lies whole in one volume is plain bytes at a known offset, so it is streamed like a tar entry ([above](#freeing-space-inside-a-large-file)), with its CRC-32 checked. Inside a compressed file there is no resume point: UnRAR can't save its decoder's state. A solid archive compresses every file together with the ones before it, so it decodes only front to back in one pass. It could only be freed once all of it is out, and since UnRAR can't save its state, a crash would have to start over with part of the archive possibly gone. So it is not extracted in place: it is one unit holding every file, extracted only while the archive is kept (`--keep-archive`), and refused in place before anything is touched, with a message that says so. Passwords are proven before anything is touched, as for ZIP. A RAR 5 file carries a 64-bit check value derived from its password with PBKDF2-HMAC-SHA256; HatchZip derives it from the given password and compares, for every distinct salt in the archive, without decoding anything. RAR 4 has no check value: a wrong password only shows in the CRC-32 at the end of a file, so the smallest encrypted file with data is decoded whole into nothing, and the command line says so when that file is 64 MiB or more. An archive made with "Encrypt file names" (`rar -hp`) encrypts its headers as well, so nothing of it can be read without the password, not even its list: opening it without one asks for it, as for a 7z with an encrypted header. `archive/rar/crypt.rs` decrypts the headers as UnRAR does (`crypt3.cpp`, `crypt5.cpp`, `ReadHeader15` and `ReadHeader50` in `arcread.cpp`), and the same parser reads them: - RAR 5 puts an archive encryption header right after the signature, in the clear: a salt, the log2 of an iteration count, and usually a password check value. Every header after it is preceded by a 16-byte IV and encrypted with AES-256-CBC, under the key PBKDF2-HMAC-SHA256 derives from the password and the salt. A wrong password fails the check value at once; without one, it fails the checksum of the first header. - RAR 4 says so with a flag in its main header, which stays in the clear. Every header after it is preceded by an 8-byte salt and encrypted with AES-128-CBC; the key and IV come from RAR 3's derivation, 2^18 rounds of SHA-1 over the password (in UTF-16) and that salt. That SHA-1 keeps a quirk of RAR 2.9, which rewrites the input buffer of a whole 64-byte block it hashes, so a password of 29 characters or more only works if it is reproduced. Keys are kept by salt, and RAR 6.24 writes the same salt in front of every header of a set, so the derivation runs once. A wrong password fails the checksum of the first encrypted header; so does damage to it, which is then reported as a wrong password, as UnRAR does. Once one header has passed, a header that fails is damage. Either way, a header is padded to a whole number of 16-byte blocks. Every offset is the real one on disk: a file starts at the IV or salt in front of its header, which is where a truncation cuts, and its data starts after the padding. The volumes of a set must agree on whether their headers are encrypted, as UnRAR requires. UnRAR gets the password for its own listing too, so every name is still compared with what it reads. The file data of such an archive is encrypted as well, and proven as above. `restore` refuses an archive with encrypted file names: it takes no password, and without one it can neither list what is left nor encrypt the end of archive header it would append. Such an extraction can only be completed, with the password. Refused before anything is touched: symbolic links, hard links and file copies, dictionaries over 4 GB (UnRAR refuses them by default, and decoding one needs that much memory), compression versions UnRAR doesn't know, a set with a volume missing or out of order, a stray volume after the last one (the last says no more follow), and any header that fails its checksum, as damage (exit 4). `restore` appends an end of archive header, saying no volume follows, to the last volume left, in that volume's format; everything before it is untouched files with their headers. Archives with encrypted file names are refused, as above. Measured on a 192 MiB tmpfs holding a 146.5 MB RAR 5 set made by RAR 6.24 (`-m3 -v40m`, four volumes) from 190.2 MB in 512 files (seven 20 MB random files, five 10 MB text files, 500 small JSON files), with 54.8 MB free: UnRAR 6.24 stopped with a write error (exit 5). `hatchzip check` asked for 45.6 MB plus the 5 MB kept free, where freeing a whole volume at a time asked for 129.4 MB; during the extraction the lowest free space, sampled every 10 ms, was 5.3 MB, a peak of 49.5 MB. It finished in 0.61 s with 9.3 MB free, as the check said, and every file's SHA-256 matched. The same files as one 146.4 MB `.rar` on the same disk: the check asked for 45.7 MB, where a single volume freed as a whole asked for 192.1 MB; the peak was 43.8 MB, in 0.45 s, every file matching. `crates/cli/tests/rar.rs` extracts sets made by RAR 6.24 (RAR 5 and RAR 4, with passwords, with encrypted file names in one volume and in several, solid, and with quick open data, a recovery record and a comment) and archives written by the test itself (stored RAR 5 and RAR 4, single files and sets, hostile names, files over many volumes, an 8 MB stored file). It kills the process at both crash points after every number of commits a run makes, for RAR 5 and RAR 4, single files and sets; checks after each commit that what is on disk is exactly what the journal says is left and that it shrank; kills a streamed stored file at each step of a resume point and checks its space was given back; and checks that solid archives, wrong and missing passwords, hostile names, missing and stray volumes and damaged headers are refused with every volume intact. `crates/cli/tests/no_rar.rs` checks the refusal of a build without the feature. ## Crash safety Each step can be interrupted. The order above makes every interruption recoverable: | Interrupted during | State on disk | What a resume does | |---|---|---| | Choosing or extracting | Some part files or finished files exist, the journal doesn't list them, the archive is intact | Deletes those files (nothing was at their paths when the run started, or it was moved aside first, so they are HatchZip's) and extracts the batch again | | Flushing | Same as above | Same as above | | Committing | The atomic rename either happened or not | Either case above or below | | Truncating | The journal lists the batch, the archive is still longer than `committed_len` | Truncates the archive, continues with the next batch | | Creating links at the end | All units committed | Recreates every link, replacing any that exist. A link that can't be made (a folder in its place, which only an extraction started by a release without the checks in [Rejected before starting](#rejected-before-starting) can lead to, or a link on its way) stops the run with every file extracted; the error lists each such link and why, and the journal keeps them. Once what is in the way is moved, a resume makes them and finishes. Nothing is ever created through a link. | | Setting folders at the end | All units committed, links made | Sets every saved folder again, then finishes | | Deleting the archive and side files | The archive is gone, the journal is still there | Recreates the links, sets the folders and removes the side files | The rule behind the table: **the journal never claims a file before it is durable, and the archive is never truncated past a unit before the journal claims it.** The test suites kill the process at steps 3 and 5 at many positions for every format, including split archives with volume boundaries inside entries, and check that the resumed run produces identical output (`crates/cli/tests/extract.rs`, `crates/cli/tests/formats.rs`). ## Keeping the archive `--keep-archive` (in the app, clearing "Remove the archive during extraction"; in the right-click menu, "Extract here (keep archive)") is a normal extraction through the same engine: the same units, batches, part files, checks, flushes and journal commits, so path safety, collisions, CRCs, passwords and resuming work exactly as above. What changes: - **Nothing in the archive changes.** The journal records `keep`, and from the moment it is written the archive's `Storage` is protected: truncating, punching, writing and deleting all refuse, so no code path can change a kept archive even by mistake. A commit (step 4) still advances `committed_len`, but only as a record of progress; step 5 is skipped. Nothing is streamed or punched, as for an output on another disk. At the end, links are made and only the journal and the saved tail are deleted. - **The space check counts every file.** The replay runs as if the output were on another disk: each unit adds its files and frees nothing, plus the saved tail. That is the whole output, rounded up to 4 KiB per file, plus the archive's index, and it is what `check --keep-archive` reports. When it does not fit but extracting in place would, `check` and the app say what extracting in place requires. When planning in place, they mention when keeping the archive would fit too. - **Resuming keeps keeping.** A resumed run follows the journal, whatever it is asked: an extraction that keeps the archive goes on keeping it, and one that started in place cannot switch to keeping it, since part of the archive is already gone. That refusal happens before the archive is opened for writing, so it doesn't even finish a truncation the last run left pending. On resume the archive must still have its original length, or the run stops without reading it. - **Restore discards.** `restore` on a kept extraction has no index to rebuild: it deletes what a crashed batch left half written, as a resume would, then the journal, the saved tail and the saved folders. Files already extracted stay. The app shows no "Restore archive" for such a run. The interruption table above holds with one row fewer: a crash after the journal write leaves nothing pending. `crates/cli/tests/keep.rs` extracts ZIP, 7z (solid and not), tar, `.tar.gz` and split ZIP, 7z and `.tar.gz` sets while keeping them, checks that every volume is byte for byte unchanged, then extracts the same archive in place and compares both outputs with the source. It kills kept runs at both commit points and, for a `.tar.gz`, at each step of a resume point, resumes them without `--keep-archive`, and checks the archive is unchanged after the crash and after the resume. `HATCHZIP_FREE` (debug builds only) sets the free space so the suite can put a disk between the two plans. ## Replacing existing files A resume deletes whatever it finds at the paths of uncommitted entries, which is safe only because the run put it there. So by default a file or link at the path of any entry refuses the extraction before anything is touched. With `--replace` (in the app, "Replace" when it asks), `crates/core/src/replace.rs` clears those paths before the first entry is extracted: 1. When the journal is written, it records `replace` without `moved`. Until `moved` is set, the run has extracted nothing, so a resume deletes nothing at the paths of entries (`Job::uncommitted` returns nothing) and the space check counts nothing there as freed. 2. At the start of the run, every file or link at the path of an entry left to do is renamed into `/.hatchzip-replaced/`, under the same relative path. A rename on the same filesystem copies no data and needs no space, and it moves a link itself, never its target. A resume repeats this step: a file already moved is no longer at its path, so it moves once. 3. The renames are flushed (one `syncfs` on Linux, every folder involved on macOS and Windows), then the journal is saved with `moved`. From here on the run is an ordinary one: every path of an entry was empty, so whatever a crash leaves there is HatchZip's. 4. Once every unit is committed and the links are made, the files the person chose to keep go back. For "Skip" (`--skip-existing`), each is renamed from the folder over what was extracted at its path. For "Keep both", the extracted file is first renamed beside it, to `name (2).ext` or the first such name not taken, then the kept one is renamed back. Each step is a rename a rerun finds done or not, so a crash in the middle finishes on resume; nothing is deleted until every kept file is back. 5. Then the folder is deleted, before the journal, so a crash in between finishes the deletion on resume. The choices are in the journal's `replace.keep` and `replace.both`, as paths relative to the output, each checked to stay inside it when the journal is loaded. A journal with choices is version 6, or 7 with `case_folders`, or 8 with `folders`: an older build would finish it by deleting the files the person chose to keep, so it refuses it. The replaced files keep their space until step 5, so the space check is the same as for an empty folder. Refused before anything is touched, besides the usual: an archive with an entry named `.hatchzip-replaced` at its root (ignoring case), a `.hatchzip-replaced` already in the output folder (an earlier restore can leave one), a replaced path that would become longer than the system allows inside the folder, and replaced paths that differ only in case, or sit in folders that do, unless the output folder is known to keep such names apart: the folder of replaced files takes after the output, however the folders they are in now behave. `restore` first saves the journal with `moved` cleared, so an interrupted restore never deletes a file it put back, then moves every replaced file back to its path if nothing holds it. After an extraction in place, a committed entry's file is the only copy of that entry, so the file it replaced stays in the folder and restore reports how many did; the folder is deleted when none do. After an extraction that keeps the archive, the archive still holds every entry, so each replaced file goes back over the extracted one. `crates/cli/tests/replace.rs` kills runs while moving files aside and at both commit points, resumes them, restores them, and checks every replaced file byte for byte. ## Restore `hatchzip restore` turns a half-extracted archive back into a normal one, for when an extraction should stop for good. What is left of the archive is the first `committed_len` bytes of the original, and those hold exactly the units not yet extracted, untouched. Only the index is missing, so restore appends a new one after them. Nothing below `committed_len` moves, so it needs only the free space the new index takes. Files already extracted stay in the output folder. Restore never needs the saved folders, so it works with `.hatchzip-folders` deleted or damaged, where a resume refuses. | Format | What restore writes | |---|---| | ZIP | A central directory holding the saved records of the entries left, copied as they are (their local header offsets still point at the same bytes), then the end records, with ZIP64 records when the original had them or the counts or offsets need them. A self-extractor stub stays in front, and the offsets keep whatever convention the original used. | | 7z | A new header for the blocks left, which are the first ones, right after the signature header, and their files. It is encoded by sevenz-rust2's writer, which is handed each block raw and writes nothing for it, since the block already sits where the writer would put it. The signature header is rewritten to point at the new header. | | tar | Entries for hard links already cut off whose targets are still in the archive, then the two zero blocks that end a tar archive. | | RAR | An end of archive header in the last volume left, in its format (RAR 5 or RAR 4), saying no volume follows. | The steps, in order: 1. Undo any earlier restore that was interrupted (below), and finish a truncation a crashed run left pending, so the archive is exactly `committed_len` bytes. 2. Build the new index in memory. Anything restore refuses, it refuses here, before writing. 3. Delete output a crashed batch wrote but never committed, as a resume does, since those entries are still in the archive. Create the links the journal holds for entries already cut off. 4. Save the journal with a `restoring` field holding the bytes step 6 will overwrite. 5. Append the new index at `committed_len` and flush it. 6. Overwrite the bytes below `committed_len` that change (only the 7z signature header) and flush them. 7. Open the result as a fresh archive with the same parsers extraction uses and check that it lists exactly the entries left. If it doesn't, undo and stop. 8. If the extraction replaced files, move them back (see [Replacing existing files](#replacing-existing-files)). 9. Delete the journal, then the saved tail. Any run that opens an archive whose journal has `restoring` set, `restore` and `extract` alike, first cuts the archive back to `committed_len`, including a volume the new index made longer than it was, and puts back the bytes the field holds. So an interrupted restore can be run again, or the extraction continued instead. The journal goes before the tail because a journal without its tail can't be loaded; a tail left behind by a crash between the two is never read again. The test suite kills restore after steps 5 and 6 for every format and checks both ways on (`crates/cli/tests/restore.rs`). ## The space check Before anything is written, HatchZip replays the whole extraction one unit at a time, counting what each step adds (the unit's files, each rounded up to a 4 KiB block) and what it frees (the unit's span in the archive), plus the saved tail written at the start. The highest point of that running total is the free space the run needs. HatchZip refuses to start unless the disk has that much free plus `--keep-free`. A unit that can be streamed counts at most one step of output (256 MiB) plus a 1 MiB read buffer, plus what the filesystem may keep allocated at each end of a punched range (64 KiB, 1 MiB on NTFS, 16 MiB on ZFS, 128 MiB on btrfs, whose extents are freed whole), plus how far its output outgrows its compressed data, never more than the whole unit. For deflate that last part is bounded by its value at the end, since encoders fall back to stored blocks rather than expand data, with a 1/1024 margin for block headers. Measured on a 950 MiB tmpfs holding a ZIP with one 600 MiB entry and three small ones: stored, the check asked for 269.6 MB where the previous release asked for 629.1 MB and refused, and the lowest free space seen while it ran (sampled every 10 ms) was 98.5 MB of 367.0 MB, a peak of 268.5 MB; deflated (mostly incompressible data, 498.1 MB archive), it asked for 401.3 MB against 629.1 MB, and the peak was 324.4 MB. Both finished with the file byte-identical. The real run never needs more than the check promised: it uses larger batches only when there is room for them, since each batch is sized against the free space measured at that moment. If something else on the machine fills the disk during the run, HatchZip stops before the next unit, saves its progress, and says how much space to free. When the output folder is on a different disk from the archive, freeing the archive does not help the output disk, and the check says so. ## Rejected before starting These are refused while the archive is still untouched: - Encrypted entries or blocks (not supported yet). - Paths that resolve outside the output folder (`..`, absolute ZIP paths, drive letters). Leading slashes in tar names are dropped, as tar itself does. - Two entries with the same path. For ZIP the names are read from the central directory itself, since the zip library keeps only the last of several entries with one name. - Entries that can't all be created where they say: a file or link at a path where another entry needs a folder (`d` as a symlink next to `d/inner.txt`), which includes any entry inside a link, and a tar hard link to anything but a file in the archive. Links are made at the very end, once the archive is gone, so such an archive would otherwise fail only after it was consumed. - Either of these between paths that differ only in case, unless the folder where the names meet is known to keep them apart. Elsewhere they would be one file, and would race inside a parallel batch. The nearest existing folder decides, since new folders take after it, and it can be a mount of its own. On Linux, btrfs always keeps them apart, ext4, f2fs and tmpfs do in a folder that reports no casefold flag, and XFS does unless it was made with ASCII case-insensitive names. On macOS, APFS and HFS+ volumes are asked, and only paths in plain ASCII qualify, since both treat names equal after Unicode normalization as one file even when case-sensitive. Any other filesystem, and Windows, stays refused, as does an archive with such names in over 1000 folders. `list` and `test` write nothing, so they don't ask. - ZIP entries whose bytes overlap other entries. Overlapping archives are a known zip bomb technique, and truncating one entry's span would destroy another's data. - tar entry types HatchZip can't create safely: device files, FIFOs, sparse files. - 7z anti-items (deletions recorded by archive updates). - In a `.tar.gz`: damage anywhere in the stream, a gzip member whose CRC-32 or length doesn't match, data after the end of the tar or after the gzip stream other than zeros (it would be lost with the archive), and header records over 1 MiB. - In a `.tar.xz`: damage anywhere in a block, a block whose CRC-32 or CRC-64 doesn't match, a container the index or footer can't be read from, blocks that don't decode to their recorded sizes, SHA-256 or unknown block checks, unknown filters, blocks using an LZMA2 dictionary over 1 GiB, more than one xz stream in the file, and, for in-place extraction, a single-block archive or one made with `--check=none` (`--keep-archive` extracts both). - Gzip and bzip2 files that don't hold a tar, xz files whose stream doesn't hold one, and compressed tar other than gzip, bzip2 and xz (`.tar.zst`). - In RAR: solid archives (unless the archive is kept), links, dictionaries over 4 GB, a volume missing or a stray one after the last, names UnRAR reads differently, a wrong password, and headers that fail their checksum (see [RAR](#rar)). Without the `rar` feature, every RAR archive. - A file or link at the path of an entry, so that a resume can safely delete anything it finds for uncommitted units, unless the run replaces existing files (see [Replacing existing files](#replacing-existing-files)). A folder where the archive has a file, a file where it needs a folder, and a link on the way to any entry are refused either way. - An archive, or one volume of a set, with another hard link, unless the archive is kept. Shrinking a file shrinks it under every name it has, so the other name would be emptied. - Anything at the name an entry's file is first written under, `.hatchzip-part`, replacing files or not: it would be in the way, and a resume would delete it as its own. An entry whose name, or the name of a folder on its path, ends in `.hatchzip-part`, ignoring case, since it would share that name with another entry's file. ## Operating systems Everything that differs between systems lives in `crates/core/src/platform/`, one file per system: | | Linux | macOS | Windows | |---|---|---|---| | Flushing a batch | One `syncfs` | `F_FULLFSYNC` per file | `FlushFileBuffers` per file | | Directory flush | `fsync` on the directory | `fsync` on the directory | Not possible; NTFS journals renames | | Same disk check | Device ID | Device ID | Volume prefix of the canonical path | | Unix permissions | Restored for files and folders, without setuid, setgid and sticky | Same | Ignored | | Symlinks | Created | Created | Written as a small text file holding the target, as Git does without developer mode | | Hard links | Created, or copied where the filesystem has none (FAT32, exFAT) | Same | Same |