dolphin/docs/WiaAndRvz.md

WIA file format description

This document describes the WIA disc image format, version 1.00, as implemented in wit v2.40a. A few notes about Dolphin's implementation of the format are also included, where Dolphin differs from wit. The unique features of WIA compared to older formats like GCZ are:

• Support for the compression algorithms bzip2, LZMA, and LZMA2
• Wii partition data is stored decrypted and without hashes, making it compressible

Like essentially all compressed GC/Wii disc image formats, WIA divides the data into blocks (called chunks in wit). Each chunk is compressed separately, making random access of compressed data possible.

The struct names and variable names below are taken directly from wit. Data in WIA files can be stored in any order unless otherwise noted. All integers are big endian unless otherwise noted. The type sha1_hash_t refers to an array of 20 bytes.

wia_file_head_t

This struct is stored at offset 0x0 and is 0x48 bytes long. The wit source code says its format will never be changed.

A short note from the wit source code about how version numbers are encoded:

//-----------------------------------------------------
// Format of version number: AABBCCDD = A.BB | A.BB.CC
// If D != 0x00 && D != 0xff => append: 'beta' D
//-----------------------------------------------------

Type and name	Description
char magic[4]	Always contains "WIA\x1".
u32 version	The WIA format version.
u32 version_compatible	If the reading program supports the version of WIA indicated here, it can read the file. version can be higher than version_compatible (wit v2.40a sets the former to 0x01000000 and the latter to 0x00090000).
u32 disc_size	The size of the wia_disc_t struct. wit v2.40a always includes the full 7 bytes of compr_data when writing this.
sha1_hash_t disc_hash	The SHA-1 hash of the wia_disc_t struct. The number of bytes to hash is determined by disc_size. For instance, you may have to hash all 7 bytes of compr_data regardless of what compr_data_len says.
u64 iso_file_size	The original size of the disc (or in other words, the size of the ISO file that has the same contents as this WIA file).
u64 wia_file_size	The size of this file.
sha1_hash_t file_head_hash	The SHA-1 hash of this struct, up to but not including file_head_hash itself.

wia_disc_t

This struct is stored at offset 0x48, immediately after wia_file_head_t.

Type and name	Description
u32 disc_type	wit sets this to 0 for "unknown" (does this ever happen in practice?), 1 for GameCube discs, 2 for Wii discs.
u32 compression	0 for NONE, 1 for PURGE (see the wia_segment_t section), 2 for BZIP2, 3 for LZMA, 4 for LZMA2.
u32 compr_level	The compression level used by the compressor. The possible values are compressor-specific. For informational purposes only.
u32 chunk_size	The size of the chunks that data is divided into. Must be a multiple of 2 MiB.
u8 dhead[0x80]	The first 0x80 bytes of the disc image.
u32 n_part	The number of wia_part_t structs.
u32 part_t_size	The size of one wia_part_t struct. If this is smaller than sizeof(wia_part_t), fill the missing bytes with 0x00.
u64 part_off	The offset in the file where the wia_part_t structs are stored (uncompressed).
sha1_hash_t part_hash	The SHA-1 hash of the wia_part_t structs. The number of bytes to hash is determined by n_part * part_t_size.
u32 n_raw_data	The number of wia_raw_data_t structs.
u64 raw_data_off	The offset in the file where the wia_raw_data_t structs are stored (compressed).
u32 raw_data_size	The total compressed size of the wia_raw_data_t structs.
u32 n_groups	The number of wia_group_t structs.
u64 group_off	The offset in the file where the wia_group_t structs are stored (compressed).
u32 group_size	The total compressed size of the wia_group_t structs.
u8 compr_data_len	The number of used bytes in the compr_data array.
u8 compr_data[7]	Compressor specific data (see below).

If the compression method is NONE, PURGE or BZIP2, compr_data_lenis 0. If the compression method is LZMA or LZMA2, the compressor specific data is stored in the format used by the 7-Zip SDK. It needs to be converted if you are using e.g. liblzma.

For LZMA, the data is 5 bytes long. The first byte encodes the lc, pb, and lp parameters, and the four other bytes encode the dictionary size in little endian. The first byte can be decoded as follows (code from the 7-Zip SDK):

d = data[0];
if (d >= (9 * 5 * 5))
  return SZ_ERROR_UNSUPPORTED;

p->lc = d % 9;
d /= 9;
p->pb = d / 5;
p->lp = d % 5;

For LZMA2, the data consists of a single byte that encodes the dictionary size. It can be decoded as follows (code from the 7-Zip SDK):

#define LZMA2_DIC_SIZE_FROM_PROP(p) (((UInt32)2 | ((p) & 1)) << ((p) / 2 + 11))

if (prop > 40)
  return SZ_ERROR_UNSUPPORTED;
dicSize = (prop == 40) ? 0xFFFFFFFF : LZMA2_DIC_SIZE_FROM_PROP(prop);

Preset dictionaries are not used for any compression method.

wia_part_data_t

Type and name	Description
u32 first_sector	The sector on the disc at which this data starts. One sector is 32 KiB (or 31 KiB excluding hashes).
u32 n_sectors	The number of sectors on the disc covered by this struct. One sector is 32 KiB (or 31 KiB excluding hashes).
u32 group_index	The index of the first wia_group_t struct that points to the data covered by this struct. The other wia_group_t indices follow sequentially.
u32 n_groups	The number of wia_group_t structs used for this data.

wia_part_t

This struct is used for keeping track of Wii partition data that on the actual disc is encrypted and hashed. This does not include the unencrypted area at the beginning of partitions that contains the ticket, TMD, certificate chain, and H3 table. So for a typical game partition, pd[0].first_sector * 0x8000 would be 0x0F820000, not 0x0F800000.

Wii partition data is stored decrypted and with hashes removed. For each 0x8000 bytes on the disc, 0x7C00 bytes are stored in the WIA file (prior to compression). If the hashes are desired, the reading program must first recalculate the hashes as done when creating a Wii disc image from scratch (see https://wiibrew.org/wiki/Wii_Disc), and must then apply the hash exceptions which are stored along with the data (see the wia_except_list_t section).

Type and name	Description
u8 part_key[16]	The title key for this partition (128-bit AES), which can be used for re-encrypting the partition data. This key can be used directly, without decrypting it using the Wii common key.
wia_part_data_t pd[2]	To quote the wit source code: segment 0 is small and defined for management data (boot .. fst). segment 1 takes the remaining data. The point at which wit splits the two segments is the FST end offset rounded up to the next 2 MiB. Giving the first segment a size which is not a multiple of 2 MiB is likely a bad idea (unless the second segment has a size of 0).

wia_raw_data_t

This struct is used for keeping track of disc data that is not stored as wia_part_t. The data is stored as is (other than compression being applied).

The first wia_raw_data_t has raw_data_off set to 0x80 and raw_data_size set to 0x4FF80, but despite this, it actually contains 0x50000 bytes of data. (However, the first 0x80 bytes should be read from wia_disc_t instead.) This should be handled by rounding the offset down to the previous multiple of 0x8000 (and adding the equivalent amount to the size so that the end offset stays the same), not by special casing the first wia_raw_data_t.

Type and name	Description
u64 raw_data_off	The offset on the disc at which this data starts.
u64 raw_data_size	The number of bytes on the disc covered by this struct.
u32 group_index	The index of the first wia_group_t struct that points to the data covered by this struct. The other wia_group_t indices follow sequentially.
u32 n_groups	The number of wia_group_t structs used for this data.

wia_group_t

This struct points directly to the actual disc data, stored compressed. The data is interpreted differently depending on whether the wia_group_t is referenced by a wia_part_data_t or a wia_raw_data_t (see the wia_part_t section for details).

A wia_group_t normally contains chunk_size bytes of decompressed data (or chunk_size / 0x8000 * 0x7C00 for Wii partition data when not counting hashes), not counting any wia_except_list_t structs. However, the last wia_group_t of a wia_part_data_t or wia_raw_data_t contains less data than that if n_sectors * 0x8000 (for wia_part_data_t) or raw_data_size (for wia_raw_data_t) is not evenly divisible by chunk_size.

Type and name	Description
u32 data_off4	The offset in the file where the compressed data is stored, divided by 4.
u32 data_size	The size of the compressed data, including any wia_except_list_t structs, and including any padding that is required after the wia_except_list_t structs when using the compression method NONE or PURGE. 0 is a special case meaning that every byte of the decompressed data is 0x00 and the wia_except_list_t structs (if there are supposed to be any) contain 0 exceptions.

wia_exception_t

This struct represents a 20-byte difference between the recalculated hash data and the original hash data. (See also wia_except_list_t below.)

When recalculating hashes for a wia_group_t with a size which is not evenly divisible by 2 MiB (with the size of the hashes included), the missing bytes should be treated as zeroes for the purpose of hashing. (wit's writing code seems to act as if the reading code does not assume that these missing bytes are zero, but both wit's and Dolphin's reading code treat them as zero. Dolphin's writing code assumes that the reading code treats them as zero.)

wit's writing code only outputs wia_exception_t structs for mismatches in the actual hash data, not in the padding data (which normally only contains zeroes). Dolphin's writing code outputs wia_exception_t structs for both hash data and padding data. When Dolphin needs to write wia_exception_t structs for a padding area which is 32 bytes long, it writes one which covers the first 20 bytes of the padding area and one which covers the last 20 bytes of the padding area, generating 12 bytes of overlap between the wia_exception_t structs.

Type and name	Description
u16 offset	The offset among the hashes. The offsets 0x0000-0x0400 here map to the offsets 0x0000-0x0400 in the full 2 MiB of data, the offsets 0x0400-0x0800 here map to the offsets 0x8000-0x8400 in the full 2 MiB of data, and so on. The offsets start over at 0 for each new wia_except_list_t.
sha1_hash_t hash	The hash that the automatically generated hash at the given offset needs to be replaced with. The replacement should happen after calculating all hashes for the current 2 MiB of data but before encrypting the hashes.

wia_except_list_t

Each wia_group_t of Wii partition data contains one or more wia_except_list_t structs before the actual data, one for each 2 MiB of data in the wia_group_t. The number of wia_except_list_t structs per wia_group_t is always chunk_size / 0x200000, even for a wia_group_t which contains less data than normal due to it being at the end of a partition.

For memory management reasons, programs which read WIA files might place a limit on how many exceptions there can be in a wia_except_list_t. Dolphin's reading code has a limit of 52×64=3328 (unless the compression method is NONE or PURGE, in which case there is no limit), which is enough to cover all hashes and all padding. wit's reading code seems to be written as if 47×64=3008 is the maximum it needs to be able to handle, which is enough to cover all hashes but not any padding. However, because wit allocates more memory than needed, it seems to be possible to exceed 3008 by some amount without problems. It should be safe for writing code to assume that reading code can handle at least 3328 exceptions per wia_except_list_t.

Type and name	Description
u16 n_exceptions	The number of wia_exception_t structs.
wia_exception_t exception[n_exceptions]	Each wia_exception_t describes one difference between the hashes obtained by hashing the partition data and the original hashes.

Somewhat ironically, there are exceptions to how wia_except_list_t structs are handled depending on what compression method is used for the group's data:

• For the compression method PURGE, the wia_except_list_t structs are stored uncompressed (in other words, before the first wia_segment_t). For BZIP2, LZMA and LZMA2, they are compressed along with the rest of the data.
• For the compression methods NONE and PURGE, if the end offset of the last wia_except_list_t is not evenly divisible by 4, padding is inserted after it so that the data afterwards will start at a 4 byte boundary. This padding is not inserted for the other compression methods.

wia_segment_t

This struct is used by the simple compression method PURGE, which stores runs of zeroes efficiently and stores other data as is.

Type and name	Description
u32 offset	The offset of data within the decompressed data. (Any wia_except_list_t structs are not counted as part of the decompressed data.)
u32 size	The number of bytes in data.
u8 data[size]	Data.

Each PURGE chunk contains zero or more wia_segment_t structs stored in order of ascending offset, followed by a SHA-1 hash (0x14 bytes) of the wia_except_list_t structs (if any) and the wia_segment_t structs. Bytes in the decompressed data that are not covered by any wia_segment_t struct are set to 0x00.

RVZ file format description

RVZ is a file format which is closely based on WIA. The differences are as follows:

• Zstandard has been added as a compression method. compression in wia_disc_t is set to 5 when Zstandard is used, and there is no compressor specific data. compr_level in wia_disc_t should be treated as signed instead of unsigned because Zstandard supports negative compression levels.
• PURGE has been removed as a compression method.
• Chunk sizes smaller than 2 MiB are supported. The following applies when using a chunk size smaller than 2 MiB:
  • The chunk size must be at least 32 KiB and must be a power of two. (Just like with WIA, sizes larger than 2 MiB do not have to be a power of two, they just have to be an integer multiple of 2 MiB.)
  • For Wii partition data, each chunk contains one wia_except_list_t which contains exceptions for that chunk (and no other chunks). Offset 0 refers to the first hash of the current chunk, not the first hash of the full 2 MiB of data.
• The wia_group_t struct has been expanded. See the rvz_group_t section below.
• Pseudorandom padding data is stored losslessly using an encoding scheme described in the RVZ packing section below.

rvz_group_t

Compared to wia_group_t, rvz_group_t changes the meaning of the most significant bit of data_size and adds one additional attribute.

"Compressed data" below means the data as it is stored in the file. When the compression method is NONE, this "compressed data" is actually not compressed.

Type and name	Description
u32 data_off4	The offset in the file where the compressed data is stored, divided by 4.
u32 data_size	The most significant bit is 1 if the data is stored using the compression method indicated in wia_disc_t, and 0 if it is stored using the compression method NONE. The lower 31 bits are the size of the compressed data, including any wia_except_list_t structs, and including any padding that is required after the wia_except_list_t structs when using the compression method NONE. The lower 31 bits being 0 is a special case meaning that every byte of the decompressed and unpacked data is 0x00 and the wia_except_list_t structs (if there are supposed to be any) contain 0 exceptions.
u32 rvz_packed_size	The size after decompressing but before decoding the RVZ packing. If this is 0, RVZ packing is not used for this group.

RVZ packing

The RVZ packing encoding scheme can be applied to wia_group_t data, with any bzip2/LZMA/Zstandard compression being applied on top of it. (In other words, when reading an RVZ file, bzip2/LZMA/Zstandard decompression is done before decoding the RVZ packing.) RVZ packed data can be decoded as follows:

1. Read 4 bytes of data and interpret it as a 32-bit unsigned big endian integer. Call this size.
2. If the most significant bit of size is not set, read size bytes and output them unchanged. If the most significant bit of size is set, unset the most significant bit of size, then read 68 bytes of PRNG seed data and output size bytes using the PRNG algorithm described below.
3. Repeat until all input has been read.

PRNG algorithm

The PRNG algorithm used for generating padding data on GameCube and Wii discs is a Lagged Fibonacci generator with the parameters f = xor, j = 32, k = 521.

Start by allocating a buffer of 521 32-bit words.

u32 buffer[521];

Copy the 68 bytes (17 words) of seed data into the start of the buffer. This seed data is stored in big endian in RVZ files, so remember to byteswap each word if the system is not big endian. Then, use the following code to fill the remaining part of the buffer:

for (size_t i = 17; i < 521; i++)
    buffer[i] = (buffer[i - 17] << 23) ^ (buffer[i - 16] >> 9) ^ buffer[i - 1];

The following code is used for advancing the state of the PRNG by a full buffer length. You must run it 4 times before you can start outputting data, and must then run it once after every 521 words of data you output.

for (size_t i = 0; i < 32; i++)
    buffer[i] ^= buffer[i + 521 - 32];

for (size_t i = 32; i < 521; i++)
    buffer[i] ^= buffer[i - 32];

After running the above code 4 times, you are ready to output data from the buffer -- but only if the offset (relative to the start of the disc for wia_raw_data_t and relative to the start of the partition data for wia_part_t) at which you are outputting data is evenly divisible by 32 KiB. Otherwise, you first have to advance the state of the PRNG by offset % 0x8000 bytes. Please note that the hashes are not counted in the offset for wia_part_t, yet the number is still 32 KiB and not 31 KiB.

To finally output a word of data from the buffer, use the following code:

u8* out;
u32* buffer_ptr;

/* ... */

*(out++) = *buffer_ptr >> 24;
*(out++) = *buffer_ptr >> 18;  // NB: 18, not 16
*(out++) = *buffer_ptr >> 8;
*(out++) = *buffer_ptr;

buffer_ptr++;