Difference between revisions of "Extdata"

From 3dbrew
Jump to navigation Jump to search
 
(44 intermediate revisions by 13 users not shown)
Line 1: Line 1:
−
This page describes the format and encryption of extdata, "extra data" stored on [[SD_Filesystem|SD card]] and [[Flash_Filesystem|NAND]].
+
This page describes the format and encryption of extdata, "extra data" stored on SD card and NAND, at:
−
At:
 
−
* nand/data/<ID>/extdata/<ExtdataID-High>
 
−
* sdmc/Nintendo 3DS/<ID0>/<ID1>/extdata/<ExtdataID-High>
 
  
−
(ExtdataID-High is always 00000000 for SD, and always 00048000 for NAND) Some titles can have Quota.dat stored in these directories. The directory-name for these directories is the ExtdataID-Low. Then there's a sub-directory 00000000, which contains the actual extdata. Size and number of files in this dir varies per title.
+
* <code>nand/data/&lt;ID&gt;/extdata/&lt;ExtdataID-High&gt;</code>
−
NAND stores the shared extdata and is structured exactly the same way, see [[Flash Filesystem]].
+
* <code>sdmc/Nintendo 3DS/&lt;ID0&gt;/&lt;ID1&gt;/extdata/&lt;ExtdataID-High&gt;</code>
  
−
Extdata image 00000001 contains a VSXE partition for the FST, the actual file data is stored in the subsequent extdata images.
+
ExtdataID-High is always 00000000 for SD, and always 00048000 for NAND. Regular apps can only mount SD extdata using the same extdataID which is stored in the CXI exheader. Therefore, regular apps which have the exheader extdataID set to zero can't use extdata. This restriction doesn't apply for shared extdata with extdataID high bitmask 0x48000 stored on NAND. System apps with a certain access right can mount arbitrary extdata. All NAND extdata is shared extdata, while all SD extdata is normal extdata.
  
−
Regular apps can only mount SD extdata using the same extdataID which is stored in the [[NCCH#CXI|CXI]] exheader. Therefore, regular apps which have the exheader extdataID set to zero can't use extdata. This restriction doesn't apply for shared extdata with extdataID high bitmask 0x48000 stored on NAND. System apps with a certain access right can mount arbitrary extdata.
+
All data in this page is little-endian. All &quot;unused / padding&quot; fields can contain uninitialized data unless otherwise specified.
−
All NAND extdata is shared extdata, while all SD extdata is normal extdata. Thus, normal extdata doesn't exist on NAND, and shared extdata doesn't exist on SD. The extdataID high excluding that bitmask is always zero for shared extdata.
 
  
−
=== Encryption ===
+
= Format =
  
−
These files are [[AES|encrypted]] with AES-CTR, the keyslot is initialized by [[nand/private/movable.sed|movable.sed]]. The same keyslot is used for the NAND/SD extdata MAC. The NAND extdata images are stored in cleartext. The WCHAR LowPath "/extdata/<ExtdataIDHigh>/<ExtdataIDLow>/<PathToImage>" text path is hashed with SHA-256, including the WCHAR null-terminator. A separate hash is used for Quota.dat. The base CTR seems to be then generated by XORing the calculated hash: CTRword[i] = Hashword[i] ^ Hashword[4+i].
+
To avoid confusion, the terms '''device directory / file''' and '''virtual directory / file''' are used with the following meanings:
  
−
The base CTR is fixed therefore the CTR never changes after each write. Thus it is possible to obtain some cleartext by XORing one file(like newly created extdata) with a newer file, where the newer file overwrote zeros in the original file with non-zero data.
+
* '''Device directory / file''' are the real directory / file stored on SD / NAND that can be seen under path <code>nand/data/&lt;ID&gt;/extdata/</code> or <code>sdmc/Nintendo 3DS/&lt;ID0&gt;/&lt;ID1&gt;/extdata/</code>.
 +
* '''Virtual directory / file''' are directory / file stored inside extdata virtual file system, which can be seen by applications in the mounted extdata archives.
  
−
=== Format ===
+
An extdata consists of several device directories and files, which forms a file system consisting of multiple virtual directories and files.
  
−
Extdata uses dual 'partitions' of IVFC hash trees to store data. The order of data in Extdata is as follows:
+
An extdata with ID <code>ExtdataId</code> has the following device files:
  
−
* AES MAC
+
* <code>.../extdata/&lt;ExtdataID-High&gt;/&lt;ExtdataId-Low&gt;/Quota.dat</code> (optional)
−
* DIFF Header
+
* <code>.../extdata/&lt;ExtdataID-High&gt;/&lt;ExtdataId-Low&gt;/&lt;SubDirID&gt;/&lt;SubFileID&gt;</code>
−
* Secondary DIFI Partition descriptor
 
−
* Primary DIFI Partition descriptor
 
−
* Secondary Partition IVFC Hash Tree
 
−
* Primary Partition IVFC Hash Tree
 
−
* DATA Partition (If applicable)
 
  
−
Only one Partition is active at a given time, this is determined by the DIFF header. Normally the 'data' contained in extdata is stored at level4 of the IVFC hash tree, and hence there are two versions of the 'data' stored in the Extdata image (although only one is 'active'). However if DIFI flags[0] is set, this indicates it is a DATA partition and the 'data' is stored outside the IVFC hash tree, at a relative offset defined by the DIFI partition (in this case there will be only one version of the 'data' stored in the Extdata image).
+
Note:
  
−
==== Chain Of Trust ====
+
* All device files are [[DISA and DIFF|DIFF containers]]. '''All format description below is about the inner content of the containers'''. Please unwrap these files first according to the DIFF format description before reading them using the extdata format description below.
 +
* <code>Quota.dat</code> is only observed existing for NAND shared extdata.
 +
* <code>&lt;SubDirID&gt;</code> and <code>&lt;SubFileID&gt;</code> are 8-digit hex strings.
 +
* Device file with <code>SubDirID = SubFileID = 00000000</code> doesn't exist. Other ID combinations can exists.
 +
* Device file with <code>SubDirID = 00000000</code> and <code>SubFileID = 00000001</code> is the VSXE metadata file and must exist.
 +
* Other files, besides <code>Quota.dat</code> and <code>00000000/00000001</code>, are normal sub files, are these device files one-to-one correspond to virtual files. They contain raw virtual file data in the DIFF inner content.
 +
* <code>SubDirID = 00000000</code> is usually the only one device directory that can be seen. See [[#Device Directory Capacity]] for more information.
  
−
The chain of trust in extdata is as follows:
+
== Quota File ==
  
−
* MAC verifies DIFF Header
+
The inner data of <code>Quota.dat</code> is 0x48 bytes with the following format. The file seems to limit the extdata total size.
−
* DIFF Selects and verifies via Active DIFI partition descriptor
 
−
* Active DIFI partition descriptor points to the location of active IVFC tree (and data if applicable), and provides the hash blob to verify Level 1 of the IVFC hash tree
 
−
* Each IVFC level verifies the next level, until Level 4(data).
 
  
−
=== Filesystem ===
+
{| class="wikitable" border="1"
−
 
+
! Offset
−
Title extdata contains a series of extdata images which comprise an independent file system. The first image (00000001) contains the VSXE (FST) partition, and subsequent images each containing a single file. Other extdata images, such as Quota.dat and [[Title Database|database extdata]], exist independent of a FS.
 
−
 
 
−
==== VSXE ====
 
−
{| class="wikitable"
 
−
|-
 
−
! Start
 
 
! Length
 
! Length
 
! Description
 
! Description
 
|-
 
|-
−
| 0x0
+
| 0x00
 +
| 4
 +
| Magic &quot;QUOT&quot;
 +
|-
 +
| 0x04
 
| 4
 
| 4
−
| Database Magic ("VSXE")
+
| Magic 0x30000
 
|-
 
|-
−
| 0x4
+
| 0x08
 
| 4
 
| 4
−
| Magic Number (0x30000)
+
| 0x1000, block size
 
|-
 
|-
−
| 0x8
+
| 0x0C
−
| 8
+
| 4
−
| Data Table Offset
+
| Always 126. Probably device directory capacity. See the [[#Device Directory Capacity]] more information.
 
|-
 
|-
 
| 0x10
 
| 0x10
−
| 8
+
| 4
−
| File Size, divided by the value at 0x18
+
| Always 0?
 +
|-
 +
| 0x14
 +
| 4
 +
| Max number of blocks
 
|-
 
|-
 
| 0x18
 
| 0x18
−
| 8
+
| 4
−
| Usually 0x1000
+
| Always 0?
 +
|-
 +
| 0x1C
 +
| 4
 +
| Free blocks remained
 
|-
 
|-
 
| 0x20
 
| 0x20
−
| 8
+
| 4
−
| Unknown
+
| Always 0?
 +
|-
 +
| 0x24
 +
| 4
 +
| Always 0?
 
|-
 
|-
 
| 0x28
 
| 0x28
 
| 4
 
| 4
−
| 'Action' made on most recently mounted Extdata image
+
| Free blocks remained + (blocks occupied by the recently mounted file, specified by the ID below (0 if recently deleted))
 
|-
 
|-
 
| 0x2C
 
| 0x2C
 
| 4
 
| 4
−
| Unknown
+
| Always 0?
 
|-
 
|-
 
| 0x30
 
| 0x30
 
| 4
 
| 4
−
| ID of most recently mounted Extdata image
+
| ID of most recently mounted file. Same as the one in [[Inner_FAT#Filesystem Header]]
 
|-
 
|-
 
| 0x34
 
| 0x34
 
| 4
 
| 4
−
| Unknown
+
| Always 0?
 
|-
 
|-
 
| 0x38
 
| 0x38
−
| 0x100
+
| 4
−
| Mount path, from most recently mounted Extdata image
+
| Always 0?
−
|}
 
−
 
 
−
Data Table:
 
−
 
 
−
{| class="wikitable"
 
 
|-
 
|-
−
! Start
+
| 0x3C
−
! Length
+
| 4
−
! Description
+
| Always 0?
 
|-
 
|-
−
| 0x0
+
| 0x40
−
| 0x38
+
| 4
−
| Unknown
+
| Size in bytes of most recently mounted file (device file size). 0 if recently deleted
−
|-
 
−
| 0x38
 
−
| 8
 
−
| Folder Table Offset
 
−
|}
 
−
 
 
−
===== Folder Table =====
 
−
 
 
−
Header:
 
−
{| class="wikitable"
 
 
|-
 
|-
−
! Start
+
| 0x44
−
! Length
 
−
! Description
 
−
|-
 
−
| 0x0
 
−
| 0x4
 
−
| Equivalent to the Used Folder Entries + 1
 
−
|-
 
−
| 0x4
 
 
| 4
 
| 4
−
| Equivalent to the Maximum Folder Entries + 1
+
| Always 0?
−
|-
 
−
| 0x8
 
−
| 0x20
 
−
| Unused
 
 
|}
 
|}
  
−
Folder Entry:
+
== Device Directory Capacity ==
−
{| class="wikitable"
 
−
|-
 
−
! Start
 
−
! Length
 
−
! Description
 
−
|-
 
−
| 0x0
 
−
| 0x4
 
−
| Parent Folder Index
 
−
|-
 
−
| 0x4
 
−
| 0x10
 
−
| Folder Name (ASCII)
 
−
|-
 
−
| 0x14
 
−
| 0x4
 
−
| Previous Folder's Index
 
−
|-
 
−
| 0x18
 
−
| 0x4
 
−
| Last Folder Index Entry
 
−
|-
 
−
| 0x1C
 
−
| 0x4
 
−
| Last File Index Entry
 
−
|-
 
−
| 0x20
 
−
| 0x4
 
−
| Unknown
 
−
|-
 
−
| 0x2C
 
−
| 0x4
 
−
| Unknown
 
−
|}
 
−
* The folder id/index for the current entry is related to it's position in the Folder table. The folder table is accessed like an array of 0x28 byte chunks, with the header consuming index = 0, root directory at index = 1, and the subsequent folder entries following.
 
  
−
===== File Table =====
+
A device directory in an extdata (a <code>&lt;SubDirID&gt;</code> directory) seems to have a maximum number of device files it can contain. For SD extdata, this maximum number seems to be hard-coded as 126. For NAND extdata, the number is probably indicated by a field in Quota.dat, which is, again, always 126 as observed. 3DS FS tries to put all device files in the device directory <code>00000000</code> if possible, and only when more than 126 files needed to add, a second device directory <code>00000001</code> and so on are created. However, few extdata have such amount of files to store, so the behavior lacks of use cases to confirm.
  
−
The location of the File table is calculated by aligning the end offset of the folder table to 0x1000 bytes.
+
The number 126 is probably from some kind of other capacity of 128 with <code>&quot;.&quot;</code> and <code>&quot;..&quot;</code> entries reserved. It is theorized that this is to keep a FAT directory table, with 0x20 bytes for each entry, in one 0x1000 cluster. The motivation is unclear.
  
−
Header:
+
== VSXE Filesystem ==
−
{| class="wikitable"
 
−
|-
 
−
! Start
 
−
! Length
 
−
! Description
 
−
|-
 
−
| 0x0
 
−
| 0x4
 
−
| Equivalent to the Used File Entries + 1
 
−
|-
 
−
| 0x4
 
−
| 4
 
−
| Equivalent to the Maximum File Entries + 1
 
−
|-
 
−
| 0x8
 
−
| 0x28
 
−
| Unused
 
−
|}
 
  
−
Folder Entry:
+
This is one variant of the [[Inner FAT|FAT filesystem]]. Please refer to its page for the description of the filesystem. In general, device file <code>00000000/00000001</code> contains the metadata of the filesystem, while other device files (except for the Quota file) contains normal sub-files
−
{| class="wikitable"
 
−
|-
 
−
! Start
 
−
! Length
 
−
! Description
 
−
|-
 
−
| 0x0
 
−
| 0x4
 
−
| Parent Folder Index
 
−
|-
 
−
| 0x4
 
−
| 0x10
 
−
| File Name (ASCII)
 
−
|-
 
−
| 0x14
 
−
| 0x4
 
−
| Previous File's Index
 
−
|-
 
−
| 0x18
 
−
| 0x4
 
−
| Unknown
 
−
|-
 
−
| 0x1C
 
−
| 0x4
 
−
| Unknown
 
−
|-
 
−
| 0x20
 
−
| 0x8
 
−
| Unique Extdata ID
 
−
|-
 
−
| 0x28
 
−
| 0x4
 
−
| Unknown
 
−
|-
 
−
| 0x2C
 
−
| 0x4
 
−
| Unknown
 
−
|}
 
−
* The file id/index for the current entry is related to it's position in the File Table, much like the folder entries in the Folder Table. The file table is accessed like an array of 0x30 byte chunks, with the header consuming index = 0, and the subsequent file entries following. The relationship between the index value of the file entry, and the actual file name of the extdata image that contains it it = index+1. For instance icon (the only file in every extdata), comes right after the header, with an index value of '1', and the icon is stored in extdata image '00000002'.
 
  
−
* The Unique Extdata ID, is the same value found in the DIFF header of the referenced extdata image for that file. The value changes most times the file in question is modified. When mounting an extdata image in the VSXE filesystem, if the file's extdata image doesn't have the expected Unique Extdata ID, it won't be mounted.
+
Each non-dummy file entry corresponds to a device file. The path to the device file is generated by the following computation:
  
−
==== VSXE Filesystem structure ====
+
<pre>// See previous section about this capacity
 +
const uint32_t device_dir_capacity = 126;
  
−
When extdata is created, these are *always* created regardless of whether the title actually uses them.
+
// entry index is the index in the file entry table, with the first dummy entry as
 +
// index = 0, which is never used for a real file.
 +
// file_index = 1 is reserved for the VSXE Filesystem Metadata itself, so real files
 +
// started from file_index = 2.
 +
uint32_t file_index = entry_index + 1;
  
−
* /icon This file contains the extdata [[SMDH|icon]] displayed in data management. This icon can only be written to by titles when creating extdata, titles would have to recreate extdata to change the icon. This file can't be read directly, instead it is read via [[FS:ReadExtSaveDataIcon]].
+
uint32_t SubDirID = file_index / device_dir_capacity;
−
* /user/ Contains the title's actual extdata files.
+
uint32_t SubFileID = file_index % pdevice_dir_capacity;
−
* /boss/ Can contain [[SpotPass]] content. SpotPass content can only be downloaded to this /boss directory.
 
  
−
User extdata and SpotPass extdata use separate [[FS:OpenArchive|mount]] points at /user and /boss. Therefore one mount can't access the other directory, and also can't access /icon.(The title's SpotPass extdata can be mounted by the title itself, if it uses SpotPass)
+
char extdata_path[...]; // &quot;.../extdata/&lt;ExtdataID-High&gt;/&lt;ExtdataId-Low&gt;&quot;
 +
char device_path[...]; // output path
 +
sprintf(device_path, &quot;%s/%08x/%08x&quot;, extdata_path, SubDirID, SubFileID);
 +
</pre>
 +
When mounting extdata, the unique identifier is used to match the ID stored in subfile's [[DISA and DIFF#DIFF header|DIFF header]]. If the ID doesn't match, mounting will fail.
  
 +
== Virtual File System Structure ==
  
−
Other optional but notable directories include:
+
When extdata is created, these are ''always'' created regardless of whether the title actually uses them.
−
* /user/ExBanner This directory can optionally store [[Extended_Banner| extended banners]]. When this is available, this banner is displayed instead of the [[CXI]] ExeFS banner. COMMON.bin stores the common exbanner, while <regionlang_code>.bin stores an optional separate region/language specific banner.(regionlang_code can be "JPN_JP", "USA_EN", etc)
 
  
−
=== Extdata without an independent FS ===
+
* <code>/icon</code> This virtual file contains the extdata icon displayed in data management. This icon can only be written to by titles when creating extdata, titles would have to recreate extdata to change the icon. This file can't be read directly, instead it is read via FS:ReadExtSaveDataIcon.
 +
* <code>/user/</code> This virtual directory contains the title's actual extdata files.
 +
* <code>/boss/</code> This virtual directory can contain SpotPass content. SpotPass content can only be downloaded to this <code>/boss</code> virtual directory.
  
−
==== Quota.dat ====
+
User extdata and SpotPass extdata use separate mount points at <code>/user</code> and <code>/boss</code>. Therefore one mount can't access the other virtual directory, and also can't access <code>/icon</code>.(The title's SpotPass extdata can be mounted by the title itself, if it uses SpotPass)
  
−
* This is contained in the Quota.dat extdata image.
+
Other optional but notable directories include:
  
−
{| class="wikitable"
+
* <code>/user/ExBanner</code> This virtual directory can optionally store extended banners. When this is available, this banner is displayed instead of the CXI ExeFS banner. <code>COMMON.bin</code> stores the common exbanner, while <code>&lt;regionlang_code&gt;.bin</code> stores an optional separate region/language specific banner.(regionlang_code can be &quot;JPN_JP&quot;, &quot;USA_EN&quot;, etc)
−
|-
 
−
! Start
 
−
! Length
 
−
! Description
 
−
|-
 
−
| 0x0
 
−
| 4
 
−
| Magic ("QUOT")
 
−
|-
 
−
| 0x4
 
−
| 4
 
−
| Magic Number (0x30000)
 
−
|-
 
−
| 0x8
 
−
| 8
 
−
| Unknown
 
−
|-
 
−
| 0x10
 
−
| 0x38
 
−
| Unknown
 
−
|}
 
−
It's unknown what this is used for.
 
−
 
 
−
==== Database Extdata ====
 
  
−
See [[Title Database|here]].
+
== SD Extdata ==
−
 
 
−
=== SD Extdata ===
 
 
Usually the ExtdataID low is in the format '00<Unique ID>'
 
Usually the ExtdataID low is in the format '00<Unique ID>'
  
Line 326: Line 196:
 
| 00000219
 
| 00000219
 
| 00000229
 
| 00000229
−
| [[eShop]]
+
| [[eShop]], contains store music in AAC format.
 +
|
 +
|-
 +
| 0000020b
 +
| 0000021b
 +
| 0000022b
 +
| Nintendo Zone
 
|  
 
|  
 
|-
 
|-
Line 335: Line 211:
 
|  
 
|  
 
|-
 
|-
−
| 0000020b
+
| 000002cc
−
| 0000021b
+
| 000002cd
−
| 0000022b
+
| 000002ce
−
| Nintendo Zone
+
| [[Home Menu]] theme
 
|  
 
|  
 
|-
 
|-
Line 397: Line 273:
 
|-
 
|-
 
| ?
 
| ?
 +
| 00000517
 +
| 00000518
 +
| Swapnote
 +
|
 +
|-
 +
| 0000055d
 +
| 0000055d
 +
| 0000055d
 +
| Pokémon X<br>Pokémon Y
 +
|
 +
|-
 
| ?
 
| ?
 +
| 00000725
 +
| 00000724
 +
| Ambassador Certificate
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 000007af
 +
| New Super Mario Bros. 2
 +
|
 +
|-
 +
| ?
 +
| 00000863
 +
| 00000864
 +
| Animal Crossing: New Leaf
 +
|
 +
|-
 +
| ?
 +
| 00000a85
 
| 00000a86
 
| 00000a86
−
| Professor Layton and the Miracle Mask
+
| Professor Layton and the Miracle Mask<br>Professor Layton and the Azran Legacy
 +
German Version ExtdataID is 00000a87
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00000b4f
 +
| Fullblox / Crashmo
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00000ba9
 +
| Pokémon Mystery Dungeon: Gates to Infinity
 
|  
 
|  
 
|-
 
|-
Line 417: Line 336:
 
| ?
 
| ?
 
| 00000d9a
 
| 00000d9a
−
| Donkey Kong Country Returns 3d Trailer
+
| Donkey Kong Country™<br>Returns 3D: Trailer
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00000ea6
 +
| Etrian Odyssey IV
 +
|
 +
|-
 +
| ?
 +
| 00000edf
 +
| 00000ee0
 +
| Super Smash Bros. for Nintendo 3DS
 +
|
 +
|-
 +
| ?
 +
| 00000f14
 +
| 00000f1e
 +
| Phoenix Wright: Ace Attorney - Dual Destinies
 +
|
 +
|-
 +
| ?
 +
| 00001007
 +
| 00001005
 +
| Professor Layton vs Phoenix Wright: Ace Attorney
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00001062
 +
| Nintendo Pocket Football Club
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 0000111c
 +
| Yoshi's New Island
 +
|
 +
|-
 +
| ?
 +
| 00001132
 +
| 00001131
 +
| Fantasy Life
 +
|
 +
|-
 +
| 000011c5
 +
| 000011c5
 +
| 000011c5
 +
| Pokémon Omega Ruby<br>Pokémon Alpha Sapphire
 +
|
 +
|-
 +
| ?
 +
| 000012c8
 +
| 000012ca
 +
| Mario vs. Donkey Kong: Tipping Stars
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00001499
 +
| Korg DSN-12
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 000014f2
 +
| Animal Crossing: Happy Home Designer
 +
|
 +
|-
 +
| 000014d1
 +
| 000014d1
 +
| 000014d1
 +
| [[Home Menu]] badge
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00001632
 +
| Fullblox / Stretchmo
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00001646
 +
| Pokémon Rumble World
 +
|
 +
|-
 +
| 00001648
 +
| 00001648
 +
| 00001648
 +
| Pokémon Sun<br>Pokémon Moon
 +
|
 +
|-
 +
| 0000165c
 +
| 0000165c
 +
| 0000165c
 +
| [[Home Menu]] saved theme layouts
 +
|
 +
|-
 +
| 000016C6
 +
| ?
 +
| 00001678
 +
| Yo-kai Watch
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 000018fa
 +
| Phoenix Wright: Ace Attorney - Spirit of Justice
 +
|
 +
|-
 +
| ?
 +
| 0000198e
 +
| 0000198f
 +
| Animal Crossing: New Leaf - Welcome amiibo
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00001a05
 +
| Super Mario Maker
 +
|
 +
|-
 +
| ?
 +
| ?
 +
| 00001a2e
 +
| Swapdoodle
 
|  
 
|  
 
|}
 
|}
  
−
=== NAND Shared Extdata ===
+
== NAND Shared Extdata ==
 
{| class="wikitable" border="1"
 
{| class="wikitable" border="1"
 
|-
 
|-
 
!  ExtdataID
 
!  ExtdataID
 
!  Description
 
!  Description
 +
|-
 +
| 0xe0000000
 +
| Home Menu attempts to open this archive during boot, if [[FS:OpenArchive]] doesn't return an error Home Menu seems to then launch the [[System Transfer]] application. Home Menu doesn't actually use this archive at all except for checking whether it exists.
 
|-
 
|-
 
|  0xf0000001
 
|  0xf0000001
Line 434: Line 482:
 
|-
 
|-
 
|  0xf0000009
 
|  0xf0000009
−
|  ?
+
|  Used for [[BOSS_Services|SpotPass]] content storage for [[News_Services|notifications]].
 
|-
 
|-
 
|  0xf000000b
 
|  0xf000000b
Line 443: Line 491:
 
|-
 
|-
 
|  0xf000000d
 
|  0xf000000d
−
|  ?
+
|  Home Menu SpotPass content data [[BOSS_Services|storage]].
 +
|-
 +
|  0xf000000e
 +
|  Contains [[VersionList|versionlist.dat]], used by Home Menu for the software update notification added with [[7.0.0-13]].
 
|}
 
|}
  
Line 455: Line 506:
 
| 0x0
 
| 0x0
 
| 0x4
 
| 0x4
−
| ?
+
| Magic number: 0x4F00
 
|-
 
|-
 
| 0x4
 
| 0x4
Line 486: Line 537:
 
|}
 
|}
  
−
The above date stores the last time new Play Coin(s) were obtained. The contents of this file is updated by home-menu. [[PTM:GetTotalStepCount]] is not checked constantly, after home-menu boot this is only checked when waking from sleep-mode.
+
The above date stores the last time new Play Coin(s) were obtained. The contents of this file is updated by home-menu. [[PTM:GetTotalStepCount]] is not checked constantly, after home-menu boot this is only checked when waking from sleep-mode. Each time home-menu updates the contents of this file, home-menu will set the Play Coin total to 300 if it's higher than the 300 Play Coin limit.
 +
 
 +
[[Home Menu]] loads this file / opens this archive during [[Home Menu|startup]]. When accessing this file fails, like when the file/archive is corrupted(or at least on older system-versions), the result is a brick due to Home Menu using [[SVC|svcBreak]]. [[User:Yellows8|Yellows8]] bricked a 3DS this way due to corruption via invalid [[FSFile:Write]] flush flags. When opening this extdata archive(0xf000000b) fails, Home Menu executes svcBreak.
 +
 
 +
==== Shared Extdata 0xf000000b ubll.lst ====
 +
List of blocked users.
  
−
=== Tools ===
+
Empty space is filled with 0xC-long sequences of 00 00 ... 07
  
−
* [https://github.com/ps3hen/ctr_toolkit/tree/master/extdata_tool extdata_tool] - Extract/verifies standalone extdata images and title extdata FS.
+
== Tools ==
 +
* [https://github.com/wwylele/3ds-save-tool 3ds-save-tool] - Extract/verifies extdata

Latest revision as of 19:55, 22 March 2024

This page describes the format and encryption of extdata, "extra data" stored on SD card and NAND, at:

  • nand/data/<ID>/extdata/<ExtdataID-High>
  • sdmc/Nintendo 3DS/<ID0>/<ID1>/extdata/<ExtdataID-High>

ExtdataID-High is always 00000000 for SD, and always 00048000 for NAND. Regular apps can only mount SD extdata using the same extdataID which is stored in the CXI exheader. Therefore, regular apps which have the exheader extdataID set to zero can't use extdata. This restriction doesn't apply for shared extdata with extdataID high bitmask 0x48000 stored on NAND. System apps with a certain access right can mount arbitrary extdata. All NAND extdata is shared extdata, while all SD extdata is normal extdata.

All data in this page is little-endian. All "unused / padding" fields can contain uninitialized data unless otherwise specified.

Format[edit]

To avoid confusion, the terms device directory / file and virtual directory / file are used with the following meanings:

  • Device directory / file are the real directory / file stored on SD / NAND that can be seen under path nand/data/<ID>/extdata/ or sdmc/Nintendo 3DS/<ID0>/<ID1>/extdata/.
  • Virtual directory / file are directory / file stored inside extdata virtual file system, which can be seen by applications in the mounted extdata archives.

An extdata consists of several device directories and files, which forms a file system consisting of multiple virtual directories and files.

An extdata with ID ExtdataId has the following device files:

  • .../extdata/<ExtdataID-High>/<ExtdataId-Low>/Quota.dat (optional)
  • .../extdata/<ExtdataID-High>/<ExtdataId-Low>/<SubDirID>/<SubFileID>

Note:

  • All device files are DIFF containers. All format description below is about the inner content of the containers. Please unwrap these files first according to the DIFF format description before reading them using the extdata format description below.
  • Quota.dat is only observed existing for NAND shared extdata.
  • <SubDirID> and <SubFileID> are 8-digit hex strings.
  • Device file with SubDirID = SubFileID = 00000000 doesn't exist. Other ID combinations can exists.
  • Device file with SubDirID = 00000000 and SubFileID = 00000001 is the VSXE metadata file and must exist.
  • Other files, besides Quota.dat and 00000000/00000001, are normal sub files, are these device files one-to-one correspond to virtual files. They contain raw virtual file data in the DIFF inner content.
  • SubDirID = 00000000 is usually the only one device directory that can be seen. See #Device Directory Capacity for more information.

Quota File[edit]

The inner data of Quota.dat is 0x48 bytes with the following format. The file seems to limit the extdata total size.

Offset Length Description
0x00 4 Magic "QUOT"
0x04 4 Magic 0x30000
0x08 4 0x1000, block size
0x0C 4 Always 126. Probably device directory capacity. See the #Device Directory Capacity more information.
0x10 4 Always 0?
0x14 4 Max number of blocks
0x18 4 Always 0?
0x1C 4 Free blocks remained
0x20 4 Always 0?
0x24 4 Always 0?
0x28 4 Free blocks remained + (blocks occupied by the recently mounted file, specified by the ID below (0 if recently deleted))
0x2C 4 Always 0?
0x30 4 ID of most recently mounted file. Same as the one in Inner_FAT#Filesystem Header
0x34 4 Always 0?
0x38 4 Always 0?
0x3C 4 Always 0?
0x40 4 Size in bytes of most recently mounted file (device file size). 0 if recently deleted
0x44 4 Always 0?

Device Directory Capacity[edit]

A device directory in an extdata (a <SubDirID> directory) seems to have a maximum number of device files it can contain. For SD extdata, this maximum number seems to be hard-coded as 126. For NAND extdata, the number is probably indicated by a field in Quota.dat, which is, again, always 126 as observed. 3DS FS tries to put all device files in the device directory 00000000 if possible, and only when more than 126 files needed to add, a second device directory 00000001 and so on are created. However, few extdata have such amount of files to store, so the behavior lacks of use cases to confirm.

The number 126 is probably from some kind of other capacity of 128 with "." and ".." entries reserved. It is theorized that this is to keep a FAT directory table, with 0x20 bytes for each entry, in one 0x1000 cluster. The motivation is unclear.

VSXE Filesystem[edit]

This is one variant of the FAT filesystem. Please refer to its page for the description of the filesystem. In general, device file 00000000/00000001 contains the metadata of the filesystem, while other device files (except for the Quota file) contains normal sub-files

Each non-dummy file entry corresponds to a device file. The path to the device file is generated by the following computation: 

// See previous section about this capacity
const uint32_t device_dir_capacity = 126;

// entry index is the index in the file entry table, with the first dummy entry as
// index = 0, which is never used for a real file.
// file_index = 1 is reserved for the VSXE Filesystem Metadata itself, so real files
// started from file_index = 2.
uint32_t file_index = entry_index + 1;

uint32_t SubDirID = file_index / device_dir_capacity;
uint32_t SubFileID = file_index % pdevice_dir_capacity;

char extdata_path[...]; // ".../extdata/<ExtdataID-High>/<ExtdataId-Low>"
char device_path[...]; // output path
sprintf(device_path, "%s/%08x/%08x", extdata_path, SubDirID, SubFileID);

When mounting extdata, the unique identifier is used to match the ID stored in subfile's DIFF header. If the ID doesn't match, mounting will fail.

Virtual File System Structure[edit]

When extdata is created, these are always created regardless of whether the title actually uses them.

  • /icon This virtual file contains the extdata icon displayed in data management. This icon can only be written to by titles when creating extdata, titles would have to recreate extdata to change the icon. This file can't be read directly, instead it is read via FS:ReadExtSaveDataIcon.
  • /user/ This virtual directory contains the title's actual extdata files.
  • /boss/ This virtual directory can contain SpotPass content. SpotPass content can only be downloaded to this /boss virtual directory.

User extdata and SpotPass extdata use separate mount points at /user and /boss. Therefore one mount can't access the other virtual directory, and also can't access /icon.(The title's SpotPass extdata can be mounted by the title itself, if it uses SpotPass)

Other optional but notable directories include:

  • /user/ExBanner This virtual directory can optionally store extended banners. When this is available, this banner is displayed instead of the CXI ExeFS banner. COMMON.bin stores the common exbanner, while <regionlang_code>.bin stores an optional separate region/language specific banner.(regionlang_code can be "JPN_JP", "USA_EN", etc)

SD Extdata[edit]

Usually the ExtdataID low is in the format '00<Unique ID>'

JPN ExtdataID USA ExtdataID EUR ExtdataID Description Extdata images
00000082 0000008f 00000098 Home Menu extdata, this contains home-menu savedata and cached icons for applications.
00000200 00000210 00000220 System Settings extdata added with 2.0.0-2.
00000207 00000217 00000227 Mii Maker, contains an ExBanner cleartext
00000208 00000218 00000228 Streetpass Mii Plaza 11 mb big!
00000209 00000219 00000229 eShop, contains store music in AAC format.
0000020b 0000021b 0000022b Nintendo Zone
0000020d 0000021d 0000022d Face Raiders, likely contains an ExBanner
000002cc 000002cd 000002ce Home Menu theme
? 000004aa 000004ab Nintendo Video Extra Data

This is where the video files are stored, and includes the thumbnail, the description, and possibly some checksum info in each video file stored in the extdata images. There are always 9 files within the subdirectory "00000000" of this folder, even without any videos downloaded. The files are "00000001" - "00000009", and "00000003" - "00000008" have the same filesize of 50.7 MB. It is possible to restore the older videos by overwriting all the files within this directory. Provided of course you have made a backup of the files before hand, by copying all the files within this directory to your computer. As far I'm aware its not possible to mix and match the files in order to get certain videos in one grouping, ie. having all 3 Zelda orchestral recordings in one group of 4 Nintendo videos.

00000306 00000308 00000307 Mario Kart 7
0000030b 0000030d 0000030c Nintendogs + Cats
00000326 00000326 00000326 Pokédex 3D
00000305 0000032d 0000033c Super Street Fighter IV 3D
00000328 00000358 0000033b Ridge Racer 3D
? 0000034d 00000402 Samurai Warriors Chronicles
? 0000034f 0000038a Dead or Alive Dimensions
00000481 N/A N/A Monster Hunter Tri G (Download-Quests)
? 00000517 00000518 Swapnote
0000055d 0000055d 0000055d Pokémon X
Pokémon Y
? 00000725 00000724 Ambassador Certificate
? ? 000007af New Super Mario Bros. 2
? 00000863 00000864 Animal Crossing: New Leaf
? 00000a85 00000a86 Professor Layton and the Miracle Mask
Professor Layton and the Azran Legacy

German Version ExtdataID is 00000a87

? ? 00000b4f Fullblox / Crashmo
? ? 00000ba9 Pokémon Mystery Dungeon: Gates to Infinity
? ? 00000c24 Denpa men
00000c73 00000c73 00000c73 Save Data Transfer Tool
? ? 00000d9a Donkey Kong Country™
Returns 3D: Trailer
? ? 00000ea6 Etrian Odyssey IV
? 00000edf 00000ee0 Super Smash Bros. for Nintendo 3DS
? 00000f14 00000f1e Phoenix Wright: Ace Attorney - Dual Destinies
? 00001007 00001005 Professor Layton vs Phoenix Wright: Ace Attorney
? ? 00001062 Nintendo Pocket Football Club
? ? 0000111c Yoshi's New Island
? 00001132 00001131 Fantasy Life
000011c5 000011c5 000011c5 Pokémon Omega Ruby
Pokémon Alpha Sapphire
? 000012c8 000012ca Mario vs. Donkey Kong: Tipping Stars
? ? 00001499 Korg DSN-12
? ? 000014f2 Animal Crossing: Happy Home Designer
000014d1 000014d1 000014d1 Home Menu badge
? ? 00001632 Fullblox / Stretchmo
? ? 00001646 Pokémon Rumble World
00001648 00001648 00001648 Pokémon Sun
Pokémon Moon
0000165c 0000165c 0000165c Home Menu saved theme layouts
000016C6 ? 00001678 Yo-kai Watch
? ? 000018fa Phoenix Wright: Ace Attorney - Spirit of Justice
? 0000198e 0000198f Animal Crossing: New Leaf - Welcome amiibo
? ? 00001a05 Super Mario Maker
? ? 00001a2e Swapdoodle

NAND Shared Extdata[edit]

ExtdataID Description
0xe0000000 Home Menu attempts to open this archive during boot, if FS:OpenArchive doesn't return an error Home Menu seems to then launch the System Transfer application. Home Menu doesn't actually use this archive at all except for checking whether it exists.
0xf0000001 NAND JPEG/MPO files and phtcache.bin from the camera application are stored here. This also contains UploadData.dat.
0xf0000002 NAND M4A files from the sound application are stored here
0xf0000009 Used for SpotPass content storage for notifications.
0xf000000b Contains idb.dat, idbt.dat, gamecoin.dat, ubll.lst, CFL_DB.dat, and CFL_OldDB.dat. These files contain cleartext Miis and some data relating (including cached ICN data) to Play/Usage Records.
0xf000000c Contains bashotorya.dat and bashotorya2.dat.
0xf000000d Home Menu SpotPass content data storage.
0xf000000e Contains versionlist.dat, used by Home Menu for the software update notification added with 7.0.0-13.

Shared Extdata 0xf000000b gamecoin.dat[edit]

Offset Size Description
0x0 0x4 Magic number: 0x4F00
0x4 0x2 Total Play Coins
0x6 0x2 Total Play Coins obtained on the date stored below. When the below date does not match the current date, this field is reset to zero, then the date(and other fields) are updated. Once this value is >=10, no more Play Coins can be obtained until the current date changes.
0x8 0x4 Total step count at the time a new Play Coin was obtained.
0xC 0x4 Step count for the day the last Play Coin was obtained, for that day's step count(same as the step count displayed by home-menu when this file was updated).
0x10 0x2 Year
0x12 0x1 Month
0x13 0x1 Day

The above date stores the last time new Play Coin(s) were obtained. The contents of this file is updated by home-menu. PTM:GetTotalStepCount is not checked constantly, after home-menu boot this is only checked when waking from sleep-mode. Each time home-menu updates the contents of this file, home-menu will set the Play Coin total to 300 if it's higher than the 300 Play Coin limit.

Home Menu loads this file / opens this archive during startup. When accessing this file fails, like when the file/archive is corrupted(or at least on older system-versions), the result is a brick due to Home Menu using svcBreak. Yellows8 bricked a 3DS this way due to corruption via invalid FSFile:Write flush flags. When opening this extdata archive(0xf000000b) fails, Home Menu executes svcBreak.

Shared Extdata 0xf000000b ubll.lst[edit]

List of blocked users.

Empty space is filled with 0xC-long sequences of 00 00 ... 07

Tools[edit]

Retrieved from "https://www.3dbrew.org/w/index.php?title=Extdata&oldid=22521"