Kryptel 8 Storage

Container Format

Although Kryptel 8 uses virtually the same format of data storage as Kryptel 7, there are quite a number of differences:

1. Five key-related fields starting from 'Key ID' and ending with 'Key verification block' have been removed; Kryptel 8 replaced them with three fields - the size of the key block, the key block itself, and the key block HMAC. The key block incapsulates the container's key(s); the storage handler handles it as a black box.
2. Init vectors have been removed from container header and from segment header. Both container directory and fixup list use the same non-random init vector produced as described below in 'Notes on key usage'.
3. Container header and fixup segment header now contain MD5 checksums that the previous versions stored in the corresponding trailers. Those checksums are used only for quick verification without key; the storage handler ignores them and verifies the corresponding HMACs instead.
4. Trailers no more contain verification data (MD5 checksums). Kryptel 8 stores there component and key data that are used for data recovery if the container header gets corrupted. As before, trailers are not used in normal operation as all those data are contained in the container header.
5. Header copy is not stored as all the data required for recovery can now be found in trailers.

Base Segment Format

[Container Header]      Always present
[Data Area]             Optional, contains data blocks
[Agent Data Area]       Optional
[Directory]             Always present
[Trailer]               Always present
[Alignment Data]        If required (see 'Segment Alignment')
		

Container Header

Container header contains IDs of the components used for encryption, component initialization values, and other critical data required for opening the container.

Size	Description
4	Container tag
2	Header size including header hash
2	Version of handler that created the container
2	Version of handler required to process the container
16	Storage Handler component ID (must be CID_STORAGE_7)
16	Agent component ID
16	Cipher component ID
16	Compressor component ID
16	Hash function component ID
20	Cipher parameter block
4	Compressor parameter block
12	Hash function parameter block
2	Size of key block
. . .	Key block
HashSize	Key block HMAC
4	Size of agent data field (0 if no agent data)
6	Agent data address (points to directory if no agent data)
HashSize	Agent data HMAC (0 if no agent data)
6	Size of the directory area
HashSize	Directory HMAC (computed on unencrypted data)
STR	Cipher name string
STR	Cipher scheme string
STR	Compressor name string
STR	Compressor scheme string
STR	Hash function name string
STR	Hash function scheme string
16	MD5 hash of the data area
16	MD5 hash of the agent data area
16	MD5 hash of the directory area
16	MD5 hash of the header

STR represents a UTF-16 string; first word (2 bytes) contains the string length in characters. The string does not contain trailing zero. Each UNICODE character occupies 2 bytes.

The container tag, the offset to the handler component ID, and the offset to the agent component ID are constants. The container tag goes first; the position of the agent/handler component IDs must remain the same for the given tag in order for the GetContainerHandlers function to work correctly.

Note that the cipher, the compressor, and the hash function cannot be null components.

All MD5 hashes are computed on the encrypted data so that the user could verify the container integrity without entering the password. If an area is empty, the corresponding hash must contain zeroes.

Data Area

Data area is a set of objects' data streams. Each data stream is an independent encrypted stream. Data area is never accessed sequentally so gaps are allowed.

Data recovery block

A data stream may be followed by an optional data recovery block. Note that recovery blocks are created but never accessed except during recovery scan.

Size	Description
4	Random data
4	Recovery block tag
2	Size of the recovery block
6	Size of the data stream
. . .	Initialization vector
16	Data stream MD5 hash
2	Size of agent-owned recovery data
. . .	Agent-owned recovery data
. . .	Bytes 0xE8 to align to BlockSize boundary - 4
4	CRC-32 of the previous data

Unlike data streams and the directory area, recovery blocks are encrypted as a simple sequence of blocks in CBC chaining mode, using zero initialization vector.

Agent Data Area

This field contains agent-specific data. The handler does not use these data, it just keeps them for the associated agent. The agent data are memory-based, so the agent should keep the data size reasonable.

Container Directory

The container directory is an encrypted stream. The directory data is compressed before encryption. It contains exactly one directory entry for the upper-level object (which in turn may contain any number of child objects). In some sense this object may be considered as the container itself.

Directory Entry

Size	Description
2	Object start tag
16	Object ID
6	Size of object's data block (0 if no data)
6	Size of uncompressed data block (present if data size > 0)
6	Address of data block (present if data size > 0)
2	Size of associated recovery block (present if data size > 0)
BlockSize	Initialization vector (present if data size > 0)
16	Data block MD5 hash (present if data size > 0)
4	Size of object's attribute block (0 if no attributes)
. . .	Attribute block
. . .	[Optional directory entries for child objects]
2	Object end tag

If the object has a data block, it is an encrypted stream. The stream is always compressed, however the storage client (i.e. an agent) can specify the compression level when calling IEncryptedObject::CreateStream, and that level may be set to CT_NO_COMPRESSION. The Kryptel agents use this method to turn off compression for specific files.

Trailer

Trailers are used for data recovery only. The standard Open operation does not read it.

Note that there in fact is only one trailer as all the fixup segment trailers are exact copies of the base segment trailer.

Size	Description
4	Trailer tag
2	Trailer size including trailer MD5 hash
16	Cipher component ID
16	Compressor component ID
16	Hash function component ID
20	Cipher parameter block
4	Compressor parameter block
12	Hash function parameter block
2	Size of key block
. . .	Key block
HashSize	Key block HMAC
16	MD5 hash of the preceding fields

Fixup Segments

Fixup segment contains corrections that should be applied to previous segments.

[Segment Header]        Always present
[Data Area]             Optional, contains data blocks
[Agent Data Area]       Optional
[Fixup List]            Optional
[Trailer]               Always present
[Alignment Data]        If required (see 'Segment Alignment')
		

Segment Header

Size	Description
4	Fixup tag
4	Size of agent data field (see note below)
6	Agent data address (points to fixup list if no agent data)
HashSize	Agent data HMAC
6	Size of the fixup list in bytes (0 if no fixup records)
HashSize	Fixup list HMAC
16	MD5 hash of the segment data area
16	MD5 hash of the agent data area
16	MD5 hash of the fixup list area
16	MD5 hash of the preceding fields

If neither agent data nor fixup list is present, then the pointer at offset 8 points to the trailer. An empty fixup segment is considered as normal; for example, a segment, which only purpose is removing the agent data, will consist of a header and a trailer only. Unused fields (HMACs) are ignored and must be filled with zeros.

All MD5 hashes are computed on the encrypted data so the user could verify the container integrity without entering the password. If an area is empty, the corresponding hash must contain zeroes.

Note about fixup tag

There are two fixup tags: in-progress and complete tag. The in-progress tag means that the previous container modification has failed and the segment is incomplete. Such a segment should be thrown away silently. When the handler completes a modification session, the last operation it performs is replacing the in-progress tag with the complete segment tag. The complete tag does not guarantee that the segment is valid, it just informs the handler that the segment has been properly closed and should be parsed.

Note also that the other header fields do not contain any meaningful data if the tag field contains in-progress. The actual header will be written later, along with the regular segment tag (see the next note).

Note about order of writes

When the handler closes a fixup session, it must write the segment parts down in the following order:

Fixup list -> Segment trailer -> Alignment data -> Segment header
		

This way the handler can distinguish incomplete segments from corrupted ones, that is

If the tag is in-progress, the handler fixes the container silently by simply throwing this segment away (i.e. by setting EOF at the tag position). There is no risk of data loss as the previous session has not been completed (and so source files have not been deleted).

If the tag is complete, but any of the hashes does not match, then the segment is corrupted, and the user must be notified.

Fixup Agent Data Area

Depending on the value of the size field in the header, there are three possible cases:

• Size is 0. The fixup segment contains no agent data. The next field points to the fixup list, and the agent data HMAC field is ignored.
• Size is -1 (0xFFFFFFFF). The fixup segment contains no agent data, and the existing agent data must be removed. The next field points to the fixup list, and the agent data HMAC field is ignored.
• Size is in the range 1 to 0xFFFFFFFE. The fixup segment contains agent data that is to replace the existing data (if present). The next field is the agent data address followed by agent data HMAC.

Fixup Data Area

Fixup data area is identical to base segment's data area, that is, it is a collection of encrypted streams (gaps are allowed). A stream may be followed by a corresponding recovery block.

Fixup List

Fixup list is a sequence of fixup records.

'Create Object' Record

Size	Description
2	'Create Object' tag
2	Number of IDs in object's path (including object's own ID)
16*n	Object path

'Attach Attributes' Record

Size	Description
2	'Attach Attributes' tag
2	Number of IDs in object's path (including object's own ID), may be zero
16*n	Object path (if the previous field > 0)
4	Size of the attribute block
. . .	Attribute block (present if size > 0)

If size of attribute block is 0, this record removes existing attribute block.

'Attach Data' Record

Size	Description
2	'Attach Data' tag
2	Number of IDs in object's path (including object's own ID), may be zero
16*n	Object path (if the previous field > 0)
6	Size of data block (0 if no data)
6	Size of uncompressed data block (present if size > 0)
6	Address of data block (present if size > 0)
2	Size of associated recovery block (present if size > 0)
BlockSize	Initialization vector (present if size > 0)
16	Data block MD5 hash (present if size > 0)

If size of data block is 0, this record removes existing data block.

'Add Object' Record

Size	Description
2	'Add Object' tag
2	Number of IDs in object's path (including object's own ID)
16*n	Object path
6	Size of data block (0 if no data)
6	Size of uncompressed data block (present if size > 0)
6	Address of data block (present if size > 0)
2	Size of associated recovery block (present if size > 0)
BlockSize	Initialization vector (present if size > 0)
16	Data block MD5 hash (present if size > 0)
4	Size of the attribute block
. . .	Attribute block (present if size > 0)

'Move Object' Record

Size	Description
2	'Move Object' tag
2	Number of IDs in source object's path (including object's own ID)
16*n	Source object path
2	Number of IDs in target object's path (including object's own ID)
16*n	Target object path

'Delete Object' Record

Size	Description
2	'Delete Object' tag
2	Number of IDs in object's path (including object's own ID)
16*n	Object path

'Undelete Object' Record

Size	Description
2	'Undelete Object' tag
2	Number of IDs in object's path (including object's own ID)
16*n	Object path
1	1 if recursive undelete, 0 otherwise

Note about 'Attach Attributes' and 'Attach Data' records

Those records may be used on objects, which already have attributes / data streams. In this case these fixup records replace the existing attributes / data stream.

Note also that those two records allow modification of the root object, so their object path fields may be empty.

Note about 'Add Object' record

The 'Add Object' record is excessive and can be replaced with 'Create Object', possibly followed by 'Attach Attributes' and/or 'Attach Data'. The only purpose for introducing that record is reducing the size of the fixup list. It is the responsibility of the storage handler to recognize the typical sequence 'Create Object' => 'Attach Attributes' => 'Attach Data' and pack it into a single 'Add Object' record.

Note about object replacement

There are two ways to replace an existing object: the recommended one is to issue a pair of fixup records - first 'remove object', then 'add object' with a different object ID. This way the deleted object will remain accessible and recoverable.

The other method is to replace object's attribute and data; while the old object's data still remain recoverable, it is not a natural operation for a file agent (as there is nothing to 'undelete').

Segment Trailer

It is a copy of the container trailer. It does not depend on the segment contents and just contains the data required for recovery decryption.

About Trailers

The reason for introducing trailer in v6 container was to provide a simple and easy-to-locate (always the last 96 bytes of the container) data structure for container verification. In v7 trailer was not easy-to-locate anymore; even in a single-segment container the trailer was followed by an unknown number of alignment bytes. So verification of v7 container required partial parsing of all the segment headers and there was not much sense in retaining an excessive data structure. However it had been decided to keep trailer in v7 containers for the sake of format commonality.

Kryptel 8 moved verification data to headers; trailers now are just reserve copies of the data required for decryption.

Segment Alignment

Fixup segments always start on a segment alignment boundary, which is 4096. This is the size of the largest disk sector; aligning a new segment on a disk sector boundary helps to avoid overwriting the previous segment's data when a new segment is created. Even if overwritten with the same data, the sector still might get corrupted if a disk error occurs. Aligning will exclude any write operation on the last sector of the preceding segment, making container corruption during modification much less likely.

The handler adds alignment data to align the container size to the 4k boundary. This way, the size of the resulting container is always a multiple of 4096; if it is not, then the last segment is assumed to be corrupted (see also the discussion of the 'in-progress' tag above).

Although the alignment data are not important, it is highly recommended to use zeros or another simple regular pattern. Using random data for segment alignment may raise a suspicion that the alignment block contains a backdoor or a lockpick. Kryptel 7 storage handler uses repeating hex string 'EDC7'.

Tag Values

Container tag                   0x07AA050B
Trailer tag                     0x0B0508AA

Recovery block tag              0xE0EAF0C8

Fixup in progress tag           0x70ADB050
Fixup segment tag               0x70AEB050

Object start tag                0x050B
Object end tag                  0x07AA

Create object tag               0xAE50
Attach attributes tag           0xAE54
Attach stream tag               0xAE55
Add object tag                  0xAE59
Move object tag                 0xAE5D
Delete object tag               0xAE5E
Undelete object tag             0xAE5F
		

Note About Deleted Objects

For version 7 encrypted storage, deletion of an object (i.e. calling IEncryptedObject::DeleteChildObject) does not actually removes the object. This operation just sets object's 'EFL_OBJECT_DELETED' flag and creates a new 'Delete Object' fixup record. The object remains fully accessible but becomes read-only; any attempt to modify the object causes an exception. The only modification operation allowed for a deleted object is undeletion (i.e. calling IEncryptedObject::UndeleteChildObject).

The storage is not guaranteed to keep objects that was added and then deleted in the same session (actually keeping ANY deleted object is not guaranteed; it is just a useful side effect of the storage organization). Specifically, Kryptel 7 storage handler discards objects, deleted during base segment creation, but keeps the objects, added and then deleted in the same fixup session.

Key Usage Notes

The cipher key, the HMAC key (for those HMACs that are computed for agent data and directory), and the directory/fixup init vector are produced from the HashSize-long base key.

Cipher uses first KeySize bytes of the key. If KeySize is greater than HashSize, the base key is padded with zeroes to KeySize;

In order to produce HMAC key revert the base key's bytes and invert them. For example:

Base key: 01 02 03 ... D0 E0 F0
HMAC key: 0F 1F 2F ... FC FD FE

The size of HMAC key is also HashSize.

Initialization vector is produced the same way as HMAC key. After that the HashSize long result is either truncated to BlockSize or extended with bytes 0xA5. On the last step the result is encrypted; the encrypted BlockSize-long vector is used as the init vector for container directory and fixup lists.