ezhikstract

CLI tool to extract playable video from EZVIZ and Hikvision SD cards' proprietary round-robin storage format.

It works specifically with security camera or video doorbell SD cards that contain files named hiv<xxxxx>.mp4 and an index00.bin metadata/index file. The tool parses the index file, validates video segments, extracts the raw MPEG-PS / HEVC streams, and remuxes them into standard .mp4 containers, merging segments from the same day into a single daily video file formatted in your local timezone.

No full video re-encoding is performed; video streams (HEVC) are copied directly. Audio streams (AAC / PCM G.711) are transcoded to Opus to ensure broad compatibility with standard media players.

Installation

Ensure you have Python 3.10 or higher installed. You can install the package directly:

pip install ezhikstract

Usage

The CLI provides two primary command groups: list and extract. Both groups support operations on either videos or pictures.

1. The list Command Group

Used to inspect valid records on the SD card without extracting any files. Timestamps are formatted in your system's local timezone.

Videos

Lists all valid active video segments detected on the SD card.

ezhikstract list videos INPUT_DIR

• INPUT_DIR (Required): Root directory of the SD card containing the index00.bin file.

Pictures

Lists all valid picture/thumbnail records detected on the SD card.

ezhikstract list pictures INPUT_DIR

• INPUT_DIR (Required): Root directory of the SD card containing the index00p.bin file.

2. The extract Command Group

Used to retrieve and process records from the SD card.

Videos

Extracts raw video segments from active container files, filters them by timestamp if requested, remuxes/transcodes them into standard .mp4 containers, and merges segments from the same calendar day into a single daily file named after the local start time (DDMMYYYY HHMMSS.mp4).

ezhikstract extract videos INPUT_DIR [OPTIONS]

• INPUT_DIR (Required): Root directory of the SD card containing the index00.bin file.

Options:

• -o, --output PATH: The output directory for the daily merged .mp4 files. Default is ./recordings.
• --from DATETIME: Start time filter (inclusive) in local time, format: "YYYY-MM-DD HH:MM:SS".
• --to DATETIME: End time filter (exclusive) in local time, format: "YYYY-MM-DD HH:MM:SS".
• --replace / --no-replace: Overwrite existing files in the output directory. Default is --replace.

Pictures

Extracts raw snapshot thumbnails from picture container files, filters them by timestamp if requested, and writes them out as standard JPEG (.jpg) files named by local timestamp.

ezhikstract extract pictures INPUT_DIR [OPTIONS]

• INPUT_DIR (Required): Root directory of the SD card containing the index00p.bin file.

Options:

• -o, --output PATH: The output directory for the extracted .jpg files. Default is ./pictures.
• --from DATETIME: Start time filter (inclusive) in local time, format: "YYYY-MM-DD HH:MM:SS".
• --to DATETIME: End time filter (exclusive) in local time, format: "YYYY-MM-DD HH:MM:SS".
• --replace / --no-replace: Overwrite existing files in the output directory. Default is --replace.

How it Works

The SD cards of these cameras use a pre-allocated round-robin storage file format:

1. index00.bin (and the backup copy index01.bin) contains pointers, timestamps, offsets, per-file header records, and checksums for the recorded video segments.
2. The video data is written to pre-allocated hivxxxxx.mp4 files, which are all exactly 268.4 MB (as are the index files).
3. ezhikstract parses the 32-byte per-file header records in index00.bin to filter out inactive / unwritten round-robin container files (segment_count == 65535).
4. It aligns segment start offsets past proprietary headers to the MPEG-PS / HEVC start codes (0x000001).
5. It executes an intelligent 3-tier fallback extraction chain:
   • MPEG-PS + AAC/Opus: Decodes MPEG-PS streams with AAC/PCM audio transcoded to Opus.
   • MPEG-PS Video-Only: Falls back to video-only (-an) if audio headers are missing or corrupted.
   • Raw HEVC Stream: Uses -f hevc if the segment is a raw Annex-B HEVC stream without container headers.
6. Daily segments are grouped by local calendar day and concatenated using FFmpeg concat demuxer (-map 0:v -map 0:a?) to preserve audio wherever present.

Architecture and Design Decisions

The repository is built around several design choices to maintain clean separation, high performance, and robustness:

1. Stream-Piped Concurrency & Non-Blocking I/O

• Piped I/O: Segments are read in memory chunks and passed to the standard input of ffmpeg subprocesses using proc.communicate(input=segment_data, timeout=30) to eliminate OS pipe deadlocks.
• Bounded Multithreading: Uses a ThreadPoolExecutor to process segments in parallel. Concurrency limits are tuned to avoid high disk latency (limited to 4 workers for videos and 8 for pictures).

2. Lossless Remuxing & Transcoding

• Video Quality: Video streams (HEVC) are copied directly (-c:v copy) using the -tag:v hvc1 format to ensure native, lossless rendering on iOS, macOS, and desktop media players.
• Audio Compatibility: AAC / PCM G.711 audio tracks are transcoded on the fly to Opus (-c:a libopus), resolving compatibility issues without modifying the underlying video track.
• Concat Demuxer: Uses the FFmpeg concat demuxer (-f concat -map 0:v -map 0:a?) to combine chronological daily segments into a single file. This is a stream-copy operation, meaning no full video re-encoding is executed.

3. Integrity and Validation

• Active Round-Robin Filtering: Reads the per-file 32-byte header records in index00.bin to ignore unwritten round-robin files (segment_count == 65535), preventing stale clips from past recording cycles from being extracted.
• Dummy Timestamp Filtering: Automatically discards zeroed / pre-2020 dummy segment slots (start_time_raw < 1577836800).
• MPEG-PS Validation: Checks the first 2KB of each raw sector boundary for MPEG Program Stream start codes (0x000001). Any sectors corrupt from sudden power loss or circular buffer overwrites are ignored.
• JPEG Verification: Validates the Start of Image (SOI) magic bytes (0xFF 0xD8 0xFF) for all picture/thumbnail files before parsing.
• Local Timezone Support: Converts all timestamps using the system's local timezone (.astimezone()) so merged daily files correspond 100% to local calendar days.

Storage Format

The security camera SD cards (typically EZVIZ / Hikvision) use a pre-allocated, round-robin storage format. All files (index00.bin, index01.bin, and the video files hivxxxxx.mp4) are pre-allocated to the exact same size of 268.4 MB (281,444,352 bytes). The same pre-allocation strategy applies to the mobile app thumbnails (index00p.bin and hivxxxxx.pic).

Files

1. index00.bin: The primary index file containing pointers, timestamps, offsets, and checksums for the recorded video segments.
2. index01.bin: A redundant copy of index00.bin used for backup and reliability.
3. hivxxxxx.mp4: Pre-allocated video containers (numbered from hiv00000.mp4 upwards) containing segment-based raw MPEG-PS streams.
4. index00p.bin: Metadata and index for the pictures/thumbnails used for mobile app push notifications (human/motion detection).
5. hivxxxxx.pic: Pre-allocated picture containers (numbered from hiv00000.pic upwards) containing segment-based raw JPEG images.

Index File Layout (index00.bin / index00p.bin)

An index file consists of:

1. A 1280-byte header (HEADER_BUFFER_LENGTH).
2. An array of AV-File records (each 32 bytes).
3. An array of Segment records (80 bytes for video, 96 bytes for pictures).

1. File Header (1280 Bytes)

The first 1280 bytes of the index file contains general settings, the number of files, and state flags. Both index00.bin and index00p.bin share this exact structure.

2. AV-File Records Section

Immediately following the header at offset 1280, there is a contiguous array of AV-File records.

• Record Size: 32 bytes (FILE_RECORD_LENGTH).
• Total Records: Equal to av_files from the header.
• Span: Offset 1280 to 1280 + (av_files * 32).
• Note: In index00p.bin, these blocks are primarily empty placeholder values (0xffff...).

3. Segment Records Section

The segment records list begins directly after the AV-File records section.

Video Segments (index00.bin)

Each segment record is exactly 80 bytes (SEGMENT_RECORD_LENGTH). For each pre-allocated video file, the index contains space for up to 256 segment entries (MAX_SEGMENTS_PER_SOURCE_FILE).

Picture Segments (index00p.bin)

Each segment record is exactly 96 bytes. Unused segment slots are padded with zero bytes (0x0000...).

Media Container Structures

Video File Structure (hivxxxxx.mp4)

Despite the .mp4 file extension, these files are not standard MP4 containers. Instead, they are pre-allocated byte blocks containing raw MPEG Program Streams (MPEG-PS):

1. Streams:
   • Video Codec: HEVC (H.265).
   • Audio Codec: PCM (G.711 A-law / pcm_alaw). Note: Original recordings do not use AAC.
2. Segment Storage:
   • Multiple recordings are written sequentially into these containers, mapped by the offsets defined in the segment tables in index00.bin.
3. MPEG-PS Validation:
   • Valid segments start with the standard MPEG Program Stream Pack Start Code 0x000001BA (with MPEG-2 marker prefix byte 0x40 at offset 4).
   • A System Header sequence 0x000001BB is present within the first 2KB of any valid segment.
   • When a device is abruptly powered down or files are overwritten, segment blocks can become corrupted or partially written, which requires checking these magic bytes to prevent loading corrupt recordings.

Picture File Structure (hivxxxxx.pic)

These files act identically to the .mp4 files but are used exclusively for storing raw JPEG images (snapshots/thumbnails).

1. Format:
   • Standard JPEG binary images containing SOI (Start of Image) magic bytes 0xFF 0xD8 0xFF.
2. Segment Storage:
   • Successive snapshots are dumped sequentially into the 256.4 MB block.
   • Offsets mapping to exact byte locations in the .pic block are found within index00p.bin.