ibd-parser 0.1.5

A Python tool for parsing and analyzing InnoDB .ibd files

InnoDB IBD File Parser

A Python tool for parsing and analyzing InnoDB .ibd files. This tool helps database administrators and developers understand the internal structure of InnoDB tablespace files.

Features

• Parse InnoDB page headers
• Analyze index pages
• Extract record contents
• Support for various page types:
  • Index pages (B-tree nodes)
  • File space header
  • XDES (Extent descriptor)
  • BLOB pages
  • And more...

Installation

From PyPI:

pip install ibd-parser

From source:

pip install git+https://github.com/likeyiyy/ibd-parser.git

Usage

Command Line Interface

# Show page header
ibd-parser -f /path/to/table.ibd header --page 4

# Dump records from an index page
ibd-parser -f /path/to/table.ibd records --page 4

Python API

from ibd_parser import IBDFileParser

# Initialize parser with your .ibd file
parser = IBDFileParser("/path/to/your/table.ibd")

# Analyze a specific page
page_info = parser.analyze_page(page_no=4)

# Access page information
print(f"Page Type: {page_info['header'].page_type}")
if 'index_header' in page_info:
    print(f"Number of records: {page_info['index_header'].n_recs}")

# Get records from an index page
records = parser.get_records(page_no=4)
for record in records:
    print(record.data)

# Hex dump of a page
parser.hex_dump(page_no=4, length=128)

Project Structure

ibd_parser/
├── ibd_parser/
│   ├── __init__.py
│   ├── constants.py    # Constants and enums
│   ├── page.py        # Page structure definitions
│   ├── record.py      # Record parsing
│   ├── parser.py      # Main parser implementation
│   ├── cli.py         # Command line interface
│   └── utils.py       # Utility functions
├── tests/
├── README.md
└── pyproject.toml     # Project metadata and dependencies

Page Structure

An InnoDB page (default 16KB) consists of:

1. File Header (38 bytes)
2. Page Header (56 bytes)
3. Infimum/Supremum Records
4. User Records
5. Free Space
6. Page Directory
7. File Trailer (8 bytes)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Based on the InnoDB storage engine documentation
• Inspired by various InnoDB internals research papers and blog posts

References

• InnoDB Page Structure
• InnoDB File Space Management

Future Plans

I will continue to enhance this project to make it more practical and valuable. Planned improvements include:

• Support for more page types
• Enhanced record analysis
• Data recovery features
• More comprehensive CLI tools
• Better documentation and examples

Contributions and suggestions are welcome!

Testing and Development

Creating Test Data

The project includes a script to create a test database with various column types:

# Install MySQL Connector
pip install mysql-connector-python

# Set MySQL connection environment variables (if needed)
export MYSQL_USER=your_user
export MYSQL_PASSWORD=your_password
export MYSQL_HOST=localhost
export MYSQL_PORT=3306  # Or your Docker mapped port

# Create test database and table
python examples/create_test_data.py

The test table includes common MySQL data types:

• Integer types (TINYINT, SMALLINT, INT, BIGINT)
• Floating point types (FLOAT, DOUBLE, DECIMAL)
• String types (CHAR, VARCHAR, TEXT)
• Date and Time types (DATE, TIME, DATETIME, TIMESTAMP)
• Other types (BOOLEAN, ENUM, BINARY, BLOB)

After creating the test database, you can find the .ibd file at:

/data/docker/mysql/data/ibd_parser_test/test_table.ibd