Strip BOM

Strip UTF-8 byte order mark (BOM) from strings, bytes, streams, and files. Inspired by the popular strip-bom npm package.

Features

• Multiple input types: Strip BOM from strings, bytes, bytearrays, streams, and files
• Smart validation: Validates UTF-8 encoding before processing buffers
• Memory efficient: Handles large files and streams without loading everything into memory
• Zero dependencies: Lightweight with no external dependencies
• Type safe: Full type hints for excellent IDE support
• Robust: Graceful error handling and Unicode support (emojis, CJK characters, etc.)

Why Strip BOM?

The UTF-8 Byte Order Mark (BOM) can cause issues when:

• ❌ Processing files from different sources (some have BOM, others don't)
• ❌ Comparing strings that should be identical but differ only by BOM
• ❌ Working with APIs that don't expect BOM characters
• ❌ Parsing JSON, CSV, or other structured data formats

Note: The Unicode Standard permits BOM in UTF-8 but doesn't require or recommend it, since byte order is irrelevant for UTF-8.

Installation

pip install strip-bom

Usage Examples

Strings

from strip_bom import strip_bom

text_with_bom = '\ufeffunicorn'
clean_text = strip_bom(text_with_bom)
print(clean_text)  # 'unicorn'

# Text without BOM remains unchanged
normal_text = 'Hello World'
print(strip_bom(normal_text))  # 'Hello World'

Bytes and Buffers

from strip_bom import strip_bom_buffer

bytes_with_bom = b'\xef\xbb\xbfunicorn'
clean_bytes = strip_bom_buffer(bytes_with_bom)
print(clean_bytes)  # b'unicorn'

# Invalid UTF-8 is left unchanged (safety first!)
invalid_utf8 = b'\xef\xbb\xbf\xff\xfe'
result = strip_bom_buffer(invalid_utf8)
print(result == invalid_utf8)  # True (no changes made)

Streams (Memory Efficient)

from strip_bom import strip_bom_stream
import io

# Process large streams without loading everything into memory
stream = io.BytesIO(b'\xef\xbb\xbfLarge file content here...')

# Process in chunks
for chunk in strip_bom_stream(stream, chunk_size=8192):
    # Process each chunk as needed
    print(chunk)

# Or get all content at once
stream.seek(0)
content = b''.join(strip_bom_stream(stream))

Files

from strip_bom import strip_bom_file

# Text mode (reads as UTF-8)
content = strip_bom_file('data.txt', mode='r')
print(f"File content: {content}")

# Binary mode
binary_content = strip_bom_file('data.txt', mode='rb')
print(f"Binary content: {binary_content}")

API Reference

strip_bom(text: str) -> str

Remove BOM from Unicode string.

strip_bom_buffer(buffer: Union[bytes, bytearray]) -> bytes

Remove BOM from bytes/bytearray if valid UTF-8.

strip_bom_stream(stream: BinaryIO, chunk_size: int = 8192) -> Iterator[bytes]

Remove BOM from binary stream, yielding chunks.

strip_bom_file(file_path: str, mode: str = 'r') -> Union[str, bytes]

Remove BOM from file content. Mode can be 'r'/'rt' for text or 'rb' for binary.

Learn More

• W3C: The byte-order mark (BOM) in HTML
• Unicode FAQ: UTF-8, UTF-16, UTF-32 & BOM
• Wikipedia: Byte Order Mark

Acknowledgments

Inspired by Sindre Sorhus's strip-bom npm package.

Changelog

See CHANGELOG.md for a detailed list of changes and version history.

Contributing

We welcome contributions! Please see our Contributing Guide for details.

Support

If you find this library helpful:

• ⭐ Star the repository
• 🐛 Report issues
• 🔀 Submit pull requests
• 💝 Sponsor on GitHub

License

MIT © Y. Siva Sai Krishna - see LICENSE file for details.

Author's GitHub • Author's LinkedIn • Report Issues • Package on PyPI