zos-ccsid-converter 0.1.11

z/OS CCSID converter using fcntl for code page detection and conversion

z/OS CCSID Converter

A Python package for working with z/OS file code pages (CCSID) and converting files between different encodings on z/OS systems.

Overview

This package provides a robust, high-performance solution for code page detection and file conversion on z/OS. It uses IBM's zos-util C extension module exclusively for all file tag operations. File tag detection uses zos_util.get_tag_info() for all file types (regular files, named pipes, special files like /dev/stdin). File tag setting uses zos_util.chtag() exclusively.

Self-Contained Package: This package bundles the zos_util shared library, making it completely self-contained with no external dependencies. When you install from PyPI, everything you need is included in the package.

While currently focused on ASCII (ISO8859-1) and EBCDIC (IBM-1047) conversion, the architecture supports extension to additional code pages.

Installation

From PyPI (Recommended)

pip install zos-ccsid-converter

No additional dependencies or prerequisites required - the package is completely self-contained!

From Source

On z/OS, the build process automatically bundles the zos-util shared library into the package.

# Clone or download the package
cd zos_ccsid_converter

# Build and install (zos-util installed automatically on z/OS)
make install

# Or use pip directly
pip install .

For Development

# Install in editable mode (zos-util installed automatically on z/OS)
make install-dev

# Or use pip directly
pip install -e .

Building from Source

The package bundles the zos-util shared library during the build process:

make build

This will:

• Clone and build zos-util from https://github.com/IBM/zos-util.git
• Bundle the compiled shared library into the package
• Create wheel and source distributions

See BUNDLING.md for detailed information about the bundling process.

Running Without Installation (Development)

If you want to run the CLI directly from your cloned git directory without installing:

# From the project root directory
python3 -m zos_ccsid_converter.cli --help

# Examples:
python3 -m zos_ccsid_converter.cli input.txt output.txt
python3 -m zos_ccsid_converter.cli --info input.txt
python3 -m zos_ccsid_converter.cli --version
cat input.txt | python3 -m zos_ccsid_converter.cli --info --stdin

Alternatively, you can use the Makefile for development setup:

# Create virtual environment and install in development mode
make install-dev

# Then use the CLI
zos-ccsid-converter --help

Usage

Command Line Interface

After installation, use the zos-ccsid-converter command:

# Convert a file to EBCDIC
zos-ccsid-converter input.txt output.txt

# Convert with verbose output
zos-ccsid-converter -v input.txt output.txt

# Get file encoding information
zos-ccsid-converter --info input.txt

# Convert from stdin to file
cat input.txt | zos-ccsid-converter --stdin output.txt

# Get encoding info from stdin (no output file needed)
cat input.txt | zos-ccsid-converter --info --stdin

# Show version
zos-ccsid-converter --version

# Show help
zos-ccsid-converter --help

Python API

Import and use the package in your Python code:

from zos_ccsid_converter import CodePageService, __version__

# Get package version
print(f"Version: {__version__}")

# Initialize service
service = CodePageService()

# Detect code page
ccsid = service.get_ccsid('/path/to/file')
encoding = service.get_encoding_name('/path/to/file')

# Check file type
if service.is_ascii('/path/to/file'):
    print("File is ASCII")

# Convert data
ebcdic_bytes = service.convert_to_ebcdic(ascii_bytes)
ascii_bytes = service.convert_to_ascii(ebcdic_bytes)

# Convert files or pipes
stats = service.convert_input('/input.txt', '/output.txt')

# Also available: convert_file() for backward compatibility
stats = service.convert_file('/input.txt', '/output.txt')

Features

Core Capabilities:

• Direct fcntl system calls for file tag detection (F_CONTROL_CVT with f_cnvrt structure)
• Uses chtag command for reliable file tag setting
• Uses Python ctypes.BigEndianStructure for proper z/OS big-endian byte order
• Support for regular files, named pipes (FIFOs), and streams
• Graceful handling of unconvertible characters
• Detailed conversion statistics
• Minimal subprocess overhead (only for tag setting)
• Tested and verified on z/OS with 11/11 tests passing

API Reference

CodePageService Class

The main service class for code page operations:

from zos_ccsid_converter import CodePageService

# Initialize service
service = CodePageService()

# Detect code page
ccsid = service.get_ccsid('/path/to/file')
encoding = service.get_encoding_name('/path/to/file')

# Check file type
if service.is_ascii('/path/to/file'):
    print("File is ASCII")

# Convert data
ebcdic_bytes = service.convert_to_ebcdic(ascii_bytes)
ascii_bytes = service.convert_to_ascii(ebcdic_bytes)

# Convert files
stats = service.convert_file('/input.txt', '/output.txt')

Initialization

service = CodePageService(verbose=False)

Code Page Detection Methods

get_ccsid(path: str) -> int

• Returns CCSID value (819=ISO8859-1, 1047=IBM-1047, 0=untagged)
• Example: ccsid = service.get_ccsid('/tmp/file.txt')

get_encoding_name(path: str) -> str

• Returns encoding name: 'ISO8859-1', 'IBM-1047', or 'untagged'
• Example: encoding = service.get_encoding_name('/tmp/file.txt')

is_ascii(path: str) -> bool

• Returns True if file is ASCII/ISO8859-1
• Example: if service.is_ascii('/tmp/file.txt'): ...

is_ebcdic(path: str) -> bool

• Returns True if file is EBCDIC/IBM-1047
• Example: if service.is_ebcdic('/tmp/file.txt'): ...

is_untagged(path: str) -> bool

• Returns True if file is untagged
• Example: if service.is_untagged('/tmp/file.txt'): ...

Data Conversion Methods

convert_bytes(data: bytes, source_encoding: str, target_encoding: str) -> bytes

• Convert bytes from one encoding to another
• Example: ebcdic = service.convert_bytes(ascii_data, 'ISO8859-1', 'IBM-1047')

convert_to_ebcdic(data: bytes, source_encoding: str = 'ISO8859-1') -> bytes

• Convert bytes to EBCDIC (IBM-1047)
• Example: ebcdic = service.convert_to_ebcdic(b"Hello World")

convert_to_ascii(data: bytes, source_encoding: str = 'IBM-1047') -> bytes

• Convert bytes to ASCII (ISO8859-1)
• Example: ascii = service.convert_to_ascii(ebcdic_data)

convert_input(input_path: str, output_path: str, source_encoding: Optional[str] = None, target_encoding: str = 'IBM-1047') -> Dict

• Convert file or named pipe from one encoding to another
• Automatically detects whether input is a regular file or named pipe (FIFO)
• Auto-detects source encoding for files; defaults to ISO8859-1 for pipes
• Returns dictionary with conversion statistics including 'input_type' field
• Example:

  # Works with both files and pipes
  stats = service.convert_input('/input.txt', '/output.txt')
  stats = service.convert_input('/tmp/mypipe', '/output.txt', source_encoding='ISO8859-1')
  if stats['success']:
      print(f"Converted {stats['bytes_read']} bytes from {stats['input_type']}")
  

convert_file(input_path: str, output_path: str, source_encoding: Optional[str] = None, target_encoding: str = 'IBM-1047') -> Dict

• Convert regular file from one encoding to another
• Auto-detects source encoding if not specified
• Returns dictionary with conversion statistics
• Note: Use convert_input() for unified file/pipe handling
• Example:

  stats = service.convert_file('/input.txt', '/output.txt')
  if stats['success']:
      print(f"Converted {stats['bytes_read']} bytes")
  

Convenience Functions

from zos_ccsid_converter import detect_code_page, detect_encoding, convert_data

# Detect code page without instantiating service
ccsid = detect_code_page('/tmp/file.txt')

# Detect encoding name
encoding = detect_encoding('/tmp/file.txt')

# Convert data
ebcdic = convert_data(b"Hello", 'ISO8859-1', 'IBM-1047')

Package Structure

zos_ccsid_converter/
├── setup.py                    # Package configuration
├── pyproject.toml              # Modern Python packaging metadata
├── MANIFEST.in                 # File inclusion rules
├── LICENSE                     # Apache 2.0 license
├── README.md                   # This file
├── zos_ccsid_converter/       # Main package
│   ├── __init__.py            # Package exports
│   ├── converter.py           # Core conversion logic
│   └── cli.py                 # Command-line interface
├── tests/                      # Test suite
│   └── test_ebcdic_converter.py
└── examples/                   # Usage examples
    └── example_service_usage.py

Building and Distribution

Using the Makefile (Recommended)

The package includes a Makefile for easy building, testing, and publishing:

# Show all available targets
make help

# Build the package
make build

# Run tests
make test

# Build and test
make all

# Install locally
make install

# Install in development mode
make install-dev

# Publish to TestPyPI (for testing)
make publish-test

# Publish to PyPI (production)
make publish

Manual Build

# Install build tools
pip install build

# Build distribution packages
python -m build

# This creates:
# - dist/zos_ccsid_converter-1.0.0-py3-none-any.whl
# - dist/zos_ccsid_converter-1.0.0.tar.gz

Install from Built Package

# Install from wheel
pip install dist/zos_ccsid_converter-1.0.0-py3-none-any.whl

# Or install from source distribution
pip install dist/zos_ccsid_converter-1.0.0.tar.gz

Testing

Run the comprehensive test suite:

# From package directory
cd tests
python3 test_ebcdic_converter.py

# With verbose output
python3 test_ebcdic_converter.py --verbose

# Keep test files for inspection
python3 test_ebcdic_converter.py --keep-files

Test Coverage:

• ISO8859-1 encoded file conversion
• IBM-1047 encoded file handling (no conversion)
• Untagged file handling (treated as EBCDIC)
• Empty file conversion
• Special characters conversion
• Large file conversion (~100KB)
• ISO8859-1 pipe conversion (stream API)
• IBM-1047 pipe conversion (stream API)
• CodePageService.convert_input() with ISO8859-1 pipe
• File tag operations (get/set)
• Error handling (nonexistent files)

Examples

See examples/example_service_usage.py for complete working examples:

1. Detecting code pages
2. Checking file types
3. Converting bytes
4. Converting files
5. Auto-detection and conversion
6. Batch processing multiple files
7. Integration patterns for existing code

Technical Details

z/OS File Tagging Implementation

The package uses z/OS-specific methods for file tagging:

Encoding Detection (F_CONTROL_CVT):

# Uses F_CONTROL_CVT (13) with f_cnvrt structure
qcvt = f_cnvrt(3, 0, 0)  # cvtcmd=3 for query
result = fcntl.fcntl(fd, F_CONTROL_CVT, qcvt)
cvt_result = f_cnvrt.from_buffer_copy(result)

# Get file CCSID directly
ccsid = cvt_result.fccsid

File Tag Setting (chtag command):

# Uses chtag command for reliable tag setting
subprocess.run(['chtag', '-t', '-c', str(ccsid), path])

Advantages:

• F_CONTROL_CVT: Direct system call (no subprocess), works with file descriptors
• Uses ctypes.BigEndianStructure for proper z/OS big-endian byte order
• chtag: Reliable file tag setting (F_SETTAG through Python's fcntl is unreliable)
• Native z/OS API usage
• Correct handling of z/OS-specific structures
• Tested and verified on z/OS

z/OS fcntl File Tagging

CCSID Mappings

CCSID	Encoding	Description
819	ISO8859-1	ASCII/Latin-1
1047	IBM-1047	EBCDIC
0	untagged	No tag set

fcntl Constants

F_SETTAG = 12       # Set file tag information
F_CONTROL_CVT = 13  # Control conversion (query/set file CCSID)

Note: z/OS does not have F_GETTAG. Use F_CONTROL_CVT with the f_cnvrt structure to query file tags.

File Tag Structures

Important: Both structures use ctypes.BigEndianStructure to match z/OS big-endian byte order.

f_cnvrt Structure (for F_CONTROL_CVT)

Used to query file conversion settings (CCSID detection):

struct f_cnvrt {
    int cvtcmd;      // Command: 3=query, others for setting
    short pccsid;    // Process CCSID
    short fccsid;    // File CCSID (output when querying)
}

Total size: 8 bytes (4+2+2)

# Python ctypes definition with big-endian byte order
class f_cnvrt(ctypes.BigEndianStructure):
    _fields_ = [
        ("cvtcmd", ctypes.c_int32),   # 4 bytes
        ("pccsid", ctypes.c_int16),   # 2 bytes
        ("fccsid", ctypes.c_int16),   # 2 bytes
    ]

# Usage example:
qcvt = f_cnvrt(3, 0, 0)  # cvtcmd=3 for query
result = fcntl.fcntl(fd, F_CONTROL_CVT, qcvt)
cvt_result = f_cnvrt.from_buffer_copy(result)
file_ccsid = cvt_result.fccsid  # Get file CCSID

Status: ✅ Fully working and tested on z/OS

File Tag Setting (chtag command)

The package uses the chtag command for setting file tags, as F_SETTAG through Python's fcntl is unreliable:

# Set text file tag
chtag -t -c 1047 /path/to/file

# Set binary file tag
chtag -b /path/to/file

Python usage:

import subprocess

# Set text file to IBM-1047
subprocess.run(['chtag', '-t', '-c', '1047', '/path/to/file'])

# Set binary file
subprocess.run(['chtag', '-b', '/path/to/file'])

Status: ✅ Reliable and fully working on z/OS

Note: F_SETTAG through Python's fcntl has known issues:

• Always returns errno 121 (Invalid argument)
• Fails to change tags when setting to a different CCSID
• See test_f_settag_issue.py for detailed demonstration of the issue

Requirements

• Python 3.6 or higher
• z/OS operating system
• Access to z/OS fcntl system calls

License

Apache License 2.0 - See LICENSE file for details.

Contributing

Contributions are welcome! Please ensure all tests pass before submitting changes.

Support

For issues or questions, please refer to the project documentation or contact the maintainers.

File: myfile.txt
  CCSID: 819
  Encoding: ISO8859-1
  Text: True

Convert from stdin

# Pipe data through converter
cat input.txt | ./ebcdic_converter_fcntl.py --stdin output.txt

Python API Usage

from ebcdic_converter_fcntl import convert_to_ebcdic_fcntl

# Convert a file
stats = convert_to_ebcdic_fcntl('input.txt', 'output.txt', verbose=True)

if stats['success']:
    print(f"Converted {stats['bytes_read']} bytes")
    print(f"Encoding detected: {stats['encoding_detected']}")
    print(f"Conversion needed: {stats['conversion_needed']}")
else:
    print(f"Error: {stats['error_message']}")

Stream Conversion

from ebcdic_converter_fcntl import convert_stream_to_ebcdic
import sys

# Convert stdin to file
with open('output.txt', 'wb') as f_out:
    stats = convert_stream_to_ebcdic(
        sys.stdin.buffer,
        f_out,
        source_encoding='iso8859-1'
    )

Running Tests

Run All Tests

./test_ebcdic_converter.py

Run with Verbose Output

./test_ebcdic_converter.py --verbose

Keep Test Files for Inspection

./test_ebcdic_converter.py --keep-files

Expected Output

======================================================================
EBCDIC Converter Test Suite
Testing: ebcdic_converter_fcntl.py
======================================================================

Running file conversion tests...
✓ PASS: ISO8859-1 file conversion
✓ PASS: IBM-1047 file handling
✓ PASS: Untagged file handling
✓ PASS: Empty file conversion
✓ PASS: Special characters conversion
✓ PASS: Large file conversion

Running pipe conversion tests...
✓ PASS: ISO8859-1 pipe conversion
✓ PASS: IBM-1047 pipe conversion

Running file tag operation tests...
✓ PASS: File tag operations

Running error handling tests...
✓ PASS: Nonexistent file error handling

======================================================================
TEST SUMMARY
======================================================================
Total tests: 11
Passed: 11
Failed: 0
======================================================================

Key Behavioral Differences

Untagged Files

Original: Treats untagged files as EBCDIC (copies as binary) New: Same behavior - treats untagged files as IBM-1047

Error Handling

Original: May fail silently or with generic errors New: Returns detailed error information in stats dictionary

Unconvertible Characters

Original: May fail on unconvertible characters New: Uses errors='replace' to gracefully handle unconvertible characters, leaving them with their initial value or replacing with a substitute character

Performance

Original: ~10-50ms overhead per file (subprocess spawn) New: ~1-5ms overhead per file (direct system call)

Technical Details

File Tag Detection Algorithm

1. Open file (or use provided file descriptor)
2. Prepare attrib_t structure initialized with zeros
3. Call fcntl(fd, F_GETTAG, buffer)
4. Unpack result to extract att_ccsid and att_txtflag
5. Map CCSID to encoding name

File Tag Setting Algorithm

1. Open file with read/write access
2. Prepare attrib_t structure:
   • Set att_filetagchg = 1 (indicate change)
   • Set att_txtflag = 1 for text, 0 for binary
   • Set att_ccsid to desired CCSID
   • Set reserved fields to 0
3. Call fcntl(fd, F_SETTAG, buffer)
4. Close file

Conversion Algorithm

For ISO8859-1 → IBM-1047:

1. Detect encoding using fcntl
2. Read file as ISO8859-1 text with errors='replace'
3. Write file as IBM-1047 text with errors='replace'
4. Tag output file as IBM-1047 using fcntl

For IBM-1047 or untagged:

1. Detect encoding using fcntl
2. Copy file as binary (no conversion)
3. Tag output file as IBM-1047 if untagged

Troubleshooting

"fcntl failed" Error

If fcntl operations fail, the converter falls back to treating files as untagged. This can happen if:

• File system doesn't support tagging
• Insufficient permissions
• File is on a non-z/OS file system

Pipe Conversion

Named pipes (FIFOs) cannot be tagged with fcntl. The convert_input() method automatically detects pipes and uses the appropriate stream conversion method, then tags the output file. For direct stream access, use convert_stream_to_ebcdic().

Example:

# Automatic pipe detection and conversion
stats = service.convert_input('/tmp/mypipe', '/output.txt', source_encoding='ISO8859-1')

# Or use stream API directly
with open('/tmp/mypipe', 'rb') as pipe_in:
    with open('/output.txt', 'wb') as file_out:
        stats = convert_stream_to_ebcdic(pipe_in, file_out, source_encoding='iso8859-1')

Character Conversion Errors

The converter uses errors='replace' to handle unconvertible characters. Characters that cannot be converted are replaced with a substitute character (usually '?'). Check the errors field in the stats dictionary to see if any errors occurred.

References

• IBM z/OS fcntl Documentation
• z/OS File Tagging
• Python fcntl Module

License

Apache License Version 2.0

Author

Mike Fulton