zos-ccsid-converter 0.1.4

z/OS CCSID converter using fcntl for code page detection and conversion

z/OS CCSID Converter

A Python package for working with z/OS file code pages (CCSID) and converting files between different encodings on z/OS systems using native fcntl system calls.

Overview

This package provides a robust, high-performance solution for code page detection and file conversion on z/OS. It uses z/OS-specific fcntl system calls for file tagging instead of external commands, providing better performance, reliability, and direct access to file metadata. While currently focused on ASCII (ISO8859-1) and EBCDIC (IBM-1047) conversion, the architecture supports extension to additional code pages.

Installation

From Source

# Clone or download the package
cd zos_ccsid_converter

# Install the package
pip install .

For Development

# Install in editable mode with development dependencies
pip install -e .

Usage

Command Line Interface

After installation, use the zos-ccsid-converter command:

# Convert a file to EBCDIC
zos-ccsid-converter input.txt output.txt

# Convert with verbose output
zos-ccsid-converter -v input.txt output.txt

# Show help
zos-ccsid-converter --help

Python API

Import and use the package in your Python code:

from zos_ccsid_converter import CodePageService

# Initialize service
service = CodePageService()

# Detect code page
ccsid = service.get_ccsid('/path/to/file')
encoding = service.get_encoding_name('/path/to/file')

# Check file type
if service.is_ascii('/path/to/file'):
    print("File is ASCII")

# Convert data
ebcdic_bytes = service.convert_to_ebcdic(ascii_bytes)
ascii_bytes = service.convert_to_ascii(ebcdic_bytes)

# Convert files
stats = service.convert_file('/input.txt', '/output.txt')

Features

Core Capabilities:

• Direct fcntl system calls for file tag detection (F_CONTROL_CVT with f_cnvrt structure)
• Direct fcntl system calls for file tag setting (F_SETTAG with attrib_t structure - may not be supported)
• Uses Python ctypes.BigEndianStructure for proper z/OS big-endian byte order
• Support for both regular files and streams/pipes
• Graceful handling of unconvertible characters
• Detailed conversion statistics
• No subprocess overhead
• Tested and verified on z/OS with 10/10 tests passing

API Reference

CodePageService Class

The main service class for code page operations:

from zos_ccsid_converter import CodePageService

# Initialize service
service = CodePageService()

# Detect code page
ccsid = service.get_ccsid('/path/to/file')
encoding = service.get_encoding_name('/path/to/file')

# Check file type
if service.is_ascii('/path/to/file'):
    print("File is ASCII")

# Convert data
ebcdic_bytes = service.convert_to_ebcdic(ascii_bytes)
ascii_bytes = service.convert_to_ascii(ebcdic_bytes)

# Convert files
stats = service.convert_file('/input.txt', '/output.txt')

Initialization

service = CodePageService(verbose=False)

Code Page Detection Methods

get_ccsid(path: str) -> int

• Returns CCSID value (819=ISO8859-1, 1047=IBM-1047, 0=untagged)
• Example: ccsid = service.get_ccsid('/tmp/file.txt')

get_encoding_name(path: str) -> str

• Returns encoding name: 'ISO8859-1', 'IBM-1047', or 'untagged'
• Example: encoding = service.get_encoding_name('/tmp/file.txt')

is_ascii(path: str) -> bool

• Returns True if file is ASCII/ISO8859-1
• Example: if service.is_ascii('/tmp/file.txt'): ...

is_ebcdic(path: str) -> bool

• Returns True if file is EBCDIC/IBM-1047
• Example: if service.is_ebcdic('/tmp/file.txt'): ...

is_untagged(path: str) -> bool

• Returns True if file is untagged
• Example: if service.is_untagged('/tmp/file.txt'): ...

Data Conversion Methods

convert_bytes(data: bytes, source_encoding: str, target_encoding: str) -> bytes

• Convert bytes from one encoding to another
• Example: ebcdic = service.convert_bytes(ascii_data, 'ISO8859-1', 'IBM-1047')

convert_to_ebcdic(data: bytes, source_encoding: str = 'ISO8859-1') -> bytes

• Convert bytes to EBCDIC (IBM-1047)
• Example: ebcdic = service.convert_to_ebcdic(b"Hello World")

convert_to_ascii(data: bytes, source_encoding: str = 'IBM-1047') -> bytes

• Convert bytes to ASCII (ISO8859-1)
• Example: ascii = service.convert_to_ascii(ebcdic_data)

convert_file(input_path: str, output_path: str, source_encoding: Optional[str] = None, target_encoding: str = 'IBM-1047') -> Dict

• Convert entire file from one encoding to another
• Auto-detects source encoding if not specified
• Returns dictionary with conversion statistics
• Example:

  stats = service.convert_file('/input.txt', '/output.txt')
  if stats['success']:
      print(f"Converted {stats['bytes_read']} bytes")
  

Convenience Functions

from zos_ccsid_converter import detect_code_page, detect_encoding, convert_data

# Detect code page without instantiating service
ccsid = detect_code_page('/tmp/file.txt')

# Detect encoding name
encoding = detect_encoding('/tmp/file.txt')

# Convert data
ebcdic = convert_data(b"Hello", 'ISO8859-1', 'IBM-1047')

Package Structure

zos_ccsid_converter/
├── setup.py                    # Package configuration
├── pyproject.toml              # Modern Python packaging metadata
├── MANIFEST.in                 # File inclusion rules
├── LICENSE                     # Apache 2.0 license
├── README.md                   # This file
├── zos_ccsid_converter/       # Main package
│   ├── __init__.py            # Package exports
│   ├── converter.py           # Core conversion logic
│   └── cli.py                 # Command-line interface
├── tests/                      # Test suite
│   └── test_ebcdic_converter.py
└── examples/                   # Usage examples
    └── example_service_usage.py

Building and Distribution

Using the Makefile (Recommended)

The package includes a Makefile for easy building, testing, and publishing:

# Show all available targets
make help

# Build the package
make build

# Run tests
make test

# Build and test
make all

# Install locally
make install

# Install in development mode
make install-dev

# Publish to TestPyPI (for testing)
make publish-test

# Publish to PyPI (production)
make publish

Manual Build

# Install build tools
pip install build

# Build distribution packages
python -m build

# This creates:
# - dist/zos_ccsid_converter-1.0.0-py3-none-any.whl
# - dist/zos_ccsid_converter-1.0.0.tar.gz

Install from Built Package

# Install from wheel
pip install dist/zos_ccsid_converter-1.0.0-py3-none-any.whl

# Or install from source distribution
pip install dist/zos_ccsid_converter-1.0.0.tar.gz

Testing

Run the comprehensive test suite:

# From package directory
cd tests
python3 test_ebcdic_converter.py

# With verbose output
python3 test_ebcdic_converter.py --verbose

# Keep test files for inspection
python3 test_ebcdic_converter.py --keep-files

Test Coverage:

• ISO8859-1 encoded file conversion
• IBM-1047 encoded file handling (no conversion)
• Untagged file handling (treated as EBCDIC)
• Empty file conversion
• Special characters conversion
• Large file conversion (~100KB)
• ISO8859-1 pipe conversion
• IBM-1047 pipe conversion
• File tag operations (get/set)
• Error handling (nonexistent files)

Examples

See examples/example_service_usage.py for complete working examples:

1. Detecting code pages
2. Checking file types
3. Converting bytes
4. Converting files
5. Auto-detection and conversion
6. Batch processing multiple files
7. Integration patterns for existing code

Technical Details

z/OS fcntl Implementation

The package uses z/OS-specific fcntl system calls for file tagging:

Encoding Detection:

# Uses F_CONTROL_CVT (13) with f_cnvrt structure
qcvt = f_cnvrt(3, 0, 0)  # cvtcmd=3 for query
    result = fcntl.fcntl(fd, F_CONTROL_CVT, qcvt)
    cvt_result = f_cnvrt.from_buffer_copy(result)
    
    # Get file CCSID directly
    ccsid = cvt_result.fccsid

Advantages:

• Direct system call (no subprocess)
• Uses ctypes.BigEndianStructure for proper z/OS big-endian byte order
• Lower overhead
• Works with file descriptors
• Native z/OS API usage
• Correct handling of z/OS-specific structures
• Tested and verified on z/OS

z/OS fcntl File Tagging

CCSID Mappings

CCSID	Encoding	Description
819	ISO8859-1	ASCII/Latin-1
1047	IBM-1047	EBCDIC
0	untagged	No tag set

fcntl Constants

F_SETTAG = 12       # Set file tag information
F_CONTROL_CVT = 13  # Control conversion (query/set file CCSID)

Note: z/OS does not have F_GETTAG. Use F_CONTROL_CVT with the f_cnvrt structure to query file tags.

File Tag Structures

Important: Both structures use ctypes.BigEndianStructure to match z/OS big-endian byte order.

f_cnvrt Structure (for F_CONTROL_CVT)

Used to query file conversion settings (CCSID detection):

struct f_cnvrt {
    int cvtcmd;      // Command: 3=query, others for setting
    short pccsid;    // Process CCSID
    short fccsid;    // File CCSID (output when querying)
}

Total size: 8 bytes (4+2+2)

# Python ctypes definition with big-endian byte order
class f_cnvrt(ctypes.BigEndianStructure):
    _fields_ = [
        ("cvtcmd", ctypes.c_int32),   # 4 bytes
        ("pccsid", ctypes.c_int16),   # 2 bytes
        ("fccsid", ctypes.c_int16),   # 2 bytes
    ]

# Usage example:
qcvt = f_cnvrt(3, 0, 0)  # cvtcmd=3 for query
result = fcntl.fcntl(fd, F_CONTROL_CVT, qcvt)
cvt_result = f_cnvrt.from_buffer_copy(result)
file_ccsid = cvt_result.fccsid  # Get file CCSID

Status: ✅ Fully working and tested on z/OS

attrib_t Structure (for F_SETTAG)

Used to set file tag information:

typedef struct attrib_t {
    int att_filetagchg;           // File tag change flag (1=change)
    int att_rsvd1;                // Reserved (0)
    unsigned short att_txtflag;   // Text flag (1=text, 0=binary)
    unsigned short att_ccsid;     // CCSID
    int att_rsvd2[2];             // Reserved (0, 0)
}

Total size: 20 bytes (4+4+2+2+8)

# Python ctypes definition with big-endian byte order
class attrib_t(ctypes.BigEndianStructure):
    _fields_ = [
        ("att_filetagchg", ctypes.c_int32),      # 4 bytes
        ("att_rsvd1", ctypes.c_int32),           # 4 bytes
        ("att_txtflag", ctypes.c_uint16),        # 2 bytes
        ("att_ccsid", ctypes.c_uint16),          # 2 bytes
        ("att_rsvd2", ctypes.c_int32 * 2),       # 8 bytes
    ]

# Usage example:
tag = attrib_t()
tag.att_filetagchg = 1
tag.att_ccsid = 819  # ISO8859-1
fcntl.fcntl(fd, F_SETTAG, bytes(tag))

Status: ⚠️ May not be supported through Python's fcntl on all z/OS systems

Requirements

• Python 3.6 or higher
• z/OS operating system
• Access to z/OS fcntl system calls

License

Apache License 2.0 - See LICENSE file for details.

Contributing

Contributions are welcome! Please ensure all tests pass before submitting changes.

Support

For issues or questions, please refer to the project documentation or contact the maintainers.

File: myfile.txt
  CCSID: 819
  Encoding: ISO8859-1
  Text: True

Convert from stdin

# Pipe data through converter
cat input.txt | ./ebcdic_converter_fcntl.py --stdin output.txt

Python API Usage

from ebcdic_converter_fcntl import convert_to_ebcdic_fcntl

# Convert a file
stats = convert_to_ebcdic_fcntl('input.txt', 'output.txt', verbose=True)

if stats['success']:
    print(f"Converted {stats['bytes_read']} bytes")
    print(f"Encoding detected: {stats['encoding_detected']}")
    print(f"Conversion needed: {stats['conversion_needed']}")
else:
    print(f"Error: {stats['error_message']}")

Stream Conversion

from ebcdic_converter_fcntl import convert_stream_to_ebcdic
import sys

# Convert stdin to file
with open('output.txt', 'wb') as f_out:
    stats = convert_stream_to_ebcdic(
        sys.stdin.buffer,
        f_out,
        source_encoding='iso8859-1'
    )

Running Tests

Run All Tests

./test_ebcdic_converter.py

Run with Verbose Output

./test_ebcdic_converter.py --verbose

Keep Test Files for Inspection

./test_ebcdic_converter.py --keep-files

Expected Output

======================================================================
EBCDIC Converter Test Suite
Testing: ebcdic_converter_fcntl.py
======================================================================

Running file conversion tests...
✓ PASS: ISO8859-1 file conversion
✓ PASS: IBM-1047 file handling
✓ PASS: Untagged file handling
✓ PASS: Empty file conversion
✓ PASS: Special characters conversion
✓ PASS: Large file conversion

Running pipe conversion tests...
✓ PASS: ISO8859-1 pipe conversion
✓ PASS: IBM-1047 pipe conversion

Running file tag operation tests...
✓ PASS: File tag operations

Running error handling tests...
✓ PASS: Nonexistent file error handling

======================================================================
TEST SUMMARY
======================================================================
Total tests: 10
Passed: 10
Failed: 0
======================================================================

Key Behavioral Differences

Untagged Files

Original: Treats untagged files as EBCDIC (copies as binary) New: Same behavior - treats untagged files as IBM-1047

Error Handling

Original: May fail silently or with generic errors New: Returns detailed error information in stats dictionary

Unconvertible Characters

Original: May fail on unconvertible characters New: Uses errors='replace' to gracefully handle unconvertible characters, leaving them with their initial value or replacing with a substitute character

Performance

Original: ~10-50ms overhead per file (subprocess spawn) New: ~1-5ms overhead per file (direct system call)

Technical Details

File Tag Detection Algorithm

1. Open file (or use provided file descriptor)
2. Prepare attrib_t structure initialized with zeros
3. Call fcntl(fd, F_GETTAG, buffer)
4. Unpack result to extract att_ccsid and att_txtflag
5. Map CCSID to encoding name

File Tag Setting Algorithm

1. Open file with read/write access
2. Prepare attrib_t structure:
   • Set att_filetagchg = 1 (indicate change)
   • Set att_txtflag = 1 for text, 0 for binary
   • Set att_ccsid to desired CCSID
   • Set reserved fields to 0
3. Call fcntl(fd, F_SETTAG, buffer)
4. Close file

Conversion Algorithm

For ISO8859-1 → IBM-1047:

1. Detect encoding using fcntl
2. Read file as ISO8859-1 text with errors='replace'
3. Write file as IBM-1047 text with errors='replace'
4. Tag output file as IBM-1047 using fcntl

For IBM-1047 or untagged:

1. Detect encoding using fcntl
2. Copy file as binary (no conversion)
3. Tag output file as IBM-1047 if untagged

Troubleshooting

"fcntl failed" Error

If fcntl operations fail, the converter falls back to treating files as untagged. This can happen if:

• File system doesn't support tagging
• Insufficient permissions
• File is on a non-z/OS file system

Pipe Conversion Issues

Named pipes (FIFOs) cannot be tagged with fcntl. Use convert_stream_to_ebcdic() for pipes and tag the output file after conversion.

Character Conversion Errors

The converter uses errors='replace' to handle unconvertible characters. Characters that cannot be converted are replaced with a substitute character (usually '?'). Check the errors field in the stats dictionary to see if any errors occurred.

References

• IBM z/OS fcntl Documentation
• z/OS File Tagging
• Python fcntl Module

License

Apache License Version 2.0

Author

Mike Fulton