FastWebHDFS 0.1.0b20260131

FastWebHDFS is an async-first Python client for WebHDFS, built for secure Hadoop clusters and modern, real-time workloads. It enables high-concurrency, low-latency access to HDFS opening up possibilities to build MCP and AI-driven applications, with support for secure authentication and a roadmap toward a powerful CLI and tooling.

FastWebHDFS

FastWebHDFS is a high-performance, asynchronous Python client for interacting with Hadoop clusters via WebHDFS, designed for modern, real-time, and AI-driven workloads.

Built with async-first principles, FastWebHDFS provides non-blocking access to HDFS, making it well-suited for MCP (Model Context Protocol) integrations, streaming pipelines, and real-time AI applications that require low-latency and scalable file system operations over Hadoop. The library supports secure Hadoop environments, including Kerberos-enabled and HTTPS-secured WebHDFS clusters, enabling seamless authentication and authorization in enterprise deployments. FastWebHDFS abstracts the complexity of secure WebHDFS communication while preserving fine-grained control over connections, timeouts, and concurrency.

Key Features

• Asynchronous WebHDFS client built for high concurrency and low latency
• Secure cluster support, including Kerberos and HTTPS-enabled Hadoop environments
• Optimized for AI and real-time workloads, such as MCP-based systems and streaming inference pipelines
• Modern Python design, compatible with asyncio and popular async frameworks
• Extensible architecture, with planned support for a rich CLI and advanced tooling
• Production-ready foundation for large-scale data access over HDFS

Intended Use Cases

• Real-time AI systems reading and writing large datasets in HDFS
• MCP-enabled applications requiring fast, contextual data access
• Streaming and event-driven pipelines over Hadoop
• Async microservices interacting with enterprise Hadoop clusters
• Future CLI-based data exploration and automation workflows
• FastWebHDFS aims to bridge traditional Hadoop ecosystems with modern, async, and AI-native Python applications—bringing WebHDFS into the era of real-time intelligence.

Example

from FastWebHDFS import FastWebHDFSClient

# Instantiate a client, defaults user to current system user name
webhdfs = FastWebHDFSClient()

# Synchronous
webhdfs.liststatus('/').json()

# Asynchronous
import asyncio
asyncio.run(webhdfs.aliststatus('/')).json()