PyQueue Client
A Python library for managing message queues with full AWS SQS-style features for both local JSON files and remote PyQueue servers. Features include visibility timeouts, receipt handles, message tracking, and flexible consumption patterns.
Features
Core Features
- Dual Mode Operation: Seamless switch between local (file-based) and remote (HTTP API) queues
- SQS-Style API: Familiar AWS SQS-like interface with visibility timeouts and receipt handles
- Full Feature Parity: Both local and remote modes support the complete feature set
- Visibility Timeout: Messages become temporarily invisible after receive to prevent duplicate processing
- Receipt Handles: Unique tokens for safe message deletion (prevents accidental deletion)
- Message Tracking: Track receive count and first received timestamp for each message
- Flexible Consumption: Filter messages by "new only", auto-delete, or manual acknowledgment
- Backward Compatible: Automatically upgrades old messages to new format
Local Queue Features
- File-Based Storage: Simple JSON file storage for development and testing
- No External Dependencies: Works offline without any server infrastructure
- Automatic Cleanup: Expired visibility timeouts are automatically cleaned up
- Thread-Safe Design: Safe concurrent access with proper file handling
Remote Queue Features
- HTTP REST API: Connect to remote PyQueue servers
- API Key Authentication: Secure access with X-API-Key header
- Connection Pooling: Persistent HTTP sessions for better performance
- Health Monitoring: Built-in health check endpoint
- Configurable Timeouts: Control request timeout behavior
Installation
pip install pyqueue-client
Quick Start
Local Queue (Development & Testing)
from pyqueue_client import PyQueue
# Initialize local queue
queue = PyQueue(queue_type="local", queue_file="queue.json")
# Add a message
queue.add_message({"task": "send_email", "to": "user@example.com"})
# Receive messages with visibility timeout
messages = queue.receive_messages(max_messages=5, visibility_timeout=30)
# Process and delete
for msg in messages:
print(f"Processing: {msg['message_body']}")
queue.delete_message(msg['ReceiptHandle'])
Remote Queue (Production)
from pyqueue_client import PyQueue
# Initialize remote queue with authentication
queue = PyQueue(
queue_type="remote",
server_url="https://api.pyqueue.com",
queue_name="production-tasks",
api_key="your-api-key-here",
timeout=30
)
# Same API as local queue!
queue.add_message({"task": "process_order", "order_id": 12345})
messages = queue.receive_messages(max_messages=10, visibility_timeout=60)
Detailed Usage
Adding Messages
# Add message with auto-generated ID
queue.add_message({
"task": "process_payment",
"amount": 99.99,
"currency": "USD"
})
# Add message with custom ID
queue.add_message({
"task": "send_notification",
"user_id": 12345
}, item_id="notification-user-12345")
Receiving Messages (SQS-Style)
The receive_messages() method supports multiple modes:
# Standard receive with visibility timeout
messages = queue.receive_messages(
max_messages=10, # Maximum messages to receive
visibility_timeout=30, # Seconds until message becomes visible again
delete_after_receive=False, # Auto-delete after receive?
only_new=False # Only messages never received before?
)
# Each message has this structure:
# {
# "Id": "message-uuid",
# "ReceiptHandle": "receipt_xyz...", # Use this to delete
# "message_body": { ... }, # Your data
# "timestamp": "2026-01-22T12:00:00",
# "receive_count": 1, # How many times received
# "first_received_at": "2026-01-22T12:00:00"
# }
Visibility Timeout Behavior
When you receive a message, it becomes invisible to other consumers for the specified timeout period:
# Consumer 1 receives a message
messages = queue.receive_messages(visibility_timeout=60)
# Consumer 2 won't see this message for 60 seconds
# (unless Consumer 1 deletes it with delete_message())
# If Consumer 1 fails to process, message automatically
# becomes visible again after 60 seconds
Only New Messages
Filter messages that have never been received before:
# Get only messages that are being received for the first time
messages = queue.receive_messages(only_new=True)
# Useful for ensuring first-time processing
# Messages with receive_count > 0 are filtered out
Auto-Delete on Receive
Automatically delete messages after receiving (no manual acknowledgment needed):
messages = queue.receive_messages(
max_messages=5,
delete_after_receive=True
)
# Messages are immediately deleted from queue
# No need to call delete_message()
Deleting Messages
# Using receipt handle (recommended - SQS-style)
messages = queue.receive_messages(max_messages=1)
queue.delete_message(messages[0]['ReceiptHandle'])
# Using message ID directly (alternative)
queue.remove_message("message-uuid")
Updating Messages
queue.update_message("message-uuid", {
"task": "send_email",
"status": "updated",
"priority": "high"
})
Getting Messages (Without Visibility Changes)
# Get all messages (doesn't affect visibility)
all_messages = queue.get_messages()
# Get limited number of messages
first_10 = queue.get_messages(max_messages=10)
Checking Message Existence
# Check single message
if queue.has_message("message-uuid"):
print("Message exists")
# Batch check multiple messages (optimized)
existing_ids = queue.check_existence([
"msg-1", "msg-2", "msg-3", "msg-4"
])
print(f"Found: {existing_ids}")
Queue Information & Statistics
info = queue.get_queue_info()
print(info)
# Local queue returns:
# {
# "queue_type": "local",
# "queue_file": "queue.json",
# "message_count": 15,
# "visible_messages": 10, # Currently visible
# "invisible_messages": 5, # Hidden by visibility timeout
# "never_received": 8, # Messages never received
# "total_receives": 22, # Sum of all receive_counts
# "file_size": 4096
# }
Queue Maintenance
# Clear all messages
queue.clear_queue()
# Manual cleanup of expired visibility timeouts (local only)
# Note: This happens automatically on every read operation
queue.cleanup_expired_messages()
# Health check
if queue.health_check():
print("Queue is accessible and healthy")
Consumer Pattern Example
Here's a production-ready consumer pattern:
import time
import logging
from pyqueue_client import PyQueue
# Setup logging
logging.basicConfig(level=logging.INFO)
# Initialize queue
queue = PyQueue(
queue_type="remote",
server_url="https://api.pyqueue.com",
queue_name="tasks",
api_key="your-api-key",
timeout=30
)
def process_message(message_body):
"""Your business logic here"""
print(f"Processing: {message_body}")
# Simulate work
time.sleep(2)
return True
# Main consumer loop
while True:
try:
# Receive up to 5 messages with 60-second visibility timeout
messages = queue.receive_messages(
max_messages=5,
visibility_timeout=60,
only_new=False
)
if not messages:
# No messages available, wait before polling again
time.sleep(5)
continue
for msg in messages:
try:
# Process the message
success = process_message(msg['message_body'])
if success:
# Delete message after successful processing
queue.delete_message(msg['ReceiptHandle'])
print(f"✅ Processed message {msg['Id']}")
else:
print(f"⚠️ Failed to process {msg['Id']}")
# Message will become visible again after timeout
except Exception as e:
print(f"❌ Error processing {msg['Id']}: {e}")
# Message will become visible again for retry
except KeyboardInterrupt:
print("Shutting down consumer...")
break
except Exception as e:
print(f"Consumer error: {e}")
time.sleep(10) # Wait before retrying
Message Structure
Internal Message Format (in queue)
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2026-01-22T12:00:00.000000",
"message_body": {
"task": "send_email",
"user_id": 12345,
"custom_field": "any value"
},
"receive_count": 3,
"first_received_at": "2026-01-22T11:00:00.000000",
"invisible_until": "2026-01-22T12:01:00.000000",
"current_receipt_handle": "receipt_abc123..."
}
Received Message Format (after receive_messages)
{
"Id": "550e8400-e29b-41d4-a716-446655440000",
"ReceiptHandle": "receipt_abc123...",
"message_body": {
"task": "send_email",
"user_id": 12345,
"custom_field": "any value"
},
"timestamp": "2026-01-22T12:00:00.000000",
"receive_count": 3,
"first_received_at": "2026-01-22T11:00:00.000000"
}
API Reference
Initialization
PyQueue(
queue_type="local", # "local" or "remote"
queue_file="queue.json", # Local only: file path
server_url=None, # Remote only: server URL
queue_name="default", # Remote only: queue name
api_key=None, # Remote only: authentication
timeout=30 # Remote only: request timeout
)
Methods
| Method | Parameters | Returns | Description |
|---|---|---|---|
add_message(message, item_id=None) |
message (dict), item_id (str, optional) |
str (message ID) |
Add message to queue |
get_messages(max_messages=None) |
max_messages (int, optional) |
List[dict] |
Get messages without affecting visibility |
get_message(item_id) |
item_id (str) |
dict or None |
Get specific message by ID |
has_message(item_id) |
item_id (str) |
bool |
Check if message exists |
check_existence(item_ids) |
item_ids (List[str]) |
List[str] |
Batch check message existence |
receive_messages(...) |
See below | List[dict] |
Receive messages SQS-style |
delete_message(receipt_handle) |
receipt_handle (str) |
bool |
Delete message via receipt handle |
remove_message(item_id) |
item_id (str) |
bool |
Delete message via ID |
update_message(item_id, new_message) |
item_id (str), new_message (dict) |
bool |
Update message content |
clear_queue() |
None | bool |
Remove all messages |
get_queue_info() |
None | dict |
Get queue statistics |
health_check() |
None | bool |
Check queue accessibility |
cleanup_expired_messages() |
None | bool |
Manual cleanup (local only) |
receive_messages() Parameters
receive_messages(
max_messages=10, # Maximum messages to receive (1-10)
visibility_timeout=30, # Seconds until message visible again
delete_after_receive=False, # Auto-delete after receive?
only_new=False # Only never-received messages?
)
Feature Comparison: Local vs Remote
| Feature | Local Queue | Remote Queue |
|---|---|---|
| Visibility Timeout | ✅ Full support | ✅ Full support |
| Receipt Handles | ✅ Full support | ✅ Full support |
| Only New Filter | ✅ Full support | ✅ Full support |
| Receive Tracking | ✅ Full support | ✅ Full support |
| Auto-delete on Receive | ✅ Full support | ✅ Full support |
| Message Updates | ✅ Full support | ✅ Full support |
| Batch Operations | ✅ Full support | ✅ Optimized |
| API Authentication | N/A | ✅ API Key |
| Connection Pooling | N/A | ✅ Yes |
| Health Checks | ✅ File access | ✅ HTTP endpoint |
| Concurrent Access | ⚠️ File-based | ✅ Server-side |
| Performance | Fast (local I/O) | Network-dependent |
| Use Case | Dev/Testing | Production |
Logging
PyQueue Client uses Python's standard logging module:
import logging
# Basic logging
logging.basicConfig(level=logging.INFO)
# Debug mode (shows all HTTP requests, file operations, etc.)
logging.getLogger("pyqueue_client").setLevel(logging.DEBUG)
# Example output:
# INFO:pyqueue_client.queue_manager:✅ Message added: abc-123
# INFO:pyqueue_client.queue_manager:🗑 Message deleted via receipt handle: receipt_xyz
# DEBUG:pyqueue_client.remote_queue:🔗 POST http://localhost:8000/api/v1/queues/tasks/messages
Configuration
Environment Variables
QUEUE_FILE_PATH: Default path for local queue file (default:"queue.json")
export QUEUE_FILE_PATH="/var/data/myqueue.json"
from pyqueue_client import PyQueue
# Will use /var/data/myqueue.json by default
queue = PyQueue(queue_type="local")
Migration Guide
Upgrading to v1.3.0
Version 1.3.0 adds full SQS-style features to local queues:
- Visibility timeout support for local queues
- Receipt handles for safe message deletion
- Message tracking (receive_count, first_received_at)
- Enhanced queue statistics
only_newnow works for local queues too
Old messages in an existing queue file are upgraded automatically when read.
Breaking changes for local queues (remote queues are unchanged):
receive_messages()now returns the received-message format: the message ID is underId(notid), and each message has aReceiptHandle. Code that readsmsg["id"]must switch tomsg["Id"].delete_message(receipt_handle)now expects a receipt handle fromreceive_messages(). To delete by message ID, useremove_message(item_id).receive_messages()withoutdelete_after_receive=Truehides the returned messages forvisibility_timeoutseconds: delete them once processed, or they come back after the timeout.
Recommended Changes:
# Old way (still works)
messages = queue.receive_messages(max_messages=5, delete_after_receive=True)
for msg in messages:
process(msg)
# New way (recommended)
messages = queue.receive_messages(max_messages=5, visibility_timeout=60)
for msg in messages:
try:
process(msg['message_body'])
queue.delete_message(msg['ReceiptHandle']) # Use receipt handle
except Exception:
pass # Message will retry after timeout
Development & Testing
Running Tests
# Basic local queue test
python test_local.py
# Advanced feature tests
python test_local_advanced.py
# Remote queue tests (requires server)
python test_remote.py
Test Coverage
The library includes comprehensive tests:
- Basic CRUD operations
- Visibility timeout behavior
- Receipt handle validation
- Only new message filtering
- Receive count tracking
- Auto-delete functionality
- Queue statistics
- Backward compatibility
Requirements
- Python >= 3.6
- requests >= 2.25.0
- urllib3 >= 1.26.0
Use Cases
Development & Testing
Use local queues for:
- Unit tests
- Local development
- Offline development
- Simple task queues
- Prototyping
Production
Use remote queues for:
- Distributed systems
- Microservices communication
- Background job processing
- Event-driven architectures
- Scalable task processing
Best Practices
- Use visibility timeouts: Set visibility_timeout based on your processing time
- Always delete processed messages: Use receipt handles for safe deletion
- Handle failures gracefully: Messages auto-retry when visibility timeout expires
- Monitor queue statistics: Use
get_queue_info()for observability - Use only_new for deduplication: Ensure first-time processing when needed
- Enable logging in production: Debug issues with structured logs
- Set appropriate timeouts: Configure HTTP timeouts for remote queues
Troubleshooting
Message not being deleted
- Ensure you're using the correct
ReceiptHandlefromreceive_messages() - Receipt handles expire when visibility timeout expires
- Use
delete_message()notremove_message()for SQS-style deletion
Messages not becoming visible
- Check if visibility timeout has expired
- Use
get_queue_info()to see invisible message count - Call
cleanup_expired_messages()for manual cleanup (local only)
Remote connection errors
- Verify server URL is correct and accessible
- Check API key is valid
- Increase timeout parameter if network is slow
- Use
health_check()to verify server status
PyQueue ecosystem
| Project | What it is |
|---|---|
| pyqueue-server | Self-hosted queue server with an AWS SQS-style REST API (FastAPI, SQLite or JSON) |
| pyqueue_client | Python client library (pip install pyqueue-client) for a local JSON queue or a remote server |
| pyqueue-ui | Web dashboard to browse queues and messages |
| pynotiq | Worker that delivers queue messages as Telegram notifications |
| keystorebot | Telegram bot that stores key-value pairs in a PyQueue queue |
License
MIT License - see LICENSE file for details
Contributing
Contributions are welcome! Please feel free to submit issues or pull requests.
Support
For issues, questions, or feature requests, please open an issue on GitHub.
Made with ❤️ for developers who need simple, reliable message queues
Metadata
Release files for pyqueue-client 1.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pyqueue_client-1.3.0.tar.gz | 19.1 kB | Without provenance | Details for pyqueue_client-1.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pyqueue_client-1.3.0-py3-none-any.whl | Python 3 | none | any | Details for pyqueue_client-1.3.0-py3-none-any.whl |
Total release size: 33.9 kB
Release files / pyqueue_client-1.3.0.tar.gz
| Download URL | pyqueue_client-1.3.0.tar.gz |
|---|---|
| Size | 19.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
efc486b9513a130d5fd1bcb4e5bf66a82f900e1d3f16eb914b360894d0efb009
|
|
BLAKE2b-256 checksum How to use checksums |
925723979bafb70098657cc33fb269d8aae4e7ec812a27c2b4a8ac3f617bf871
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.21 {"installer":{"name":"uv","version":"0.9.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pyqueue_client-1.3.0-py3-none-any.whl
| Download URL | pyqueue_client-1.3.0-py3-none-any.whl |
|---|---|
| Size | 14.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bfa606b4032946019e968532b8284bba03bf95d1bc0fa6aa530f19056fde7ed2
|
|
BLAKE2b-256 checksum How to use checksums |
d287499992230d9f8f173bb8ccfad45c6694f72d001a810a5cea3d7429c71336
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.21 {"installer":{"name":"uv","version":"0.9.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pyqueue-client 1.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.