No project description provided
Project description
PyPageCache
Introduction
Welcome to PyPageCache, a tool designed to help you manage and understand disk page caching in your Python applications.
Whether you're working on a data-intensive application or building a database, PyPageCache provides insights into how your code interacts with the disk and OS page cache.
Installation
You can install PyPageCache using pip:
pip install pypagecache
Getting Started
View how many pages are in cache
To start using PyPageCache, import the module and create an instance by specifying the path to the file or directory you want to monitor:
from pypagecache import PyPageCache
cache_stats = PyPageCache("/path/to/your/file_or_directory").stats()
print(cache_stats)
This should print something like this:
=> Page cache stats: [14359/14359] (100.0%)
This indicates that this directory/file contains 14359 pages and all of it are in page cache.
Warming the cache
Want to reduce the page faults in your app by prewarming the page cache? Simply touch all files you care about.
PyPageCache("/path/to/files").evict()
Obviously caching beyond your physical memory limits will result in eviction of older pages.
Evicting file pages from cache
Let's say you want to start afresh and see how your app performs when there are no data pages lying in your page cache. You can evict your data files & perform the test.
PyPageCache("/path/to/files").evict()
Obviously caching beyond your physical memory limits will result in eviction of older pages.
Recipes
Now that we know what are the basic operations that we can leverage. Let's do something more interesting. Let's see if we can include this as part of our specs.
Testing Cached pages by your app
import unittest
from pypagecache import PyPageCache
class TestPyPageCache(unittest.TestCase):
def setUp(self):
# Set up PyPageCache for a specific file
self.test_file_path = "/path/to/test/file.txt"
self.ppc = PyPageCache(self.test_file_path)
# Perform cache eviction to ensure a clean slate
self.ppc.evict()
def tearDown(self):
# Clean up resources if needed
pass
def my_io_intensive_function(self):
# Execute code that performs disk I/O on the specified file
with open(self.test_file_path, 'r') as file:
# Read the first 5 lines from the file
return '\n'.join(file.readline() for _ in range(5))
def test_disk_io_impact(self):
# replace this with something that you care about
self.my_io_intensive_function()
# Retrieve and analyze page cache statistics
stats = self.ppc.stats()
print(f"File Size: {stats.filesize} bytes")
print(f"Page Size: {stats.pagesize} bytes")
print(f"Cached Pages: {stats.cached_pages}")
print(f"Total Pages: {stats.total_pages}")
# Add your assertions based on the obtained statistics
self.assertLess(stats.cached_pages, 20, "Cached pages should be under 20 for a data read")
self.assertLess(stats.cached_percent(), 1.0,
"Cache ratio should be low since we're only reading top 5 lines." +
"It should not require the entire file to be paged")
self.assertGreaterEqual(stats.cached_pages, 1, "It should atleast read 1 page")
if __name__ == '__main__':
unittest.main()
Dynamic Cache Monitoring in a Loop
If you want to continuously monitor cache statistics while your application is running, you can incorporate PyPageCache into a loop.
import time
from pypagecache import PyPageCache
file_path = "/path/to/your/file.txt"
ppc = PyPageCache(file_path)
while True:
cache_stats = ppc.stats()
print(f"Current cache stats: {cache_stats}")
time.sleep(60) # Sleep for 1 minute before checking again
Contributing
First off, thanks for considering contributing to PyPageCache! 👏
PyPageCache is an early project, and I'd love to see it grow with the help of the community. Whether you're a seasoned developer or just starting, there are plenty of ways to contribute. Here's how you can get involved:
-
Bug Reports: Found a bug? Open an issue on GitHub. Be sure to provide as much detail as possible to help squash that pesky bug.
-
Feature Requests: Have an idea for a new feature? Share it! Open an issue and let's discuss how we can make PyPageCache even better.
-
Pull Requests: Feel free to submit pull requests for bug fixes, enhancements, or new features. I'll review them with a smile (and maybe a virtual high-five 🙌).
License
PyPageCache is released under the MIT License. Feel free to check out the full license for the legal details, but in a nutshell, you're free to use, modify, and distribute PyPageCache as long as you include the original license and disclaimer.
Acknowledgements
I want to express my gratitude to the vmtouch project, which served as a significant source of inspiration for PyPageCache. Their innovative approach to page caching has been a guiding light for this project.
Author
Hi there! I'm the lone hacker behind PyPageCache. If you have questions, suggestions, or just want to chat about page caching in Python, feel free to reach out:
- lokeshdevnani (Github)
- lokesh.me (Blog)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pypagecache-0.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 154816c2e8ab4e2096f73868bd04642b2a064103f0b3f763019b12f5a3fc7468 |
|
MD5 | 35ddae710a1af082b33768d2baf0b933 |
|
BLAKE2b-256 | 00f7e4cad52cca2220ca6107bd7d76cc983609bd5c1faf950b8c99da7538b8a9 |