Skip to main content

Calculate the number of CPUs available taking into account system constraints

Project description

cpu_count

project_badge version_badge python_version license_badge

Modified version of python's cpu_count that takes into account system constraints to calculate the number of available CPUs

Motivation

The Python standard library offers an implementation of cpu_count that returns the real number of CPUs even when they are not actually available to be used by the python process (due to constraints such as CPU affinity or CPU scheduler configurations). This is the preferred behaviour for most applications. However, when the interest is the amount of CPUs available for data processing, that approach could be misleading. Especially when it is used behind the scenes such as by the concurrent.futures.Executor when defining its defaults.

The purpose of this module is to provide this functionality in an API equal to the standard implementation. By taking into account the described constraints, this implementation attempts to return the amount of usable CPUs that are available. If no constraint is identified the result will be the same as the standard implementation.

How to install

pip install cpu_count

How to use

As an external module

This is the standard way. Just import and call cpu_count

from cpu_count import cpu_count

print(cpu_count())
# $> 8

Monkey-patch standard lib

This an alternative way that replaces python's standard cpu_count with the one from this module (Affected internal modules are posix, os and multiprocessing). The advantage of this approach is not needing to port any code, just import and call setup_monkey_patch ate the begin of your application and everything will just work™.

Note: This will also have implications in the behaviour of standard libraries that use this function

import os
from cpu_count.monkey_patch import setup_monkey_patch

print(os.cpu_count())
# $> 12

setup_monkey_patch()

print(os.cpu_count())
# $> 8

Limitation

This approach has one limitation: it can't replace previous code that imported the standard implementation using from os import cpu_count.

from os import cpu_count
from cpu_count.monkey_patch import setup_monkey_patch

print(cpu_count())
# $> 12

setup_monkey_patch()

print(cpu_count())
# $> 12

System wide monkey-patch

This approach also replaces python's standard cpu_count. However instead of calling this module's setup_monkey_patch in your application code, it will be called at python startup. For this to work you need to create a file called cpu_count_monkey_patch.pth at your python's global or local site-package's folder with the following content:

import cpu_count; cpu_count.monkey_patch.setup_monkey_patch()

NOTE: This approach is specially useful when creating container images of python applications. An example of using this on a Dockerfile can be found here

TODO

  • Add logic for Realtime Scheduler constraint
  • Create unit tests

Contributions

All contributions are very welcome.

Code styles is defined in the Editorconfig file. Besides that I use black and isort for auto-format, their configurations are defined in the Editorconfig and the black.toml files respectively.

License

BSD 3-Clause “New” or “Revised” License

See LICENSE

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cpu_count-1.0.0.tar.gz (6.2 kB view details)

Uploaded Source

Built Distribution

cpu_count-1.0.0-py3-none-any.whl (7.4 kB view details)

Uploaded Python 3

File details

Details for the file cpu_count-1.0.0.tar.gz.

File metadata

  • Download URL: cpu_count-1.0.0.tar.gz
  • Upload date:
  • Size: 6.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for cpu_count-1.0.0.tar.gz
Algorithm Hash digest
SHA256 9edd8796a7fd14143f821acd4b4fc7f8e933fe9c064805892137ff9ff72a5f8b
MD5 ae0a208ae80b73d0a2b5bd3b824de379
BLAKE2b-256 2ee3dc921e26a47b7c549795e934008b9683f0bd1ddd6aa1a08fbc2825d25822

See more details on using hashes here.

File details

Details for the file cpu_count-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: cpu_count-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 7.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for cpu_count-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 7cfc4130d0e44b4bee357b96d4479010d9a9fed68fb3e74ba8e87c549c0fab88
MD5 ef2807eb36164b295da9f874eae54681
BLAKE2b-256 b52057fd25f1a9bd90a56a75fe1976d0c147d7a5167a2baf28a85ba9afa961cf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page