Minimalistic benchmarking of computers with general data/ modeling and simulation purposes
Project description
Minimalistic benchmarking of computers with general data/ modeling and simulation purposes
The two problems this package solve:
- What machine/ configuration should I buy to do mathematical modeling/ data analysis/ machine learning?
- I don't want to spend more money, how should I configurate parallelism/ backends to make the same code run faster?
Both seem to have answers: Why not O complexities? Why not look for FLOPs? Why not just do divisions on frequencies?
The answer is: time and memory consumptions, given a task, are metrics, they are not symbolic values, but distributions.
Hence:
- We can run same tasks on different machines (or let others do) to compare configs.
- We can re-run same tasks on different configs
But it seems we can just time and verbose it, why bother?
The thing is:
- You may need to test on some tasks that you don't have working code on--maybe I have that.
- Some benchmarking will take too long and may fail, you need to handle that.
- You may want to compare your task to others
- You may be lazy
Key functions: machine mode and task mode
In machine mode you expand the tasks and let it gradually reach the limit of the pc in a sequence, and get a dataframe of time, space etc. w.r.t tasks
In task mode you tune the HPOs and compute compromise.
But both returns dataframes, so it's easy to build and analyze the relations.
Quick start
Benchmarking:
bench([matrices, DFTs, ls_pred,# matrix based predictor
sgd_pred, gbdt_pred, symbolic_strings,
iters,
pdes, black_box_opts])
Get machine and environment infos:
detect_board()
detect_torch_availability()
detect_dask_availability()
detect_pycuda_availability()
Search time/ memory checkpoints along with metrics w.r.t. HPO
search_s(task = my_task,#an iterator, yielding one task at a time
reset_func = my_task_reset, timeout = 1000, intra = 0, inter = 0)
Fit relationships based on complexity curves.
What if the program running is not in py?
bench_kernel(task_command_dict, write_path)
A benchmarking software to test computer performance. 2 modes:
- machine-centric mode: given one machine, enlist performace score for all tasks in a tasklist
- task-centric mode: given one task, benchmark the machine on different configurations of the tasks
Installation:
Any environment with tensorflow and torch. Although it is generally easier with Docker, considering the incompatibility of KVM on some older machines, and the different configuration of CUDA (some are within the containers, while others are not) we give this non-docker way of installation
conda create --name mlgpu python==3.8
conda activate mlgpu
conda install tensorflow-gpu==2.7.1 -c conda-forge
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
This solves the problem on all my machines and is able to run torch and tf in the same environment.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for minibenchmark-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a240549f360df2fc00a8a6e8222b0ef575e9235514e5ec33ff0bf81aaa560bc9 |
|
MD5 | be70dc485f8009ca39f46f83b42ca89d |
|
BLAKE2b-256 | 59d114277c7600a5bcaa0a5b59988c1d2d80091c17fcd9ab01f1a78a28f35cd7 |