module-dependencies 0.2.4

Gather module dependencies of source code

Module Dependencies

Brief overview

The module_dependencies Python module allows you to gather the dependencies of specific modules in source code. It has been split into two main sections: Module and Source.

The former, Module, supports functionality for mapping a module name to the usage of that module within open source repositories. This is very useful when we are interested in determining which sections of a Python module is most frequently used. For example:

from module_dependencies import Module
from pprint import pprint

# Attempt to find 1000 imports of the "nltk" module
# in both Python files and Jupyter Notebooks each
module = Module("nltk", count="1000")
print(module.usage())
module.plot()

This program outputs:

[2022-01-03 14:14:39,127] [module_dependencies.module.session] [INFO    ] - Fetching Python source code containing imports of `nltk`...
[2022-01-03 14:14:42,824] [module_dependencies.module.session] [INFO    ] - Fetched Python source code containing imports of `nltk` (status code 200)
[2022-01-03 14:14:42,825] [module_dependencies.module.session] [INFO    ] - Parsing 6,830,859 bytes of Python source code as JSON...
[2022-01-03 14:14:42,865] [module_dependencies.module.session] [INFO    ] - Parsed 6,830,859 bytes of Python source code as JSON...
[2022-01-03 14:14:42,866] [module_dependencies.module.session] [INFO    ] - Extracting dependencies of 725 files of Python source code...
Parsing Files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 725/725 [00:02<00:00, 258.48files/s]
[2022-01-03 14:14:45,702] [module_dependencies.module.session] [INFO    ] - Extracted dependencies of 725 files of Python source code.
[2022-01-03 14:14:45,703] [module_dependencies.module.session] [INFO    ] - Fetching Jupyter Notebook source code containing imports of `nltk`...
[2022-01-03 14:14:48,726] [module_dependencies.module.session] [INFO    ] - Fetched Jupyter Notebook source code containing imports of `nltk` (status code 200)
[2022-01-03 14:14:48,726] [module_dependencies.module.session] [INFO    ] - Parsing 25,713,281 bytes of Jupyter Notebook source code as JSON...
[2022-01-03 14:14:48,886] [module_dependencies.module.session] [INFO    ] - Parsed 25,713,281 bytes of Jupyter Notebook source code as JSON...
[2022-01-03 14:14:48,888] [module_dependencies.module.session] [INFO    ] - Extracting dependencies of 495 files of Jupyter Notebook source code...
Parsing Files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 495/495 [00:02<00:00, 167.09files/s]
[2022-01-03 14:14:51,851] [module_dependencies.module.session] [INFO    ] - Extracted dependencies of 495 files of Jupyter Notebook source code.
[('nltk.tokenize.word_tokenize', 327),
('nltk.download', 298),
('nltk.corpus.stopwords.words', 257),
('nltk.tokenize.sent_tokenize', 126),
('nltk.stem.porter.PorterStemmer', 115),
('nltk.stem.wordnet.WordNetLemmatizer', 99),
('nltk.tag.pos_tag', 75),
('nltk.stem.snowball.SnowballStemmer', 48),
('nltk.data.path.append', 42),
('nltk.probability.FreqDist', 42),
('nltk.tokenize.RegexpTokenizer', 42),
('nltk.tokenize.TweetTokenizer', 35),
('nltk.corpus.wordnet.synsets', 33),
('nltk.data.load', 32),
('nltk.translate.bleu_score.corpus_bleu', 29)]

And then opens an interactive version of the following plot:

(Note that the true plot is interactive, but this copy for GitHub is just a png)

With the methods provided in the Module class, it becomes elementary to see which sections of code are frequently used, allowing you to prioritise these sections over rarely used sections.

---

module_dependencies also provides Source, wich implements functionality for mapping Python source code to the dependencies and imports within that file. For example:

from module_dependencies import Source
from pprint import pprint

# This creates a Source instance for this file itself
src = Source.from_file(__file__)

pprint(src.dependencies())
pprint(src.imports())

This program outputs:

['module_dependencies.Source.from_file', 'pprint.pprint']
['module_dependencies', 'pprint']

---

Documentation

More detailed documentation, including examples and an API Reference, can be found in the online documentation.

Installation

module_dependencies can be installed directly via pip. It is recommended to set up a virtualenvironment before installation, although this is not strictly a requirement.

The command to install module_dependencies is:

pip install module_dependencies

Note that module_dependencies requires Python 3.8 onwards.