Tools for analysing Zipf's law from text samples
Tools in python for analysing Zipf’s law from text samples.
This can be installed as a package from the python3 package library using the terminal command:
>>> pip3 install zipfanalysis
The package can be used from within python scripts to estimate Zipf exponents, assuming a simple power law model for word frequencies and ranks. To use the pacakge import it using
The easiest way to carry out an analysis on a book or text file, using different estimators, is:
alpha_clauset = zipfanalysis.clauset("path_to_book.txt") alpha_pdf = zipfanalysis.ols_pdf("path_to_book.txt", min_frequency=3) alpha_cdf = zipfanalysis.ols_cdf("path_to_book.txt", min_frequency=3) alpha_abc = zipfanalysis.abc("path_to_book.txt")
In Depth Method
Convert a book or text file to the frequency of words, ranked from highest to lowest:
word_counts = zipfanalysis.preprocessing.preprocessing.get_rank_frequency_from_text("path_to_book.txt")
Carry out different types of analysis to fit a power law to the data:
# Clauset et al estimator alpha_clauset = zipfanalysis.estimators.clauset.clauset_estimator(word_counts) # Ordinary Least Squares regression on log(rank) ~ log(frequency) # Optional low frequency cut-off alpha_pdf = zipfanalysis.estimators.ols_regression_pdf.ols_regression_pdf_estimator(word_counts, min_frequency=2) # Ordinary least squares regression on the complemantary cumulative distribution function of ranks # OLS on log(P(R>rank)) ~ log(rank) # Optional low frequency cut-off alpha_cdf = zipfanalysis.estimators.ols_regression_cdf.ols_regression_cdf_estimator(word_counts) # Approximate Bayesian computation (regression method) # Assumes model of p(rank) = C prob_rank^(-alpha) # prob_rank is a word's rank in an underlying probability distribution alpha_abc = zipfanalysis.estimators.approximate_bayesian_computation.abc_estimator(word_counts)
Development - Next Steps
- Speed up abc. Current bottleneck is sampling from infinite power law. Could be sped up by considering we only need the frequency vector of ranks, not the whole sample. So for example could sample from unoform distribution then drop values into interger ranked buckets based on inverse CDF.
- Build in frequency rank analysis. Convert to frequency counts representation, then carry out fit on that.
- Add significance testing
- Add ability to calcaulte x_min and truncated power laws.
- Speed up OLS on the cdf
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size zipfanalysis-0.5.tar.gz (9.2 kB)||File type Source||Python version None||Upload date||Hashes View|