Pre-requisites

See setup.py for a full list of Python packages.

occupationcoder is built on top of NLTK and uses ‘Wordnet’ (a corpora, number 82 on their list) and the Punkt Tokenizer Models (number 106 on their list). When the coder is run, it will expect to find these in their usual directories. If you have nltk installed, you can get them corpora using nltk.download() which will install them in the right directories or you can go to http://www.nltk.org/nltk_data/ to download them manually (and follow the install instructions).

A couple of the other packages, such as fuzzywuzzy do not come with the Anaconda distribution of Python. You can install these via pip (if you have access to the internet) or download the relevant binaries and install them manually.

File and folder description

• occupationcoder/coder applies SOC codes to job descriptions

• occupationcoder/createdictionaries turns the ONS’ index of SOC code into dictionaries used by occupationcoder/coder

• occupationcoder/dictionaries contains the dictionaries used by occupationcoder/coder

• occupationcoder/outputs is the default output directory

• occupationcoder/testvacancies contains ‘test’ vacancies to run the code on

• occupationcoder/utilities contains helper functions which mostly manipulate strings

Installation via terminal using pip

Download the package and navigate to the download directory. Then use

python setup.py sdist
cd dist
pip install occupationcoder-version.tar.gz

The first line creates the .tar.gz file, the second navigates to the directory with the packaged code in, and the third line installs the package. The version number to use will be evident from the name of the .tar.gz file.

Running the code as a python script

Importing, and creating an instance, of the coder

import pandas as pd
from occupationcoder.coder import coder
myCoder = coder.Coder()

To run the code on a single job, use the following syntax with the codejobrow(job_title,job_description,job_sector) method:

if __name__ == '__main__':
    myCoder.codejobrow('Physicist','Calculations of the universe','Professional scientific')

The if statement is required because the code is parallelised. Note that you can leave some of the fields blank and the algorithm will still return a SOC code.

To run the code on a file (eg csv name ‘job_file.csv’) with structure

use

df = pd.read_csv('path/to/foo.csv')
df = myCoder.codedataframe(df)

This will return a new dataframe with SOC code entries appended in a new column:

Running the code from the command line

If you have all the relevant packages in requirements.txt, download the code and navigate to the occupationcoder folder (which contains the README). Then run

python -m occupationcoder.coder.coder path/to/foo.csv

This will create a ‘processed_jobs.csv’ file in the outputs/ folder which has the original text and an extra ‘SOC_code’ column with the assigned SOC codes.

Testing

To run the tests in your virtual environment, use

python -m unittest

in the top level occupationcoder directory. Look in test_occupationcoder.py for what is run and examples of use. The output appears in the ‘processed_jobs.csv’ file in the outputs/ folder.

Acknowledgements

We are very grateful to Emmet Cassidy for testing this algorithm.

Disclaimer

This code is provided ‘as is’. We would love it if you made it better or extended it to work for other countries. All views expressed are our personal views, not those of any employer.