Notes for Development install

Important note: the package ships Fortran-based extensions that are used to provide fast ILT methods. We believe this is a useful feature. Unfortunately, while the instructions below work for most cases, not everyone’s system is set up equally well for Fortran compilation. If you experience difficulties, and are interested in messing with the code, please don’t hesitate to reach out to us at jmf356 [at] cornell.edu. We’re always happy to test the code for new applications! Fortunately if you just want to use the library, not develop/tweak it, pip install pyspecdata should now satisfy your needs (i.e. you don’t need the development install).

we would be happy for the opportunity to test distribution on new platforms! In all situations, note that this is a development library that works very well in our hands – we are happy to hear from you and work with you to try to broaden its applicability!

On Windows with Anaconda 3.X, just run:

• Remember that for new installs of anaconda, you need to turn off the python “execution alias” in windows to prevent windows from overriding all the importnat commands! Also, all of below are tested in the “anaconda prompt” in admin mode.

• conda install -y -c anaconda numpy scipy sympy pyside6 pytables matplotlib h5py libpython

• conda install -y -c conda-forge gcc_win-64 gfortran_win-64 binutils_win-64

  • If you’ve installed before, you might need to first do conda remove -y m2w64-toolchain (this is an older toolchain that breaks with numpy 2.0)

  • (the libpython and all the packages on this line are for building compiled extensions such as the ILT).

  • On an ASUS Vivobook, conda looks like it’s completely dead, but it’s not → just be patient and give it an hour. (If you look online, you’ll find plenty of info about how sloooooow conda is now, especially for conda forge, and can find some advice on using the mamba drivers to speed it up, though on a cheap new computer with windows 11 JMF has had very little luck.)

• conda install -y meson meson-python ninja (the new build system to replace distutils)

• Note: pip --version must be greater than 21.3 (this is what you need for an editable install using meson).

  • If it’s not, run conda update pip

Then follow the installation for developers below. We have a package on pip, but it currently lags behind the github repo.

On CentOS7, we’ve tested yum install python-matplotlib python-devel sympy h5py python-tables scipy (after running yum install epel-release to install the EPEL distribution). Install PySide6 separately if you want the Qt GUIs, then follow the installation for developers below.

On Debian, we’ve tested sudo apt-get install -y python3 python3-matplotlib libpython3.7 python3-dev python3-sympy python3-h5py python3-tables python3-scipy python3-setuptools gfortran pip. Then follow the installation for developers below.

On Ubuntu, we’ve tested: sudo apt update

• If you are not using miniconda or something like that, then do: sudo apt install -y python3 python3-venv python3-dev gfortran libhdf5-dev meson ninja-build - This allows you to: python3 -m venv ~/.venv to create and environment and then switch to it with source ~/.venv/bin/activate (b/c more recent versions of python really don’t like to distribute packages over apt, and prefer to distribute them over pip)

• If you are using miniconda, etc, we’ll rely on that, but you still need the modern build tools: sudo apt install -y gfortran libhdf5-dev meson ninja-build

• Finally, in either case, make sure you have recent versions of all the packages we need: pip install 'numpy>=2.0' matplotlib sympy h5py tables scipy pint meson-python

On MacOS, if you want to install as a developer your python distribution needs to have a working Fortran compiler, since some of the modules use Fortran. We have tested conda install -c conda-forge fortran-compiler, followed by conda install -y -c anaconda numpy scipy sympy pyside6 pytables matplotlib h5py. However due to a problem with more recent versions of MacOS/xcode, you need to modify setup.py to tell it where to find the system libraries. At about line 27, you need to add something like following as a keyword arg for the Extension function: library_dirs = ["/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib"], (we recommmend just using keyword completion to find a similar directory that exists). (Feel free to contact us if you have issues with this or would like to test deployment on pip for a Mac).

More generally, these instructions are based on the fact that it’s Highly Recommended that you install the following packages using a good package-management system (conda or linux package manager), rather than relying on pip or setuptools to install them:

• numpy

• scipy

• sympy

• pyside6 (for the Qt GUIs)

• pytables (in future work, we hope to eliminate dependence on this package)

• matplotlib

• h5py

• lmfit

• The python libraries, and a Fortran compiler. Under anaconda, these are supplied by libpython and mingw, respectively.

• If you plan on building the documentation, you also want sphinx_rtd_theme sphinx-gallery

(If you don’t install these packages with your system pip will try to install them, and there is a good chance it will fail – it’s known not to work great with several of these; setuptools should error out and tell you to install the packages.)

mayavi: Mayavi can be used (and gives very nice graphics), but frequently lags behind common Python distros. Therefore, this package was written so that it doesn’t depend on mayavi. Rather, you can just import mayavi.mlab and pass it to any figure list that you initialize: figlist_var(mlab = mayavi.mlab)

Installation for developers

To install pySpecData from github, just git clone https://github.com/jmfranck/pyspecdata.git.

Then switch over to the anaconda prompt (as administrator, if you are installing system-wide), move to the directory where pyproject.toml lives (root directory of repository), and type pip install --editable . --no-build-isolation. Make sure that this terminates with a successful message, and without any compilation errors. In particular:

• If it gives an error about permissions (will happen for a system-wide anaconda install), you need to load the anaconda prompt as admin (right click and run as administrator).

• Near the end (above EXT compiler optimization) it should tell you that you can run pyspecdata_dataconfig. You should do this, unless you’ve installed pyspecdata before on the computer you are working at.

Important notes for conda on Windows:

• Warning Before running the installation for developers, you must first check that the output of conda info on your git bash terminal matches the output of your anaconda prompt.

• For reasons that we don’t understand, the Fortran compiler can give odd errors, depending on which terminal you are using to install. This appears to be Windows’ fault, rather than conda’s (?). We highly recommend trying both the Anaconda prompt, as well as the standard dos prompt (press start: type cmd) if you experience errors related to compilation.

• If you want to build the documentation, run: conda install -y -c conda-forge sphinx sphinx-gallery

Setting up your _pyspecdata configuration file

New 0.9.45 (Nov ‘25): as long as you have pyqt installed, pyspecdata_dataconfig will now open a GUI that helps manage all the following.

Part of the pySpecData package is the datadir module, allowing the user to run the same code on different machines - even thought the location of the raw spectral data might change. This is controlled by the ~/.pyspecdata (unix-like) or ~/_pyspecdata (windows) config file, which looks like the following.

[General]
data_directory = /home/jmfranck/exp_data
qesr conversion = 162.66
qesr diameter = 0.704
qesr q = 4700

[ExpTypes]
odnp_nmr_comp/odnp = /home/jmfranck/exp_data/NMR_comp/ODNP

[mode]
figures = standard

[RcloneRemotes]
nmr_comp/odnp = jmf_teams:General/exp_data/NMR_comp/ODNP/

The General section points to the directory with the datasets of interest whether that is the direct path to the drive with the datasets or if you prefer Rclone, this data_directory points to your local folder of datasets. (This is also a good spot to include, e.g. proportionality constants for QESR, which we have done here, and which are utilized in the proc_scripts repo.)

The ExpTypes section gives the various locations to folders containing the appropriate data sets - either pointing to the cloud storage or pointing to the local directory your rclone adds files to. So when you call odnp_nmr_comp/odnp this will point to the actual location, /home/jmfranck/exp_data/NMR_comp/ODNP

Note that it’s possible to point the different exp_type directly to shared drives or to synced folders (google drive, onedrive, etc.), pySpecData also offers a (we think superior) method that downloads local copies of files on-demand using rclone. The new GUI makes it easier to set up different rclone remotes (e.g. different cloud resources at different schools or shared by different people) for different experiment times.

Obviously, you need to install rclone and add it to your path to do this (see next subsection). Rclone is an amazing tool that can be configured to talk to virtually any type of cloud storage (Google Drive accounts, OneDrive and SharePoint accounts, etc.)

Setting up Rclone

To get set up with Rclone, download Rclone and follow the documentation which should include running the command rclone config enabling you to set up the location and name of the cloud drive you wish to pull from. The documentation of rclone is pretty straightforward and can walk you through this. If you are at an academic institution, we highly recommend asking your IT department for a protocol for connecting rclone to your cloud storage of choice.

As of Nov 0.9.45 (Nov 2025), the GUI presents all the rclone remotes in columns right next to your experiment types. Just pick an rclone remote from the drop-down and then give the directory on the remote in the text entry box to the right.

But, to explain how this works in the config file: Inside the RcloneRemote section of your config, each key/variable points to a properly configured remote that was set up with rclone– e.g., jmf_teams here is a properly configured remote that shows up in response to the shell command rclone config. Note: as you require datasets from other folders you will need to make new folders locally to match for Rclone. You will receive error messages that guide you to do this, and you should follow them. For example, if you required a dataset from exp_data/francklab_esr/alex you will need to go into your local exp_data folder and add a new folder called francklab_esr/alex

Further installation notes

Upon upgrading from Python 2.X to 3.X, we made some notes in conda_upgrade.md; this includes some useful (but possibly dated) instructions on how to implement different environments in anaconda, how to deal with AppLocker permissions, and Windows permissions generally, if you run into any of these issues.