pelican_nlp

pelican_nlp stands for "Preprocessing and Extraction of Linguistic Information for Computational Analysis - Natural Language Processing". This package enables the creation of standardized and reproducible language processing pipelines, extracting linguistic features from various tasks like discourse, fluency, and image descriptions.

Installation

Create conda environment

conda create --name pelican-nlp --channel defaults python=3.10

Activate environment

conda activate pelican-nlp

From this repository (default for development and pre-PyPI testing). This makes pelican-run use the code on disk, not a published wheel:

cd /path/to/PELICAN-nlp
pip install -e '.[dev]'

Confirm the command is bound to the checkout (run this from any directory, not only the repo):

python -c "import pelican_nlp.cli; print(pelican_nlp.cli.__file__)"

The printed path should be .../PELICAN-nlp/pelican_nlp/cli.py. If it contains site-packages, the env is still on an installed wheel; rerun the editable install.

From PyPI (released package only):

pip install 'pelican_nlp[dev]'

Optional extras

pip install pelican_nlp still installs the full stack. Extra names document which libraries a YAML config needs. They do not shrink that default install today: pip install 'pelican_nlp[transcription]' is not smaller while torch, fastText, and the other heavy libraries remain required dependencies. The names are the install contract if the default is slimmed later.

pip install 'pelican_nlp[transcription]'
pip install 'pelican_nlp[embeddings]'
pip install 'pelican_nlp[all]'

Extra	YAML that needs it
`transcription`	`input_file: audio` with a `transcription:` block
`acoustic`	`opensmile_feature_extraction` / `prosogram_extraction`
`embeddings`	`metrics_to_extract` embeddings, logits, or perplexity
`nlp`	`pipeline_options.normalize_text`
`topic`	`metrics_to_extract` includes `topic_modeling`
`all`	union of the extras above (includes BERTopic)
`dev`	pytest (`pelican-run --run-tests`)

Usage

To run pelican_nlp, you need a configuration.yml file in your main project directory. This file defines the settings and parameters used for your project.

Sample configuration files are available here: https://github.com/ypauli/pelican_nlp/tree/main/examples

Adapt a sample configuration to your needs.
Save your personalized configuration.yml in the root of your project directory.

Running pelican_nlp

You can run pelican_nlp via the command line or a Python script.

From the command line:

Navigate to your project directory (must contain your participants/ folder and configuration.yml), then run:

conda activate pelican-nlp
pelican-run

Running tests

After the editable install above, run the suite from any directory (same conda env):

pelican-run --run-tests

That uses local tests/ and examples/. You do not need to publish to PyPI first.

Include example golden tests (needs models; the same set as pytest --run-examples):

pelican-run --run-tests --examples
pelican-run --run-tests --examples fluency,discourse

Example runs print a short progress summary. For the full pipeline dump:

pelican-run --run-tests --examples --example-logs

Forward extra pytest flags after --:

pelican-run --run-tests -- -k test_model_registry -q

pelican-test-examples is an alias for pelican-run --run-tests --examples.

To optimize performance, close other programs and limit GPU usage during language processing.

Data Format Requirements: LPDS

For reliable operation, your data must follow the Language Processing Data Structure (LPDS), inspired by brain imaging data structures like BIDS.

Main Concepts (Quick Guide)

Project Root: Contains a participants/ folder plus optional files like participants.tsv, dataset_description.json, and README.
Participants: Each participant has a folder named part-<ID> (e.g., part-01).
Sessions (Optional): For longitudinal studies, use ses-<ID> subfolders inside each participant folder.
Tasks/Contexts: Each session (or directly in the participant folder for non-longitudinal studies) includes subfolders for specific tasks (e.g., interview, fluency, image-description).
Data Files: Named with structured metadata, e.g.: part-01_ses-01_task-fluency_cat-semantic_acq-baseline_transcript.txt

Filename Structure

Filenames follow this format:

part-<id>[_ses-<id>]_task-<label>[_<key>-<value>...][_suffix].<extension>

Required Entities: part, task
Optional Entities Examples: ses, cat, acq, proc, metric, model, run, group, param
Suffix Examples: transcript, audio, embeddings, logits, annotations

Example Project Structure

my_project/
├── participants/
│   ├── part-01/
│   │   └── ses-01/
│   │       └── interview/
│   │           └── part-01_ses-01_task-interview_transcript.txt
│   └── part-02/
│       └── fluency/
│           └── part-02_task-fluency_audio.wav
├── configuration.yml
├── dataset_description.json
├── participants.tsv
└── README.md

Features

Feature 1: Cleaning text files
- Handles whitespaces, timestamps, punctuation, special characters, and case-sensitivity.
Feature 2: Linguistic Feature Extraction
- Extracts semantic embeddings, logits, distance from optimality, perplexity and semantic similarity.
Feature 3: Acoustic Feature Extraction
- Extracts prosogram and openSMILE feature.

Examples

You can find example setups on the github repository in the examples folder.

Contributing

Contributions are welcome! Please check out the contributing guide.

License

This project is licensed under Attribution-NonCommercial 4.0 International. See the LICENSE file for details.

Citation

If you use this project, please cite:

Pauli Y, Marsman J-B, Rabe F, et al. Standardising the NLP Workflow: A Framework for Reproducible Linguistic Analysis. arXiv preprint arXiv:2511.15512 [cs.CL] 2025. https://doi.org/10.48550/arXiv.2511.15512

Metadata

Release files for pelican-nlp 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for pelican-nlp 0.4.0
File	Interpreter	ABI	Platform	Reset
pelican_nlp-0.4.0-py3-none-any.whl 83.4 MB Sep 8, 2026	Python 3	none	any	Details

Release files / pelican_nlp-0.4.0-py3-none-any.whl

Download URL	pelican_nlp-0.4.0-py3-none-any.whl
Size	83.4 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`1c8eadb01de46600b7e77ac5eb00c2f0b22a0b68a257dc2b0232d51637abae90`
BLAKE2b-256 checksum How to use checksums	`7247b5c6ecf3db8569c4f62570115c8890fa2c23e69e7e6d15140ff6e49ae560`
Upload date	Sep 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.10.16`