pybiber 0.3.0

Extract Biber features from a document parsed and annotated by spaCy.

The pybiber package provides tools for extracting 67 lexicogrammatical and functional features described by Biber (1988) and widely used for text-type, register, and genre classification tasks in corpus linguistics.

Key Features:

• 67 Linguistic Features: Automated extraction of tense markers, pronouns, subordination patterns, modal verbs, and more

• Multi-Dimensional Analysis: Complete implementation of Biber’s MDA methodology for register analysis

• Principal Component Analysis: Alternative dimensionality reduction approaches with visualization tools

• High Performance: Built on spaCy and Polars for efficient text processing

• End-to-End Pipeline: From raw text files to statistical analysis in just a few lines of code

• Comprehensive Visualization: Built-in plotting functions for exploratory data analysis

Applications:

• Register and genre analysis in corpus linguistics

• Text classification and machine learning preprocessing

• Diachronic language change studies

• Cross-linguistic variation research

• Academic writing analysis and pedagogical applications

• Stylometric analysis and authorship attribution

The package uses spaCy part-of-speech tagging and dependency parsing with Polars DataFrames for high-performance analytics.

Accuracy Note: Feature extraction builds from probabilistic taggers, so accuracy depends on model quality. Texts with irregular spellings or non-standard punctuation may produce unreliable outputs unless taggers are specifically tuned for those domains.

See the documentation for comprehensive guides and API reference.

See pseudobibeR for the R implementation.

Quick Start

One-line processing from a folder of text files:

import pybiber as pb

# Process all .txt files in a directory
pipeline = pb.PybiberPipeline(model="en_core_web_sm")
features = pipeline.run_from_folder("path/to/texts")

Multi-Dimensional Analysis with visualization:

# Create analyzer for statistical analysis
analyzer = pb.BiberAnalyzer(features)

# Perform MDA and generate scree plot
mda_results = analyzer.mda()
analyzer.mdaviz_screeplot()

# Plot group means by dimension
analyzer.mdaviz_groupmeans(grouping_var="register")

Installation

You can install the released version of pybiber from PyPI:

pip install pybiber

Install a spaCY model:

python -m spacy download en_core_web_sm

Usage

Data Requirements

The pybiber package works with corpora structured as DataFrames with: - doc_id column: Unique document identifiers - text column: Raw text content

This follows conventions from readtext and quanteda.

Step-by-Step Workflow

1. Import libraries and load spaCy model:

import spacy
import pybiber as pb
from pybiber.data import micusp_mini  # Sample corpus

nlp = spacy.load("en_core_web_sm")

2. Parse corpus with spaCy:

# Parse texts to extract linguistic annotations (modern approach)
processor = pb.CorpusProcessor()
tokens_df = processor.process_corpus(micusp_mini, nlp)

3. Extract Biber features:

# Aggregate 67 linguistic features per document
features_df = pb.biber(tokens_df)

4. Advanced Analysis (optional):

# Statistical analysis and visualization
analyzer = pb.BiberAnalyzer(features_df)

# Multi-Dimensional Analysis
mda_results = analyzer.mda()

# Principal Component Analysis
pca_results = analyzer.pca()

# Visualization options
analyzer.mdaviz_screeplot()           # Eigenvalue plot
analyzer.pcaviz_contrib()             # Feature contributions
analyzer.mdaviz_groupmeans(group_var="genre")  # Group comparisons

Pipeline Convenience Functions

For streamlined processing, use the high-level pipeline:

from pybiber import PybiberPipeline

pipeline = PybiberPipeline(model="en_core_web_sm", disable_ner=True)

# From folder of .txt files
features_df = pipeline.run_from_folder("/path/to/texts")

# From in-memory corpus
features_df, tokens_df = pipeline.run(corpus_df, return_tokens=True)

# One-liner convenience functions
features_df = pb.run_biber_from_folder("/path/to/texts")
features_df = pb.run_biber(corpus_df)

Feature Categories

The package extracts 67 linguistic features across 16 categories:

• Tense & Aspect: Past tense, perfect aspect, present tense

• Adverbials: Place and time adverbials

• Pronouns: 1st/2nd/3rd person, demonstrative, indefinite pronouns

• Questions: Direct wh-questions

• Nominal Forms: Nominalizations, gerunds, nouns

• Passives: Agentless and by-passives

• Stative Forms: be as main verb, existential there

• Subordination: 18 different clause types (that-clauses, wh-clauses, infinitives, relatives, etc.)

• Modification: Prepositional phrases, attributive/predicative adjectives, adverbs

• Lexical Specificity: Type-token ratio, word length

• Lexical Classes: Conjuncts, hedges, amplifiers, emphatics, discourse particles

• Modals: Possibility, necessity, and predictive modals

• Specialized Verbs: Public, private, suasive verbs

• Reduced Forms: Contractions, deletions, split constructions

• Coordination: Phrasal and clausal coordination

• Negation: Synthetic and analytic negation

See the full feature list for detailed descriptions.

Performance & Requirements

System Requirements: - Python 3.10+ - spaCy model with POS tagging and dependency parsing (e.g., en_core_web_sm)

Performance Notes: - Built on Polars for fast DataFrame operations - Supports multiprocessing for large corpora - Memory-efficient processing with configurable batch sizes - Processing time: ~20-30 seconds for small corpora (e.g., 500 documents)

License

Code licensed under the MIT License. See the LICENSE file.