depccg

Codebase for A* CCG Parsing with a Supertag and Dependency Factored Model

Changes in v3

• The English and Japanese pretrained models run on PyTorch.
• Chainer and AllenNLP are no longer runtime dependencies.
• The package supports Python 3.10 and newer.

Requirements

• Python >= 3.10
• A C++ compiler supporting C++11 standard (in case of gcc, must be >= 4.8)

Installation

Using pip:

➜ pip install depccg

Development

Project metadata and dependencies are defined in pyproject.toml and locked in uv.lock. To create a reproducible development environment and run the tests:

➜ uv sync --frozen
➜ uv run --frozen pytest

To build the source distribution and wheel:

➜ uv build

setup.py is retained only for the native Cython/C++ extension definitions; package metadata and dependencies must be changed in pyproject.toml.

Usage

Using a pretrained English parser

The following English models run on PyTorch without Chainer or AllenNLP:

The basic model is available by:

➜ depccg_en download

Downloaded models are stored under ~/.cache/depccg by default. Set DEPCCG_HOME to use a different directory.

To use:

➜ echo "this is a test sentence ." | depccg_en
ID=1, Prob=-0.0006299018859863281
(<T S[dcl] 0 2> (<T S[dcl] 0 2> (<L NP XX XX this NP>) (<T S[dcl]\NP 0 2> (<L (S[dcl]\NP)/NP XX XX is (S[dcl]\NP)/NP>) (<T NP 0 2> (<L NP[nb]/N XX XX a NP[nb]/N>) (<T N 0 2> (<L N/N XX XX test N/N>) (<L N XX XX sentence N>) ) ) ) ) (<L . XX XX . .>) )

The ELMo and Rebank models can be downloaded and selected by name:

➜ depccg_en download elmo
➜ echo "this is a test sentence ." | depccg_en --model elmo

➜ depccg_en download rebank
➜ echo "this is a test sentence ." | depccg_en --model rebank

The old elmo_rebank file is no longer available and is not included in v3. You can also pass a PyTorch model directory with --model.

Using a GPU (by --gpu option) is recommended if possible.

There are several output formats (see below).

➜ echo "this is a test sentence ." | depccg_en --format deriv
ID=1, Prob=-0.0006299018859863281
 this        is           a      test  sentence  .
  NP   (S[dcl]\NP)/NP  NP[nb]/N  N/N      N      .
                                ---------------->
                                       N
                      -------------------------->
                                  NP
      ------------------------------------------>
                      S[dcl]\NP
------------------------------------------------<
                     S[dcl]
---------------------------------------------------<rp>
                      S[dcl]

By default, the input is expected to be pre-tokenized. If you want to process untokenized sentences, you can pass --tokenize option.

The POS and NER tags in the output are filled with XX by default. You can replace them with ones predicted using SpaCy:

➜ echo "this is a test sentence ." | depccg_en --annotator spacy
ID=1, Prob=-0.0006299018859863281
(<T S[dcl] 0 2> (<T S[dcl] 0 2> (<L NP DT DT this NP>) (<T S[dcl]\NP 0 2> (<L (S[dcl]\NP)/NP VBZ VBZ is (S[dcl]\NP)/NP>) (<T NP 0 2> (<L NP[nb]/N DT DT a NP[nb]/N>) (<T N 0 2> (<L N/N NN NN test N/N>) (<L N NN NN sentence N>) ) ) ) ) (<L . . . . .>) )

The parser uses a SpaCy's en_core_web_sm model.

Orelse, you can use POS/NER taggers implemented in C&C, which may be useful in some sorts of parsing experiments:

➜ export CANDC=/path/to/candc
➜ echo "this is a test sentence ." | depccg_en --annotator candc
ID=1, log prob=-0.0006299018859863281
(<T S[dcl] 0 2> (<T S[dcl] 0 2> (<L NP DT DT this NP>) (<T S[dcl]\NP 0 2> (<L (S[dcl]\NP)/NP VBZ VBZ is (S[dcl]\NP)/NP>) (<T NP 0 2> (<L NP[nb]/N DT DT a NP[nb]/N>) (<T N 0 2> (<L N/N NN NN test N/N>) (<L N NN NN sentence N>) ) ) ) ) (<L . . . . .>) )

By default, depccg expects the POS and NER models are placed in $CANDC/models/pos and $CANDC/models/ner, but you can explicitly specify them by setting CANDC_MODEL_POS and CANDC_MODEL_NER environmental variables.

It is also possible to obtain logical formulas using ccg2lambda's semantic parsing algorithm.

➜ echo "This is a test sentence ." | depccg_en --format ccg2lambda --annotator spacy
ID=0 log probability=-0.0006299018859863281
exists x.(_this(x) & exists z1.(_sentence(z1) & _test(z1) & (x = z1)))

Using a pretrained Japanese parser

The best performing model is available by:

➜ depccg_ja download

It can be downloaded directly here (56M).

The parser provides the almost same interface as with the English one, with slight differences including the default output format, which is now one compatible with the Japanese CCGbank:

➜ echo "これはテストの文です。" | depccg_ja
ID=1, Prob=-53.98793411254883
{< S[mod=nm,form=base,fin=t] {< S[mod=nm,form=base,fin=f] {< NP[case=nc,mod=nm,fin=f] {NP[case=nc,mod=nm,fin=f] これ/これ/**} {NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f] は/は/**}} {< S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f] {< NP[case=nc,mod=nm,fin=f] {< NP[case=nc,mod=nm,fin=f] {NP[case=nc,mod=nm,fin=f] テスト/テスト/**} {NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f] の/の/**}} {NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f] 文/文/**}} {(S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f])\NP[case=nc,mod=nm,fin=f] です/です/**}}} {S[mod=nm,form=base,fin=t]\S[mod=nm,form=base,fin=f] 。/。/**}}

You can pass pre-tokenized sentences as well:

➜ echo "これ は テスト の 文 です 。" | depccg_ja --pre-tokenized
ID=1, Prob=-53.98793411254883
{< S[mod=nm,form=base,fin=t] {< S[mod=nm,form=base,fin=f] {< NP[case=nc,mod=nm,fin=f] {NP[case=nc,mod=nm,fin=f] これ/これ/**} {NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f] は/は/**}} {< S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f] {< NP[case=nc,mod=nm,fin=f] {< NP[case=nc,mod=nm,fin=f] {NP[case=nc,mod=nm,fin=f] テスト/テスト/**} {NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f] の/の/**}} {NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f] 文/文/**}} {(S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f])\NP[case=nc,mod=nm,fin=f] です/です/**}}} {S[mod=nm,form=base,fin=t]\S[mod=nm,form=base,fin=f] 。/。/**}}

Available output formats

• auto - the most standard format following AUTO format in the English CCGbank
• auto_extended - extension of auto format with combinator info and POS/NER tags
• deriv - visualized derivations in ASCII art
• xml - XML format compatible with C&C's XML format (only for English parsing)
• conll - CoNLL format
• html - visualized trees in MathML
• prolog - Prolog-like format
• jigg_xml - XML format compatible with Jigg
• ptb - Penn Treebank-style format
• ccg2lambda - logical formula converted from a derivation using ccg2lambda
• jigg_xml_ccg2lambda - jigg_xml format with ccg2lambda logical formula inserted
• json - JSON format
• ja - a format adopted in Japanese CCGbank (only for Japanese)

Programmatic Usage

Please look into depccg/__main__.py.

Evaluation in terms of predicate-argument dependencies

The standard CCG parsing evaluation can be performed with the following script:

➜ cat ccgbank/data/PARG/00/* > wsj_00.parg
➜ export CANDC=/path/to/candc
➜ python -m depccg.tools.evaluate wsj_00.parg wsj_00.predicted.auto

The script is dependent on C&C's generate program, which is only available by compiling the C&C program from the source.

(Currently, the above page is down. You can find the C&C parser here or here)

Miscellaneous

Diff tool

In error analysis, you must want to see diffs between trees in an intuitive way. depccg.tools.diff does exactly this:

➜ python -m depccg.tools.diff file1.auto file2.auto > diff.html

which outputs:

where trees in the same lines of the files are compared and the diffs are marked in color.

Citation

If you make use of this software, please cite the following:

    @inproceedings{yoshikawa:2017acl,
      author={Yoshikawa, Masashi and Noji, Hiroshi and Matsumoto, Yuji},
      title={A* CCG Parsing with a Supertag and Dependency Factored Model},
      booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
      publisher={Association for Computational Linguistics},
      year={2017},
      pages={277--287},
      location={Vancouver, Canada},
      doi={10.18653/v1/P17-1026},
      url={http://aclweb.org/anthology/P17-1026}
    }

Licence

MIT Licence

Contact

For questions and usage issues, please contact yoshikawa@tohoku.jp.

Acknowledgement

In creating the parser, I owe very much to:

• EasyCCG: from which I learned everything
• NLTK: for nice pretty printing for parse derivation