Skip to main content

Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation (ACL 2020 Findings) Stacked Ensemble Framework and DeepCut as Baseline model

Project description

OSKut (Out-of-domain StacKed cut for Word Segmentation)

Open In Collab

Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation (ACL 2021 Findings)
Stacked Ensemble Framework and DeepCut as Baseline model

Read more:


    title = "Handling Cross- and Out-of-Domain Samples in {T}hai Word Segmentation",
    author = "Limkonchotiwat, Peerat  and
      Phatthiyaphaibun, Wannaphong  and
      Sarwar, Raheem  and
      Chuangsuwanich, Ekapol  and
      Nutanong, Sarana",
    booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "",
    doi = "10.18653/v1/2021.findings-acl.86",
    pages = "1003--1016",


pip install OSKut

How To use


  • python >= 3.6
  • tensorflow >= 2.0


Load Engine & Engine Mode

  • ws, tnhc, and BEST !!
    • ws: The model trained on Wisesight-1000 and test on Wisesight-160
    • ws-augment-60p: The model trained on Wisesight-1000 augmented with top-60% entropy
    • tnhc: The model trained on TNHC (80:20 train&test split with random seed 42)
    • BEST: The model trained on BEST-2010 Corpus (NECTEC)
    • SCADS: The model trained on VISTEC-TP-TH-2021 Corpus (VISTEC)
    # OR
    # OR
    # OR
    # OR
    # OR
  • tl-deepcut-XXXX
    • We also provide transfer learning of deepcut on 'Wisesight' as tl-deepcut-ws, 'TNHC' as tl-deepcut-tnhc, and 'LST20' as tl-deepcut-lst20
    # OR
  • deepcut
    • We also provide the original deepcut

Segment Example

You need to read the paper to understand why we have $k$ value!

  • Tokenize with default k-value
    ['เบียร์', 'ยู', 'ไม่', 'อร่อย', 'สัด', 'ๆ', 'ๆ', 'ๆ', 'ๆ', 'ๆฟ', 'ๆ']
    ['เบียร์', 'ยู', 'ไม่', 'อร่อย', 'สัด', 'ๆ', 'ๆ', 'ๆ', 'ๆ', 'ๆฟ', 'ๆ']
  • Tokenize with a various k-value
    print(oskut.OSKut('เบียร์ยูไม่อร่อยสัดๆๆๆๆๆฟๆ',k=5)) # refine only 5% of character number
    print(oskut.OSKut('เบียร์ยูไม่อร่อยสัดๆๆๆๆๆฟๆ',k=100)) # refine 100% of character number
    ['เบียร์', 'ยู', 'ไม่', 'อร่อย', 'สัด', 'ๆ', 'ๆ', 'ๆ', 'ๆ', 'ๆฟๆ']
    ['เบียร์', 'ยู', 'ไม่', 'อร่อย', 'สัด', 'ๆ', 'ๆ', 'ๆ', 'ๆ', 'ๆฟ', 'ๆ']

New datasets!!

VISTEC-TP-TH-2021 (VISTEC), which consists of 49,997 text samples from Twitter (2017-2019).
VISTEC corpus contains 49,997 sentences with 3.39M words where the collection was manually annotated by linguists on four tasks, namely word segmentation, misspelling detection and correction, and named entity recognition.
For more information and download click here



Without Data Augmentation

With Data Augmentation

Thank you many code from

  • Deepcut (Baseline Model) : We used some of code from Deepcut to perform transfer learning

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

OSKut-1.3.tar.gz (17.6 kB view hashes)

Uploaded Source

Built Distribution

OSKut-1.3-py3-none-any.whl (44.2 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page