word-segment-grpc-client 0.1.0

Word segment grpc service

Development

Make sure you have installed uv and protoc and pre-commit

curl -LsSf https://astral.sh/uv/install.sh | sh

brew install protobuf

pip install pre-commit

Setup

We use BERT based model to do word segment job, the F1 score is better than Jieba. When we start to develop on this project, we can directly run below command.

make prepare_dev

The above command will set up virtual env for Python, and install the requirements.

Download model

We can download the model from huggingface, or Kaggle.

https://huggingface.co/datasets/mikehug24/knowledge_graph_datasets/resolve/main/nlp_models.zst

gRPC golang client

In order to get the gRPC client, we need to make the client package public, hence, we use git submodule to manage it. Most of the time, we don't need to update the client.

TODO

• Rename to NLPPythonService