sentencepiece/README.en.md

# sentencepiece

#### Description
An unsupervised text tokenizer and detokenizer.

#### Software Architecture
Software architecture description

#### Installation

1.  Python module

    SentencePiece provides Python wrapper that supports both SentencePiece training and segmentation. You can install Python binary package of SentencePiece with.

    % pip install sentencepiece

2.  Build and install SentencePiece command line tools from C++ source

    The following tools and libraries are required to build SentencePiece:

    * cmake

    * C++11 compiler

    * gperftools library (optional, 10-40% performance improvement can be obtained.)

    On Ubuntu, the build tools can be installed with apt-get:

    % sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev

    Then, you can build and install command line tools as follows.

    % git clone https://github.com/google/sentencepiece.git

    % cd sentencepiece

    % mkdir build

    % cd build

    % cmake ..

    % make -j $(nproc)

    % sudo make install

    % sudo ldconfig -v

    On OSX/macOS, replace the last command with sudo update_dyld_shared_cache.

3.  Build and install using vcpkg

    You can download and install sentencepiece using the vcpkg dependency manager:

    git clone https://github.com/Microsoft/vcpkg.git

    cd vcpkg

    ./bootstrap-vcpkg.sh

    ./vcpkg integrate install

    ./vcpkg install sentencepiece

    The sentencepiece port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.


#### Instructions

1.  Train SentencePiece Model

    % spm_train --input=< input > --model_prefix=<model_name> --vocab_size=8000 --character_coverage=1.0 --model_type=<type>

    * --input: one-sentence-per-line raw corpus file. No need to run tokenizer, normalizer or preprocessor. By default, SentencePiece normalizes the input with Unicode NFKC. You can pass a comma-separated list of files.

    * --model_prefix: output model name prefix. <model_name>.model and <model_name>.vocab are generated.

    * --vocab_size: vocabulary size, e.g., 8000, 16000, or 32000

    * --character_coverage: amount of characters covered by the model, good defaults are: 0.9995 for languages with rich character set like Japanese or Chinese and 1.0 for other languages with small character set.

    * --model_type: model type. Choose from unigram (default), bpe, char, or word. The input sentence must be pretokenized when using word type.

2.  Encode raw text into sentence pieces/ids

    % spm_encode --model=<model_file> --output_format=piece < input > output

    % spm_encode --model=<model_file> --output_format=id < input > output

    Use --extra_options flag to insert the BOS/EOS markers or reverse the input sequence.

    % spm_encode --extra_options=eos (add </s> only)

    % spm_encode --extra_options=bos:eos (add <s> and </s>)

    % spm_encode --extra_options=reverse:bos:eos (reverse input and add <s> and </s>)

    SentencePiece supports nbest segmentation and segmentation sampling with --output_format=(nbest|sample)_(piece|id) flags.

    % spm_encode --model=<model_file> --output_format=sample_piece --nbest_size=-1 --alpha=0.5 < input > output

    % spm_encode --model=<model_file> --output_format=nbest_id --nbest_size=10 < input > output

3.  Decode sentence pieces/ids into raw text

    % spm_decode --model=<model_file> --input_format=piece < input > output

    % spm_decode --model=<model_file> --input_format=id < input > output

    Use --extra_options flag to decode the text in reverse order.

    % spm_decode --extra_options=reverse < input > output

4.  End-to-End Example

    % spm_train --input=data/botchan.txt --model_prefix=m --vocab_size=1000

    unigram_model_trainer.cc(494) LOG(INFO) Starts training with :

    input: "../data/botchan.txt"

    ... <snip>

    unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=1 size=1100 obj=10.4973 num_tokens=37630 num_tokens/piece=34.2091

    trainer_interface.cc(272) LOG(INFO) Saving model: m.model

    trainer_interface.cc(281) LOG(INFO) Saving vocabs: m.vocab

    % echo "I saw a girl with a telescope." | spm_encode --model=m.model

      ▁I ▁saw ▁a ▁girl ▁with ▁a ▁ te le s c o pe .

    % echo "I saw a girl with a telescope." | spm_encode --model=m.model --output_format=id

      9 459 11 939 44 11 4 142 82 8 28 21 132 6

    % echo "9 459 11 939 44 11 4 142 82 8 28 21 132 6" | spm_decode --model=m.model --input_format=id

      I saw a girl with a telescope.

    You can find that the original input sentence is restored from the vocabulary id sequence.

5.  Export vocabulary list

    % spm_export_vocab --model=<model_file> --output=<output file>

    <output file> stores a list of vocabulary and emission log probabilities. The vocabulary id corresponds to the line number in this file.

6.  Redefine special meta tokens

    By default, SentencePiece uses Unknown (<unk>), BOS (<s>) and EOS (</s>) tokens which have the ids of 0, 1, and 2 respectively. We can redefine this mapping in the training phase as follows.

    % spm_train --bos_id=0 --eos_id=1 --unk_id=5 --input=... --model_prefix=... --character_coverage=...

    When setting -1 id e.g., bos_id=-1, this special token is disabled. Note that the unknow id cannot be disabled. We can define an id for padding (<pad>) as --pad_id=3.

7.  Vocabulary restriction

    spm_encode accepts a --vocabulary and a --vocabulary_threshold option so that spm_encode will only produce symbols which also appear in the vocabulary (with at least some frequency).

    The usage is basically the same as that of subword-nmt. Assuming that L1 and L2 are the two languages (source/target languages), train the shared spm model, and get resulting vocabulary for each:

    % cat {train_file}.L1 {train_file}.L2 | shuffle > train

    % spm_train --input=train --model_prefix=spm --vocab_size=8000 --character_coverage=0.9995

    % spm_encode --model=spm.model --generate_vocabulary < {train_file}.L1 > {vocab_file}.L1

    % spm_encode --model=spm.model --generate_vocabulary < {train_file}.L2 > {vocab_file}.L2

    shuffle command is used just in case because spm_train loads the first 10M lines of corpus by default.

    Then segment train/test corpus with --vocabulary option

    % spm_encode --model=spm.model --vocabulary={vocab_file}.L1 --vocabulary_threshold=50 < {test_file}.L1 > {test_file}.seg.L1

    % spm_encode --model=spm.model --vocabulary={vocab_file}.L2 --vocabulary_threshold=50 < {test_file}.L2 > {test_file}.seg.L2

#### Contribution

1.  Fork the repository
2.  Create Feat_xxx branch
3.  Commit your code
4.  Create Pull Request


#### Gitee Feature

1.  You can use Readme\_XXX.md to support different languages, such as Readme\_en.md, Readme\_zh.md
2.  Gitee blog [blog.gitee.com](https://blog.gitee.com)
3.  Explore open source project [https://gitee.com/explore](https://gitee.com/explore)
4.  The most valuable open source project [GVP](https://gitee.com/gvp)
5.  The manual of Gitee [https://gitee.com/help](https://gitee.com/help)
6.  The most popular members  [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/)