sentencepiece (textproc/sentencepiece) Updated: 8 months ago Add to my watchlist
Unsupervised text tokenizer for Neural Network-based text generation.SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) (Sennrich et al.) and unigram language model (Kudo)) with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing.
Version: 0.2.0 License: Apache-2
GitHub
| Maintainers | No Maintainer |
| Categories | textproc |
| Homepage | https://github.com/google/sentencepiece |
| Platforms | darwin |
| Variants |
|
"sentencepiece" depends on
lib (2)
build (2)
Ports that depend on "sentencepiece"
lib (6)Port Health:
Loading Port Health
Installations (30 days)
5
Requested Installations (30 days)
0
Livecheck results
sentencepiece seems to have been updated (port version: 0.2.0, new version: 0.2.0pre1)
livecheck ran: 12 hours ago