sentencepiece (textproc/sentencepiece) Updated: 11 months, 1 week ago Add to my watchlist 0

SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) (Sennrich et al.) and unigram language model (Kudo)) with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing.

SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) (Sennrich et al.) and unigram language model (Kudo)) with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing.

+ Show more

GitHub

• Details
• Builds
• Installation Stats
• Trac Tickets
• Report an Issue with this port
• API
• Stats FAQ

Statistics for selected duration

2025-Jan-24 to 2025-Feb-23

Variants table

Monthly Statistics

Can remain cached for up to 24 hours