Per-byte lookup table describing which byte values may begin a dictionary key. More...

#include <src/Utf8SkipScan.hpp>

Public Member Functions
bool	CharLevel () const
void	EnableCharLevel ()
void	DisableCharLevel ()
	Permanently falls back to lead-byte filtering, e.g.
void	MarkCharCandidate (uint32_t codePoint)
bool	IsCharCandidate (uint32_t codePoint) const
void	MarkAllCandidates ()
void	Finalize ()
	Must be called after candidate[] is filled and before the table is used.

Public Attributes
bool	candidate [256] = {}
	candidate[b] is true when byte value b may begin a dictionary key (or is the lead byte of an ideographic description operator), so the conversion loop must run a full prefix lookup at that position.
bool	asciiHasCandidates = false
	True when any ASCII byte value is a candidate; disables the bulk ASCII-run scan.
std::vector< uint64_t >	bmpCandidates
	Optional character-level refinement: one bit per BMP code point (U+0000..U+FFFF, 8 KiB).

Detailed Description

Per-byte lookup table describing which byte values may begin a dictionary key.

The conversion hot loop uses it to consume runs of characters that cannot possibly match any key without paying for a trie lookup per character, and to scan the common all-ASCII case a word at a time.

Member Function Documentation

◆ DisableCharLevel()

void opencc::internal::Utf8SkipTable::DisableCharLevel ( )

inline

Permanently falls back to lead-byte filtering, e.g.

when a key's first character is truncated or invalid and cannot be represented as a code point. Byte-level candidates accumulated so far remain valid.

◆ Finalize()

void opencc::internal::Utf8SkipTable::Finalize ( )

inline

Must be called after candidate[] is filled and before the table is used.

Ideographic description operators must always stop the scan: the conversion loop groups IDS sequences without consulting the dictionary, and skipping over an operator would lose that grouping. The operator set is defined by UTF8Util::IdeographicDescriptionOperatorArity() and bounded by UTF8Util::kFirst/kLastIdeographicDescriptionOperator.

Member Data Documentation

◆ bmpCandidates

std::vector<uint64_t> opencc::internal::Utf8SkipTable::bmpCandidates

Optional character-level refinement: one bit per BMP code point (U+0000..U+FFFF, 8 KiB).

When non-empty, 2- and 3-byte UTF-8 characters are filtered by their exact code point instead of their lead byte, so e.g. a rare CJK character sharing its lead byte with common dictionary keys can still be skipped. 4-byte characters keep lead-byte filtering. Character-level mode is on exactly when this vector is non-empty; there is deliberately no separate flag to keep in sync.

The documentation for this struct was generated from the following file:

Utf8SkipScan.hpp