CLemmatizer Class Reference

#include <Lemmatizers.h>

Inheritance diagram for CLemmatizer:

Collaboration diagram for CLemmatizer:

List of all members.

Public Member Functions

• CLemmatizer (MorphLanguageEnum Language)
• virtual ~CLemmatizer ()
• MorphLanguageEnum GetLanguage () const
• const CStatistic & GetStatistic () const
• bool CheckABC (const string &WordForm) const
• bool IsHyphenPostfix (const string &Postfix) const
• bool IsHyphenPrefix (const string &Prefix) const
• bool initIconv (const string &enc_internal="", const string &enc_external="UTF8")
• std::string recode_ext2int (const std::string s_ext) const
• std::string recode_int2ext (const std::string s_int) const
• bool LoadDictionariesRegistry (string &strError)
• bool LoadStatisticRegistry (SubjectEnum subj)
• bool CreateParadigmCollection (bool bNorm, string &WordStr, bool capital, bool bUsePrediction, vector< CFormInfo > &Result) const
• void GetAllAncodesQuick (const BYTE *WordForm, bool capital, BYTE *OutBuffer, bool bUsePrediction) const
• bool GetAllAncodesAndLemmasQuick (string &InputWordStr, bool capital, char *OutBuffer, size_t MaxBufferSize, bool bUsePrediction) const
• bool CreateParadigmFromID (DWORD id, CFormInfo &Result) const
• bool ProcessHyphenWords (CGraphmatFile *piGraphmatFile) const

Public Attributes

• bool m_bLoaded
• bool m_bMaximalPrediction
• bool m_bUseStatistic
• bool m_bAllowRussianJo
• std::string m_enc_int

  internal encoding (default=""=none)
  

• std::string m_enc_ext

  external encoding (default=""=none)
  

• ddcIconv * m_ic_ext2int

  iconv converter from user encoding to morph-internal encoding
  

• ddcIconv * m_ic_int2ext

  iconv converter from morph-internal encoding to user encoding
  

Protected Member Functions

• virtual void FilterSrc (string &src) const =0
• string GetRegistryString () const
• string GetPath () const
• void ReadOptions (string FileName)
• bool LemmatizeWord (string &InputWordStr, const bool cap, const bool predict, vector< CAutomAnnotationInner > &results, bool bGetLemmaInfos) const
• void AssignWeightIfNeed (vector< CAutomAnnotationInner > &FindResults) const
• bool CheckAbbreviation (string InputWordStr, vector< CAutomAnnotationInner > &FindResults, bool is_cap) const
• CAutomAnnotationInner ConvertPredictTupleToAnnot (const CPredictTuple &input) const
• void PredictByDataBase (string InputWordStr, vector< CAutomAnnotationInner > &results, bool is_cap) const
• bool IsPrefix (const string &Prefix) const

Protected Attributes

• string m_Registry
• set< string > m_HyphenPostfixes
• set< string > m_HyphenPrefixes
• CStatistic m_Statistic
• CPredictBase m_Predict
• set< string > m_PrefixesSet

---

Constructor & Destructor Documentation

CLemmatizer::CLemmatizer

(

MorphLanguageEnum

Language

)

References CMorphDict::InitAutomat(), initIconv(), m_bAllowRussianJo, m_bLoaded, m_bMaximalPrediction, m_bUseStatistic, m_enc_ext, m_enc_int, and MorphAnnotChar.

Here is the call graph for this function:

CLemmatizer::~CLemmatizer

(

)

[virtual]

References m_ic_ext2int, and m_ic_int2ext.

---

Member Function Documentation

virtual void CLemmatizer::FilterSrc

(

string &

src

)

const [protected, pure virtual]

Implemented in CLemmatizerRussian, CLemmatizerEnglish, and CLemmatizerGerman.

Referenced by CreateParadigmCollection(), GetAllAncodesAndLemmasQuick(), and GetAllAncodesQuick().

Here is the caller graph for this function:

string CLemmatizer::GetRegistryString

(

)

const [inline, protected]

References m_Registry.

Referenced by GetPath().

Here is the caller graph for this function:

string CLemmatizer::GetPath

(

)

const [protected]

References GetRegistryString().

Referenced by LoadDictionariesRegistry(), and LoadStatisticRegistry().

Here is the call graph for this function:

Here is the caller graph for this function:

void CLemmatizer::ReadOptions

(

string

FileName

)

[protected]

References LoadFileToString(), m_bAllowRussianJo, and Trim().

Referenced by LoadDictionariesRegistry().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::LemmatizeWord	(	string &	InputWordStr,
		const bool	cap,
		const bool	predict,
		vector< CAutomAnnotationInner > &	results,
		bool	bGetLemmaInfos
	)			const [protected]

References CMorphAutomat::GetInnerMorphInfos(), GetLanguage(), CMorphDict::GetLemmaInfos(), IsPrefix(), CFlexiaModel::m_Flexia, CMorphDict::m_FlexiaModels, CMorphForm::m_FlexiaStr, CAutomAnnotationInner::m_ItemNo, CAutomAnnotationInner::m_LemmaInfoNo, CMorphDict::m_LemmaInfos, CAutomAnnotationInner::m_ModelNo, CMorphDict::m_NPSs, CMorphDict::m_pFormAutomat, PredictByDataBase(), CMorphDict::PredictBySuffix(), RmlMakeUpper(), and UnknownPartOfSpeech.

Referenced by CreateParadigmCollection(), GetAllAncodesAndLemmasQuick(), GetAllAncodesQuick(), and ProcessHyphenWords().

Here is the call graph for this function:

Here is the caller graph for this function:

void CLemmatizer::AssignWeightIfNeed

(

vector< CAutomAnnotationInner > &

FindResults

)

const [protected]

References CStatistic::get_HomoWeight(), CAutomAnnotationInner::GetParadigmId(), m_bUseStatistic, CAutomAnnotationInner::m_ItemNo, CAutomAnnotationInner::m_nWeight, and m_Statistic.

Referenced by CreateParadigmCollection().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::CheckAbbreviation	(	string	InputWordStr,
		vector< CAutomAnnotationInner > &	FindResults,
		bool	is_cap
	)			const [protected]

References ConvertPredictTupleToAnnot(), CPredictBase::Find(), CABCEncoder::GetCriticalNounLetterPack(), GetLanguage(), is_upper_consonant(), CMorphDict::m_pFormAutomat, and m_Predict.

Referenced by PredictByDataBase().

Here is the call graph for this function:

Here is the caller graph for this function:

CAutomAnnotationInner CLemmatizer::ConvertPredictTupleToAnnot

(

const CPredictTuple &

input

)

const [protected]

References CPredictTuple::m_ItemNo, CAutomAnnotationInner::m_ItemNo, CPredictTuple::m_LemmaInfoNo, CAutomAnnotationInner::m_LemmaInfoNo, CMorphDict::m_LemmaInfos, CAutomAnnotationInner::m_ModelNo, CAutomAnnotationInner::m_nWeight, and CAutomAnnotationInner::m_PrefixNo.

Referenced by CheckAbbreviation(), and PredictByDataBase().

Here is the caller graph for this function:

void CLemmatizer::PredictByDataBase	(	string	InputWordStr,
		vector< CAutomAnnotationInner > &	results,
		bool	is_cap
	)			const [protected]

References CheckAbbreviation(), CheckABC(), ConvertPredictTupleToAnnot(), CPredictBase::Find(), CABCEncoder::GetCriticalNounLetterPack(), GetLanguage(), m_bMaximalPrediction, CMorphDict::m_LemmaInfos, CPredictBase::m_ModelFreq, CMorphDict::m_pFormAutomat, m_Predict, and morphGerman.

Referenced by LemmatizeWord().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::IsPrefix

(

const string &

Prefix

)

const [protected]

References m_PrefixesSet.

Referenced by LemmatizeWord().

Here is the caller graph for this function:

MorphLanguageEnum CLemmatizer::GetLanguage

(

)

const [inline]

References CABCEncoder::m_Language, and CMorphDict::m_pFormAutomat.

Referenced by CheckAbbreviation(), CFormInfo::GetAccentedVowel(), LemmatizeWord(), PredictByDataBase(), ProcessHyphenWords(), and CPlmLineCollection::ProcessPlmLines().

Here is the caller graph for this function:

const CStatistic & CLemmatizer::GetStatistic

(

)

const

References m_Statistic.

Referenced by CFormInfo::GetHomonymWeightWithForm(), CFormInfo::GetWordWeight(), and CFormInfo::SetParadigmId().

Here is the caller graph for this function:

bool CLemmatizer::CheckABC

(

const string &

WordForm

)

const

References CABCEncoder::CheckABCWithoutAnnotator(), and CMorphDict::m_pFormAutomat.

Referenced by PredictByDataBase().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::IsHyphenPostfix

(

const string &

Postfix

)

const

References m_HyphenPostfixes.

Referenced by CreateParadigmCollection().

Here is the caller graph for this function:

bool CLemmatizer::IsHyphenPrefix

(

const string &

Prefix

)

const

References m_HyphenPrefixes.

Referenced by CreateParadigmCollection().

Here is the caller graph for this function:

bool CLemmatizer::initIconv	(	const string &	enc_internal = "",
		const string &	enc_external = "UTF8"
	)

References ddcIconv::good(), m_enc_ext, m_enc_int, m_ic_ext2int, and m_ic_int2ext.

Referenced by CLemmatizer(), and CLemmatizerGerman::CLemmatizerGerman().

Here is the call graph for this function:

Here is the caller graph for this function:

std::string CLemmatizer::recode_ext2int

(

const std::string

s_ext

)

const [inline]

References ddcIconv::convert(), and m_ic_ext2int.

Referenced by GetWordForms().

Here is the call graph for this function:

Here is the caller graph for this function:

std::string CLemmatizer::recode_int2ext

(

const std::string

s_int

)

const [inline]

References ddcIconv::convert(), and m_ic_int2ext.

Referenced by GetWordForms().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::LoadDictionariesRegistry

(

string &

strError

)

References GetPath(), GetRmlVariable(), CPredictBase::Load(), CStatistic::Load(), CMorphDict::Load(), m_bLoaded, m_bUseStatistic, CMorphDict::m_FlexiaModels, CMorphDict::m_LemmaInfos, CPredictBase::m_ModelFreq, m_Predict, CMorphDict::m_Prefixes, m_PrefixesSet, m_Statistic, CExpc::m_strCause, MORPH_MAIN_FILES, OPTIONS_FILE, PREDICT_BIN_PATH, and ReadOptions().

Referenced by CMorphologyHolder::LoadGraphanAndLemmatizer().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::LoadStatisticRegistry

(

SubjectEnum

subj

)

References GetPath(), CStatistic::Load(), m_Statistic, subjComputer, subjFinance, and subjLiterature.

Here is the call graph for this function:

bool CLemmatizer::CreateParadigmCollection	(	bool	bNorm,
		string &	WordStr,
		bool	capital,
		bool	bUsePrediction,
		vector< CFormInfo > &	Result
	)			const

References AssignWeightIfNeed(), CFormInfo::Create(), CreateDecartProduction(), FilterSrc(), IsFound(), IsHyphenPostfix(), IsHyphenPrefix(), LemmatizeWord(), and CAutomAnnotationInner::m_ItemNo.

Referenced by GetParadigmCollection(), CMorphologyHolder::GetParadigmIdsByNormAndAncode(), GetWordForms(), CPlmLineCollection::ProcessPlmLines(), and CMorphologyHolder::string_to_ids().

Here is the call graph for this function:

Here is the caller graph for this function:

void CLemmatizer::GetAllAncodesQuick	(	const BYTE *	WordForm,
		bool	capital,
		BYTE *	OutBuffer,
		bool	bUsePrediction
	)			const

References FilterSrc(), LemmatizeWord(), CLemmaInfo::m_CommonAncode, CFlexiaModel::m_Flexia, CMorphDict::m_FlexiaModels, CAutomAnnotationInner::m_ItemNo, CAutomAnnotationInner::m_LemmaInfoNo, CMorphDict::m_LemmaInfos, and CAutomAnnotationInner::m_ModelNo.

Referenced by GetGramInfosFromWord().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::GetAllAncodesAndLemmasQuick	(	string &	InputWordStr,
		bool	capital,
		char *	OutBuffer,
		size_t	MaxBufferSize,
		bool	bUsePrediction
	)			const

References FilterSrc(), LemmatizeWord(), CFlexiaModel::m_Flexia, CMorphDict::m_FlexiaModels, CMorphForm::m_FlexiaStr, CAutomAnnotationInner::m_ItemNo, CAutomAnnotationInner::m_ModelNo, and CMorphForm::m_PrefixStr.

Here is the call graph for this function:

bool CLemmatizer::CreateParadigmFromID	(	DWORD	id,
		CFormInfo &	Result
	)			const

References CFormInfo::AttachLemmatizer(), and CFormInfo::SetParadigmId().

Referenced by CMorphologyHolder::id_to_paradigm(), and CMorphologyHolder::id_to_string().

Here is the call graph for this function:

Here is the caller graph for this function:

bool CLemmatizer::ProcessHyphenWords

(

CGraphmatFile *

piGraphmatFile

)

const

References GetLanguage(), CUnitHolder::GetToken(), CGraphmatFile::GetTokenLanguage(), CUnitHolder::GetTokensCount(), CUnitHolder::GetUnits(), CUnitHolder::HasDescr(), LemmatizeWord(), CUnitHolder::MakeOneWord(), OHyp, OLw, OSentEnd, CUnitHolder::PSoft(), and CGraphmatFile::StartsFixedOborot().

Referenced by CMorphologyHolder::GetMorphology().

Here is the call graph for this function:

Here is the caller graph for this function:

---

Member Data Documentation

string CLemmatizer::m_Registry [protected]

Referenced by CLemmatizerEnglish::CLemmatizerEnglish(), CLemmatizerGerman::CLemmatizerGerman(), CLemmatizerRussian::CLemmatizerRussian(), and GetRegistryString().

set<string> CLemmatizer::m_HyphenPostfixes [protected]

Referenced by CLemmatizerRussian::CLemmatizerRussian(), and IsHyphenPostfix().

set<string> CLemmatizer::m_HyphenPrefixes [protected]

Referenced by CLemmatizerRussian::CLemmatizerRussian(), and IsHyphenPrefix().

CStatistic CLemmatizer::m_Statistic [protected]

Referenced by AssignWeightIfNeed(), GetStatistic(), LoadDictionariesRegistry(), and LoadStatisticRegistry().

CPredictBase CLemmatizer::m_Predict [protected]

Referenced by CheckAbbreviation(), LoadDictionariesRegistry(), and PredictByDataBase().

set<string> CLemmatizer::m_PrefixesSet [protected]

Referenced by IsPrefix(), and LoadDictionariesRegistry().

bool CLemmatizer::m_bLoaded

Referenced by CLemmatizer(), and LoadDictionariesRegistry().

bool CLemmatizer::m_bMaximalPrediction

Referenced by CLemmatizer(), and PredictByDataBase().

bool CLemmatizer::m_bUseStatistic

Referenced by AssignWeightIfNeed(), CLemmatizer(), and LoadDictionariesRegistry().

bool CLemmatizer::m_bAllowRussianJo

Referenced by CLemmatizer(), CLemmatizerRussian::FilterSrc(), and ReadOptions().

std::string CLemmatizer::m_enc_int

internal encoding (default=""=none)

Referenced by CLemmatizer(), and initIconv().

std::string CLemmatizer::m_enc_ext

external encoding (default=""=none)

Referenced by CLemmatizer(), and initIconv().

ddcIconv* CLemmatizer::m_ic_ext2int

iconv converter from user encoding to morph-internal encoding

Referenced by initIconv(), recode_ext2int(), and ~CLemmatizer().

ddcIconv* CLemmatizer::m_ic_int2ext

iconv converter from morph-internal encoding to user encoding

Referenced by initIconv(), recode_int2ext(), and ~CLemmatizer().

---

The documentation for this class was generated from the following files:

• Lemmatizers.h
• Lemmatizers.cpp