Abstract class for token input.

Inheritance diagram for moot::TokenReader:

[legend]

Collaboration diagram for moot::TokenReader:

[legend]

Public Member Functions
Constructors etc.
	TokenReader (int fmt=tiofUnknown, const std::string &name="TokenReader")

virtual	~TokenReader (void)

void	tr_clear (void)

Input Selection
virtual void	from_mstream (mootio::mistream *mistreamp)

virtual void	from_mstream (mootio::mistream &mis)

virtual void	from_filename (const char *filename)

virtual void	from_file (FILE *file)

virtual void	from_fd (int fd)

virtual void	from_buffer (const void *buf, size_t len)

virtual void	from_string (const char *s)

virtual void	from_cxxstream (std::istream &is)

virtual void	close (void)

virtual bool	opened (void)

Token-Level Access
mootToken *	token (void)

mootSentence *	sentence (void)

virtual mootTokenType	get_token (void)

virtual mootTokenType	get_sentence (void)

Diagnostics
virtual void	reader_name (const std::string &myname)

virtual size_t	line_number (void)

virtual size_t	line_number (size_t n)

virtual size_t	column_number (void)

virtual size_t	column_number (size_t n)

virtual mootio::ByteOffset	byte_number (void)

virtual mootio::ByteOffset	byte_number (size_t n)

virtual void	carp (const char *fmt,...)

Public Attributes
int	tr_format

std::string	tr_name

mootio::mistream *	tr_istream

bool	tr_istream_created

mootToken *	tr_token

mootSentence *	tr_sentence

void *	tr_data

Static Public Attributes
static const size_t	TR_DEFAULT_BUFSIZE = 256

Additional Inherited Members
Static Public Member Functions inherited from moot::TokenIO
static int	parse_format_string (const std::string &fmtString)

static int	guess_filename_format (const char *filename)

static bool	is_empty_format (int fmt)

static int	sanitize_format (int fmt, int fmt_implied=tiofNone, int fmt_default=tiofNone)

static int	parse_format_request (const char request, const char filename=__null, int fmt_implied=tiofNone, int fmt_default=tiofNone)

static std::string	format_canonical_string (int fmt)

static class TokenReader *	new_reader (int fmt)

static class TokenWriter *	new_writer (int fmt)

static class TokenReader *	file_reader (const char filename, const char fmt_request=__null, int fmt_implied=tiofNone, int fmt_default=tiofNone)

static class TokenWriter *	file_writer (const char filename, const char fmt_request=__null, int fmt_implied=tiofNone, int fmt_default=tiofNone)

static size_t	pipe_tokens (class TokenReader reader, class TokenWriter writer)

static size_t	pipe_sentences (class TokenReader reader, class TokenWriter writer)

Constructor & Destructor Documentation

◆ TokenReader()

moot::TokenReader::TokenReader	(	int	fmt = `tiofUnknown`,
		const std::string &	name = `"TokenReader"`
	)

inline

Default constructor

Parameters

fmt	bitmask of moot::TokenIOFormat flags
name	name of current input source

◆ ~TokenReader()

virtual moot::TokenReader::~TokenReader ( void )

inlinevirtual

Default destructor : override in descendant classes

References close().

Member Function Documentation

◆ tr_clear()

void moot::TokenReader::tr_clear ( void )

inline

Clear TokenReader-relevant construction buffers, if they exist.

References moot::mootToken::clear().

◆ from_mstream() [1/2]

virtual void moot::TokenReader::from_mstream ( mootio::mistream * mistreamp )

inlinevirtual

Select input from a mootio::mistream pointer. This is the basic case. Descendendant classes may want to override this method.

Reimplemented in moot::TokenBuffer, moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

Referenced by moot::TokenReaderExpat::from_mstream().

◆ from_mstream() [2/2]

virtual void moot::TokenReader::from_mstream ( mootio::mistream & mis )

inlinevirtual

Select input from a mootio::mistream object, reference version. Default implementation just calls from_mstream(&mis).

Reimplemented in moot::TokenReaderExpat.

◆ from_filename()

virtual void moot::TokenReader::from_filename ( const char * filename )

inlinevirtual

Select input from a named file. Descendants using named file input may override this method. The filename "-" may be used to specify stdin. Default implementation calls from_mstream().

Reimplemented in moot::TokenReaderExpat.

References mootio::mstream::valid().

Referenced by moot::TokenReaderExpat::from_filename().

◆ from_file()

virtual void moot::TokenReader::from_file ( FILE * file )

inlinevirtual

Select input from a C stream. Caller is responsible for opening and closing the stream. Descendants using C stream input may override this method. Default implementation calls from_mstream().

Reimplemented in moot::TokenReaderExpat.

Referenced by moot::TokenReaderExpat::from_file().

◆ from_fd()

virtual void moot::TokenReader::from_fd ( int fd )

inlinevirtual

Select input from a file descriptor. Caller is responsible for opening and closing the stream. Descendants using file descriptor input may override this method. No default implementation.

Reimplemented in moot::TokenReaderExpat.

Referenced by moot::TokenReaderExpat::from_fd().

◆ from_buffer()

virtual void moot::TokenReader::from_buffer	(	const void *	buf,
		size_t	len
	)

inlinevirtual

Select input from a C memory-buffer. Caller is responsible for allocation and de-allocation. Descendants using C memory-buffer input may override this method. Default implementation calls from_mstream().

Reimplemented in moot::TokenReaderExpat.

Referenced by moot::TokenReaderExpat::from_buffer().

◆ from_string()

virtual void moot::TokenReader::from_string ( const char * s )

inlinevirtual

Select input from a NUL-terminated C string. Caller is responsible for allocation and de-allocation. Descendants using C string input may override this method. Default implementation calls from_cbuffer(s,len).

◆ from_cxxstream()

virtual void moot::TokenReader::from_cxxstream ( std::istream & is )

inlinevirtual

Select input from a C++ stream. Caller is responsible for allocation and de-allocation. Descendants using C++ stream input may override this method. Default implementation calls from_mstream().

Reimplemented in moot::TokenReaderExpat.

Referenced by moot::TokenReaderExpat::from_cxxstream().

◆ close()

virtual void moot::TokenReader::close ( void )

inlinevirtual

Finish input from currently selected source & perform any required cleanup operations. This method should always be called before selecting a new input source. The current input stream is only closed if it was created locally.

Descendants may override this method.

Reimplemented in moot::TokenBuffer, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

References mootio::mstream::close().

Referenced by ~TokenReader().

◆ opened()

virtual bool moot::TokenReader::opened ( void )

inlinevirtual

Test whether this reader is currently opened. Default just checks tr_istream && tr_istream->valid()

Reimplemented in moot::TokenBuffer.

References mootio::mstream::valid().

◆ token()

mootToken* moot::TokenReader::token ( void )

inline

Get pointer to the current input token. Returns NULL if no token is available.

Warning: The contents of the token returned may be overwritten on the next call to any other TokenReader method.

◆ sentence()

mootSentence* moot::TokenReader::sentence ( void )

inline

Get a pointer to the current input sentence. Returns NULL if no sentence is available.

Warning: The contents of the sentence returned may be overwritten on the next call to any other TokenReader method.

◆ get_token()

virtual mootTokenType moot::TokenReader::get_token ( void )

inlinevirtual

Get the next token from the buffer. On completion, current token (if any) is in *tr_token. Descendants must override this method.

Reimplemented in moot::TokenBuffer, moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

◆ get_sentence()

virtual mootTokenType moot::TokenReader::get_sentence ( void )

virtual

Read in next sentence. On completion, current sentence (if any) is in *tr_sentence. Descendants may override this method for sentence-wise input.

Reimplemented in moot::TokenBuffer, moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

◆ reader_name()

virtual void moot::TokenReader::reader_name ( const std::string & myname )

inlinevirtual

Set reader subtype name to use for diagnostics. Descendants may override this method.

Reimplemented in moot::TokenReaderExpat.

Referenced by moot::TokenReaderExpat::reader_name().

◆ line_number() [1/2]

virtual size_t moot::TokenReader::line_number ( void )

inlinevirtual

Get current line number. Descendants may override this method.

Reimplemented in moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

Referenced by moot::wasteLexerReader::line_number().

◆ line_number() [2/2]

virtual size_t moot::TokenReader::line_number ( size_t n )

inlinevirtual

Set current line number. Descendants may override this method.

Reimplemented in moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

◆ column_number() [1/2]

virtual size_t moot::TokenReader::column_number ( void )

inlinevirtual

Get current column number. Descendants may override this method.

Reimplemented in moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

Referenced by moot::wasteLexerReader::column_number().

◆ column_number() [2/2]

virtual size_t moot::TokenReader::column_number ( size_t n )

inlinevirtual

Set current column number. Descendants may override this method.

Reimplemented in moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

◆ byte_number() [1/2]

virtual mootio::ByteOffset moot::TokenReader::byte_number ( void )

inlinevirtual

Get current byte number. Descendants may override this method.

Reimplemented in moot::TokenReaderNative, moot::wasteLexerReader, moot::TokenReaderExpat, and moot::wasteTokenScanner.

Referenced by moot::wasteLexerReader::byte_number().

◆ byte_number() [2/2]

virtual mootio::ByteOffset moot::TokenReader::byte_number ( size_t n )

inlinevirtual

Get current byte number. Descendants may override this method.

Reimplemented in moot::wasteLexerReader, and moot::wasteTokenScanner.

◆ carp()

virtual void moot::TokenReader::carp	(	const char *	fmt,
			...
	)

virtual

Complain, giving verbose information

Member Data Documentation

◆ TR_DEFAULT_BUFSIZE

const size_t moot::TokenReader::TR_DEFAULT_BUFSIZE = 256

static

Default size of input buffer.

◆ tr_format

int moot::TokenReader::tr_format

Format flags: bitmask of moot::TokenIOFormat flags

◆ tr_name

std::string moot::TokenReader::tr_name

Name of TokenReader subtype.

◆ tr_istream

mootio::mistream* moot::TokenReader::tr_istream

Currently selected input stream (may be NULL)

◆ tr_istream_created

bool moot::TokenReader::tr_istream_created

Whether we created tr_istream locally

◆ tr_token

mootToken* moot::TokenReader::tr_token

Pointer to a real internal 'current token' buffer, used as return value for token() method.

Descendant implementations are responsible for allocation, population, manipulation, and destruction of the data it points to.

◆ tr_sentence

mootSentence* moot::TokenReader::tr_sentence

Pointer to a real internal 'current sentence' buffer, used as return value for sentence() method.

Descendant implementations are responsible for allocation, population, maniuplation, and destruction of the data it points to.

◆ tr_data

void* moot::TokenReader::tr_data

User data associated with this reader object. Guaranteed to be unused by internal library routines. Useful e.g. for perl wrappers etc

The documentation for this class was generated from the following file:

mootTokenIO.h

Public Member Functions

Public Attributes

Static Public Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ TokenReader()

◆ ~TokenReader()

Member Function Documentation

◆ tr_clear()

◆ from_mstream() [1/2]

◆ from_mstream() [2/2]

◆ from_filename()

◆ from_file()

◆ from_fd()

◆ from_buffer()

◆ from_string()

◆ from_cxxstream()

◆ close()

◆ opened()

◆ token()

◆ sentence()

◆ get_token()

◆ get_sentence()

◆ reader_name()

◆ line_number() [1/2]

◆ line_number() [2/2]

◆ column_number() [1/2]

◆ column_number() [2/2]

◆ byte_number() [1/2]

◆ byte_number() [2/2]

◆ carp()

Member Data Documentation

◆ TR_DEFAULT_BUFSIZE

◆ tr_format

◆ tr_name

◆ tr_istream

◆ tr_istream_created

◆ tr_token

◆ tr_sentence

◆ tr_data