oofem::Tokenizer Class Reference

#include <tokenizer.h>

Public Member Functions
	Tokenizer ()
	Constructor. Creates tokenizer with given character as separator.
void	tokenizeLine (const std ::string &line)
	Tokenizes given record (string).
int	giveNumberOfTokens ()
	returns the number of tokens.
const char *	giveToken (int i)
	Returns pointer to i-th token.

Protected Member Functions
std::string	readSimpleToken (std ::size_t &pos, const std ::string &line)
std::string	readToken (std ::size_t &pos, const std ::string &line, char sep)
std::string	readStructToken (std ::size_t &pos, const std ::string &line)
std::string	readStringToken (std ::size_t &pos, const std ::string &line)
std::string	readSimpleExpressionToken (std ::size_t &pos, const std ::string &line)

Private Attributes
std ::vector< std ::string >	tokens
	Array of tokens.

Detailed Description

String bracket- and quotation-aware string tokenizer. This class splits given record (represented as string) to particular tokens, which are separated by white spaces. Tokenizer recognizes "quoted strings" and structured tokens that are bounded by '{}, $$' pairs, can be nested and represent single token.

Definition at line 50 of file tokenizer.h.

Constructor & Destructor Documentation

Tokenizer()

oofem::Tokenizer::Tokenizer

(

)

Constructor. Creates tokenizer with given character as separator.

Definition at line 43 of file tokenizer.C.

References tokens.

Member Function Documentation

giveNumberOfTokens()

int oofem::Tokenizer::giveNumberOfTokens

(

)

returns the number of tokens.

Definition at line 140 of file tokenizer.C.

References tokens.

giveToken()

const char * oofem::Tokenizer::giveToken

(

int

i

)

Returns pointer to i-th token.

Definition at line 146 of file tokenizer.C.

References tokens.

readSimpleExpressionToken()

std::string oofem::Tokenizer::readSimpleExpressionToken ( std ::size_t & pos, const std ::string & line )

protected

Reads next simple expression token (section identified by starting with '$' and finishing with '$').

Parameters

pos	Position (index) in token buffer.
line	Record from which token is parsed.

Definition at line 75 of file tokenizer.C.

References OOFEM_WARNING, and readToken().

Referenced by tokenizeLine().

readSimpleToken()

std::string oofem::Tokenizer::readSimpleToken ( std ::size_t & pos, const std ::string & line )

protected

Reads next simple token (stops when whitespace character is reached)

Parameters

pos	Starting position.
line	Record from which token is parsed.

Definition at line 89 of file tokenizer.C.

Referenced by tokenizeLine().

readStringToken()

std::string oofem::Tokenizer::readStringToken ( std ::size_t & pos, const std ::string & line )

protected

Reads next string token (quoted).

Parameters

pos	Position (index) in token buffer.
line	Record from which token is parsed.

Definition at line 49 of file tokenizer.C.

References OOFEM_WARNING, and readToken().

Referenced by tokenizeLine().

readStructToken()

std::string oofem::Tokenizer::readStructToken ( std ::size_t & pos, const std ::string & line )

protected

Reads next structured token (bounded by '{' '}' pairs, possibly nested).

Parameters

pos	Starting position (should point to a '{').
line	Record from which token is parsed.

Definition at line 63 of file tokenizer.C.

References OOFEM_WARNING, and readToken().

Referenced by tokenizeLine().

readToken()

std::string oofem::Tokenizer::readToken ( std ::size_t & pos, const std ::string & line, char sep )

protected

Reads next token (stops when separator is reached)

Parameters

pos	Starting position.
line	Record from which token is parsed.
sep	Separator.

Definition at line 100 of file tokenizer.C.

Referenced by readSimpleExpressionToken(), readStringToken(), and readStructToken().

tokenizeLine()

void oofem::Tokenizer::tokenizeLine

(

const std ::string &

line

)

Tokenizes given record (string).

Definition at line 110 of file tokenizer.C.

References readSimpleExpressionToken(), readSimpleToken(), readStringToken(), readStructToken(), and tokens.

Member Data Documentation

tokens

std :: vector< std :: string > oofem::Tokenizer::tokens

private

Array of tokens.

Definition at line 54 of file tokenizer.h.

Referenced by giveNumberOfTokens(), giveToken(), tokenizeLine(), and Tokenizer().

---

The documentation for this class was generated from the following files:

• tokenizer.h
• tokenizer.C