tribble.io
Interface TokenI

All Known Implementing Classes:

TokenAdapter

public interface TokenI

Generic lexical token interface.

This interface is used to implement any kind of lexical token object, which is constructed by some form of lexical analyzer (lexer) such as any that implements the LexerI interface. A token object represents a single lexical entity read from an input source text stream. A lexer provides a stream of tokens read from such an input stream, which can then be used by a syntactical parser (such as any that implements the ParserI interface) for parsing sequences of source tokens into a symbol table or parse tree.

A token object has the following attributes:

text

The textual contents of the token, which is typically the characters it is composed of as it occurs in the source text stream.

type code

A code used to indicate what kind of token it is.

line number

The source line number, corresponding to the physical text line that the token was read from.

column number

The source column number, corresponding to the physical text column that the token was read from. This can be either the column number at which the token starts or ends.

filename

The source filename, corresponding to the name of the physical file or stream that the token was read from.

Since:

2001-04-22

Version:

$Revision: 1.3 $ $Date: 2003/02/23 03:09:58 $

Author:

David R. Tribble, david@tribble.com.
Copyright ©2001-2003 by David R. Tribble, all rights reserved.
Permission is granted to freely use and distribute this source code provided that the original copyright and authorship notices remain intact.

See Also:

TokenAdapter, LexerI, ParserI

Field Summary

static java.lang.String

REV Revision information.

Method Summary

int	getColumnNumber() Retrieve the source column number of this token.
java.lang.String	getFileName() Retrieve the source filename of this token.
int	getLineNumber() Retrieve the source line number of this token.
java.lang.String	getText() Retrieve the textual contents of this token.
int	getType() Retrieve the type code of this token.
void	setColumnNumber(int colNo) Establish the source column number of this token.
void	setFileName(java.lang.String fname) Establishe the source filename of this token.
void	setLineNumber(int lineNo) Establish the source line number number of this token.
void	setText(java.lang.String text) Establish the textual contents of this token.
void	setType(int type) Establish the token type code of this token.

Field Detail

REV

static final java.lang.String REV

Revision information.

See Also:

Constant Field Values

Method Detail

setText

void setText(java.lang.String text)

Establish the textual contents of this token.

Parameters:

text - The string containing the textual contents for this token.

Since:

1.1, 2001-04-22

getText

java.lang.String getText()

Retrieve the textual contents of this token.

Returns:

A string containing the textual contents for this token.

Since:

1.1, 2001-04-22

setType

void setType(int type)

Establish the token type code of this token.

Parameters:

type - The token type code for this token.

Since:

1.1, 2001-04-22

getType

int getType()

Retrieve the type code of this token.

Returns:

The type code for this token.

Since:

1.1, 2001-04-22

setLineNumber

void setLineNumber(int lineNo)

Establish the source line number number of this token.

Parameters:

lineNo - The source line number for this token.

Since:

1.3, 2003-02-22

getLineNumber

int getLineNumber()

Retrieve the source line number of this token.

Returns:

The source line number for this token.

Since:

1.3, 2003-02-22

setColumnNumber

void setColumnNumber(int colNo)

Establish the source column number of this token.

Parameters:

colNo - The source column number for this token.

Since:

1.3, 2003-02-22

getColumnNumber

int getColumnNumber()

Retrieve the source column number of this token.

Returns:

The source column number for this token.

Since:

1.3, 2003-02-22

setFileName

void setFileName(java.lang.String fname)

Establishe the source filename of this token.

Parameters:

fname - The source filename for this token.

Since:

1.1, 2001-04-22

getFileName

java.lang.String getFileName()

Retrieve the source filename of this token.

Returns:

The source filename for this token.

Since:

1.1, 2001-04-22