tribble.io
Interface TokenI

All Known Implementing Classes:
TokenAdapter

public interface TokenI

Generic lexical token interface.

This interface is used to implement any kind of lexical token object, which is constructed by some form of lexical analyzer (lexer) such as any that implements the LexerI interface. A token object represents a single lexical entity read from an input source text stream. A lexer provides a stream of tokens read from such an input stream, which can then be used by a syntactical parser (such as any that implements the ParserI interface) for parsing sequences of source tokens into a symbol table or parse tree.

A token object has the following attributes:

text
The textual contents of the token, which is typically the characters it is composed of as it occurs in the source text stream.

type code
A code used to indicate what kind of token it is.

line number
The source line number, corresponding to the physical text line that the token was read from.

column number
The source column number, corresponding to the physical text column that the token was read from. This can be either the column number at which the token starts or ends.

filename
The source filename, corresponding to the name of the physical file or stream that the token was read from.

Since:
2001-04-22
Version:
$Revision: 1.3 $ $Date: 2003/02/23 03:09:58 $
Author:
David R. Tribble, david@tribble.com.
Copyright 2001-2003 by David R. Tribble, all rights reserved.
Permission is granted to freely use and distribute this source code provided that the original copyright and authorship notices remain intact.
See Also:
TokenAdapter, LexerI, ParserI

Field Summary
static java.lang.String REV
          Revision information.
 
Method Summary
 int getColumnNumber()
          Retrieve the source column number of this token.
 java.lang.String getFileName()
          Retrieve the source filename of this token.
 int getLineNumber()
          Retrieve the source line number of this token.
 java.lang.String getText()
          Retrieve the textual contents of this token.
 int getType()
          Retrieve the type code of this token.
 void setColumnNumber(int colNo)
          Establish the source column number of this token.
 void setFileName(java.lang.String fname)
          Establishe the source filename of this token.
 void setLineNumber(int lineNo)
          Establish the source line number number of this token.
 void setText(java.lang.String text)
          Establish the textual contents of this token.
 void setType(int type)
          Establish the token type code of this token.
 

Field Detail

REV

static final java.lang.String REV
Revision information.

See Also:
Constant Field Values
Method Detail

setText

void setText(java.lang.String text)
Establish the textual contents of this token.

Parameters:
text - The string containing the textual contents for this token.
Since:
1.1, 2001-04-22

getText

java.lang.String getText()
Retrieve the textual contents of this token.

Returns:
A string containing the textual contents for this token.
Since:
1.1, 2001-04-22

setType

void setType(int type)
Establish the token type code of this token.

Parameters:
type - The token type code for this token.
Since:
1.1, 2001-04-22

getType

int getType()
Retrieve the type code of this token.

Returns:
The type code for this token.
Since:
1.1, 2001-04-22

setLineNumber

void setLineNumber(int lineNo)
Establish the source line number number of this token.

Parameters:
lineNo - The source line number for this token.
Since:
1.3, 2003-02-22

getLineNumber

int getLineNumber()
Retrieve the source line number of this token.

Returns:
The source line number for this token.
Since:
1.3, 2003-02-22

setColumnNumber

void setColumnNumber(int colNo)
Establish the source column number of this token.

Parameters:
colNo - The source column number for this token.
Since:
1.3, 2003-02-22

getColumnNumber

int getColumnNumber()
Retrieve the source column number of this token.

Returns:
The source column number for this token.
Since:
1.3, 2003-02-22

setFileName

void setFileName(java.lang.String fname)
Establishe the source filename of this token.

Parameters:
fname - The source filename for this token.
Since:
1.1, 2001-04-22

getFileName

java.lang.String getFileName()
Retrieve the source filename of this token.

Returns:
The source filename for this token.
Since:
1.1, 2001-04-22