All Packages  Class Hierarchy  This Package  Previous  Next  Index

Class sdsu.util.SimpleTokenizer

java.lang.Object
   |
   +----sdsu.util.SimpleTokenizer

public class SimpleTokenizer
extends Object
This class performs some simple parsing of strings or streams. The input is a sequence of ascii characters. The sequence is divided into tokens, whitespace, and comments. Comments start with the comment character and continue to the next newline (\n ) character. Comments are removed from the input characters and not returned as part of a token. A token is string from the current location to the next separator or whitespace character. Characters defined as whitespace (tab, newline, and space default values ) help delineate tokens but are not part of tokens. That is whitespace characters are removed after finding a token. If a token must contain whitespace character, a possible separator, or comment character, the token must be placed between two quote characters. A quoted token can contain a quote character.

Author:
Roger Whitney (whitney@cs.sdsu.edu)
See Also:
Stringizer, TokenCharacters

Constructor Index

 o SimpleTokenizer(InputStream)
Create a SimpleTokenizer on tokenSource with default settings
 o SimpleTokenizer(InputStream, TokenCharacters)
Create a SimpleTokenizer on tokenSource
 o SimpleTokenizer(String)
Create a SimpleTokenizer on string with default settings
 o SimpleTokenizer(String, TokenCharacters)
Create a SimpleTokenizer on string

Method Index

 o hasMoreElements()
Returns true if not at end of source stream or source string
 o hasMoreTokens()
Returns true if not at end of source stream or source string
 o nextToken()
Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found.
 o nextToken(String)
Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found.
 o separator()
Returns the separator found by the last call to nextToken
 o setEatEscapeChar(boolean)
Enable/Disable the eating of the escape character.

Constructors

 o SimpleTokenizer
 public SimpleTokenizer(String parsable)
Create a SimpleTokenizer on string with default settings

 o SimpleTokenizer
 public SimpleTokenizer(String parsable,
                        TokenCharacters charTable)
Create a SimpleTokenizer on string

Parameters:
commentChar - character used to indicate start of a comment
quoteChar - character used to quote a string containing special characters
whitespace - characters used for whitespace. Use null or empty string for no whitespace characters
 o SimpleTokenizer
 public SimpleTokenizer(InputStream tokenSource)
Create a SimpleTokenizer on tokenSource with default settings

 o SimpleTokenizer
 public SimpleTokenizer(InputStream tokenSource,
                        TokenCharacters charTable)
Create a SimpleTokenizer on tokenSource

Parameters:
commentChar - character used to indicate start of a comment
beginQuoteChar - character used to start a quote of a string containing special characters
endQuoteChar - character used to end a quote of a string containing special characters
whitespace - characters used for whitespace. Use null or empty string for no whitespace characters

Methods

 o hasMoreTokens
 public boolean hasMoreTokens()
Returns true if not at end of source stream or source string

 o hasMoreElements
 public boolean hasMoreElements()
Returns true if not at end of source stream or source string

 o separator
 public char separator()
Returns the separator found by the last call to nextToken

 o setEatEscapeChar
 public void setEatEscapeChar(boolean value)
Enable/Disable the eating of the escape character.

 o nextToken
 public String nextToken(String newSeparators) throws IOException
Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found. The separator is removed from the stream, but not returned as part of token.

Parameters:
separator - can be any character except the current comment or quote character
Throws: IOException
If separator or EOF does not follow this token
 o nextToken
 public String nextToken() throws IOException
Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found. The separator is removed from the stream, but not returned as part of token.

Parameters:
separator - set of characters to be used as separator after token. Can be any non-null or nonempty string of characters except the current comment or quote character
Throws: IOException
If separator or EOF does not follow this token

All Packages  Class Hierarchy  This Package  Previous  Next  Index