Descriptors FAQ

Well firstly, what is TLex?

It's a lexical analysis class and the lexer example code, which comes with the Symbian OS SDK, gives one example of how to use it. You can find a brief description of the lexer example in the Symbian Developer Library documentation which comes with each SDK; it is also available on the Symbian website (see the References section for details).

And, in case you're in some doubt as to what the lexer example is doing - I did - [reference 8] is a very useful resource about Reverse Polish Notation.

The rest of this article will summarise the main features of TLex into groups of useful API calls. I'd like to create another, simple, example of how to use the class, but am a bit stuck for ideas of what would be useful. If anyone's got any thoughts on how best to illustrate the main features, please do add a comment to this post and I'll see what I can do.

The main class, TLex, is like the descriptors in that two classes are available, TLex8 and TLex16 and the neutral version, TLex, is a typedef to TLex16 for today's UNICODE builds of Symbian OS. You can find the Symbian API reference documentation for TLex16 in [reference 9].

As the documentation states, TLex "...provides general string-parsing functions suitable for numeric format conversions and syntactical-element parsing...An instance of this class stores a string, maintaining an extraction mark to indicate the current lexical element being analysed and a pointer to the next character to be examined..."

TLex can be constructed with the data for lexical analysis or constructed empty and later assigned the data. Both construction and assignment take either another TLex object, a 16 bit non-modifiable descriptor or a TUint16* pointer to string data.

At the very simplest level, when the string contains just numerical data, the descriptor contents can be converted to an integer by using the Val() function of TLex. For example:

_LIT(KTestString1, "54321");

TLex lex(KTestString1());
TInt value = 0;
User::LeaveIfError(lex.Val(value));
ASSERT(value==54321);

The Val() function is overloaded for different signed integer types: TInt, TInt8, TInt16, TInt32, TInt64 - with or without limit checking. Likewise, there are Val() overloads for the unsigned integer types, passing in a radix type (decimal, hexadecimal, binary or octal). There are also overloads of Val() for TReal.

However, there's a lot more you can do with TLex. The Symbian Developer Library has more information about lexical analysis with TLex - see [reference 10] for more information.

For example, you can move through the string using the Inc() functions, or just inspect each character using Peek(). Calling Get() will both increment the position and return the character - it can be reversed using UnGet().

You can skip whitespace using SkipSpace() or characters using SkipCharacters(). The end of the string is hit when EoS() returns ETrue.

You can put an extraction marker at position in the string, using the Mark() overloads. Marking is also useful if you want to reverse (UnGetToMark() or skip using SkipAndMark() or SkipSpaceAndMark()).

Tokens are used to describe a character string which is deliminated by white space. There are a number of token methods available, such as TokenLength(), MarkedToken() - which extracts a token - and NextToken(). If you don't use tokens, you can use offsets and remainders instead - there are a number of methods for navigating the string in this way.