ficl parse steps
Description: Ficl - embedded scripting with object oriented programming
Keywords: scripting prototyping tcl OOP Forth interpreter C

| ficl parse steps
| Index ANS API Debugger Download Licensing Links Locals OOP In Ficl Parse Steps Release History Upgrading To 4.0 |  |
| Parse Steps
Unlike every other FORTH we know of, Ficl features an extensible parser chain. The Ficl parser is not a monolithic function; instead, it is comprised of a simple tokenizer and a series of parse steps. A parse step is a step in the parser chain that handles a particular kind of token, acting on the token as appropriate. Example parse steps, in terms of traditional FORTH lore, would be the "number runner" and the "colon compiler". The Ficl parser works like this: Read in a new token (string of text with no internal whitespace). For each parse step in the chain, call the parse step, passing in the token. If the parse step returns `FICL_TRUE`, that parse step must have handled the token appropriately; move on to the next token. If the parser tries all the parse steps and none of them return `FICL_TRUE`, the token is illegal—print an error and reset the virtual machine. Parse steps can be written as native functions, or as Ficl script functions. New parse steps can be appended to the chain at any time.
| The Default Ficl Parse Chain
These is the default Ficl parser chain, shown in order. `?word` If compiling and local variable support is enabled, attempt to find the token in the local variable dictionary. If found, execute the token's compilation semantics and return `FICL_TRUE`. Attempt to find the token in the system dictionary. If found, execute the token's semantics (may be different when compiling than when interpreting) and return `FICL_TRUE`. `?prefix` This parse step is only active if prefix support is enabled, setting `FICL_WANT_PREFIX` in `ficl.h` to a non-zero value. Attempt to match the beginning of the token to the list of known prefixes. If there's a match, execute the associated prefix method and return `FICL_TRUE`. `?number` Attempt to convert the token to a number in the present `BASE`. If successful, push the value onto the stack if interpreting, otherwise compile it, then return `FICL_TRUE`. `?float` This parse step is only active if floating-point number support is enabled, setting `FICL_WANT_FLOAT` in `ficl.h` to a non-zero value. Attempt to convert the token to a floating-point number. If successful, push the value onto the floating-point stack if interpreting, otherwise compile it, then return `FICL_TRUE`.
| Adding A Parse Step From Within Ficl
> You can add a parse step in two ways. The first is to write a Ficl word that has the correct stack signature for a parse step:
>     MY-PARSE-STEP   ( c-addr u -- x*i flag )
> where `c-addr u` are the address and length of the incoming token, and `flag` is `FICL_TRUE` if the parse step processed the token and `FICL_FALSE` otherwise. Install the parse step using `add-parse-step`. A trivial example:
>     : ?silly   ( c-addr u -- flag )
>        ." Oh no! Not another  " type cr  true ;
>     ' ?silly add-parse-step
>     parse-order
| Adding A Native Parse Step
> The other way to add a parse step is to write it in C and add it into the parse chain with the following function:
>     void ficlSystemAddPrimitiveParseStep(ficlSystem *system, char *name, ficlParseStep step);
> `name` is the display name of the parse step in the parse chain (as displayed by the Ficl word `PARSE-ORDER`). `step` is a pointer to the code for the parse step itself, and must match the following declaration:
>     typedef int (*ficlParseStep)(ficlVm *vm, ficlString s);
When a native parse step is run, `si` points to the incoming token. The parse step must return `FICL_TRUE` if it succeeds in handling the token, and `FICL_FALSE` otherwise. See `ficlVmParseNumber()` in `system.c` for an example.
| Prefixes
> What's a prefix, anyway? A prefix (contributed by Larry Hastings) is a token that's recognized as the beginning of another token. Its presence modifies the semantics of the rest of the token. An example is `0x`, which causes digits following it to be converted to hex regardless of the current value of `BASE`. Caveat: Prefixes are matched in sequence, so the more of them there are, the slower the interpreter gets. On the other hand, because the prefix parse step occurs immediately after the dictionary lookup step, if you have a prefix for a particular purpose, using it may save time since it stops the parse process. Also, the Ficl interpreter is wonderfully fast, and most interpretation only happens once, so it's likely you won't notice any change in interpreter speed even if you make heavy use of prefixes. Each prefix is a Ficl word stored in a special wordlist called `<PREFIXES>`. When the prefix parse step (`?prefix`, implemented in C as `ficlVmParsePrefix()`) is executed, it searches each word in `<PREFIXES>` in turn, comparing it with the initial characters of the incoming token. If a prefix matches, the parse step returns the remainder of the token to the input stream and executes the code associated with the prefix. This code can be anything you like, but it would typically do something with the remainder of the token. If the prefix code does not consume the rest of the token, it will go through the parse process again (which may be what you want). Prefixes are defined in `prefix.c` and in `softcore/prefix.fr`. The best way to add prefixes is by defining them in your own code, bracketed with the special words `START-PREFIXES` and `END-PREFIXES`. For example, the following code would make `.(` a prefix.
>     start-prefixes
>     : .(  .( ;
>     end-prefixes
The compile-time constant `FICL_EXTENDED_PREFIX` controls the inclusion of several additional prefixes. This is turned off in the default build, since several of these prefixes alter standard behavior, but you might like them.
| Notes
Prefixes and parser extensions are non-standard. However, with the exception of prefix support, Ficl's default parse order follows the standard. Inserting parse steps in some other order will almost certainly break standard behavior. The number of parse steps that can be added to the system is limited by the value of `FICL_MAX_PARSE_STEPS` (defined in `sysdep.h`). The default maximum number is 8. The compile-time constant `FICL_EXTENDED_PREFIX` controls the inclusion of several additional prefixes. This is turned off in the default build, since several of these prefixes alter standard behavior, but you might like them.
| Parser Glossary
`PARSE-ORDER ( -- )` Prints the list of parse steps, in the order in which they are called. `ADD-PARSE-STEP ( xt -- )` Appends a parse step to the parse chain. `xt` is the address (execution token) of a Ficl word to use as the parse step. The word must be a legal Ficl parse step (see above). `SHOW-PREFIXES ( -- )` Prints the list of all prefixes. Each prefix is a Ficl word that is executed if its name is found at the beginning of a token. `START-PREFIXES ( -- )` Declares the beginning of a series of prefix definitions. Should be followed, eventually, by `END-PREFIXES`. (All `START-PREFIXES` does is tell the Ficl virtual machine to compile into the `<PREFIXES>` wordlist.) `END-PREFIXES ( -- )` Declares the end of a series of prefix definitions. Should only be used after calling `START-PREFIXES`. (All `END-PREFIXES` does is tell the Ficl virtual machine to switch back to the wordlist that was in use before `START-PREFIXES` was called.)