Fleet Forth design considerations

[JimBoyd]

JimBoyd wrote:

I'm wondering if there is a better way to implement EVALUATE within the parameters of the Forth-83 standard?

I ask because the way it is implemented in Fleet Forth seems a bit like a kludge with the need for QUERY to set TIB to the correct address just in case an evaluated string causes an abort.
Does anyone know of a better way to do this?

[JimBoyd]

Ok. No suggestions for a better way to do this so I'll just keep it as it is. Maybe it's not really a kludge after all. QUERY is the only Forth word which stores a string ( using EXPECT ) in TIB , so having it set TIB to the correct address isn't so strange.

[barrym95838]

Even though it seems like you're talking to yourself in an empty room here, please be aware that I'm listening and following your trains of thought to the best of my abilities, and learning as much as I am able. My lack of suggestions is not due to lack of interest ... I'm just not qualified to provide much in the way of meaningful input at this stage of my Forth journey, and my feeling is that I'm not alone in this circumstance. Please continue.

[DerTrueForce]

You certainly aren't. I'm still fiddling around with rather toy-ish examples, when I'm looking at Forth at all...

[SamCoVT]

There are definitely some folks following along, so please continue.

I had to look up EXPECT, and it appears it was replaced by the word ACCEPT in newer Forths (with pretty much the same behavior). That knowledge makes your description of EVALUATE make a lot more sense to me.

If it makes you feel any better, your EVALUATE is very similar to what TaliForth2 ended up with, where you save everything, then use the string (in place) as the input buffer, and then put everything back how you found it. I also used EVALUATE to evalulate block buffers in-place for LOAD. Once a block was done being EVALUATEd, I made sure to restore the previous block (if the saved BLK was non-zero) to the same buffer, to handle the case where this block was LOADed from another block. With the other input variables restored, it would pick up in the former block right where it left off.

You should be able to EVALUATE from the keyboard, from a block, and from another evaluated string. All of those are also allowed to have a LOAD in them, which should also nest properly.

Getting a test suite running was fantastic for finding those odd corner cases, and also super useful for finding regressions when a minor change had unintended consequences. The ANS test suite already knows about many of these corner cases and many of the tests specifically probe for them to ensure proper behavior. Like you, I stripped the floating point stuff out of the tester software. I will warn you, when looking up the ANS tests for a partciular word, that some of the tests need a previous test to have been run to have a particular word defined, but that word might be found in a different place. You're welcome to peek in the TaliForth2 tests on github, as the test files there have everything in a reasonable order. You'll just need to substitute some of the newer words (like MARKER) with their Forth83 equivalents (like FORGET), and change all the brackets, but your Forth should be able to run many of the tests there.

[JimBoyd]

Thanks for the encouragement!

[JimBoyd]

SamCoVT wrote:

I also used EVALUATE to evalulate block buffers in-place for LOAD. Once a block was done being EVALUATEd, I made sure to restore the previous block (if the saved BLK was non-zero) to the same buffer, to handle the case where this block was LOADed from another block. With the other input variables restored, it would pick up in the former block right where it left off.

That could be a problem under the wrong set of circumstances. Blocks can hold data as well as source code.
I was working on my multitasker and noticed that the background tasks stop while a block is being loaded. The background task I was running is a loop that increments the C64 's border color and waits 30 jiffies ( about half a second ) just to make sure the multitasker is working. I had to put PAUSE in INTERPRET to remedy this. What if someone using Tali Forth or Fleet Forth wants to run a background task that logs data to a block. If there is only one block buffer and the background task ran during the loading of a block, wouldn't that invalidate the address returned by BLOCK ?
Fleet Forth's WORD calls 'STREAM which returns the address and remaining size of the current position in the text stream regardless of source. 'STREAM checks the variable BLK and if it is not zero, calls BLOCK with the number from BLK . That is how 'STREAM and therefore WORD makes certain that a valid address is used if a background task invalidates the address returned by the previous call to BLOCK .
Wasn't WORD superseded by PARSE-NAME in ANSI Forth?
Note that 'STREAM is not the same as ANSI Forth's SOURCE . If defined in ANSI Forth, it might be something like:

Code: Select all

: 'STREAM  ( ADR CNT -- )
   SOURCE  DUP >IN @ MIN 0 MAX /STRING ;

In fleet forth it's:

Code: Select all

SCR# 7E 
// 'STREAM
HEX
: 'STREAM  ( -- ADR N )
   BLK @ ?DUP
   IF
      BLOCK B/BUF
   ELSE
      TIB #TIB @
   THEN
   DUP >IN @ UMIN DUP NEGATE
   UNDER+ UNDER+ ;

Where UMIN returns the minimum of two unsigned numbers and UNDER+ has the following stack effect:

Code: Select all

UMIN  ( A B C -- A+C B )

And this:

Code: Select all

   BLK @ ?DUP
   IF
      BLOCK B/BUF
   ELSE
      TIB #TIB @
   THEN

Is the Forth-83 equivalent of SOURCE .

[JimBoyd]

JimBoyd wrote:

I had to put PAUSE in INTERPRET to remedy this.

Since PAUSE is set to a no-op when not multitasking, I did what I could to minimize the time penalty.
1) NOOP the no-op word just has a CFA that points to NEXT and no body.
2) DEFER is a CREATE ;CODE word. Here is its definition in Fleet Forth:

Code: Select all

: DEFER  ( -- )
   CREATE ['] -SET ,
   ;CODE
   2 # LDY,
   W )Y LDA,  PHA,  INY,  W )Y LDA,
   W 1+ STA,  PLA,  W STA, 0 # LDY,
   W 1- JMP,  END-CODE

[SamCoVT]

JimBoyd wrote:

SamCoVT wrote:

I also used EVALUATE to evalulate block buffers in-place for LOAD. Once a block was done being EVALUATEd, I made sure to restore the previous block (if the saved BLK was non-zero) to the same buffer, to handle the case where this block was LOADed from another block. With the other input variables restored, it would pick up in the former block right where it left off.

That could be a problem under the wrong set of circumstances. Blocks can hold data as well as source code.
I was working on my multitasker and noticed that the background tasks stop while a block is being loaded. The background task I was running is a loop that increments the C64 's border color and waits 30 jiffies ( about half a second ) just to make sure the multitasker is working. I had to put PAUSE in INTERPRET to remedy this. What if someone using Tali Forth or Fleet Forth wants to run a background task that logs data to a block. If there is only one block buffer and the background task ran during the loading of a block, wouldn't that invalidate the address returned by BLOCK ?

ANS-2012 (the only one I'm fairly familiar with) avoids this problem entirely by not directly discussing tasking but rather by listing all the things that could invalidate a buffer, and the list is fairly extensive. It includes the obvious things, like using BLOCK and BUFFER, but also things I would not have thought of, like inputting characters with KEY or outputting characters with EMIT (because I/O might be redirected to/from a block) or even using AT-XY.

If your tasking system is preemptive, then the ANS-2012 standard essentially says that only 1 task should be using blocks/buffers at a time, or even doing things that could implicitly use a buffer like I/O. If you are doing cooperative multitasking, where you choose when task switching happens with a word like PAUSE (Tali2 seems to be leaning in this direction, but it's not implemented yet), then you can have control over the block/buffer but have to assume that it might be invalidated when it's not your turn (eg. use BLOCK to make sure it's in a buffer again before using it the next time it's your turn). The ANS standard seems to bend over backwards to not specifically talk about tasking and buffers/blocks - I suspect that's because there are many existing implementations of tasking with varying levels of compatibility.

If you know what you are doing won't invalidate a buffer on your system, then you can totally do it even if ANS recommends against it. It's just that your code won't be as portable because that operation may not be safe on a different Forth.

One thing I liked about ANS-2012 (relative to FIG, which is my only other experience) is that they standardized the buffer size to be 1024 bytes so a block fits exactly into one buffer and a buffer holds exactly one block.

[whartung]

SamCoVT wrote:

ANS-2012 (the only one I'm fairly familiar with) avoids this problem entirely by not directly discussing tasking but rather by listing all the things that could invalidate a buffer, and the list is fairly extensive. It includes the obvious things, like using BLOCK and BUFFER, but also things I would not have thought of, like inputting characters with KEY or outputting characters with EMIT (because I/O might be redirected to/from a block) or even using AT-XY.

The list of invalidating words is pretty much specifically designed around the idea of cooperative tasking using PAUSE or something similar. The problem isn't necessarily buffer invalidation, but also words that may themselves call PAUSE underneath, so that you as a developer are aware of when you might lose control, particularly inadvertently. This is why the I/O words are on the list.

[SamCoVT]

whartung wrote:

SamCoVT wrote:

ANS-2012 (the only one I'm fairly familiar with) avoids this problem entirely by not directly discussing tasking but rather by listing all the things that could invalidate a buffer, and the list is fairly extensive. It includes the obvious things, like using BLOCK and BUFFER, but also things I would not have thought of, like inputting characters with KEY or outputting characters with EMIT (because I/O might be redirected to/from a block) or even using AT-XY.

The list of invalidating words is pretty much specifically designed around the idea of cooperative tasking using PAUSE or something similar. The problem isn't necessarily buffer invalidation, but also words that may themselves call PAUSE underneath, so that you as a developer are aware of when you might lose control, particularly inadvertently. This is why the I/O words are on the list.

That makes the list of events that can invalidate the buffer make a whole lot more sense now. Thanks for the clarification. That means you also need to keep track of your own words that have PAUSE in them and add those to the list as well.

Parsing is on the list of things that can invalidate buffers, so that seems to imply that you can put PAUSE into INTERPRET.

[JimBoyd]

Yes, the idea is to use the address returned by BLOCK before the next task switch. This is why, when loading a block, each time WORD is called it causes a call to BLOCK through 'STREAM . This is also why the buffer table is arranged from most recently used to least recently used and BLOCK starts out as a code definition.

Code: Select all

HEX
CODE BLOCK  ( BLK -- ADR )
   DEY,
   // CFA OF BUFFER POINTS HERE
   ' T/F >BODY    STY,  // SET T/F
   ' T/F >BODY 1+ STY,  //  
   ' MRU >BODY    LDA,  N    STA,
   ' MRU >BODY 1+ LDA,  N 1+ STA,
   6 # LDY,  N )Y LDA,  0 ,X CMP,
   0= IF,
      INY,  N )Y LDA,  1 ,X CMP,
      0= IF,
         INY,  N )Y LDA,  0 ,X STA,
         INY,  N )Y LDA,  1 ,X STA,
         NEXT JMP,
      THEN,
   THEN,
   >FORTH

Fleet Forth is an ITC Forth. BUFFER has no body, its CFA points one byte into the body of BLOCK ( after the DEY instruction.) T/F is a VALUE which determines if a read is performed later in the high level part of BLOCK/BUFFER . BLOCK sets it to TRUE and BUFFER sets it to FALSE . If the requested block is the most recently used then BLOCK just has to replace the block number on the stack with the buffer address in the table and jump to NEXT . If the requested block is not the most recently used, BLOCK transitions to high level to handle it. This is to make BLOCK as fast as possible while still sharing code with BUFFER ( by setting T/F ) to mitigate the speed penalty of having to call BLOCK each time the same block needs accessed.

[Edit: One correction. Originally I typed BUFFER when I meant BLOCK.]

[JimBoyd]

SamCoVT wrote:

whartung wrote:

SamCoVT wrote:

ANS-2012 (the only one I'm fairly familiar with) avoids this problem entirely by not directly discussing tasking but rather by listing all the things that could invalidate a buffer, and the list is fairly extensive. It includes the obvious things, like using BLOCK and BUFFER, but also things I would not have thought of, like inputting characters with KEY or outputting characters with EMIT (because I/O might be redirected to/from a block) or even using AT-XY.

The list of invalidating words is pretty much specifically designed around the idea of cooperative tasking using PAUSE or something similar. The problem isn't necessarily buffer invalidation, but also words that may themselves call PAUSE underneath, so that you as a developer are aware of when you might lose control, particularly inadvertently. This is why the I/O words are on the list.

That makes the list of events that can invalidate the buffer make a whole lot more sense now. Thanks for the clarification. That means you also need to keep track of your own words that have PAUSE in them and add those to the list as well.

Parsing is on the list of things that can invalidate buffers, so that seems to imply that you can put PAUSE into INTERPRET.

I must admit that when I included multitasking support, I limited it to background tasks because the Commodore 64 isn't exactly a multi user system. I had not considered the possibility of a background task accessing blocks. The Commodore REU, ram expansion unit, can be accessed with block numbers $5000 and up and it might be useful for a background task to store temporary data there. I originally had PAUSE at the beginning of the TYPE words. It is now at the end so TYPE can, for example, type a line from a BLOCK with a screen of source before the task switch and potential invalidation of the buffer address.

[whartung]

JimBoyd wrote:

SamCoVT wrote:

I had not considered the possibility of a background task accessing blocks.

A classic use case is something like a back ground print job printing out screens of source code.

[JimBoyd]

SamCoVT wrote:

One thing I liked about ANS-2012 (relative to FIG, which is my only other experience) is that they standardized the buffer size to be 1024 bytes so a block fits exactly into one buffer and a buffer holds exactly one block.

That was standardized far earlier.
Forth-83 standardized block size and buffer size to be 1024 bytes. Forth-79 states that a block must contain 1024 bytes. It also states that a block buffer is a "memory area where a mass storage block is maintained."