Storing binary data in screens.

[IamRob]

Finally got some word definitions created that helps edit screens and frankly now prefer editing screens over text files.

At first I thought screens would be just good for creating word definitions, but now am beginning to realize that it is quite easy to store data from everything from lists of words to binary data to spreadsheet data to text documents with Control characters, in each screen.

This gives much more flexibility than just plain text as one cannot store Control characters or memory pointers in a text file. Storing a zero as the first byte of the screen also prevents the screen from being accidentally loaded, but does not affect the screen being displayed, and the LIST command is modified to display Control characters in inverse video.

And if needed, a screen or series of screens, can easily be saved to a text file or binary file.

Forth is truly a blast and way more productive than Pascal or Basic.

[GARTHWILSON]

It's an interesting idea, and one I had never thought of. It should make certain operations easier, with storage in flash memories for example. One thing I had thought of, but never got around to doing anything with yet, is making screens to be bigger, rather than 64 characters by 16 lines. 64 is rather limiting for source code with lots of comments. That is of course why there were shadow screens, back when displays couldn't do 128 characters across; but how inconvenient! So I thought why not have a screen be 2K or 4K, being 128 characters by 16 or 32 lines. Storage is plenty cheap now for that. 128 columns is hardly beyond my threshold of where I really start to feel comfortable.

[IamRob]

I thought of that too. Using a full screen of 80 cols by 24 lines. But 80 x 24 is only 1920 characters and the next step up for an even number of blocks is 2048 bytes. There would be some memory wastage then, so decided to stick with the 1024 byte screen.

But one would think it would be quite easy to implement a larger screen size just by changing a few constants.

In my Forth just 3 constants would need to be changed

C/L (characters per line)
B/BLK (bytes per block)
B/SCRN (blocks per screen)

B/BUF (bytes per buffer) should get calculated as: B/BLK x B/SCRN

I don't have a "# of lines" constant. But since I am currently updating my source anyways, I might just add this constant.

Or could just make #LINES a calculated variable.

#LINES = B/BLK x B/SCRN / C/L

[GARTHWILSON]

IamRob wrote:

I thought of that too. Using a full screen of 80 cols by 24 lines.

I have the Trident 8900 video chip set on a video card on my DOS machine where I do my software development for the workbench, so I have 132 columns, and I usually choose 43 lines, although if I don't mind them being squished together with less space between them, I can go for 50 or 60 lines. In the picture below, I kept things to 80 columns for someone else; but you can see how much more space the 132 affords. When I write stuff that will be compiled by the workbench computer where I only allotted 128 bytes for TIB on my '02 Forth, I just don't use the last few places on the line. (In my '816 Forth, I made TIB $88, or 136d.) MultiEdit, BTW, lets me have dozens of files open at once, tiled and windowed any number of ways.

[IamRob]

At one time I had a Videx Ultraterm which offered 128x32 or 160x24, but no-one has implemented support for those screen sizes in any emulators yet. But yeah, I am envious of all that screen real-estate you have access to.

[pbj]

I haven't used screens since I first used Forth with my 6511AQ chips teamed up to a 3" floppy for development back in the 80's. However, I recently decided I needed config files in my Forth but I wanted an easy way to edit them on the target board via an ANSI terminal or via VGA. So I decided to write a simple screen editor that I called SED which just handled one sector at a time which in SD cards is 512 bytes. That worked well and I thought I can also edit raw sectors as well as memory and Flash etc. But then I thought I could also edit source code too if I made it larger than a sector, although 1k seemed a bit small, I settled with 2k display of 64 wide by 32 lines with an option for 4k buffers to handle the 4k page erase sizes of SPI Flash. When I edit memory or sectors it displays the hex bytes instead of characters.

This works really well over serial terminals or via VGA display on my embedded systems and I can toggle screen sizes for wider displays although there is no real need. Some mentioned having room for comments on the right but in recent years I have made a shift to having comments mostly on their own rows just above the code, since Forth source can have a line full of swaps and drops and other code, and it is handy to track this sometimes with positional comments.

So here are a few lines from a config file that I have added a few lines of source code to with comments just to illustrate that this is so much better for Forth vs the messy comment columns that only seem to line up for assembly code. The .< at the end of each indicates a CR is added, just to make it more text file like and easy to process as normal input. I use --- as a comment alias simply because it clearly separates code and comments and doesn't look like other non-comment symbols.

BTW - when it comes to "w i d e screens", I can set my target board's VGA mode to 1920x1080 and a 5x7 font can fit 320 characters on each line! But that is too wide.

Code: Select all

FILE: CONFIG                    2048 x64                        10:45:37
        0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
00000:  OWNER:  PETER JAKACKI                                          .<
00040:  BOARD:  P2D2r4                                                 .<
00080:  SPLASH: MINION                                                 .<
000C0:  SOUND:  TAQOZUP                                                .<
00100:  SYSTEM: TAQOZ DEMO                                             .<
00140:  NOTES:  Testing SED sector text editor for config              .<
00180:                                                                 .<
001C0:                                                                 .<
00200:  --- display the clock in the top right corner                  .<
00240:  pri CLOCK       63 1 XY QTIME@ .AS" ##:##:## " ;               .<
00280:  --- display the ruler as hex digits/char or bytes/hex          .<
002C0:  pri RULER       CRLF 8 SPACES tw FOR I >N .H NEXT              .<
00300:          tw 16 = IF 16 FOR SPACE I .BYTE NEXT THEN ;            .<
00340:  --- display header - reset term - plain - clear                .<
00380:  pri HEADER      OFF CURSOR 15 EMIT PLAIN CLS HDRS              .<
003C0:          tw 8 + SPACES CR                                       .<
00400:  --- FILE: SED.BLK ...  2048 x64 ... 10:45:37                   .<
00440:      PRINT" FILE: " file$ PRINT$ 3 SPACES                       .<
00480:      SECTOR? IF SPACE @FILE .L THEN SIZE CLOCK ;                .<
004C0:                                                                 .<
00500:                                                                 .<

[SamCoVT]

I had experimented with storing arrays in BLOCKS and it worked reasonably well. I ended up with the following code (stored in blocks 1-2 with the data starting at block 3) to make it go. Pardon the excessive (what is the stack doing) comments on the first word as I was still learning Forth when I wrote this.

Code: Select all

( Block Binary Data Words  1/2                 SCC 2018-12 )
( Make a defining word to create block arrays. )
: BLOCK-ARRAY: ( base_block# "name" -- ) ( index -- addr )
  CREATE ,     ( save the base block# )
  DOES> @ SWAP ( base_block# index )
    CELLS      ( Turn index into byte index )
    1024 /MOD  ( base_block# offset_into_block block# )
    ROT +      ( offset_into_block real_block# )
    BLOCK      ( offset_into_block buffer_address )
    + ;
( Create the array starting at block 3           )
( With 4 blocks, max index is 2047 - not checked )
3 BLOCK-ARRAY: MYARRAY
( Some helper words for accessing elements )
: MYARRAY@ ( index -- n ) MYARRAY @ ;
: MYARRAY! ( n index -- ) MYARRAY ! UPDATE ;  ok

( Block Binary Data Words cont. 2/2            SCC 2018-12 )
( Note: For both words below, end-index is one past the )
( last index you want to use.                           )

( A helper word to initialize values in block arrays to 0 )
: ARRAY-ZERO ( end_index start_index -- )
    ?DO 0 I MYARRAY! LOOP ;

( A helper word to view a block array )
: ARRAY-VIEW ( end_index start_index -- )
    ( Print 10 values per line with 6 digit columns. )
    ?DO I 10 MOD 0= IF CR THEN I MYARRAY @ 6 .R LOOP ;

The only note I would add about doing this is to make sure you FLUSH or SAVE-BUFFERS before powering down your computer, but that's true any time when using blocks.

[JimBoyd]

IamRob wrote:

one cannot store Control characters or memory pointers in a text file.

That would depend on the system you are using. The Commodore 64 has what is called quote mode. Control characters can be embedded in a string as this screen shot shows. Note: the word " (quote) parses the text stream until a quote character is found as a delimiter. When used while interpreting, the text is stored as a counted string at PAD .

As can be seen, the control characters to change the text color were captured in the string. If I can save it in a string, I can save it in a file.

Quote:

Storing a zero as the first byte of the screen also prevents the screen from being accidentally loaded, but does not affect the screen being displayed, and the LIST command is modified to display Control characters in inverse video.

Oh really?
Here are some excerpts from the log of my latest session at the simulated C64. In the log, everything I entered as well as the computer's responses are shown:

Code: Select all

 OK
HEX  OK
1 BUFFER U. CC00  OK
2 BUFFER U. C800  OK
3 BUFFER U. C400  OK
4 BUFFER U. C000  OK

I use BUFFER to assign buffers for blocks 1 thru 4 without reading anything from disk. I remove them later with EMPTY-BUFFERS

Code: Select all

1 LIST 
SCR# 1 
0: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
3: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
4: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
5: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
6: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
7: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
8: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
9: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
A: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
B: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
C: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
D: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
E: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
F: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 OK
WIPE  OK
L 
SCR# 1 
0: 
1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
9: 
A: 
B: 
C: 
D: 
E: 
F: 
 OK

When I list block 1, it is full of the null character (byte 0) so I wipe it and list it again.

Code: Select all

0 1 BLOCK C!  OK
L 
SCR# 1 
0: @
1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
9: 
A: 
B: 
C: 
D: 
E: 
F: 
 OK
1 LOAD 
SCR# 1 LINE# 0 
@
^
WHAT?

I store a zero at the first byte of block 1 and list. The zero is clearly seen in the listing. When I try loading the block, Fleet Forth aborts with an error. Blazin' Forth for the C64 and Gforth for the PC have similar behaviors.
Fleet Forth's LIST shows all characters in a screen. The only character that shows as whitespace is the blank character ( $20).
Here is a short word to fill block 1 with the bytes 0 to $FF inclusive:

Code: Select all

: PATTERN 
   $100 0 
   DO 
      I DUP 1 BLOCK + C! 
   LOOP ; 

And here is the result of using this word then listing block 1:

This is the source for Fleet Forth's LIST

Code: Select all

: LIST  ( SCR -- )
   RB EDITOR  R# OFF DUP SCR !
   CR ." SCR# " U. HEX  10 0
   DO
      CR I 1 .R ." : "
      I LINE -TRAILING QTYPE
      DONE? ?LEAVE
   LOOP
   CR ;

QTYPE shows the control characters as reverse video. Actually, it shows the control characters as the reverse of the normal characters. If the string is printed as reverse video ( as in the location shown for an error) the control characters will show as normal video while the normal characters show as reverse video.

[JimBoyd]

GARTHWILSON wrote:

It's an interesting idea, and one I had never thought of. It should make certain operations easier, with storage in flash memories for example. One thing I had thought of, but never got around to doing anything with yet, is making screens to be bigger, rather than 64 characters by 16 lines. 64 is rather limiting for source code with lots of comments. That is of course why there were shadow screens, back when displays couldn't do 128 characters across; but how inconvenient! So I thought why not have a screen be 2K or 4K, being 128 characters by 16 or 32 lines. Storage is plenty cheap now for that. 128 columns is hardly beyond my threshold of where I really start to feel comfortable.

I prefer using 1K blocks when using them for virtual memory on the Commodore 64. With 4 buffers I have 4 different 1K windows into different locations in virtual memory at the same time.
Here is something for consideration that I may not use myself because 1K screens seem a good fit for the C64's limited physical screen real-estate.
As for screens of 1K being annoying, whartung mentioned something in another thread about screens not being the same size as blocks on some Forth systems. What about making screens a fixed multiple of 1K blocks? They could be 128 characters long. The first 8 lines could be on the first block, the next 8 lines on the next block and so on. The editor could handle treating the 2, 4 or even 5 blocks as a single screen.
In the place of LOAD and THRU , or even in addition to them, there could be SLOAD ( screen load) and STHRU ( screen thru) to load a screen or a range of screens respectively. With these two words and the editor words handling the details, the user would not even need to know the underlying block size was still 1K.

[JimBoyd]

SamCoVT wrote:

The only note I would add about doing this is to make sure you FLUSH or SAVE-BUFFERS before powering down your computer, but that's true any time when using blocks.

When I've used the Commodore 64, whether real or a simulation, I've gotten into the habit of using BYE before shutting it down.

Code: Select all

: BYE  ( -- )
   SAVE-BUFFERS  ACLOSE
   [ 0FFFC , ] ; -2 ALLOT

[SamCoVT]

JimBoyd wrote:

IamRob wrote:

Storing a zero as the first byte of the screen also prevents the screen from being accidentally loaded, but does not affect the screen being displayed, and the LIST command is modified to display Control characters in inverse video.

Oh really?

I believe that's FIG-FORTH type behavior with the null character being the magical word "X" that stops interpreting a block or line of text. That was handy for FIG where the buffers were often smaller than a block, so the end of a buffer did not mean the end of a block, and refill would just get the next buffer and continue interpreting. FIG also would put a null at the end of the input buffer as it read in lines, so it worked well there too.

The newer-ish forths (I think starting with '94) do input differently and process things with "address count" type strings/buffers that I believe can have null and other characters in there. DPANS94 states that control characters shall be treated as a space (verified on Tali2 which lets me type null characters inside strings, but I can't use it as part of a word name because it is treated like a space). A zero at the beginning of a block/screen would then just be treated like a space and the rest of the block would continue to be evaluated if LOADed.

The zero at the beginning of a screen seems like a nice trick for FIG based forths, though.

[IamRob]

SamCoVT wrote:

I believe that's FIG-FORTH type behavior with the null character being the magical word "X" that stops interpreting a block or line of text. That was handy for FIG where the buffers were often smaller than a block, so the end of a buffer did not mean the end of a block, and refill would just get the next buffer and continue interpreting. FIG also would put a null at the end of the input buffer as it read in lines, so it worked well there too.

The newer-ish forths (I think starting with '94) do input differently and process things with "address count" type strings/buffers that I believe can have null and other characters in there. DPANS94 states that control characters shall be treated as a space (verified on Tali2 which lets me type null characters inside strings, but I can't use it as part of a word name because it is treated like a space). A zero at the beginning of a block/screen would then just be treated like a space and the rest of the block would continue to be evaluated if LOADed.

The zero at the beginning of a screen seems like a nice trick for FIG based forths, though.

If that is the case, then that is one behavior I like above other Forths.

Also, if Control Characters are getting automatically converted to an Ascii displayable character, that takes away from the ability to use pointers and Control Characters in screen blocks.

I don't see the purpose of new Forths making Control Characters into printable ascii characters. Does that mean there will need to be different word definitions for different devices?

For example, displaying text to a display screen will display the Control Characters as an ascii character, but printing the text to a printer will print the Control Character as is.

[JimBoyd]

IamRob wrote:

SamCoVT wrote:

I believe that's FIG-FORTH type behavior with the null character being the magical word "X" that stops interpreting a block or line of text. That was handy for FIG where the buffers were often smaller than a block, so the end of a buffer did not mean the end of a block, and refill would just get the next buffer and continue interpreting. FIG also would put a null at the end of the input buffer as it read in lines, so it worked well there too.

The newer-ish forths (I think starting with '94) do input differently and process things with "address count" type strings/buffers that I believe can have null and other characters in there. DPANS94 states that control characters shall be treated as a space (verified on Tali2 which lets me type null characters inside strings, but I can't use it as part of a word name because it is treated like a space). A zero at the beginning of a block/screen would then just be treated like a space and the rest of the block would continue to be evaluated if LOADed.

The zero at the beginning of a screen seems like a nice trick for FIG based forths, though.

If that is the case, then that is one behavior I like above other Forths.

Also, if Control Characters are getting automatically converted to an Ascii displayable character, that takes away from the ability to use pointers and Control Characters in screen blocks.

I don't see the purpose of new Forths making Control Characters into printable ascii characters. Does that mean there will need to be different word definitions for different devices?

For example, displaying text to a display screen will display the Control Characters as an ascii character, but printing the text to a printer will print the Control Character as is.

From what I can find in the Forth-83 Standard and the Ansi Forth Standard, the use of ASCII control characters is an environmental dependency. In other words, go ahead and support the use of control characters. Someone writing for your system, or mine, will need to make the choice between the extra functionality and convenience of using control characters versus portability. Fleet Forth is a Forth-83 Standard Forth and it does support control characters, as can be seen in my two screen shots in this thread.
Fleet Forth also has a "magic" word "X" that stops loading a block or parsing the text input buffer ( or evaluating a string). It is not an inline null. parsing text in Fleet Forth, including loading a block or evaluating a string, will stop when the text stream is exhausted or if EXIT is executed. When the input stream is exhausted, the string returned by WORD has a count of zero and a blank. This word is found in the Forth vocabulary and is an alias for EXIT that is immediate.
It is possible to add the Fig Forth behavior to stop parsing if a null is encountered. It will need to be space delimited.
Here is the session log of where I did just that.

Code: Select all

 OK
FORTH  OK
CREATE X  OK

In Fleet Forth, this basically creates a variable with no storage. Now to make the code field of X point to the body of EXIT

Code: Select all

' EXIT @ ' X !  OK
SEE X 
X
 ' (LOOP) >BODY 24 +
  2870          PLA
  2871   251    STA IP
  2873          PLA
  2874   252    STA IP 1+
  2876  2629    JMP NEXT
9 
 OK

Now to change the name field.

Code: Select all

$80C1 ' X >NAME !  OK
HEX LATEST 4 DUMP 
5721  C1 80 36  B  4 44 55 4D  50 20 54 20 20 4D 45 20   ..6KDDUMP T  ME 
 OK

And store a zero in the first byte of the block I'm using for this test.

Code: Select all

1 RAM LIST 
SCR# 8001 
0: @
1: 
2: CR .S
3: 
4: 
5: 
6: 
7: 
8: 
9: 
A: 
B: 
C: 
D: 
E: 
F: 
 OK
1 RAM LOAD  OK
2 1 RAM LINELOAD 
EMPTY  OK

As can be seen, the block doesn't load until I use LINELOAD to load the block starting at line 2, skipping the null character.
Having said that, it is not a technique I will use. When I use blocks for data, it is usually for virtual memory. The value in the first bytes of a block used for virtual memory depends on what is written to virtual memory.

[GARTHWILSON]

GARTHWILSON wrote:

It's an interesting idea, and one I had never thought of. It should make certain operations easier, with storage in flash memories for example. One thing I had thought of, but never got around to doing anything with yet, is making screens to be bigger, rather than 64 characters by 16 lines. 64 is rather limiting for source code with lots of comments. That is of course why there were shadow screens, back when displays couldn't do 128 characters across; but how inconvenient! So I thought why not have a screen be 2K or 4K, being 128 characters by 16 or 32 lines. Storage is plenty cheap now for that. 128 columns is hardly beyond my threshold of where I really start to feel comfortable.

In the facebook group "Forth2020 Users-Group" Zoom meeting yesterday (Sat, Mar 13, 2021) with Leo Brodie, author of "Starting Forth" and "Thinking Forth" which you can watch at https://www.youtube.com/watch?v=--IJEl6HV2k, he said that Charles Moore (inventor of Forth) was not concerned about what others thought about how something should be done, and that what was of interest to him was "What is the best way to do this for me?", and I seem to remember he said in another part of the video that there was nothing magical about the 1K block size. It kind of opens the way all the more for making 128x32 (or other size) blocks if you want to—not that you really needed permission, but it's easier to justify if you want to.

[BruceRMcF]

If using the first "pseudo-line" of a block as an index, with supporting tools:

\ This block contains definitions supporting BMW style --[ comments

... would be a way to make sure the index line is not interpreted.

The "Comus" equivalent to \ for "end processing source in file" would be \\ ... and then if you have a version that ends processing of a block, an index line starting with \\ would be a block that is never interpreted.

\\ This block contains binary tables used for fast multiply.

Of course, the block is not printable if it is raw binary, and so if it is assumed that the index line listing word(s) require printable characters, that means that only 960 bytes is available for raw binary storage in a 1K block.

If the C64 equivalent for \ is // then an equivalent for \\ may be ///

If it is desired for the block to be printable or transmittable as text, 7 base-32 digits is a maximum of 35 bits, so can hold 32bits of data, so with a trailing space would be 2 printable characters per 1 byte of data ... so a binary block of all bits set (2^32-1) would be:

3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV
3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV 3VVVVVV
...

Note that allocating 32 bits with 35 available means the the most significant digit is 0-3.

As a raw block, that would be 512 binary bytes of data per printable block ... with index line, 480 binary bytes per printable block.

An advantage of the format for a 16bit Forth is that the leading digit is 0-3, and three base-32 digits is at most 32,767 (VVV), so it can be processed with the leading digit, the next trio, and the last trio. So suppose you have LEAD@+ ( addr1 -- addr2 n ) TRIO@+ ( addr1 -- addr2 n ), TOPBITS gets the bottom two bits of a cell into the top two bits, TOPBIT gets the bottom bit into the top bit, and PUT-DOUBLE ( n1 n2 -- ) puts a double into a target buffer, incrementing the buffer index by four.

HEX
LEAD@+ TOPBITS >R
TRIO@+ DUP 2/ >R TOPBIT >R
TRIO@+ R> OR
R> R> OR
PUT-DOUBLE
CHAR+
... and you are ready for the next chunk of data.