FAT32 on Compact Flash project

[SamCoVT]

Now I can start implementing some of the ANS words. I checked the older ANS standards to see if there was anything "simpler", but it looks like the ANS2012 is a copy/paste of the ANS94 word set for files, and I didn't see file handling mentioned at all in the older standards. I'm going to be generally assuming that my hardware is working and not check for or report any errors (eg. the "ior" or I/O Result Code will always be 0 meaning nothing went wrong). I'm also trying just to get reading of files to work before I even think about how to write to them.

I had some issues writing some of these words (especially with doubles and single cells mixed on the stack), so I still have stack comments left in them. That was quite handy for debugging them, but I did have the lingering thought that I should make a new word for stack comments, perhaps "SC(" for Stack Comments, that prints out what I think should be on the stack and the actual values that are there - I could even make it a deferred word so that I could change it to just be silent when I'm not debugging anymore. Forth is a bit mind bending when you realize that you can make it do whatever you want that seems useful or amusing.

I didn't end up writing my stack comment helper word, but I'd be interested in how others do debugging. I found the ability to copy and paste chunks of code and then be able to inspect the stack very helpful, along with dump for checking my buffers and data structures. I also snuck a bunch of ".s" in the tricky spots.

Code: Select all

\ Continued from previous code
: bin ; ( Required by ANS, but we're ignoring it. )

( File access methods )
1 constant r/o  2 constant w/o  3 constant r/w

: findfile ( fileid 'findfile direntry_addr -- f )
( This word is used to help find a file in a directory  )
( with walkdirectory.  It needs the address of a fileid )
( structure with just the name filled in to be on the   )
( stack -- under the things walkdirectory needs.        )
   ( cr ." checking " dup 0B type cr .s ( DEBUG )
   0B ( length to compare )
   3 pick ( bring fileid struct address over )
   ( >filename -- but offset is zero)
   0B compare ;

: eof? ( fileid -- f ) ( return true if at end of file )
   dup >fileposition 2@ rot >filesize 2@ d= ;

( open-file - simplified for read-only access )
: open-file  ( c-addr u fam -- fileid ior )
   >r ( save fam )
   newfileid ( Allot memory for the file info )
   ( c-addr u fileid ) -rot 2 pick swap move ( filename )
   ( fileid ) r> over >attrib c! ( save fam )
   ( fileid ) working_dir  init_directory
   ( fileid ) ['] findfile working_dir walkdirectory
   ( fileid direntry|0 )
   ?dup 0= if
      cr ." Can't open file: " dup 0B type cr
      1 ( ior of 1 = Can't open file ) exit
   then
   ( fileid direntry ) swap >r ( save fileid )
   dup >clustLOW @ ( first cluster LSB )
   over >clustHI @ ( first cluster MSB )
   ( direntry firstcluster.d )
   2dup r@ >firstcluster 2! ( Save first cluster )
   2dup r@ >currentcluster 2! ( start at first cluster )
   ( direntry firstcluster.d )
   cluster>sector r@ >currentsector 2! ( starting sector )
   ( direntry ) dup >filesize 32bit@ r@ >filesize 2!
   ( direntry ) drop ( Done with direntry )
   r> ( bring back fileid structure )
   dup >fileposition d0!
   dup >linestart d0!
   0 ( no errors/exceptions )
;

: read-char  ( fileid -- c ) ( Note: does not check eof )
   dup >currentsector 2@ sector>buffer ( Get the sector )
   dup >fileposition 2@ ( Get the position in the file )
   drop 200 ( 512 ) mod ( Get offset into buffer )
   sectorbuff + c@ swap ( Get the character )
   ( c fileid )
   ( Move to next character )
   dup >fileposition d1+! ( increment fileposition )
   ( Check to see if we've wrapped to a new sector )
   dup >fileposition 2@ drop 200 ( 512 ) mod 0= if
      nextsector ( uses fileid ) else drop ( the fileid )
   then
;

: file-position  ( fileid -- ud ior )
   >fileposition 2@ ( Get current position )
   0 ( No error ) ;
: file-size  ( fileid -- ud ior )
   >filesize 2@ ( Get file size )
   0 ( No error ) ;

: sectorff  ( #bytes.d fileid -- )
   ( Sector fast-forward - follow FAT chain      )
   ( to determine current cluster/sector values. )
   ( Assumes starting from beginning of file.    )
   >r ( Save fileid )
   ( Move to next sector for every 512 bytes )
   ( Divide count by 512  - using shifts     )
   9 drshift
   begin
      2dup d0<>
   while
         r@ nextsector ( Move to the next sector )
         1. d- ( Reduce the count )
   repeat
   r> drop 2drop ;

: reposition-file  ( ud fileid -- ior )
   >r ( Save the fileid )
   r@ >filesize 2@ 2over ( newpos filesize newpos )
   d< if ( Check bounds )
      ." Cannot resize beyond end of file"
      r> drop 2drop exit then
   ( New position is ok.  Save it. )
   2dup  r@ >fileposition 2!
   ( Rewind the file to the first sector )
   r@ >firstcluster 2@ ( Get the first cluster )
   2dup r@ >currentcluster 2! ( Make first cluster current )
   cluster>sector ( Determine sector for this cluster )
   r@ >currentsector 2! ( Make that the current )
   r> sectorff ( Follow the FAT chains to the right sector )
   0 ( IO Result - no error ) ;

Once I got that working, it was time make some useful words:

Code: Select all

\ Continued from previous code
: catfile  ( fileid -- )  ( Copy file contents to screen )
   >r ( Save fileid ) cr ( Start on the next line )
   begin r@ eof? 0= while r@ read-char emit repeat r> drop ;

: rewind  ( fileid -- )
   ( Move fileposition back to beginning of file )
   0. rot reposition-file drop ;

And finally running it:

Code: Select all

: lcd.fs s" LCD     FS " ; ( A filename in direntry format )  ok
lcd.fs r/o open-file drop constant lcdfileid  ok
lcdfileid catfile 
( LCD Routines - SamCo - 2018-11-15 ) hex
: lcd-init ( -- ) 38 7FC0 c! 0E 7FC0 c! ;
: lcd-busy ( -- )         ( Wait until LCD is not busy )
    begin 7FC0 c@ 80 and while repeat ;
: lcd-cmd ( cmd -- )      ( Send an LCD command )
    lcd-busy 7FC0 c! ; allow-native
: lcd-char ( char -- )    ( Send an LCD character )
    lcd-busy 7FC1 c! ; allow-native
: lcd-type ( addr n -- )  ( Send a string to the LCD )
    0 ?do dup i + c@ lcd-char loop drop ;
: lcd." ( Make and send string ) postpone s" lcd-type ;
( Helper words )
: lcd-line1 ( -- ) 80 lcd-cmd ; allow-native
: lcd-line2 ( -- ) C0 lcd-cmd ; allow-native
: lcd-clear ( -- ) 01 lcd-cmd ; allow-native
decimal
 ok

It works! The LCD.FS file is 668 bytes, so it crosses over a sector change in the middle. I have another file that is 15K, and my fat32 partition has 4K clusters (8 sectors per cluster) so I've tested the cluster to cluster hopping using the FAT as well.

[GARTHWILSON]

SamCoVT wrote:

I'm going to be generally assuming that my hardware is working and not check for or report any errors (eg. the "ior" or I/O Result Code will always be 0 meaning nothing went wrong).

Off the top of my head (where there's not much left), I think what's more common in Forth is to return a TRUE flag (ie, -1) to mean success, and FALSE (ie, 0) to mean it didn't work. If an error is such that you really can't keep going, there's ABORT" . Brodie's book Thinking Forth also has CATCH and THROW which also made it into ANS for how to handle various error conditions.

Quote:

I did have the lingering thought that I should make a new word for stack comments, perhaps "SC(" for Stack Comments, that prints out what I think should be on the stack and the actual values that are there - I could even make it a deferred word so that I could change it to just be silent when I'm not debugging anymore.

That shouldn't be necessary, just as you don't need debuggers in Forth. The backslash tells Forth to ignore the rest of the line and neither interpret nor compile it (then you don't need the closing parenthesis at the end); and after the \, I put ^ to mean that what follows is what's on the data stack after that line executes. It takes fewer characters, which is valuable when you can hardly get enough display columns on your screen as it is. For the return stack, I use R^. I think ANS recommends R: (although since it's in the comments and gets thrown out, it won't affect any compilation or execution if you do it differently).

Quote:

I didn't end up writing my stack comment helper word, but I'd be interested in how others do debugging. I found the ability to copy and paste chunks of code and then be able to inspect the stack very helpful, along with dump for checking my buffers and data structures. I also snuck a bunch of ".s" in the tricky spots.

It's normal in Forth to write a little bit and then try it, write a little more and try it, etc.. If there's a problem, I can usually see immediately what it is, slap my forehead and call myself an idiot, and say, "Well, of course!", and make the change in the source code for the particular word. One of the nice things about Forth is that recompiling the word and setting it to working again is basically instant, not having to recompile the whole application, nor recompile an INCLude file and then link it, or anything like that. When there's a crash or I have to push the reset button, my reset routine gives three options: start over, keep the dictionary pointer where it was and just go to a state as if you had done an ABORT, or keep the dictionary pointer and even re-start execution with the CFA pointed to by variable INIT-AP. IOW, a crash need not take more than a couple of seconds to recover from, and you don't have to start over and re-load everything. Most crashes come from things like a loop whose exit condition is never met, rather than something that goes writing garbage all over memory; so it usually works out ok. I seldom use .S. but I do use DUP . a lot, and beeps to announce arrival at various parts of the program, or put out a pulse on an output pin to monitor on the oscilloscope. DUMP does come in handy sometimes. I seem to do very little debugging in Forth though, even in some rather complex programs like to program PIC microcontrollers.

[SamCoVT]

GARTHWILSON wrote:

Off the top of my head (where there's not much left), I think what's more common in Forth is to return a TRUE flag (ie, -1) to mean success, and FALSE (ie, 0) to mean it didn't work. If an error is such that you really can't keep going, there's ABORT" . Brodie's book Thinking Forth also has CATCH and THROW which also made it into ANS for how to handle various error conditions

It looks like this "ior" or "I/O Result" is indeed primed to be ready for a THROW. Tali2 doesn't have exceptions, but it does have ABORT" which can also use these "0 means everything is OK" values.

The first time I used ABORT", I didn't realize it took an argument and checked it for non-zero. I had put it in an IF and was just expecting it to print a message while running ABORT. Once I realized it had the "0<> IF" built in, I got it working properly.

I think my next step is to allow changing of directories. Before I can do that, I have to take a mini excursion into dealing with strings. I've currently been using S" to input string constants, however Tali2 compiles them into the dictionary where they live forever (unless using MARKER - similar to FORGET but you have to use a special word to mark the "rewind" point). Using MARKER doesn't really help me as any effects of using a file would also be lost.

The ANS2012 standard specifically mentions S" and S\" in section 11.3.4 and says that they should be able to store at least two distinct strings of 80 characters when used consecutively. For my needs, I only need short strings (eg. for 8.3 format filenames), so I'm thinking of making a special string word such as TS" for "Temporary String". This would allow me to change directories and open files without leaving their names in the dictionary. I've seen some examples that use multiple string buffers and use them round-robin, and I think that works for my application.

I also would like a string to keep track of the current directory's full path, and this one should be persistent and reasonably long. I can now see where some of the arbitrary path length limits and things come from - that's all the space a programmer reserved for it!

I also need some words to change filenames between disk format "NAME____EXT" (11 chars, space padded, with no period - I had to use underscores here to get it to display properly) and presentation format "name.ext". In the rough version I have working now, you have to enter filenames in disk format.

[GARTHWILSON]

SamCoVT wrote:

For my needs, I only need short strings (eg. for 8.3 format filenames), so I'm thinking of making a special string word such as TS" for "Temporary String". This would allow me to change directories and open files without leaving their names in the dictionary.

Is that different from " which puts the string in the pad (if you're in interpretation mode, ie, not compiling a string literal)?

Quote:

I've seen some examples that use multiple string buffers and use them round-robin, and I think that works for my application.

Over the years, the idea of trying a string stack has crossed my mind many times, but I've never actually done it.

[SamCoVT]

GARTHWILSON wrote:

Is that different from " which puts the string in the pad (if you're in interpretation mode, ie, not compiling a string literal)?

That seems like a reasonable word, but I couldn't find it anywhere in FIG or any of the ANS standards from 79 onwards.
ANS2012 requires the ability to define up to two such strings, so I ended up making my own temporary string handling words, but I did use your suggested name of just " for a temporary string.

Eventually I will need to modify my code to make S" use this temporary string behavior when interpreted and it's original "compile it into the dictionary" behavior when it's compiled. I do like how you can extend a Forth word and use the old word in the definition. I'm still playing around with getting things right, so I haven't taken that step yet.

These are the words I came up with for handling temporary strings as well as filename strings. A filename string comes in like "NAME.EXT" and needs to be expanded out to 11 chars with the . removed, which would look like "NAME(4 spaces here)EXT". This is how the file names are actually stored in a directory entry.

Code: Select all

decimal
( Temporary strings for filenames )
80 constant stringsize ( 80 char strings )
2  constant #strings
create stringsbuff stringsize #strings * allot
variable whichstring
hex

: nextstring ( -- caddr ) ( Determine next string addr )
   whichstring @ 1+ #strings mod ( Determine which is next )
   dup whichstring ! ( Update the variable for later )
   stringsize * stringsbuff + ; ( Calculate address )

: " ( ccc" -- c-addr u ) ( Put a string into stringsbuff )
   [char] " parse
   stringsize min ( No more than 80 chars )
   nextstring ( Use the next available temporary string )
   swap 2dup 2>r cmove ( Copy into the current string )
   2r> ; ( c-addr u for result left on stack )

: dotindex ( caddr u -- u2 ) ( Find the . in a filename )
   ( Return 0 if not found - or filename starts with . )
   0 ?do
      dup c@ [char] . = if drop i unloop exit then 1+
   loop ( If we made it here, no . )
   drop 0 ;

: string>filename ( caddr u -- caddr2 u2 )
   ( Convert a regular filename string into diraddr format )
   ( "FNAME.EXT" becomes "FNAME   EXT"                     )
   nextstring dup 0B blank ( Start with blank filename )
   dup >r ( Save location for later )
   -rot                                 ( ^ dest src u )
   0C min ( No more than 12 chars for filename.ext )
   2dup dotindex ( See if there is a . in the filename )
   (                                 ^ dest src u udot )
   ?dup 0= if ( No . so copy whole filename )
      rot swap move ( Copy whole string )
   else
      ( Figure out how many chars in ext and save )
      swap over - 1- >r                 ( ^ dest src udot )
      2dup + 1+ >r ( addr after .         ^ dest src udot )
      >r over r> move              ( ^ dest )
      8 + ( Move to extension in destination )
      r> swap r> ( Bring back source after .   ^ srcext destext uext )
      move
   then
   r> ( Bring back original temp string ) 0B ;

: f" ( ccc" -- caddr u ) 
   ( Accept a filename and convert to dirinfo format )
   [char] " parse string>filename ;

For now, I'm, using F" to enter filenames and have them converted to dirinfo format for my words that currently expect that. Eventually I will modify the routines that take filenames to all use STRING>FILENAME so that regular "name.ext" type names can be used directly. With the temporary strings out of the way, I can now work on changing directories:

Code: Select all

: cd ( caddr u -- ) ( Change the working directory )
   string>filename drop
   working_dir init_directory ( Start at the beginning of dir )
   ( Address of dirname in dirinfo format is on the stack )
   ['] findfile working_dir walkdirectory
   ( working_dir_fileid direntry )
   ?dup 0= if
      cr ." Can't find directory: " dup 0B type cr exit
   then
   ( working_dir_fileid direntry )
   swap drop ( Get rid of working_dir_fileid for now )
   dup >clustLOW @ swap >clustHI @ ( Get starting cluster )
   2dup d0= if ( Check for zero - meaning use root dir )
      ( Replace with correct cluster for root dir )
      2drop root_dir_first_cluster 2@
   then
   ( directory_firstcluster.d )
   ( Save everything to move to the new directory )
   working_dir >firstcluster 2! ( Fill in starting cluster )
   working_dir init_directory ; ( Fill in all the rest     )

And finally, a short demo. My LS word has been modified to show the attributes, which will show a "D" for directories. The names are still case sensitive (on my to-do list), so I have to enter the directory names in uppercase here

Code: Select all

ls 
FILENAME     ATTRIB   SIZE
LCD     .FS      A     668
REDIRECT.FS      A     978
SCREENS .FS      A    2579
BLOCK   .BLK     A   15360
FORTH   .       D        0
STUFF   .       D        0 ok
  ok
" STUFF" cd  ok
ls 
FILENAME     ATTRIB   SIZE
.       .       D        0
..      .       D        0
MORESTUF.       D        0
COPYING .TXT     A     679 ok
  ok
" MORESTUF" cd  ok
ls 
FILENAME     ATTRIB   SIZE
.       .       D        0
..      .       D        0
HEADERS .ASM     A   55122
NATIVE~1.ASM     A  369836
README  .MD      A    3299 ok
  ok
" .." cd  ok
ls 
FILENAME     ATTRIB   SIZE
.       .       D        0
..      .       D        0
MORESTUF.       D        0
COPYING .TXT     A     679 ok
" noexist" cd 
Can't find directory: noexist    
 ok

[GARTHWILSON]

SamCoVT wrote:

GARTHWILSON wrote:

Is that different from " which puts the string in the pad (if you're in interpretation mode, ie, not compiling a string literal)?

That seems like a reasonable word, but I couldn't find it anywhere in FIG or any of the ANS standards from 79 onwards.

By George, you're right! It was in another Forth I used, and I just carried it over to my '816 Forth the same way. The following is from kind of a "shadow" text file that gives more description of what's in the assembly-language source-code file:

Code: Select all

  " below can be followed by COUNT TYPE (which is all dotq says).
  " is a state-smart word.

  Compilation:  It compiles litq (above), then puts the string (delimited by " )
  into your program, then increments DP to the next available even address.
  WORD puts the string at HERE, but the PAD does not get used.

  Interpretation:  It puts the string in the PAD, and leaves the PAD's address
  on the stack.

  Execution:  (Actually it's the execution action of litq , which is compiled by
  "  .)  It only puts the address of the counted string on the stack, and
  increments IP to the next execution token.  PAD is not used.


: "             ( <text> -- addr )      \ addr is that of PAD          SF253-254
   STATE @
   IF    COMPILE litq  ,"
         ALIGN  EXIT
   THEN  ASCII " WORD
   DUP C@ 1+
   PAD SWAP CMOVE       PAD     ;  IMMEDIATE


: ,"            ( <text> -- )           \                            FD XIX/2/20
   ASCII " WORD                         \ Compile a string as for a table.
   C@ 1+ ALLOT                  ;       \ No runtime code is associated as there
                                        \ is with " above when " is compiled.


STRING in Starting Forth is like ," above but requires putting the delimiter's
ASCII value on the stack first, like  ASCII " STRING bla bla bla"  (Impractical!)

It says there " is in Starting Forth (second edition) pages 253-254; but Starting Forth says " is from the 83 model by Laxen and Perry, and it returns the address and the count. That's also what my HP-71B handheld computer's Forth does, although that seems to be mostly Forth-79.

[SamCoVT]

I found a bug in my read-char routine that shows up reading files longer than 32767 bytes (which was a big clue). My original version was:

Code: Select all

: read-char  ( fileid -- c ) ( Note: does not check eof )
   dup >currentsector 2@ sector>buffer ( Get the sector )
   dup >fileposition 2@ ( Get the position in the file )
   drop 200 ( 512 ) mod ( Get offset into buffer )
   sectorbuff + c@ swap ( Get the character )
   ( c fileid )
   ( Move to next character )
   dup >fileposition d1+! ( increment fileposition )
   ( Check to see if we've wrapped to a new sector )
   dup >fileposition 2@ drop 200 ( 512 ) mod 0= if
      nextsector ( uses fileid ) else drop ( the fileid )
   then
;

The "drop 200 mod" code is trying to be a little sneaky with doubles when trying to figure out the offset into a 512 byte (the 200 is in hex) disk buffer. I was dropping the most significant cell and just doing mod on the least significant cell. That almost worked, but the issue I ran into was that once the double value goes over $7FFF, the sign bit gets set in the least significant cell. That shouldn't affect me when doing unsigned math, but MOD in TaliForth2 returns a signed result in the range -$1FF up to 0 (I was expecting 0 to +$1FF) when the dividend is negative. I was adding this offset to my disk sector buffer address and expecting to land somewhere in the buffer, but I was going backwards instead of forwards!

Fortunately for me, I remembered a bit of a discussion in the Forth section on the quirks of / and MOD when the dividend and divisor have different signs and was able to find the bug. Once I knew it happened after 32767 characters, I was easily able to set up the exact point where things went wonky and copy and paste my routine in one line at a time, watching the stack as I went. When I saw the negative remainder, I knew what was up.

My solution was to use UM/MOD and drop the quotient. It's actually the exact right word for this situation because I'm dividing an unsigned double by an unsigned single and expecting an unsigned result. I initially avoided it, thinking my way was "faster", however I've spent a good amount of time hunting this bug down and it runs plenty fast on my SBC after being corrected. I also realized I could have done this with a bitwise AND, just because my divisor is an exact power of 2, and if speed becomes and issue I will do it that way.

My fixed version is:

Code: Select all

: read-char  ( fileid -- c ) ( Note: does not check eof )
   dup >currentsector 2@ sector>buffer ( Get the sector )
   dup >fileposition 2@ ( Get the position in the file )
   200 ( 512 ) um/mod drop ( Get offset into buffer )
   sectorbuff + c@ swap ( Get the character )
   ( c fileid )
   ( Move to next character )
   dup >fileposition d1+! ( increment fileposition )
   ( Check to see if we've wrapped to a new sector )
   dup >fileposition 2@ 200 ( 512 ) um/mod drop 0= if
      nextsector ( uses fileid ) else drop ( the fileid )
   then
;

[SamCoVT]

I have read-only access working for FAT32 and I believe I have a good start on INCLUDED and INCLUDE from the ANS2012 File Access wordset. I still need to work on making INCLUDED/INCLUDE nest properly along with LOAD and EVALUATE. I think extending SAVE-INPUT and RESTORE-INPUT to be file aware and then using those should do the trick. My READ-LINE already saves the offset to the beginning of the line it is about to read, so I should be able to re-read a line from the previous input file if needed.

I did learn some important things:
1) You can't use I in a DO loop in between >R and R> because the loop parameters are kept on the return stack.
2) You can't keep the file-id on the stack when looping through the file and interpreting, because (of course) the code you are interpreting will be putting things on the stack.

I ended up using a separate variable to keep the file-id in while looping through the file line by line. Prior to this, I've just been pasting the compact flash and FAT32 code into the terminal when I wanted to work with it.

I'm not sure if I will get adventurous enough to actually write to the FAT32 - being able to read it from my SBC is probably enough for my uses and I'll use my PC to create all the files I need.

I currently have:

Code: Select all

DECIMAL
CREATE mybuff 80 ALLOT

VARIABLE current_fileid
0 current_fileid !

: SOURCE-ID ( -- u) ( Modify source-id to return file-id if available ) 
   current_fileid @ ?DUP IF ELSE SOURCE-ID THEN ;

: INCLUDED ( caddr u -- ) ( Open and interpret file )
   R/O OPEN-FILE IF EXIT THEN ( Make sure the file opens. )
   CR
   ( fileid )
   current_fileid @ >R   current_fileid ! ( save fileid for source-id )
   BEGIN
      current_fileid @   mybuff 80 ROT READ-LINE DROP
      ( numchars_read not_at_eof_flag )
   WHILE
         ( numchars_read )
         DUP mybuff SWAP TYPE CR ( Type it to the screen. )
         mybuff SWAP EVALUATE ( Run it. )
   REPEAT
   R> current_fileid ! ( restore previous fileid )
;

: INCLUDE ( "filename" -- ) ( prefix version of included )
   PARSE-NAME ( filename comes next - with no quotes )
   STRING>FILENAME INCLUDED ;

I can now say:

Code: Select all

ls 
FILENAME     ATTRIB   SIZE
FORTH   .                0
DIRA    .       D        0
DIRB    .       D        0
LCD     .FS      A     668
SCREENS .FS      A    2579 ok
include LCD.FS

and the file LCD.FS will be loaded into TaliForth2. I'll be working on making this nest properly with LOAD and EVLUATE next and then will block-ize this (on my PC using emacs' forth-block-mode) and use my SBC to load it from blocks on the compact flash card into blocks on my I2C EEPROM. That way, I can bring up FAT32 support by LOADing it from my I2C chip, which my version of Tali2 already supports directly.