6502.org • View topic - Fleet Forth metacompiler redesign

View unanswered posts | View active topics

Board index » 6502.org Users Forum » Forth

All times are UTC

Fleet Forth metacompiler redesign

Page 4 of 4

[ 49 posts ]

Go to page Previous 1, 2, 3, 4

Previous topic | Next topic

Author

Message

JimBoyd

Post subject: Re: Fleet Forth metacompiler redesign

Posted: Sun Sep 17, 2023 6:49 pm

Joined: Fri May 05, 2017 9:27 pm
Posts: 851

JimBoyd wrote:

In solving this problem I came up with two different solutions which resulted in two versions of the metacompiler. They were the same except for what feature was added.

For reference I will refer to the version of the metacompiler without either of these solutions as the base metacompiler. It is this base upon which version A and version B are built.
First, some words in the base metacompiler were cleaned up.
The original source for PNAME and VHEADER .

Code:

: PNAME  ( -- )
   1 VALLOT  1 PNAMES +!
   COLS ?CR ." PADDED: R"
   HERE S? CR ;

: VHEADER  ( >IN -- >IN )
   VNAME TFIND NIP
   ABORT" REDEFINITION IN TARGET"
   HEAD @
   IF
      DUP >IN !
      HERE C@ THERE + 4 +
      SPLIT DROP 0=
      IF  PNAME  THEN
      WLINK VADD
      VNAME C@  THERE $80 VTOGGLE
      VALLOT  THERE $80 VTOGGLE
      1 VALLOT
      EXIT
   THEN
   THERE 1+ SPLIT DROP 0=
   IF  PNAME  THEN
   1 HEADLESS +! ;
HOST

and the improved source.

Code:

: PNAME  ( ADR -- )
   1+ SPLIT DROP ?EXIT      \ If the low byte of the address is not $FF then exit.
   1 VALLOT  1 PADDED +!    \ Pad the virtual header and increment the padded count.
   COLS ?CR ." PADDED: R"   \ If not at left column perform carriage return
   HERE S? CR ;             \ before displaying the name of the padded word.

: VHEADER  ( -- )
   >IN @  VNAME TFIND NIP           \ Save >IN. If the parsed name already exists
   ABORT" REDEFINITION IN TARGET"   \ in the target vocabulary then abort.
   HEAD @                           \ If HEAD is on
   IF                               \ create header.
      >IN !                         \ Restore >IN to parse the name again.
      HERE C@ THERE + 2+ 1+ PNAME   \ Avoid the indirect jump bug.
      WLINK VADD                    \ Create the link field.
      VNAME C@  THERE $80 VTOGGLE   \ Create the name field with high bit set
      VALLOT  THERE $80 VTOGGLE     \ for count and last character.
      1 VALLOT                      \
      EXIT
   THEN                             \ If HEAD is off
   DROP  1 HEADLESS +!              \ increment the count of headerless words
   THERE PNAME ;                    \ and avoid the indirect jump bug.

PNAME now takes an address and pads the header, if necessary, to avoid the code field straddling a page boundary.
VHEADER now no longer requires the value of >IN on the data stack and it returns nothing.

I will present the first solution, metacompiler A.
Here is the base metacompiler's vocabulary structure.

Code:

FORTH
  META
    SHADOW
      FORTH
        ASSEMBLER
  EDITOR
  ASSEMBLER

The vocabulary RUN is added to the META vocabulary.

Code:

FORTH
  META
    SHADOW
      FORTH
        ASSEMBLER
    RUN
  EDITOR
  ASSEMBLER

The variable ALIASES is added to keep a count of how many words have an alias.
The new word ALIAS creates a header in virtual memory and a target word (a handle) in the RUN vocabulary. ALIAS also switches off header creation in virtual memory for the next word with NH .

Code:

: ALIAS  ( ++ )
   ORDER@ [META] RUN DEFINITIONS     \ Save the search order and 
   HEADER NH                         \ create header in RUN vocabulary.
   ORDER!                            \ Restore the search order.
   TRUE HEADLESS +!  1 ALIASES +! ;  \ Correct the headless count and
                                     \ increment the count of aliases.

ALIAS is used like this:

Code:

ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   ...
   END-CODE

Here is the base metacompiler's MCOMPILE .

Code:

CODE MCOMPILE  ( -- )
   >FORTH                           \ Shift to high level Forth.
   ?COMP                            \ Abort if not compiling.
   R> DUP 2+ >R @                   \ Get the CFA of next word in Forth thread containing MCOMPILE
   >NAME COUNT $1F AND >HERE CLIP   \ and use the name field to create a search string at HERE .
   TFIND ?TARGET M, ;               \ Search the TARGET FORTH vocabulary and abort if not found.
                                    \ If found, compile to virtual memory with M, .

and the modified MCOMPILE .

Code:

CODE MCOMPILE  ( -- )
   >FORTH                           \ Shift to high level Forth.
   ?COMP                            \ Abort if not compiling.
   R> DUP 2+ >R @                   \ Get the CFA of next word in Forth thread containing MCOMPILE
   >NAME COUNT $1F AND >HERE CLIP   \ and use the name field to create a search string at HERE .
   TFIND TRUE <>                    \ If not found or if immediate
   IF
      DROP  HERE [META]             \ drop the result and use the string at HERE
      [ ' RUN >BODY ] LITERAL       \ to search the RUN vocabulary.
      VFIND TRUE <>                 \ If still not found
      ABORT" RUN-TIME NOT FOUND"    \ abort.
   THEN
   M, ;                             \ If found, compile to virtual memory.

The modified MCOMPILE for metacompiler A still creates a search string from the name field of the next word in it's definition.

Code:

: TEST
   MCOMPILE (TEST) ; IMMEDIATE

When TEST is executed, MCOMPILE will create the search string "(TEST)" at HERE .
The TARGET FORTH vocabulary is searched by TFIND .If the sought word is not found or if it is immediate, the result is dropped and the RUN vocabulary is searched. Once more, the result has to be for a word which is not immediate.
VFIND does not search parent vocabularies.

This version of the metacompiler allows changing the names of run-time words with ALIAS ; however, there is one downside.
Using DO and it's run-time as an example.

Code:

ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   ...
   END-CODE

There are now two handles on the host system for the same target word in virtual memory, this is not the problem. If the name of the run-time for DO on the host system does not match either one, a new alias is needed.

Code:

ALIAS DO   \ Name of run-time for new kernel. Creates handle DO in RUN vocabulary and header DO in virtual memory.
ALIAS D    \ Name of run-time in host system. Creates handle D in RUN vocabulary. header creation in virtual memory suppressed.
CODE (DO)  \ Name of run-time used in kernel source. Creates handle (DO) in TARGET FORTH vocabulary. header creation in virtual memory suppressed.
   ...
   ...
   END-CODE

This creates three handles for the same word in virtual memory, one in the TARGET FORTH vocabulary and two in the RUN vocabulary.
There is only one header created in virtual memory for this word since ALIAS suppresses header creation in the following word.

Top

JimBoyd

Post subject: Re: Fleet Forth metacompiler redesign

Posted: Sun Sep 17, 2023 8:52 pm

Joined: Fri May 05, 2017 9:27 pm
Posts: 851

I will now present the second solution, metacompiler B.
No new metacompiler vocabularies are added. This metacompiler's ALIAS does not create a target word. It creates a header in virtual memory. It does use NH to switch off header creation in virtual memory for the next word.

Code:

: ALIAS  ( -- )
   VHEADER NH
   TRUE HEADLESS +!  1 ALIASES +! ;

ALIAS is used in this metacompiler just like the other version.

Code:

ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   ...
   END-CODE

MCOMPILE was modified slightly so the search only succeeds if the word found is not immediate.

Code:

CODE MCOMPILE  ( -- )
   >FORTH                           \ Shift to high level Forth.
   ?COMP                            \ Abort if not compiling.
   R> DUP 2+ >R @                   \ Get the CFA of next word in Forth thread containing MCOMPILE
   >NAME COUNT $1F AND >HERE CLIP   \ and use the name field to create a search string at HERE .
   TFIND TRUE <>                    \ If not found or if immediate
   ABORT" RUN-TIME NOT FOUND"       \ abort.
   M, ;                             \ If found, compile to virtual memory.

With these modifications it is possible to create a kernel where each of the DO LOOP run-time words has the same name as it's compiling word. The next addition to metacompiler B will allow it to work when built on the new system.
The word :: (double colon) is an alias for the host colon. This will allow accessing colon's functionality while defining words in the META vocabulary, where colon was redefined.

Code:

: ::  ( -- ADR TRUE )
   : ;  // ACCESS TO HOST COLON

These words defined in the META vocabulary are stubs. They don't get compiled or metacompiled. MCOMPILE uses their names to find the corresponding word in the TARGET FORTH vocabulary.

Code:

META DEFINITIONS
:: (DO)   ;
:: (?DO)   ;
:: (LOOP)   ;
:: (+LOOP)   ;
HOST DEFINITIONS

These are metacompiler B's DO LOOP compiling words.

Code:

META DEFINITIONS
:: DO  ( -- >SYSL <SYSL )
   MCOMPILE (DO)
   >MARK 2+ 2+
   <MARK 2+ 2+ ; IMMEDIATE
:: ?DO  ( -- >SYSL <SYSL )
   MCOMPILE (?DO)
   >MARK 2+ 2+
   <MARK 2+ 2+ ; IMMEDIATE
:: LOOP  ( >SYSL <SYSL -- )
   MCOMPILE (LOOP)
   2- 2- <RESOLVE
   2- 2- >RESOLVE ; IMMEDIATE
:: +LOOP  ( >SYSL <SYSL -- )
   MCOMPILE (+LOOP)
   2- 2- <RESOLVE
   2- 2- >RESOLVE ; IMMEDIATE
HOST

Recall that the new semicolon will behave just like the host semicolon when metacompiling is off.
I could have defined these metacompiler DO LOOP compiling words with COMPILE since COMPILE gets patched when metacompiling.
This version of the metacompiler works well; however, there is a slight downside. If I want to rename the run-time for ." so it has the same name as ." , I will need to define a stub for the run-time and a metacompiler version of ." . The same is true for any run-time I wish to rename.

Code:

META DEFINITIONS
:: (.")   ;
:: (ABORT")   ;

:: ."  ( -- )
   COMPILE (.")
   ," ; IMMEDIATE
:: ABORT"  ( -- )
   COMPILE (ABORT")
   ," ; IMMEDIATE
HOST

Code:

META DEFINITIONS
:: (.")   ;
:: (ABORT")   ;

:: ."  ( -- )
   MCOMPILE (.")
   V," ; IMMEDIATE
:: ABORT"  ( -- )
   MCOMPILE (ABORT")
   V," ; IMMEDIATE
HOST

I thought of yet another approach to this problem and I think it will yield the best version of the metacompiler.

Top

JimBoyd

Post subject: Re: Fleet Forth metacompiler redesign

Posted: Sun Sep 17, 2023 9:53 pm

Joined: Fri May 05, 2017 9:27 pm
Posts: 851

Metacompiler C, the, hopefully, last solution to this problem.
The only differences with metacompiler B:
:: (double colon) is not defined.
MCOMPILE is different.

Code:

: CFA>STR  ( CFA -- HERE )
   >NAME COUNT $1F AND >HERE CLIP ;  \ Use CFA to generate a search string 
                                     \ from a word's name field.
CODE MCOMPILE  ( -- )
   >FORTH
   ?COMP  R> DUP 2+ >R @ CFA>STR     \ Get CFA of next word in definition
                                     \ to generate search string.
   TFIND TRUE <>                     \ If the word is not found or is
   IF                                \ immediate
      DROP  R@ 2- 2- BODY>  CFA>STR  \ Back up to CFA of the word containing
      ASCII ) OVER COUNT + C!        \ MCOMPILE to generate a search string
      DUP DUP 1+ BL MOVE             \ and modify string to be enclosed in
      ASCII ( OVER 1+ C!             \ parenthesis.
      2 OVER +!  TFIND TRUE <>       \ Search the TARGET FORTH vocabulary
      ABORT" RUN-TIME NOT FOUND"     \ and abort if not found.
   THEN
   M, ;                              \ If found, compile to virtual memory.

CFA>STR takes the CFA of a word and generates a string at HERE which is the same as the word's name.
MCOMPILE still generates a search string based on the name of the word to be metacompiled by MCOMPILE . If the search is unsuccessful or the word found is immediate, MCOMPILE will generate a search string from the name of the word which contains METACOMPILE . This search string will be the compiling word's name enclosed in parenthesis.
Using DO as an example.
Suppose the run-time for DO on the host system is still named (DO) .
MCOMPILE generates the string "(DO)" and searches for this string in the TARGET FORTH vocabulary. It will be found and the address for (DO) in virtual memory will be compiled to virtual memory.
Suppose the run-time for DO on the host has a different name. Let's suppose it is something absurd like DO.RUNTIME .
MCOMPILE tries to find DO.RUNTIME in the TARGET FORTH vocabulary and fails.
MCOMPILE then generates a string from the name field of the compiling word, in this case DO . The string will have parenthesis added.
Here is a sample test.

Code:

: TESTWORD
   MCOMPILE FOOBAR ; IMMEDIATE

FOOBAR does not exist in the TARGET FORTH vocabulary. In this example, (TESTWORD) does exist in the TARGET FORTH vocabulary.
MCOMPILE tries to find the target word FOOBAR in the TARGET FORTH vocabulary. When that fails, it tries to find (TESTWORD) in the TARGET FORTH vocabulary and succeeds.
Run-time words with the same name in the host system and the source used by the metacompiler work fine.
Examples are ?BRANCH and BRANCH used by most of the control flow words.
Run-time words with different names in the host system and the source used by the metacompiler will work if the name of the run-time in the source is a parenthesized version of the compiling word's name.
Examples are (DO) compiled by DO and (LOOP) compiled by LOOP . These run-time words can be given a different name on the target being built in virtual memory by using the word ALIAS .

At present, I don't see a downside to this version of the metacompiler

Top

JimBoyd

Post subject: Re: Fleet Forth metacompiler redesign

Posted: Tue Sep 26, 2023 9:17 pm

Joined: Fri May 05, 2017 9:27 pm
Posts: 851

I recently came up with three variations of Fleet Forth's metacompiler. The third variation seemed the best way to go so it is Fleet Forth's metacompiler; indeed, if I'd thought of it first, I would not have bothered with the other two.
As I previously mentioned, the metacompiler patches the word COMPILE to MCOMPILE . This is so I don't have to define a metacompiler version of every word which compiles ?BRANCH or BRANCH , or words which contain such words. I also don't have to define a metacompiler version of some of the other compiling words.
Although the target being built in virtual memory can have redefinitions, a target word (a handle for it's counterpart in virtual memory) in the TARGET FORTH vocabulary can not be a redefinition of another target word.
To allow redefinitions in the target being built, the word ALIAS is used.
There is one run-time word in the kernel which could not be given an alias with the way the latest MCOMPILE was written. One thing to keep in mind, what I mean by this is that any word can be given an alias and the kernel will be metacompiled successfully; however, the Forth system built on the new kernel might not be able to host the metacompiler.
The following deals with using the metacompiler on the new system (I really don't want to maintain two versions of ITC Fleet Forth's kernel).
The run-time word (") , PAREN-QUOTE, is compiled by the word " , QUOTE. The source for QUOTE.

Code:

: "  ( -- )
   STATE @
   IF  COMPILE (") ," EXIT  THEN
   ASCII " CHAR ?MEM  PAD TUCK C!
   TUCK COUNT MOVE ; IMMEDIATE

If compiling, QUOTE compiles PAREN-QUOTE, it's run-time word, and compiles an inline string with ," , COMMA-QUOTE.
If interpreting, the text string in the text stream is copied to PAD and the address of PAD is left on the data stack.

PAREN-QUOTE could not be renamed to QUOTE.
When QUOTE is executed while metacompiling, the name for QUOTE would not be found in the TARGET FORTH vocabulary. I didn't mean a non immediate version couldn't be found, although MCOMPILE checks for that, no version would be found. QUOTE is not even defined in the Forth kernel, it is in the system loader.
MCOMPILE would not be able to back up to the name field of QUOTE to generate a parenthesized version of it's name because COMPILE is not the first word in the body of QUOTE.

The latest improvement to MCOMPILE uses AFIND .
AFIND is a word which takes an address and returns that address and the closest link field address below and the closest link field address above the given address.

Code:

AFIND ( ADR -- ADR LFA1 LFA2 )

These are excerpts from the source used by the metacompiler.

Code:

ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   END-CODE

   ...

: DO  ( -- >SYSL <SYSL )
   COMPILE (DO)
   >MARK 2+ <MARK 2+ ; IMMEDIATE

The run time of DO is also named DO because of ALIAS .
When DO is executed, MCOMPILE would get the code field of the run-time DO and back up to it's name field. It would generate a search string based on that name and attempt to find a word with the name DO in the TARGET FORTH vocabulary. Failing to do so, it would get the address within the word containing it, DO , and back up to it's name field. MCOMPILE would generate a parenthesized version of the name and use it , (DO) , for the search string.
This is the latest source for MCOMPILE and two helper words.

Code:

: LFA>STR  ( LFA -- HERE )
   L>NAME COUNT $1F AND >HERE CLIP ;
: ?RT  ( FLAG -- )
   ABORT" RUN-TIME NOT FOUND" ;

CODE MCOMPILE  ( -- )
   >FORTH
   ?COMP  R> DUP 2+ >R @          // GET ADR OF NEXT WORD
   AFIND DROP TUCK LINK> = DUP    // DOES IT HAVE A HEADER?
   IF  DROP  LFA>STR TFIND  THEN  // IS IT IN TARGET FORTH
   TRUE <>                        // VOCABULARY?
   IF                             // NOT IN VOC OR HEADERLESS    
      R@ AFIND 2NIP DROP LFA>STR  // GET ADR OF WORD
      ASCII ) OVER COUNT + C!     // CONTAINING MCOMPILE
      DUP DUP 1+ BL MOVE          // USE NFA TO GENERATE
      ASCII ( OVER 1+ C!          // SEARCH STRING AND 
      2 OVER +!  TFIND TRUE <> ?RT  // PARENTHESIZE
   THEN
   M, ;                           // IF FOUND, COMPILE TO VM

LFA>STR takes a word's link field address, converts it to the name field address and copies the word's name to HERE .
?RT aborts if there is a true value on the data stack.
This version of MCOMPILE will work even if the run-time in question has no header on the system the metacompiler is built upon.
MCOMPILE gets the address of the next word in the definition containing MCOMPILE , or COMPILE since COMPILE is patched while metacompiling. AFIND is used to find the closest link field address below this word. This LFA is converted to the CFA and compared with the CFA of the run-time word. If they differ the run-time is headerless.
If it is not headerless MCOMPILE uses it's name for the search string to search the TARGET FORTH vocabulary with TFIND , target find.
If the run-time is not found or it is headerless MCOMPILE uses AFIND to get the name field address of the word which contains MCOMPILE and generate a search string which is a parenthesized version of that name.

With this version of MCOMPILE it does not matter if a run-time word is headerless in the system upon which the metacompiler is built. MCOMPILE will generate a string which is a parenthesized version of the compiling word's name.
It also does not matter if COMPILE or MCOMPILE is not the first word in the compiling words body. Once again AFIND is used to find the word's name field address.

This is an excerpt from the metacompiler source.

Code:

META DEFINITIONS
: (>ASSEM)   ;
: >ASSEM  ( -- )
   ?METACOMP
   ?COMP  MCOMPILE (>ASSEM)
   [ASSEMBLER] MEM ASSEMBLER
   [COMPILE] [ ; IMMEDIATE

The metacompiler has to define it's own version of >ASSEM , even though COMPILE is patched, because it has to set CONTEXT to the metacompiler's ASSEMBLER vocabulary.
Notice the dummy word (>ASSEM) . This stub is defined because the name (>ASSEM) is no longer found in the new system. The run-time for >ASSEM also has the name >ASSEM .
Also notice the metacompiler's version of >ASSEM has MCOMPILE in it's definition.
With the new version of MCOMPILE it is possible to redefine the metacompiler's version of >ASSEM

Code:

META DEFINITIONS
: (>ASSEM)   ;
   ?METACOMP
   ?COMP  MCOMPILE [ 0 , ]
   [ASSEMBLER] MEM ASSEMBLER
   [COMPILE] [ ; IMMEDIATE

MCOMPILE will treat the inline zero as a headerless word and generate the search string (>ASSEM) from the name field of >ASSEM .

Top

Page 4 of 4

[ 49 posts ]

Go to page Previous 1, 2, 3, 4

Board index » 6502.org Users Forum » Forth

All times are UTC

Who is online

Users browsing this forum: No registered users and 14 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum