6502.org Forum  Projects  Code  Documents  Tools  Forum
It is currently Thu Nov 21, 2024 12:21 pm

All times are UTC




Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4
Author Message
PostPosted: Sun Sep 17, 2023 6:49 pm 
Offline

Joined: Fri May 05, 2017 9:27 pm
Posts: 895
JimBoyd wrote:
In solving this problem I came up with two different solutions which resulted in two versions of the metacompiler. They were the same except for what feature was added.

For reference I will refer to the version of the metacompiler without either of these solutions as the base metacompiler. It is this base upon which version A and version B are built.
First, some words in the base metacompiler were cleaned up.
The original source for PNAME and VHEADER .
Code:
: PNAME  ( -- )
   1 VALLOT  1 PNAMES +!
   COLS ?CR ." PADDED: R"
   HERE S? CR ;

: VHEADER  ( >IN -- >IN )
   VNAME TFIND NIP
   ABORT" REDEFINITION IN TARGET"
   HEAD @
   IF
      DUP >IN !
      HERE C@ THERE + 4 +
      SPLIT DROP 0=
      IF  PNAME  THEN
      WLINK VADD
      VNAME C@  THERE $80 VTOGGLE
      VALLOT  THERE $80 VTOGGLE
      1 VALLOT
      EXIT
   THEN
   THERE 1+ SPLIT DROP 0=
   IF  PNAME  THEN
   1 HEADLESS +! ;
HOST

and the improved source.
Code:
: PNAME  ( ADR -- )
   1+ SPLIT DROP ?EXIT      \ If the low byte of the address is not $FF then exit.
   1 VALLOT  1 PADDED +!    \ Pad the virtual header and increment the padded count.
   COLS ?CR ." PADDED: R"   \ If not at left column perform carriage return
   HERE S? CR ;             \ before displaying the name of the padded word.

: VHEADER  ( -- )
   >IN @  VNAME TFIND NIP           \ Save >IN. If the parsed name already exists
   ABORT" REDEFINITION IN TARGET"   \ in the target vocabulary then abort.
   HEAD @                           \ If HEAD is on
   IF                               \ create header.
      >IN !                         \ Restore >IN to parse the name again.
      HERE C@ THERE + 2+ 1+ PNAME   \ Avoid the indirect jump bug.
      WLINK VADD                    \ Create the link field.
      VNAME C@  THERE $80 VTOGGLE   \ Create the name field with high bit set
      VALLOT  THERE $80 VTOGGLE     \ for count and last character.
      1 VALLOT                      \
      EXIT
   THEN                             \ If HEAD is off
   DROP  1 HEADLESS +!              \ increment the count of headerless words
   THERE PNAME ;                    \ and avoid the indirect jump bug.

PNAME now takes an address and pads the header, if necessary, to avoid the code field straddling a page boundary.
VHEADER now no longer requires the value of >IN on the data stack and it returns nothing.

I will present the first solution, metacompiler A.
Here is the base metacompiler's vocabulary structure.
Code:
FORTH
  META
    SHADOW
      FORTH
        ASSEMBLER
  EDITOR
  ASSEMBLER

The vocabulary RUN is added to the META vocabulary.
Code:
FORTH
  META
    SHADOW
      FORTH
        ASSEMBLER
    RUN
  EDITOR
  ASSEMBLER

The variable ALIASES is added to keep a count of how many words have an alias.
The new word ALIAS creates a header in virtual memory and a target word (a handle) in the RUN vocabulary. ALIAS also switches off header creation in virtual memory for the next word with NH .
Code:
: ALIAS  ( ++ )
   ORDER@ [META] RUN DEFINITIONS     \ Save the search order and
   HEADER NH                         \ create header in RUN vocabulary.
   ORDER!                            \ Restore the search order.
   TRUE HEADLESS +!  1 ALIASES +! ;  \ Correct the headless count and
                                     \ increment the count of aliases.

ALIAS is used like this:
Code:
ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   ...
   END-CODE

Here is the base metacompiler's MCOMPILE .
Code:
CODE MCOMPILE  ( -- )
   >FORTH                           \ Shift to high level Forth.
   ?COMP                            \ Abort if not compiling.
   R> DUP 2+ >R @                   \ Get the CFA of next word in Forth thread containing MCOMPILE
   >NAME COUNT $1F AND >HERE CLIP   \ and use the name field to create a search string at HERE .
   TFIND ?TARGET M, ;               \ Search the TARGET FORTH vocabulary and abort if not found.
                                    \ If found, compile to virtual memory with M, .

and the modified MCOMPILE .
Code:
CODE MCOMPILE  ( -- )
   >FORTH                           \ Shift to high level Forth.
   ?COMP                            \ Abort if not compiling.
   R> DUP 2+ >R @                   \ Get the CFA of next word in Forth thread containing MCOMPILE
   >NAME COUNT $1F AND >HERE CLIP   \ and use the name field to create a search string at HERE .
   TFIND TRUE <>                    \ If not found or if immediate
   IF
      DROP  HERE [META]             \ drop the result and use the string at HERE
      [ ' RUN >BODY ] LITERAL       \ to search the RUN vocabulary.
      VFIND TRUE <>                 \ If still not found
      ABORT" RUN-TIME NOT FOUND"    \ abort.
   THEN
   M, ;                             \ If found, compile to virtual memory.

The modified MCOMPILE for metacompiler A still creates a search string from the name field of the next word in it's definition.
Code:
: TEST
   MCOMPILE (TEST) ; IMMEDIATE

When TEST is executed, MCOMPILE will create the search string "(TEST)" at HERE .
The TARGET FORTH vocabulary is searched by TFIND .If the sought word is not found or if it is immediate, the result is dropped and the RUN vocabulary is searched. Once more, the result has to be for a word which is not immediate.
VFIND does not search parent vocabularies.

This version of the metacompiler allows changing the names of run-time words with ALIAS ; however, there is one downside.
Using DO and it's run-time as an example.
Code:
ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   ...
   END-CODE

There are now two handles on the host system for the same target word in virtual memory, this is not the problem. If the name of the run-time for DO on the host system does not match either one, a new alias is needed.
Code:
ALIAS DO   \ Name of run-time for new kernel. Creates handle DO in RUN vocabulary and header DO in virtual memory.
ALIAS D    \ Name of run-time in host system. Creates handle D in RUN vocabulary. header creation in virtual memory suppressed.
CODE (DO)  \ Name of run-time used in kernel source. Creates handle (DO) in TARGET FORTH vocabulary. header creation in virtual memory suppressed.
   ...
   ...
   END-CODE

This creates three handles for the same word in virtual memory, one in the TARGET FORTH vocabulary and two in the RUN vocabulary.
There is only one header created in virtual memory for this word since ALIAS suppresses header creation in the following word.


Top
 Profile  
Reply with quote  
PostPosted: Sun Sep 17, 2023 8:52 pm 
Offline

Joined: Fri May 05, 2017 9:27 pm
Posts: 895

I will now present the second solution, metacompiler B.
No new metacompiler vocabularies are added. This metacompiler's ALIAS does not create a target word. It creates a header in virtual memory. It does use NH to switch off header creation in virtual memory for the next word.
Code:
: ALIAS  ( -- )
   VHEADER NH
   TRUE HEADLESS +!  1 ALIASES +! ;

ALIAS is used in this metacompiler just like the other version.
Code:
ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   ...
   END-CODE

MCOMPILE was modified slightly so the search only succeeds if the word found is not immediate.
Code:
CODE MCOMPILE  ( -- )
   >FORTH                           \ Shift to high level Forth.
   ?COMP                            \ Abort if not compiling.
   R> DUP 2+ >R @                   \ Get the CFA of next word in Forth thread containing MCOMPILE
   >NAME COUNT $1F AND >HERE CLIP   \ and use the name field to create a search string at HERE .
   TFIND TRUE <>                    \ If not found or if immediate
   ABORT" RUN-TIME NOT FOUND"       \ abort.
   M, ;                             \ If found, compile to virtual memory.

With these modifications it is possible to create a kernel where each of the DO LOOP run-time words has the same name as it's compiling word. The next addition to metacompiler B will allow it to work when built on the new system.
The word :: (double colon) is an alias for the host colon. This will allow accessing colon's functionality while defining words in the META vocabulary, where colon was redefined.
Code:
: ::  ( -- ADR TRUE )
   : ;  // ACCESS TO HOST COLON

These words defined in the META vocabulary are stubs. They don't get compiled or metacompiled. MCOMPILE uses their names to find the corresponding word in the TARGET FORTH vocabulary.
Code:
META DEFINITIONS
:: (DO)   ;
:: (?DO)   ;
:: (LOOP)   ;
:: (+LOOP)   ;
HOST DEFINITIONS

These are metacompiler B's DO LOOP compiling words.
Code:
META DEFINITIONS
:: DO  ( -- >SYSL <SYSL )
   MCOMPILE (DO)
   >MARK 2+ 2+
   <MARK 2+ 2+ ; IMMEDIATE
:: ?DO  ( -- >SYSL <SYSL )
   MCOMPILE (?DO)
   >MARK 2+ 2+
   <MARK 2+ 2+ ; IMMEDIATE
:: LOOP  ( >SYSL <SYSL -- )
   MCOMPILE (LOOP)
   2- 2- <RESOLVE
   2- 2- >RESOLVE ; IMMEDIATE
:: +LOOP  ( >SYSL <SYSL -- )
   MCOMPILE (+LOOP)
   2- 2- <RESOLVE
   2- 2- >RESOLVE ; IMMEDIATE
HOST

Recall that the new semicolon will behave just like the host semicolon when metacompiling is off.
I could have defined these metacompiler DO LOOP compiling words with COMPILE since COMPILE gets patched when metacompiling.
This version of the metacompiler works well; however, there is a slight downside. If I want to rename the run-time for ." so it has the same name as ." , I will need to define a stub for the run-time and a metacompiler version of ." . The same is true for any run-time I wish to rename.
Code:
META DEFINITIONS
:: (.")   ;
:: (ABORT")   ;

:: ."  ( -- )
   COMPILE (.")
   ," ; IMMEDIATE
:: ABORT"  ( -- )
   COMPILE (ABORT")
   ," ; IMMEDIATE
HOST

or
Code:
META DEFINITIONS
:: (.")   ;
:: (ABORT")   ;

:: ."  ( -- )
   MCOMPILE (.")
   V," ; IMMEDIATE
:: ABORT"  ( -- )
   MCOMPILE (ABORT")
   V," ; IMMEDIATE
HOST

I thought of yet another approach to this problem and I think it will yield the best version of the metacompiler.


Top
 Profile  
Reply with quote  
PostPosted: Sun Sep 17, 2023 9:53 pm 
Offline

Joined: Fri May 05, 2017 9:27 pm
Posts: 895

Metacompiler C, the, hopefully, last solution to this problem.
The only differences with metacompiler B:
:: (double colon) is not defined.
MCOMPILE is different.
Code:
: CFA>STR  ( CFA -- HERE )
   >NAME COUNT $1F AND >HERE CLIP ;  \ Use CFA to generate a search string
                                     \ from a word's name field.
CODE MCOMPILE  ( -- )
   >FORTH
   ?COMP  R> DUP 2+ >R @ CFA>STR     \ Get CFA of next word in definition
                                     \ to generate search string.
   TFIND TRUE <>                     \ If the word is not found or is
   IF                                \ immediate
      DROP  R@ 2- 2- BODY>  CFA>STR  \ Back up to CFA of the word containing
      ASCII ) OVER COUNT + C!        \ MCOMPILE to generate a search string
      DUP DUP 1+ BL MOVE             \ and modify string to be enclosed in
      ASCII ( OVER 1+ C!             \ parenthesis.
      2 OVER +!  TFIND TRUE <>       \ Search the TARGET FORTH vocabulary
      ABORT" RUN-TIME NOT FOUND"     \ and abort if not found.
   THEN
   M, ;                              \ If found, compile to virtual memory.

CFA>STR takes the CFA of a word and generates a string at HERE which is the same as the word's name.
MCOMPILE still generates a search string based on the name of the word to be metacompiled by MCOMPILE . If the search is unsuccessful or the word found is immediate, MCOMPILE will generate a search string from the name of the word which contains METACOMPILE . This search string will be the compiling word's name enclosed in parenthesis.
Using DO as an example.
Suppose the run-time for DO on the host system is still named (DO) .
MCOMPILE generates the string "(DO)" and searches for this string in the TARGET FORTH vocabulary. It will be found and the address for (DO) in virtual memory will be compiled to virtual memory.
Suppose the run-time for DO on the host has a different name. Let's suppose it is something absurd like DO.RUNTIME .
MCOMPILE tries to find DO.RUNTIME in the TARGET FORTH vocabulary and fails.
MCOMPILE then generates a string from the name field of the compiling word, in this case DO . The string will have parenthesis added.
Here is a sample test.
Code:
: TESTWORD
   MCOMPILE FOOBAR ; IMMEDIATE

FOOBAR does not exist in the TARGET FORTH vocabulary. In this example, (TESTWORD) does exist in the TARGET FORTH vocabulary.
MCOMPILE tries to find the target word FOOBAR in the TARGET FORTH vocabulary. When that fails, it tries to find (TESTWORD) in the TARGET FORTH vocabulary and succeeds.
Run-time words with the same name in the host system and the source used by the metacompiler work fine.
Examples are ?BRANCH and BRANCH used by most of the control flow words.
Run-time words with different names in the host system and the source used by the metacompiler will work if the name of the run-time in the source is a parenthesized version of the compiling word's name.
Examples are (DO) compiled by DO and (LOOP) compiled by LOOP . These run-time words can be given a different name on the target being built in virtual memory by using the word ALIAS .

At present, I don't see a downside to this version of the metacompiler


Top
 Profile  
Reply with quote  
PostPosted: Tue Sep 26, 2023 9:17 pm 
Offline

Joined: Fri May 05, 2017 9:27 pm
Posts: 895

I recently came up with three variations of Fleet Forth's metacompiler. The third variation seemed the best way to go so it is Fleet Forth's metacompiler; indeed, if I'd thought of it first, I would not have bothered with the other two.
As I previously mentioned, the metacompiler patches the word COMPILE to MCOMPILE . This is so I don't have to define a metacompiler version of every word which compiles ?BRANCH or BRANCH , or words which contain such words. I also don't have to define a metacompiler version of some of the other compiling words.
Although the target being built in virtual memory can have redefinitions, a target word (a handle for it's counterpart in virtual memory) in the TARGET FORTH vocabulary can not be a redefinition of another target word.
To allow redefinitions in the target being built, the word ALIAS is used.
There is one run-time word in the kernel which could not be given an alias with the way the latest MCOMPILE was written. One thing to keep in mind, what I mean by this is that any word can be given an alias and the kernel will be metacompiled successfully; however, the Forth system built on the new kernel might not be able to host the metacompiler.
The following deals with using the metacompiler on the new system (I really don't want to maintain two versions of ITC Fleet Forth's kernel).
The run-time word (") , PAREN-QUOTE, is compiled by the word " , QUOTE. The source for QUOTE.
Code:
: "  ( -- )
   STATE @
   IF  COMPILE (") ," EXIT  THEN
   ASCII " CHAR ?MEM  PAD TUCK C!
   TUCK COUNT MOVE ; IMMEDIATE

If compiling, QUOTE compiles PAREN-QUOTE, it's run-time word, and compiles an inline string with ," , COMMA-QUOTE.
If interpreting, the text string in the text stream is copied to PAD and the address of PAD is left on the data stack.

PAREN-QUOTE could not be renamed to QUOTE.
When QUOTE is executed while metacompiling, the name for QUOTE would not be found in the TARGET FORTH vocabulary. I didn't mean a non immediate version couldn't be found, although MCOMPILE checks for that, no version would be found. QUOTE is not even defined in the Forth kernel, it is in the system loader.
MCOMPILE would not be able to back up to the name field of QUOTE to generate a parenthesized version of it's name because COMPILE is not the first word in the body of QUOTE.

The latest improvement to MCOMPILE uses AFIND .
AFIND is a word which takes an address and returns that address and the closest link field address below and the closest link field address above the given address.
Code:
AFIND  ( ADR -- ADR LFA1 LFA2 )

These are excerpts from the source used by the metacompiler.
Code:
ALIAS DO
CODE (DO)  ( LIMIT START -- )
   ...
   END-CODE

   ...

: DO  ( -- >SYSL <SYSL )
   COMPILE (DO)
   >MARK 2+ <MARK 2+ ; IMMEDIATE

The run time of DO is also named DO because of ALIAS .
When DO is executed, MCOMPILE would get the code field of the run-time DO and back up to it's name field. It would generate a search string based on that name and attempt to find a word with the name DO in the TARGET FORTH vocabulary. Failing to do so, it would get the address within the word containing it, DO , and back up to it's name field. MCOMPILE would generate a parenthesized version of the name and use it , (DO) , for the search string.
This is the latest source for MCOMPILE and two helper words.
Code:
: LFA>STR  ( LFA -- HERE )
   L>NAME COUNT $1F AND >HERE CLIP ;
: ?RT  ( FLAG -- )
   ABORT" RUN-TIME NOT FOUND" ;

CODE MCOMPILE  ( -- )
   >FORTH
   ?COMP  R> DUP 2+ >R @          // GET ADR OF NEXT WORD
   AFIND DROP TUCK LINK> = DUP    // DOES IT HAVE A HEADER?
   IF  DROP  LFA>STR TFIND  THEN  // IS IT IN TARGET FORTH
   TRUE <>                        // VOCABULARY?
   IF                             // NOT IN VOC OR HEADERLESS   
      R@ AFIND 2NIP DROP LFA>STR  // GET ADR OF WORD
      ASCII ) OVER COUNT + C!     // CONTAINING MCOMPILE
      DUP DUP 1+ BL MOVE          // USE NFA TO GENERATE
      ASCII ( OVER 1+ C!          // SEARCH STRING AND
      2 OVER +!  TFIND TRUE <> ?RT  // PARENTHESIZE
   THEN
   M, ;                           // IF FOUND, COMPILE TO VM

LFA>STR takes a word's link field address, converts it to the name field address and copies the word's name to HERE .
?RT aborts if there is a true value on the data stack.
This version of MCOMPILE will work even if the run-time in question has no header on the system the metacompiler is built upon.
MCOMPILE gets the address of the next word in the definition containing MCOMPILE , or COMPILE since COMPILE is patched while metacompiling. AFIND is used to find the closest link field address below this word. This LFA is converted to the CFA and compared with the CFA of the run-time word. If they differ the run-time is headerless.
If it is not headerless MCOMPILE uses it's name for the search string to search the TARGET FORTH vocabulary with TFIND , target find.
If the run-time is not found or it is headerless MCOMPILE uses AFIND to get the name field address of the word which contains MCOMPILE and generate a search string which is a parenthesized version of that name.

With this version of MCOMPILE it does not matter if a run-time word is headerless in the system upon which the metacompiler is built. MCOMPILE will generate a string which is a parenthesized version of the compiling word's name.
It also does not matter if COMPILE or MCOMPILE is not the first word in the compiling words body. Once again AFIND is used to find the word's name field address.

This is an excerpt from the metacompiler source.
Code:
META DEFINITIONS
: (>ASSEM)   ;
: >ASSEM  ( -- )
   ?METACOMP
   ?COMP  MCOMPILE (>ASSEM)
   [ASSEMBLER] MEM ASSEMBLER
   [COMPILE] [ ; IMMEDIATE

The metacompiler has to define it's own version of >ASSEM , even though COMPILE is patched, because it has to set CONTEXT to the metacompiler's ASSEMBLER vocabulary.
Notice the dummy word (>ASSEM) . This stub is defined because the name (>ASSEM) is no longer found in the new system. The run-time for >ASSEM also has the name >ASSEM .
Also notice the metacompiler's version of >ASSEM has MCOMPILE in it's definition.
With the new version of MCOMPILE it is possible to redefine the metacompiler's version of >ASSEM
Code:
META DEFINITIONS
: (>ASSEM)   ;
   ?METACOMP
   ?COMP  MCOMPILE [ 0 , ]
   [ASSEMBLER] MEM ASSEMBLER
   [COMPILE] [ ; IMMEDIATE

MCOMPILE will treat the inline zero as a headerless word and generate the search string (>ASSEM) from the name field of >ASSEM .


Top
 Profile  
Reply with quote  
PostPosted: Thu Sep 19, 2024 1:54 am 
Offline

Joined: Fri May 05, 2017 9:27 pm
Posts: 895
Take a good look at the disassembly and decompilation of the ABORT" runtime for my Forth. Yes, in my Forth, ABORT" and its runtime have the same name.
Code:
ABORT"
  7740          INX
  7741          INX
  7742   254 ,X LDA  W
  7744   255 ,X ORA  W 1+
  7746  7707 ^^ BEQ ' " >BODY 4 +
  7748  3378    JSR ' >FORTH >BODY
  7751  7921 WHERE
  7753  6778 CR
  7755  4789 R@
  7757  7169 S?
  7759  8742 ABORT
21

In the high level portion of the decompilation, the first number on a line of decompilation is the address in the decompiled word. The second number is the CFA of the word in the definition of the decompiled word.
Code:
   --------------- address within the body of ABORT" runtime
   |     --------- Code field address of ABORT
   V     V
  7759  8742 ABORT

Notice in this fragment two of the words used by the runtime are defined after the ABORT" runtime.
Code:
  7751  7921 WHERE  \ Defined after ABORT" runtime.
  7753  6778 CR
  7755  4789 R@
  7757  7169 S?
  7759  8742 ABORT  \ Defined after ABORT" runtime.

With my Forth's older version of INTERPRET , the ABORT" runtime was needed by INTERPRET to ABORT" if a string which was not found in the dictionary could not be converted to a number. INTERPRET was needed by QUIT which was needed by ABORT which was needed by the ABORT" runtime.
Looking at this fragment and noting that WHERE and ABORT are defined after the ABORT" runtime might lead one to believe this Forth was assembled. This is not the case. It was metacompiled. Here is the source for the ABORT" primitive.
Code:
ALIAS ABORT"
CODE (ABORT")  ( F -- )
   INX  INX
   $FE ,X LDA  $FF ,X ORA
   ' (") >BODY 4 +  0= BRAN
   >FORTH
   LABEL DO.WHERE NOOP
   CR R@ S?
   LABEL DO.ABORT NOOP -;

And the high level portion.
Code:
   LABEL DO.WHERE NOOP
   CR R@ S?
   LABEL DO.ABORT NOOP -;

Each label holds the value of THERE , the target system's HERE , when it is created. The word NOOP is used as a placeholder.
The following shows the resolution of the address at label DO.ABORT
Code:
: ABORT  ( -- )
   ERR SP! AP! QUIT -;
' ABORT DO.ABORT !

The resolution for DO.WHERE is similar.
That is how to handle forward references in the source used by the metacompiler without resorting to creating a deferred word in the target for something which should not change.


Top
 Profile  
Reply with quote  
PostPosted: Sun Nov 10, 2024 9:38 pm 
Offline

Joined: Fri May 05, 2017 9:27 pm
Posts: 895
My Forth, an ITC Forth, has a few words with no body and a code field which points into the body of a primitive. The definition of these words follows this pattern:
Code:
CODE <NAME>
   <ADDRESS>  LATEST NAME> !
   END-CODE

Here is an example:
Code:
CODE ON  ( ADR -- )
   DEY  TYA  DEX  DEX
   ' SWAP! @ 4 + 0= NOT BRAN
   END-CODE
CODE OFF  ( ADR -- )
   ' ON @ 1+ LATEST NAME> !
   END-CODE

This metacompiler technique does not work if the word is to be headerless. Although there is a word with a header on the host to serve as a handle, a headerless word has no header in the target, which is being built in virtual memory; therefore, the metacompiler versions of LATEST and NAME> do not work.
Since the metacompiler version of ' (tick) does not require the presence of a header in the target, the following technique works.
Code:
NH
CODE <NAME>
   END-CODE
   <ADDRESS>  ' <NAME> !

In the previous example, OFF could be defined like this:
Code:
CODE OFF  ( ADR -- )
   END-CODE
   ' ON @ 1+ ' OFF !



Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4

All times are UTC


Who is online

Users browsing this forum: No registered users and 10 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: