Meta Compiling Forth

[whartung]

Chatter in the "What is Forth" thread brought up the potential to talk about Forth meta compiling, so I figured I'd open one up

Here's a nice introduction for folks coming in who may not know what we're talking about:

http://www.ultratechnology.com/meta.html

Then here are some links to some articles about implementations.

First, is Henry Laxen's series in 4th Dimensions:

http://www.forth.org/fd/FD-V04N6.pdf
http://www.forth.org/fd/FD-V05N2.pdf
http://www.forth.org/fd/FD-V05N3.pdf

And then, there's Brad Rodriguez's treatment as well:

http://www.forth.org/fd/FD-V14N3.pdf
http://www.forth.org/fd/FD-V14N4.pdf
http://www.forth.org/fd/FD-V14N5.pdf

Also, Frank Sergeant's paper from 4D

http://www.forth.org/fd/FD-V12N6.pdf

I found these from Brad's other smaller paper on Meta Compiling:

http://www.bradrodriguez.com/papers/moving4.htm

Figured it might be nice to put them in one place.

[BigEd]

Thanks for collecting those links - some good reading ahead!

[barrym95838]

Yes, thank you very much Mr. H. I am not yet qualified to comment, but I will be checking out those links and following this thread with interest.

Mike B.

[nyef]

I came up with much of my metacompiler design before I had much of an idea as to how other metacompilers worked, and I seem to have ended up with some unusual design features.

• The target image is built in dictionary space, rather than in a separate chunk of address space. Once the metacompiler is loaded, anything added to the dictionary ends up in the target image.
• Instead of having memory operations for accessing target space, I have separate operations for working with addresses instead of data (applying a displacement to convert target pointers to host pointers and back).
• The metacompiler searches the target image dictionary directly, rather than having a separate host wordlist for target names. There used to be a post-build step that ran through the dictionary altering host pointers to target, but that was too hacky to keep and wouldn't've worked when building a target of a difference cell width than the host.

At one point I started over with a new implementation as a Linux-hosted system, instead of continuing as a standalone implementation. I used (32-bit) Jonesforth as the initial bootstrap host, which was fairly dreadful, and once I had the system up and working enough, and the target system converted to conform closely enough to the ANS spec, I added support for building from (64-bit) gforth. Supporting three host environments (including self-hosting) required some work to either paper over the differences between the host environments or to stop doing unportable things, but is far better than requiring Jonesforth as the only host system.

Adding support for a DTC target as well as the original ITC target, especially with respect to DOES>, involved some cleverness. The target configuration file is loaded before the metacompiler, since the metacompiler needs to refer to it during build, but it includes definitions which need to refer to parts of the metacompiler. So there are several places where, in the middle of a definition, we leave compilation mode to use a word that's part of the target configuration, and that word is basically a literal string and a call to INTERPRET, and the code that gets interpreted re-enters the compiler to add something to the word currently being defined.

I am hoping at some point to add support for 65816 and possibly 6502 standalone targets, producing ROMmable code, but that will need to wait until I finish getting my '816 system working.

[dwight]

Chuck Moore wanted to get away from using vocabularies in the normal sense when he wrote the Forth for the NC4000. For me, vocabularies and search order are really handy for meta compiling and also target compiling.
In the NC4000 Forth, he only has 2 vocabularies. One was for interpreting and the other was for compiling words.
To meta compile, one would recompile in ram someplace then meta compile with offset for the target using the just previously create version.
It seem strange at first but made sense later.
I have a NC4000 that I connected a couple XT boards to for hard drive and floppies. I can meta compile from the hard drive to create a completely new forth in about 45 seconds, ready to program on a EPROM.
Dwight

[larsbrinkhoff]

nyef wrote:

I came up with much of my metacompiler design before I had much of an idea as to how other metacompilers worked

• The metacompiler searches the target image dictionary directly, rather than having a separate host wordlist for target names.

I used the same design, for the same reason. The target FIND code was included in both the metacompiler and the target kernel.

In the end, I decided that the conventional target shadow vocabulary technique was simpler.

[nyef]

larsbrinkhoff wrote:

I used the same design, for the same reason. The target FIND code was included in both the metacompiler and the target kernel.

In the end, I decided that the conventional target shadow vocabulary technique was simpler.

I've been thinking about this, off and on, and ISTM that the shadow vocabulary technique makes supporting different dictionary formats a lot simpler, even if it's "just" a matter of machine word width.

Of course, now you have a problem if your host forth doesn't actually support multiple vocabularies/wordlists. I seem to be in a situation where one of my hosts doesn't support any such concept, the other probably has wordlists and might have vocabularies, and my target (which is also my third host, part of the build process is a self-build) has a "VOCABULARY" word which I've never really needed before and I'm not sure actually works properly, but no other search-order support.