In this post from Garth Wilson he discusses a little bit about how he embeds 6502 assembly language into his Forth programs. I responded mentioning that I do this in Python and was interested in hearing more about his techniques. This started getting off-topic for that particular thread (which was actually about avoiding assembly by embedding machine language), so I'm starting this thread to continue to the assembly ideas.

Unlike the other thread, this one is not limited to embedding and/or manipulating assembly in just FORTH; discussion of doing this in any higher-level language is welcome.

The next two posts are Garth's post linked above, quoted by me, and my follow-up post.

_________________
Curt J. Sampson - github.com/0cjs

I'd definitely be interested in hearing about this. I'm not sure how directly applicable it would be to my work with 6502 "assembly" of sorts in Python, but what you've shown so far is pretty much exactly what I do, and I'm contemplating how to deal with the next step. Here's a unit test that shows how I currently code things up:

I use LDA instead of LDAi for immediate load because it doesn't collide with anything else, but, just as you do, I use suffixes for other addressing modes: LDAm, LDAz, LDAmx, LDAzy, etc.

One idea I'm kicking around is to continue to use a list, use string elements for defining and referencing labels, maybe add some pseudo-ops, and feed the list into an assembler function. And one might allow 16-bit values in the list and change the LDA processing to pick an appropriate addressing mode.

So what does your system handle now, and what are your thoughts on how you'd expand it to have more functionality?

_________________
Curt J. Sampson - github.com/0cjs

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

Sadly, that's so brief that I think it may be clearly comprehensible only to FORTH programmers. :-) I have no objection to heavy discussion of FORTH in this topic; I put it under "Programming" rather than "FORTH" just because I didn't want to limit the discussion to FORTH. (For example, I think that BBC BASIC may have some good ideas that could be discussed here.)

I should also emphasise that I'm not talking about inline assembler here (i.e., assembly code interspersed with and executed as part of the host language code, though some uses of that may be related) but actually the embedding of assembly code in a higher-level language in the manner of what's usually called an embedded domain-specific language; i.e., usually using the host langauge syntax or something that can easily fit with that. This is typically used to produce reified objects that can then be manipulated as necessary.

For example, in your post I quoted above you used LDA_ABS FOOBAR (note that there's no trailing comma there!) to define an assembly language fragment, and then at that point it can be assembled in a variety of ways, and presumably various things could be done with it afterwards, if I'm understanding your explanation correctly. ("It is up to you...to use the right word to compile the operand (whether comma or C-comma).")

And in my Python example above, I use a pseudo-assembly syntax to generate a list of bytes, which I then deposit and call. That is not inline at all; if there is a way to do inline assembly in Python, I certainly don't use it, and it wouldn't even work since my Python interpreter is running on an x86_64 processor. (The 6502 code above is run in a simulator.)


Are these secondaries assembled with a separate stand-alone assembler, or within the Forth system?

_________________
Curt J. Sampson - github.com/0cjs

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

I'd quite like to understand better the distinction you're drawing here. Is it that the assembly text produces a lump of binary data which can then be used in the program? That is, it's not to be executed at the point it's described, but it's to be stored, passed around, eventually put together with other code and then executed?

Yes, that's pretty much the core of one part of it.

The second part that is the key here (and where the "manipulation" comes in) is that with what Garth and I have been showing, the pieces of assembly are first-class objects within the host language. So, for example, if I need ten NOPs I can trivially generate that using the host language: in Python, [I.NOP] * 10. A slightly more sophisticated example from my post above is the use of host language variables and functions within the assembly code, such as addr = 0x400; asm = [ I.LDA, LB(addr-1) ], which takes an instruction and the low byte of a calculated value to give you a list of binary values, [ 0xA9, 0xFF ].

The most obvious way of "embedding" assembly code in a language is to simply write code in strings, e.g. A$ = "LDA #$33" in BASIC. But with just plain old strings like that, you essentially end up writing a separate interpreter to interpret those strings, and getting the kind of power with this that you see in the examples above is a huge amount of work, rather than most of that coming for nearly free when you use an embedded DSL in a reasonably powerful language. (As Alan Perlis points out in Epigram 34, "The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.")

Currently, my embedded "assembly code" (and Graham's, I think) is pretty darn simple, hardly more than assigning names to integers. So the question I'm interested in investigating is, how does one move beyond that to start handling things such as, say, labels. (Hand-counting entries in a list and then based on that setting addr =0x40d, as in the test in my previous post, is clearly neither reliable nor scalable.) The temptation is to start writing interpreters for those lists (essentially, writing an assembler), but go too far that way in a certain direction and you end up with just another assembler that you're calling from your host language, rather than having an actual embedded assembly language.

_________________
Curt J. Sampson - github.com/0cjs

Embedding assembler into BBC Basic is "a thing" that's been there for a long time. You have to construct the 2-pass assembler using a BASIC FOR loop and the code generated can be stored and run from a byte array, or saved to disc, etc.

You can use BASIC procedures to create macros too and some quite complex systems have been built using it.

Some systems I built used the BASIC assembler to assemble and save code which was then loaded and run from disc by a 2nd program - mostly due to memory constraints, but this was really not that different from using a system that might have had a separate editor, assembler and loader - but the editor and assembler was the BASIC environment.

The BCPL system for the BBC Micro also had a separate relocatable assembler that allowed to you edit and assemble code which could then reference BCPL global variables and be called from BCPL as if it was a BCPL function. Like BBC BASIC, this was all self-hosted on the BBC Micro. I'm working on something similar for my own Ruby BCPL system - again, self hosted.

Doing it in C using cc65 and as65 is not hard either, but does require a separate system to edit, compile/assembler and link the code to produce a binary to run on the target.

-Gordon

_________________
--
Gordon Henderson.
See my Ruby 6502 and 65816 SBC projects here: https://projects.drogon.net/ruby/

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

Garth, that was an excellent reply, just the kind of thing I was looking for. (Though I did have to go and do a bit of research to remember what Forth immediate words words are. It has been many decades since I have typed even a word of Forth.)


Yes, that's pretty much exactly where I'm at, too, though perhaps with one caveat. My unit test system loads an entire assembly (which is potentially huge) into simulator's memory and also returns and the assembly's symbol table as a Python object. So I am often using those symbols in new code I assemble and add to the loaded assembly.

From looking at what you've written, it seems that your assembly system is working very much the way the standard Forth word compiler works: you start with CREATE ____ PRIMITIVE to start building a new word and then non-immediate words get compiled, immediate words to deal with things like branch targets get executed, and then , finishes off the whole thing. Is that right?

And though your examples use this along the lines of inline assembler, it seems to me that one could of course use a different dictionary and produce code for another system, in words that one would save and then load into the other system. So you could do this assembly in Forth running on a modern PC, producing words for a 6502 system, right? (Isn't this essentially how a Forth metacompiler works?)

Also, do you have any way to mix Forth and assembly compilation within a single word, or does a word have be one or the other? Can assembly code call into Forth code?

_________________
Curt J. Sampson - github.com/0cjs

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

With such clear and descriptive variable names as STATE, how could anybody get confused? :-P

Anyway, I think I see what's going on here. Let me know if I've got any major errors in the following summary of my understanding of the Forth part of this.

Essentially there are two main interpreters in Forth. The first is the runtime interpreter (let's call it "interpreter I," unless there's a more standard name for it) which reads lexical tokens, looks them up in a dictionary (I am eliding parsing of numeric literals for the moment), and immediately executes the routines to which they refer. Then there is the "compiler" interpreter ("interpreter C") which reads lexical tokens and looks them up in the dictionary, but then checks to see if they're marked "immediate" or not. If the word is marked immediate it's executed just as with interpreter I, but if not a reference to the word is instead appended to a buffer it's using to build a new word to be added to the dictionary.

So a when interpreter I reads a ":", interpreter C is started, which reads the next lexical token as a name for the new word, allocates a buffer somewhere in which to build the word, and then continues parsing as described for interpreter C above. Eventually it reaches a ";" and, upon reading that, does whatever it needs to do to finish up the generation of the new word and then returns control to interpreter I.

(As a side note, this idea of multiple interpreters all sharing lexical analysis and data about the current system but interpreting identical series of lexical tokens in various different ways is central to Lisp, too. Central even to Lisp without macros, I mean; macros add a whole new level to this of course.)

Presumably it would be possible to add further interpreters that could also take over reading of the input stream and do their own thing, if they wanted to. (This may or may not require some changes to the core of the interpretation system.) One could use such a technique to write an interpreter for an assembly language compatible with Forth's lexical analyzer.

But in your case you've decided to embed assembly in a different way, by having words read and executed by interpreter I, which set up and use their own data areas to build and register new Forth words. And "C," and "," (used to generate instructions and finalize the new word, respectively?) are nothing special to the interpreter but just words to be called like any other.

This is very interesting, and I realize now that a similar technique is available to me in Python. Currently I seem to be heading towards the "separate interpreter" approach, where I build up a program in the language and then apply an interpreter function to it, e.g.,


But an alternative would be to do the assembly on the fly using the Python interpreter itself to drive this execution:


A notable difference here is that this is latter approach seems a lot less readable in Python than it is in Forth. And of course, like Forth, this approach seems limited to being a one-pass assembler. (Though I suppose you could instead treat this as an intermediate object and ask the user to call .assemble() on it to get the assembly output.)


Well, perhaps not for optimization, but there are other good reasons to have in-line assembly, such as calling a routine written in another language or, perhaps, direct access to hardware.

_________________
Curt J. Sampson - github.com/0cjs

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

I just figured out how switching parsers works; it's explained on the Wikipedia Forth page at Parsing words and comments. Basically, words are able to consume the current input stream from the point just after they are executed, do what they will, and then when the main parser resumes it will continue after the text consumed by the parsing word.


Well, that's fair enough. Essentially, the "buffer" being filled is the target area of memory where the word will reside, then. Pretty much the same general idea.


Sure; for simple things where the equivalant of BASIC's PEEK(), POKE, IN() and OUT will do the trick, that's fine. But I was thinking of things like using the Apple II disk controller directly, where timing constraints are critical (certain loops must be a specific number of cycles), or switching in different bank of memory and copying something from it (where the Forth interpreter may no longer be mapped into the address space). For these sorts of things it may be possible to do them only with machine code.


Yup, I'm used to that. Lisp is similar; in the expression (- 5 3), the minus sign is just another variable name that happens to be bound to a particular function. You could even have fun confusing people, if that turns your crank, by using (let ((+ -)) (+ 5 3)) to do that same subtraction.

(The reader is in fact just an ordinary Lisp function, so you're free to replace it with something else if you like using, I don't know, BEGIN and END instead of ( and ), or even want a totally different syntax. Perhaps some people would find more pleasing a reader that interprets ' + ' - let 5 3 + as my let subtraction example above.)

BTW, just out of curiousity, what character set and encoding are you using on your system with those Greek characters and whatnot, and how do you deal with displaying them?

_________________
Curt J. Sampson - github.com/0cjs