Having exchanged a few posts regarding how macros should be handled, I can't stop thinking about it. I think it's worth a separate topic.

It seems that every assembler I've worked with got some part of it wrong (and I've sifted through dozens of assemblers, tossing them out as unusable due to some or other misimplementation). I am sure there were reasons for the decisions made, but I can't help thinking that at least some of them were made to workaround problems created by other poorly thought out decisions. Having written assemblers and compilers in the past, I personally have made some poor choices simply because I failed to understand the problem at hand and was invested in code that should have been re-written.

So I will propose a logical framework for working with macros. If you can think of something you do with an assembler that my method cannot handle please let me know. I would prefer to avoid a religious fight over syntax or 'that's not how assemblers do it'. I also do not mean to suggest that my way is the 'right' way. I just want to explore the simplest possible way of doing macros that makes sense.

I am going to do my experiments in python as it has adequate facilities for memory management and expressive power (I want to avoid the usual memory bugs and focus on higher-level functionality). My experience with python is limited to about 3 hours, so don't expect much. Although I generally avoid things like that, I find that python is pretty easy to understand. Python is also interactive, and allows execution of strings, which makes it very flexible as an extension language.

Perhaps the macro subsystem can be made flexible enough to implement the assembler in.

Summary

There exists a memory space initialized with the file being assembled. Macro expansion proceeds in one pass, with characters parsed from SRC, the source pointer. As words are parsed (according to parsing rules), they are inserted at SRC, unless they match a name in the macro dictionary. A macro match triggers an expansion of the macro into SRC, following parameter substitution rules. The new text will be processed as any other text. Only one pass is required. Note: the implementation may work differently, but conceptually equivalent to a memory space where expanded macro contents get inserted ahead of SRC.

Parsing rules

Identifiers are terminated with a whitespace or a special character ',' ';' [I will update this as needed]

Parameter rules

This is a little tricky. For assemblers, a comma-separated list makes sense, although what terminates it? <CR>, ';' perhaps. There should be a default value, and the possibility of skipping, as well as a variable-length list. I think named parameters are a bad idea and no assembler I've used implements it.

Error handling

The assembler will get enough information to generate error messages (as a stack of context records containing filename, line number and position within the line, at least).
EDIT: I forgot about my idea of appending the state information (file/line number/position) to the end of the comment on each line.... Although Kingswood assembler chokes without writing any of the listing, so it would not help anyhow.


Listing support

I think the macro engine should remove the comments for the assembler. The listing file can be generated by using the original source line (inserting the data fields necessary). Depending on settings, macros can be expanded into the listing as well. This requires a little more thought to get it right.

Intrinsic support

Include and other built-in keywords need to coordinate with the macro engine. Inclusion is handled as any other macro expansion.

Symbol handling

Since we are dealing with macros only, everything is text. A macro definition is stored in a Forth-like dictionary, with later definitions overloading older ones with the same name. An incomplete macro definition is not visible. A definition within a definition is handled simply: the inner definition is treated as any other text and no new macros are defined. When the outer macro is expanded, the newly-expanded text will contain the inner macro definition, and the macro will be created when that text is processed.

I think it would be useful to have a 'forget' command, perhaps? Maybe not needed.

Ideally the macro system should be completely assembler-agnostic and capable of anything (is it possible to keep the macro defining and ending words flexible, so a macro can be ended with an 'endm' or 'end macro' or '}'?). If that is not possible, I would settle on a sensible syntax that is usable.


Should the macro engine be aware of anything beyond text expansion? Should it know about lines, or fields within a line? I hope not.

I think that's enough to do just about anything. If I missed anything or if you have syntax ideas please let me know. Please remember that the point of this exercise is to keep it simple and to avoid special cases, but sufficient to do anything that we may need to do with macros.

_________________
In theory, there is no difference between theory and practice. In practice, there is. ...Jan van de Snepscheut

Take care to ensure that recursive macros are still possible. It sounds like it might be, but it should be an explicit requirement.

You need to ensure that at least line numbers will be retained throughout the pipeline for good error reporting. This means that the output of a macro system is not just expanded text, but requires annotations somehow to note what file & line number the bits came from.


If you really look at it, every assembler's macro & state system really wants to be a full programming language, managing assemble-time variables for anything from building labels & tables to full on code generation. Being able to generate code from code is the all-encompassing ideal; any deviation from that is making hopeful tradeoffs. If you're serious about a macro system that will not be constrained, and you're looking at starting to use different tools anyway, Lisp is king in that field. Python borrows a fair amount from Lisp, so it shouldn't be too foreign.

While there are very few tools around for general Lisp-based assemblers, here's a snippet I found via google: link
(Of course, Lisp implementations are generally native-speed compilers nowadays.)

This page is quite technical, but writes a macro assembler in Lisp targeting the NES. While some of the main code looks like fairly straight 6502 code, the music for the demo ends up looking like this:

That's basic macro expansion that does all the right thing and has full programming language power inside the macro expanders. It's much higher level than you'd get from other macro options, and is still straight assembly instruction & byte-level definitions under user control.


tl;dr: Write a 6502 assembler as a DSL inside an existing programming environment that already does all the cool & solid macro stuff.

_________________
WFDis Interactive 6502 Disassembler
AcheronVM: A Reconfigurable 16-bit Virtual CPU for the 6502 Microprocessor

It absolutely a requirement. However, there is a limitation: there must already be a macro by that name; otherwise the expression is nonsensical. Example:

Without the initial definition, there is no macro named test and macro expansion fails. Since the macro definition has not been closed, there is no sensible way to expand the macro.


Yes, exactly what I was thinking. The more I think about it, the less of an issue the assembler itself is - writing an assembler is tedious but not hard if the macros are taken care of.


While beautifully stated, I think I disagree just a little with the first sentence. I think it's possible to look at the macro subsystem strictly as a text transformation engine, completely unaware of the language. While it may make sense to comingle symbol tables and state information for the sake of error reporting or overall integration, the macro system should really be independent.

I have deep respect for LISP from a distance, having never delved into it. It does look very tempting at times like these...

_________________
In theory, there is no difference between theory and practice. In practice, there is. ...Jan van de Snepscheut

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

Good point, it has to be aware of strings and perhaps inline comments and skip them.

Yes, this is an issue. Definitions should be allowed to set default values, and I am inclined to use python-like 'None' to indicate that no parameter is being sent.

By named parameters I mean verilog-style named parameters. You can specify out-of-order parameters by name given in the macro definition. This makes the code long-winded, error-prone and annoying when someone else names parameters. Naming parameters inside the definition is for another day...

Arrays and fields should be handled by the assembler itself, in the expression parser.

Thanks for the comments.

_________________
In theory, there is no difference between theory and practice. In practice, there is. ...Jan van de Snepscheut

That's not a recursive macro. That's redefining a macro, just with the old one still in scope. If a macro body contains any sort of conditional, then a truly recursive macro will be able to terminate finitely. However, that means the conditional is evaluated as part of the macro system. This is why it's hard to separate out a macro system from a programming environment; the things that make the decision on what to include or exclude need to be informed and capable.



ca65 has some really good support for that, too. But what if you want to say "Expand this for each of the remaining macro parameters"? Again, the context of the macro invocation is something that contains state and invites making decision on that state, in order to determine what to expand. But I do think you are already in agreement with "macro language = capable programming language", based on your last paragraph.



The reason for the macro system having this is so that separate macro invocations can be aware of what's going on with each other. It's really the only way to have, for instance, a set of FOR/NEXT structure macros, because they need some storage to remember what was where in the source code, and how they nest. What do you think about the inability to have such macros? Is that within the scope of your tradeoffs in defining a simple but sensible macro system, or is that a capability that you'd like to have in there?

_________________
WFDis Interactive 6502 Disassembler
AcheronVM: A Reconfigurable 16-bit Virtual CPU for the 6502 Microprocessor

As I see it, it is not really sensible for a macro to invoke itself inside its own definition. That would require a multipass 'magic' that is confusing. Maybe I am wrong, but what could it mean to invoke a process that has not yet been described? The only sensible meaning I can think of is that the macro is invoking another macro with the same name, hopefully defined previously. FASM works that way and I use this feature all the time to define macros that contain the previous definition along with some new items.

I still think that arrays and structures should belong to the assembler, not the macro engine.

You do have a point about FOR/NEXT and other complex macros/keywords. It appears that there has to be a general purpose processing component to allow macros to be smart. Perhaps Python should be used for that. I have to think about how to do that before embarrassing myself even more...

Thank you for pointing out these things to me!

_________________
In theory, there is no difference between theory and practice. In practice, there is. ...Jan van de Snepscheut

White Flame, or anyone, please could you give a useful example showing why a recursive macro would be worth having?
Thanks
Ed

_________________
http://WilsonMinesCo.com/ lots of 6502 resources
The "second front page" is http://wilsonminesco.com/links.html .
What's an additional VIA among friends, anyhow?

Another feature one might accept or reject as important: forward references. I think enso has said that a macro must be defined before it is used. Is that a restriction anyone finds important?
Cheers
Ed

Forward references for regular symbols, such as calls to procedures not yet defined, are pretty much expected for an assembler. They are not technically necessary (we can write code bottom up, as in Forth) but nice. Forward references for fixed-size symbols can be resolved in a single pass using 'fixups'.

Forward references for macros are probably not wise as it's impossible to tell what will happen, and will require multiple passes (which I am trying to avoid). On the other hand if any kind of forward references are allowed, it would be consistent to allow macro forward references.

I may be wrong about the macro engine being text only and unaware of the underlying semantics. For instance, it may be good to do conditional assembly based on the location or value of symbols from the assembler domain. But that creates a paradox unless multiple passes are done and the macro expansion is just a part of the assembler pass. In this case I would need to keep track of constants and fully resolved symbols which may be used for macros, and if a macro expansion is required and relies on unresolved symbols, an error results.

My initial attraction to a single-pass macro pre-processor may just be not possible to implement for a usable system. On the other hand, if I strictly prohibit forward references it may still be as usable as Forth (which is pretty usable).

FASM will do as many passes as necessary (I've seen it do over 10) to resolve all issues. It is not very sensible - for instance 'eq' and '=' assignments are done at different times, handling of constants in expressions is almost reasonable but sometimes weird, and macro processing is usable but sometimes _really_ strange.

The whole thing, as I suspected, is a snakepit. The more I think about the less likely that a 'sensible' solution exists. In fact I am leaning more toward Forth the more I think about it.

_________________
In theory, there is no difference between theory and practice. In practice, there is. ...Jan van de Snepscheut

Regarding recursion, it's often used in place of loops for handling varying numbers of parameters, or managing numbers in table generation until they reach some conclusion. Recursion tends to have more control over what happens to each variable in each iteration than regular loops and can be much more organized, but it's usually technically equivalent to the imperative loop form.

Enso: I would avoid thinking about a fixed number of passes, and just structure it to let it work until it's done. That usually involves running more passes until no changes happen like you mention, or less often having a set of dependencies/constraints/tasks to finish. Fixed passes are a bit conceptually easier in terms of writing the structure of the program, but lock in hard limits when the users start to really push it.


It wouldn't necessarily need addresses. If it can create new unique labels, inject those into the source, and remember what the names of those labels were on the context stack, it could be completely agnostic to the actual numeric addresses. (unless that's what you meant already)

_________________
WFDis Interactive 6502 Disassembler
AcheronVM: A Reconfigurable 16-bit Virtual CPU for the 6502 Microprocessor

I know I've blathered on about a lot of high level stuff, but I just realized that I didn't describe it in the simple way I normally do:

Macros are assembly-time function calls, whose return value is source code.

Now, most macro systems have lots of syntactic sugar to allow you to write the literal source code that it should return, with some injected functionality. But you could imagine that, as a function call, it could really do anything to build the source code it's going to return, including managing internal state and building up source code from textual strings or AST pieces. Assembly time is the "runtime" of the build system, and that runtime has decisions to make & state to track just like the final assembled program runtime.

_________________
WFDis Interactive 6502 Disassembler
AcheronVM: A Reconfigurable 16-bit Virtual CPU for the 6502 Microprocessor

I don't think either functions or macros ought to be managing internal state.
That's too "meta"
Frankly I don't want an assembler to be a dynamic language.
If you want that you ought to go to something higher level
and not mess up a perfectly good assembler

But obviously this is what happens, to a point.

Many macros rely upon and set internal flags. Reseting label values, for example.

Having simple symbol substitution and basic math are pretty handy, and can go a long way.

But I agree, you don't need something really sophisticated for the vast majority of use cases.