Overall, I like your system, I'm always a fan of consistency, however I do have some thoughts & suggestions. Please make of them as you will.

1. Though I agree that using a hash prefix is not necessary for immediate operands in your syntax, I think using it makes it easier for a programmer to tell, at a glance, if something is supposed to be a literal number or a label, particularly when such is indirected through an equate, so the number does not actually appear in the instruction's source. Since I can't see it doing any harm either way, you could make it optional; would also make it slightly easier to transform source code from the traditional syntax to your new one.

2. Whilst I have no problem with keeping the existing Branch instruction mnemonics, perhaps it might be possible to have an additional syntax along the lines of



I think this syntax more closely fits with the rest of your mnemonics, where the operation is kept seperate from the operands. I've always felt that the operation of the Bxx instructions is simply "Relative Branch": the nature of any condition or length of branch is an operand of that branch, but the fundamental operation is still the same. I also, personally, find the non-straightfoward mnemonics unhelpful (e.g. what do BPL, BMI, BNE and BEQ actually mean in terms of flags? I have terrible trouble remembering and normally have to look it up in the programming manual).

3. Similarly, I would suggest XCE -> SCE, in order to match with XBA -> SH, just to keep the "Swap" nomenclature consistent.

4. As the WDM instruction is explicitly stated to be intended for future expansion (even if that's increasingly unlikely to ever happen), I suggest giving WDM an explicitly required signature byte (which could default to a NOP if not specified, if you like), whilst NOPL assembles as a 1-byte instruction for skipping the following, or even rename the latter SKIP to make it even clearer exactly what its for. That way, I feel, the instruction called WDM more closely resembles the intended purpose of the instruction of the same name in the traditional syntax.

_________________
Want to design a PCB for your project? I strongly recommend KiCad. Its free, its multiplatform, and its easy to learn!
Also, I maintain KiCad libraries of Retro Computing and Arduino components you might find useful.

At present, the main use of WDM is as a 2-cycle, 2-byte NOP, as an alternative to the official 2-cycle, 1-byte NOP. A secondary use by experienced programmers is as a "skip", avoiding the need for an additional branch over a single-byte "else" structure. There are no known concrete plans to introduce 65xx family CPUs that use WDM for any other purpose, so I chose the syntax to support these existing uses. But if and when that happens, it will be straightforward to modify the assembler to directly support the new instructions, removing the need for programmers to include kludges in their assembly source.

I already have B by itself as a replacement for BRA/BRL. It's possible that BMI/BPL could have synonyms as BNS/BNC, and BNE/BEQ as BZC/BZS. Then you only need to remember that C is Clear and S is Set if you're thinking in terms of flag states. The use of two-letter suffixes to make an instruction (usually a branch, sometimes a data move, occasionally more generally) conditional is established in both ARM and PowerPC assembly; I don't actually think of the flags as operands to the instruction. ARM has 16 such suffixes representing 14 unique conditions besides "always", many of which involve combinations of two or more physical flags, and still uses EQ/NE/PL/MI for the same purposes as the 6502. Along the same lines, I could introduce BHS/BLO as synonyms for BCS/BCC, following ARM practice.

One thing that XCE emphasises, with the unique X as its initial, is that it's the means to switch between Emulation and Native modes, which is a very unusual and significant operation - indeed that is its only purpose. Switching to an S initial would dilute that.

I don't currently plan to allow # to distinguish immediate operands. It should be sufficient to observe that the operand isn't surrounded by brackets. One advantage is that this allows using a C preprocessor as a macro engine instead of implementing one from scratch.

All your counter points seem reasonable; and your suggestion of including appropriate synonyms for the Bxx instructions I think is a decent compromise (I had already thought of it, but I realised that it doesn't actually need to be a part of the assembler itself, you can just make some macros that map one to the other, so I saw no need to mention it).

_________________
Want to design a PCB for your project? I strongly recommend KiCad. Its free, its multiplatform, and its easy to learn!
Also, I maintain KiCad libraries of Retro Computing and Arduino components you might find useful.

In every 65816 assembler I've personally used, without exception, the code above would generate absolute-long addressing. Just as opcodes are length-sensitive, so too are symbols (or, it has been for me).

The only time ambiguity has arisen for me is when performing ambiguous calculations (e.g., if addr=$A000 and you execute something like LDA addr+$10000), or if you want to use 16-bit symbol expressions and force them to be long addresses.

That said, regarding your proposal, I have no particular opinion one way or another. Many environments have defined their own assembly syntax (e.g., AT&T Unix syntax assembly vs Intel or Motorola syntaxes), and even more contemporary systems are no exception (e.g., Plan 9's assemblers, which influenced the assembler backends for the Go programming language).

I personally prefer the legacy syntax only out of familiarity and I'm comfortable with its limitations and workarounds. But, that's a subjective matter, and if I were asked to maintain code using your syntax, I probably wouldn't complain vociferously over having to change habits.

_________________
x86?  We ain't got no x86.  We don't NEED no stinking x86!