Klaus2m5: It is actually the ADC part of it; I checked that. It's failing on incorrect flag output.

Druzyek: So I wrote a program to exercise the ADC or SBC instruction in decimal mode, and I noticed that the accumulator always seemed to be set to $20. It turns out this was actually a bug in the program(I was clobbering A without realising). Having fixed that, I now have sane output. I've attached the program I used, along with its output. I wrote it for SB-Assembler, thought I can probably convert it to something else if you'd like.

The Output looks like its a plain binary ADC. In a decimal ADC if the result of a nibble is greater 9 then 6 must be added to that nibble (decimal correction). For example: binary 9 + 1 = A, decimal 9 + 1 = 10.

_________________
6502 sources on GitHub: https://github.com/Klaus2m5

You're right, it is. Thanks. I've got that fixed now. Not sure if the flags are correct, but I haven't yet run this on real hardware. Once I do that, it'll get interesting if the results differ substantially.
However, I'll have to do that tomorrow, because it's getting quite late now.

That looks like a perfect decimal ADC. However, I haven't looked at every individual result. So we are back to SBC decimal instead of chasing a ghost (we've all been there).

edit: I've spotted a flaw here9 + 6 + carry is still greater then 9 for the lower nibble ($10) so the decimal correction needs to be applied.

_________________
6502 sources on GitHub: https://github.com/Klaus2m5

It looks like it's doing a normal (non-decimal) addition, and then applying the correction if the nybble of the result is greater than 9. That's not correct: what you're really after is whether there's a decimal carry out of the nybble, and that happens when the nybble is greater than 9 or there's a binary carry.

The simple way of doing it right is to work one nybble at a time, rather than adding first and trying to correct. If you wanted to get fancy, you can check for a carry out of bit 3 by looking at the xor of A-in, mem, and A-out. If that has a 1 in bit 4, there was a carry and you need to apply the correction. (You also need to correct the high nybble if the carry out of bit 7 is 1).

My code is about as clear as mud, but that's essentially what I do (or something along that line of thinking):
I calculate the binary result first, then bias it by $66 to force BCD carry propagation. The XORs illuminate the carries in bits 8 and 4, the >> 3 puts them into bits 5 and 1, the & 0x22 masks off the extraneous nonsense in the other bits, the * 3 converts the carries to one of $00, $06, $60 or $66, and the += adjusts the result ... all in two "simple" assignment statements.

_________________
Got a kilobyte lying fallow in your 65xx's memory map? Sprinkle some VTL02C on it and see how it grows on you!

Mike B. (about me) (learning how to github)

I was doing a binary add and then correcting, because that's easily comprehensible. I used to have an ADC function that went nybble-by-nybble, and it quickly became a pain to read.

EDIT: I have switched back to a nybble-by-nybble approach to obtain this output.

I've got the results back from the the real hardware, and the diff is bigger than the source file. I've managed to cut down the sheer quantity of them by filtering out invalid BCD inputs, but that's still a bit over 10,000 lines of data. The ones I've looked at seem to exhibit incorrect overflow behaviour. I'm not sure whether V is even valid for decimal mode, so I'm not sure what to do about that. I suspect I'll have to write a program to work out what the actual differences are; see if that's all it is.

I have attached the cut-down and partly-marked-up diff. At this point, I don't particularly care about invalid inputs or outputs being true to the original, but I'm not sure that I haven't got a dud output from valid inputs in there somewhere, so I'll have to check it all one way or another.

EDIT: It's also of note that this simulation(running on a 3.4GHz Ryzen) seems to run far faster than the 1MHz real thing(over an 115200bps serial connection). The sim completes in less than a second, provided I'm redirecting output to a file, whereas the 'C02 takes several seconds.

I started writing my own 6502 simulator for my new hardware project and after looking at al those decimal ADC/SBC rants I decided to do it the right way. Instead of going "Intel" way (that's what most of 6502 simulators do), I looked at the 6502 schematic and MOS Tech "decimal" patent. There is no schematic available for 65C02, but I think the whole stuff, including the 6502/65C02 difference may be explained quite easily and without any magic.

The key notes are:
1. Two circuits are involved in decimal arithmetics: binary adder (ALU) with BCD carry propagation and a separate decimal adjust circuit.
2. For the ALU, the decimal mode is activated ONLY during ADC. During "decimal" SBC, the operation of ALU is exactly the same as in binary mode - its a standard binary addition, only the 2nd arguments is bit-negated.
3. During ADC, the ALU performs binary addition with a modified carry propagation. Carry from bit 3 to bit 4 is set if the result in bits 3..0 exceeds 9. This carry (AC signal) is used as bit 4 input carry and controls the operation of an adjuster. Carry from bit 7 is set if the result in bits 7..4 exceeds 9. This goes to C flag (same in NMOS and CMOS). This carry value, although obtained by a slightly modified binary adder, is a proper decimal carry value, assuming the source arguments were legal BCD. During SBC the AC and C values are standard binary carries (= negated borrows).
4. V flag in both NMOS and CMOS is set in a standard way - as a XOR between carry from bit 6 to 7 and the C flag. Its value has nothing to do with the final BCD result since it's NOT based on BCD result - it's generated by ALU.
5. The ALU output value (pure binary for SBC, tricky for ADC) goes to the correction circuitry, which is imply a pair of reduced binary adders, one for bits 3..0, another for 7..4 (actually bits 0 and 4 pass transparently), with NO CARRY PROPAGATION between them (since the BCD carry was already accounted for by ALU). The adders are activated respectively by AC (3..0) and C (7..4) signals from ALU.
6. During decimal addition, the adder adds 6 to a nibble if the control carry input is 1 (there is a carry out of a digit generated in ALU).
7. During decimal SBC, the adder adds 10 to a nibble if the control carry input is 0 (there is a borrow).
8. Now the NMOS/CMOS difference: N and Z in NMOS are set based on uncorrected ALU output, while in CMOS version these flags are set based on the corrected value. C and V behave in the same way in both versions.

(I did not verify the V generation algorithm yet, but looking at the diagram I can't see any other way of setting V.)

I will post my C code for ADC and SBC after I test it thoroughly.
I know, I am re-inventing the wheel, but it's a new re-invention...