Read the thread Garth mentioned. To calculate the accumulator result and the carry you use one set of rules. There is a different set of rules for the other flags. With the same inputs you get the same results every time.
In my post about halfway down the first page in the other thread there are two programs given, and a list of seven steps for determining the V flag in decimal mode. Those same seven steps can be used to predict the value of the ADC N flag result in decimal mode -- just subtitute N for V in steps 6 and 7. The other 4 undocumented cases -- the SBC N, V, and Z flag results, and the ADC Z flag result -- are the same as if the two numbers were subtracted (SBC) or added (ADC) in binary mode. In other words, the Z flag after:
Code:
SED
CLC
LDA #$50
ADC #$50
is the same as the Z flag after:
Code:
CLD
CLC
LDA #$50
ADC #$50
So, in both cases the Z flag will be zero (i.e. BNE will branch). To summarize:
After ADC:
Accumulator: determined by BCD addition result
C flag: determined by BCD addition result
N flag: see the seven step method in the other thread
V flag: see the seven step method in the other thread
Z flag: same value as when the two numbers are added in binary mode
After SBC:
Accumulator: determined by BCD subtraction result
C flag: determined by BCD subtraction result
N flag: same value as when the two numbers are subtracted in binary mode
V flag: same value as when the two numbers are subtracted in binary mode
Z flag: same value as when the two numbers are subtracted in binary mode
You can use this infomation to synthesize the decimal mode flag results using only binary arithmetic. That is what the two programs mentioned earlier do.
On the 6502, the only documented behavior is the accumulator result and the carry flag when given valid BCD numbers (i.e. both BCD digits are in the range 0-9). So, a simulator can be considered "correct" if these cases give the correct result. However, as blargg mentioned, it can be useful to replicate the undocumented behavior, since there are programs out there that use undocumented decimal mode behavior. Relying on undocumented behavior does not make for a robust program, so when such a program doesn't behave as expected on a simulator, to me, that is a problem with the program, not the simulator. Anyway, here are a couple of examples:
Example #1: distinguishing a 6502 from a 65C02.
Code:
CLC
LDA #$99
ADC #$01
BEQ THIS_IS_A_65C02 ; or it's a 65802 or a 65816
BNE THIS_IS_A_6502
The ADC clears Z flag on a 6502, but sets the Z flag on a 65C02. This is undocumented behavior on a 6502, but it
IS documented behavior on a 65C02.
Incidentally, if you always cleared the Z flag in decimal mode, this particular example would still behave correctly.
Example #2: converting a hex digit 0-F to ASCII
A straightforward way to convert $00-$09 to $30-$39 and $0A-$0F to $41-$46 is:
Code:
CMP #$0A
BCC SKIP
ADC #$06 ; this adds 7 since the carry is set
SKIP ADC #$30
A couple of bytes can be saved by resorting to undocumented behavior:
Code:
SED
CMP #$0A
ADC #$30
CLD
In this version, the undocumented behavior relied upon is addition when one of the digits in the range A-F.
I can't recall any other examples off the top of my head. There is very little existing code that would be broken by not replicating undocumented behavior.
If you have additional questions feel free to ask. I've worked out all of the cases for 8-bit and 16-bit (i.e. on the 65816 when the m flag is zero) decimal mode addition and subtraction and have verified every possible combination on the 6502 (Rockwell and Synertek), the 65C02, and the 65816 works as I've described. Now if only I'd stop writing long-winded forum posts and finish the decimal mode write-up...