Testing instruction timings for 6502 emulators

[BigEd]

Elsewhere is a discussion about how to test that an emulator has the right instruction timings - Klaus' functional test suite isn't ideal for this although it could perhaps be a confidence test. Other representative code can also be a confidence test.

I thought it might be useful to have a test specifically for this purpose, so please see
https://github.com/BigEd/6502timing
and please let me know how you get on.

Until we find a way to run a test like this on a real 6502 and count the cycles, the best we can do is compare various emulators. In case of differences we can run subsets of the test, branch and bound until we find the first discrepancy.

Obviously, to compare we need to use the same version of the program, have no interrupts running, no stolen cycles with RDY, and account for all the instructions in the program and no more. Could be fiddly.

Assembly syntax is for Michal Kowalski's simulator as found at http://www.exifpro.com/utils.html because that was the easiest way to both assemble and get timings. Edit: but see below, the cycle count model in this simulator is not accurate.

Edit: summarised results for various models here.

Edit: attached a zipfile with source, listing, and a binary file.
Edit: updated the zipfile with changes to label syntax

Edit: here's the current version of the program in srec format - it's to be loaded and run at $1000

Code: Select all

S1231000A2FF9AA2018617861886168AA898BAAAA200A0000117051709000D1717111715F2
S1231020171D1717191717CA880117111715171D1717191717E8C82117251729002D17170B
S1231040311735173D1717391717CA882117311735173D1717391717A200A0004117451740
S123106049004D1717511755175D1717591717CA884117511755175D1717591717A200A0F0
S1231080006117651769006D1717711775177D1717791717CA886117711775177D171779EC
S12310A01717A200A000A901811785178D1717911795179D1717991717CA88811791179516
S12310C0179D1717991717A200A000A117A517A900AD1717B117B517BD1717B91717CA8871
S12310E0A117B117B517BD1717B91717A200A000C117C517C900CD1717D117D517DD171735
S1231100D91717CA88C117D117D517DD1717D91717A200A000E117E517E900ED1717F11772
S1231120F517FD1717F91717CA88E117F117F517FD1717F91717A200061746170E17174EF7
S12311401717161756171E17175E1717CA161756171E17175E1717A200261766172E1717F7
S12311606E1717361776173E17177E1717CA361776173E17177E1717A000A6178617AE1762
S1231180178E1717B6179617BE171788B6179617BE1717A200C617E617CE1717EE1717D6BA
S12311A017F617DE1717FE1717CAD617F617DE1717FE1717A2001002105C30FE10FAF00289
S12311C0F056D0FEF0FAA2FF3002304E10FE30FAD002D048F0FED0FA1890029041B0FE9029
S12311E0FA38B002B03A90FEB0FAB85002503370FE50FAA97F697F7002702950FE70FAC805
S123120088E8CAEA0A2A4A6A202612486808287858F8D84C3B1210A6F0AC30B4D0BA90C13C
S1231220B0C850CF70D9203412AD3312A20081176C17006068186901484840A20024172C92
S12312401717A4178417AC17178C1717B4179417BC1717CAB4179417BC1717C017C417CC6A
S10F12601717E017E417EC17174C0010E8

[BigEd]

I realised that visual6502 should be more of a gold standard. To start the program at $1000 I need to start with a jump, so the execution trace looks like this:

Code: Select all

cycle    ab   db	rw	Fetch	   pc	 a	 x	 y	 s	  p
0	   0000	4c	1	JMP Abs	0000	aa	00	00	fd	nv‑BdIZc
3	   1000	a2	1	LDX #	  1000	aa	00	00	fd	nv‑BdIZc
5	   1002	9a	1	TXS	    1002	aa	ff	00	fd	Nv‑BdIzc
...
1135	1262	e0	1	CPX #	  1262	60	ff	01	ff	nv‑BdiZC
1137	1264	e4	1	CPX zp	 1264	60	ff	01	ff	nv‑BdiZC
1140	1266	ec	1	CPX Abs	1266	60	ff	01	ff	Nv‑BdizC
1144	1269	4c	1	JMP Abs	1269	60	ff	01	ff	Nv‑BdizC

Which means visual6502 says this test program takes 1141 cycles, compared to Kowalski's verdict of 1130. Hmm.

Maybe the page-crossing branches are not taking the extra cycle that they should, in the Kowalski simulator.

Actually, the first discrepancy is opcode 1d (ORA Abs,X) taking 5 cycles where it should take 4, in the case of not crossing a page boundary.

Conclusion: don't trust the cycle counts from this version of the Kowalski simulator. (Or the v1.2.12 version either.)

[JimDrew]

Thanks for starting this thread!

How about a binary version that can be incbin'd? I can't assemble the code with anything from the C64 or the CBM prg Studio. I guess I could convert the syntax. Most of the assemblers don't support local labels, and I am not sure what the .db $bb is suppose to do, or where the end of the test actually resides.

[JimDrew]

OK, I converted the code to a CBM prg Studio project and I get 1141 cycles up to (but not including) the JMP $1269. I would like to recommend that you remove the local label references and make those real label names. That will open up the number of different assemblers that can assemble this code. I just changed the .1/.2 to r(plus the return number. ie. r11/r12, r21/r22, r31/r32, etc.).

[BigEd]

Great - I'll attach a zip file to the head post, but it sounds like you don't need it. Your run was with a real physical C64?
Edit: oops, it seems CBM prg Studio is an IDE which includes its own emulator, so probably that's what you tested!

It seems then that these models agree:

• visual6502
  CBM prg Studio (thanks Jim)
  jsbeeb (thanks to Matt for the assist)
  T65 FPGA core (thanks Dave)
  µDrive (PIC 6502 emulator) (thanks Jim)
  VICE (thanks Jim)

This model does not agree with the above set:

• Kowalski simulator up to v1.2.12

Cheers
Ed

Edit: updating the table of results.

[JimDrew]

It was Arthur's CBM prg Studio, see: http://www.ajordison.co.uk

I am going to move this test code to a fake ROM that I can incbin into my PIC 6502 emulator to check the cycle time as well, but I think it is going to be the same as Arthur's as I helped correct a lot of his debugger code.

[hoglet]

ICE-T 65 (aka Atom Bus Monitor) to the rescue

Attached is the complete program single stepped (299 instructions).

The clock rate is 1MHz, so the timing are in <seconds>.<micro seconds> and are logged before the instruction executes.

It you subtract the first from the last, you get:
= 04.785946 - 04.784805
= 00.001141
= 1141 cycles

That matches Ed's Visual 6502 result.

This is with the T65 core, not a real 6502. I could repeat this with a real 6502, but we would only get the instruction addresses logged, not a live disassembly.

I'm also going to add a command to allow the cycle counter to be reset to zero.

Dave

Code: Select all

6502 free running...
Ex Breakpoint hit at 1000
04.784805: 1000 : LDX #FF     
>> s299
Stepping 299 instructions
04.784807: 1002 : TXS         
04.784809: 1003 : LDX #01     
04.784811: 1005 : STX 17      
04.784814: 1007 : STX 18      
04.784817: 1009 : STX 16      
04.784820: 100B : TXA         
04.784822: 100C : TAY         
04.784824: 100D : TYA         
04.784826: 100E : TSX         
04.784828: 100F : TAX         
04.784830: 1010 : LDX #00     
04.784832: 1012 : LDY #00     
04.784834: 1014 : ORA (17,X)  
04.784840: 1016 : ORA 17      
04.784843: 1018 : ORA #00     
04.784845: 101A : ORA 1717    
04.784849: 101D : ORA (17),Y  
04.784854: 101F : ORA 17,X    
04.784858: 1021 : ORA 1717,X  
04.784862: 1024 : ORA 1717,Y  
04.784866: 1027 : DEX         
04.784868: 1028 : DEY         
04.784870: 1029 : ORA (17,X)  
04.784876: 102B : ORA (17),Y  
04.784882: 102D : ORA 17,X    
04.784886: 102F : ORA 1717,X  
04.784891: 1032 : ORA 1717,Y  
04.784896: 1035 : INX         
04.784898: 1036 : INY         
04.784900: 1037 : AND (17,X)  
04.784906: 1039 : AND 17      
04.784909: 103B : AND #00     
04.784911: 103D : AND 1717    
04.784915: 1040 : AND (17),Y  
04.784920: 1042 : AND 17,X    
04.784924: 1044 : AND 1717,X  
04.784928: 1047 : AND 1717,Y  
04.784932: 104A : DEX         
04.784934: 104B : DEY         
04.784936: 104C : AND (17,X)  
04.784942: 104E : AND (17),Y  
04.784948: 1050 : AND 17,X    
04.784952: 1052 : AND 1717,X  
04.784957: 1055 : AND 1717,Y  
04.784962: 1058 : LDX #00     
04.784964: 105A : LDY #00     
04.784966: 105C : EOR (17,X)  
04.784972: 105E : EOR 17      
04.784975: 1060 : EOR #00     
04.784977: 1062 : EOR 1717    
04.784981: 1065 : EOR (17),Y  
04.784986: 1067 : EOR 17,X    
04.784990: 1069 : EOR 1717,X  
04.784994: 106C : EOR 1717,Y  
04.784998: 106F : DEX         
04.785000: 1070 : DEY         
04.785002: 1071 : EOR (17,X)  
04.785008: 1073 : EOR (17),Y  
04.785014: 1075 : EOR 17,X    
04.785018: 1077 : EOR 1717,X  
04.785023: 107A : EOR 1717,Y  
04.785028: 107D : LDX #00     
04.785030: 107F : LDY #00     
04.785032: 1081 : ADC (17,X)  
04.785038: 1083 : ADC 17      
04.785041: 1085 : ADC #00     
04.785043: 1087 : ADC 1717    
04.785047: 108A : ADC (17),Y  
04.785052: 108C : ADC 17,X    
04.785056: 108E : ADC 1717,X  
04.785060: 1091 : ADC 1717,Y  
04.785064: 1094 : DEX         
04.785066: 1095 : DEY         
04.785068: 1096 : ADC (17,X)  
04.785074: 1098 : ADC (17),Y  
04.785080: 109A : ADC 17,X    
04.785084: 109C : ADC 1717,X  
04.785089: 109F : ADC 1717,Y  
04.785094: 10A2 : LDX #00     
04.785096: 10A4 : LDY #00     
04.785098: 10A6 : LDA #01     
04.785100: 10A8 : STA (17,X)  
04.785106: 10AA : STA 17      
04.785109: 10AC : STA 1717    
04.785113: 10AF : STA (17),Y  
04.785119: 10B1 : STA 17,X    
04.785123: 10B3 : STA 1717,X  
04.785128: 10B6 : STA 1717,Y  
04.785133: 10B9 : DEX         
04.785135: 10BA : DEY         
04.785137: 10BB : STA (17,X)  
04.785143: 10BD : STA (17),Y  
04.785149: 10BF : STA 17,X    
04.785153: 10C1 : STA 1717,X  
04.785158: 10C4 : STA 1717,Y  
04.785163: 10C7 : LDX #00     
04.785165: 10C9 : LDY #00     
04.785167: 10CB : LDA (17,X)  
04.785173: 10CD : LDA 17      
04.785176: 10CF : LDA #00     
04.785178: 10D1 : LDA 1717    
04.785182: 10D4 : LDA (17),Y  
04.785187: 10D6 : LDA 17,X    
04.785191: 10D8 : LDA 1717,X  
04.785195: 10DB : LDA 1717,Y  
04.785199: 10DE : DEX         
04.785201: 10DF : DEY         
04.785203: 10E0 : LDA (17,X)  
04.785209: 10E2 : LDA (17),Y  
04.785215: 10E4 : LDA 17,X    
04.785219: 10E6 : LDA 1717,X  
04.785224: 10E9 : LDA 1717,Y  
04.785229: 10EC : LDX #00     
04.785231: 10EE : LDY #00     
04.785233: 10F0 : CMP (17,X)  
04.785239: 10F2 : CMP 17      
04.785242: 10F4 : CMP #00     
04.785244: 10F6 : CMP 1717    
04.785248: 10F9 : CMP (17),Y  
04.785253: 10FB : CMP 17,X    
04.785257: 10FD : CMP 1717,X  
04.785261: 1100 : CMP 1717,Y  
04.785265: 1103 : DEX         
04.785267: 1104 : DEY         
04.785269: 1105 : CMP (17,X)  
04.785275: 1107 : CMP (17),Y  
04.785281: 1109 : CMP 17,X    
04.785285: 110B : CMP 1717,X  
04.785290: 110E : CMP 1717,Y  
04.785295: 1111 : LDX #00     
04.785297: 1113 : LDY #00     
04.785299: 1115 : SBC (17,X)  
04.785305: 1117 : SBC 17      
04.785308: 1119 : SBC #00     
04.785310: 111B : SBC 1717    
04.785314: 111E : SBC (17),Y  
04.785319: 1120 : SBC 17,X    
04.785323: 1122 : SBC 1717,X  
04.785327: 1125 : SBC 1717,Y  
04.785331: 1128 : DEX         
04.785333: 1129 : DEY         
04.785335: 112A : SBC (17,X)  
04.785341: 112C : SBC (17),Y  
04.785347: 112E : SBC 17,X    
04.785351: 1130 : SBC 1717,X  
04.785356: 1133 : SBC 1717,Y  
04.785361: 1136 : LDX #00     
04.785363: 1138 : ASL 17      
04.785368: 113A : LSR 17      
04.785373: 113C : ASL 1717    
04.785379: 113F : LSR 1717    
04.785385: 1142 : ASL 17,X    
04.785391: 1144 : LSR 17,X    
04.785397: 1146 : ASL 1717,X  
04.785404: 1149 : LSR 1717,X  
04.785411: 114C : DEX         
04.785413: 114D : ASL 17,X    
04.785419: 114F : LSR 17,X    
04.785425: 1151 : ASL 1717,X  
04.785432: 1154 : LSR 1717,X  
04.785439: 1157 : LDX #00     
04.785441: 1159 : ROL 17      
04.785446: 115B : ROR 17      
04.785451: 115D : ROL 1717    
04.785457: 1160 : ROR 1717    
04.785463: 1163 : ROL 17,X    
04.785469: 1165 : ROR 17,X    
04.785475: 1167 : ROL 1717,X  
04.785482: 116A : ROR 1717,X  
04.785489: 116D : DEX         
04.785491: 116E : ROL 17,X    
04.785497: 1170 : ROR 17,X    
04.785503: 1172 : ROL 1717,X  
04.785510: 1175 : ROR 1717,X  
04.785517: 1178 : LDY #00     
04.785519: 117A : LDX 17      
04.785522: 117C : STX 17      
04.785525: 117E : LDX 1717    
04.785529: 1181 : STX 1717    
04.785533: 1184 : LDX 17,Y    
04.785537: 1186 : STX 17,Y    
04.785541: 1188 : LDX 1717,Y  
04.785545: 118B : DEY         
04.785547: 118C : LDX 17,Y    
04.785551: 118E : STX 17,Y    
04.785555: 1190 : LDX 1717,Y  
04.785560: 1193 : LDX #00     
04.785562: 1195 : DEC 17      
04.785567: 1197 : INC 17      
04.785572: 1199 : DEC 1717    
04.785578: 119C : INC 1717    
04.785584: 119F : DEC 17,X    
04.785590: 11A1 : INC 17,X    
04.785596: 11A3 : DEC 1717,X  
04.785603: 11A6 : INC 1717,X  
04.785610: 11A9 : DEX         
04.785612: 11AA : DEC 17,X    
04.785618: 11AC : INC 17,X    
04.785624: 11AE : DEC 1717,X  
04.785631: 11B1 : INC 1717,X  
04.785638: 11B4 : LDX #00     
04.785640: 11B6 : BPL 11BA    
04.785643: 11BA : BMI 11BA    
04.785645: 11BC : BPL 11B8    
04.785648: 11B8 : BPL 1216    
04.785652: 1216 : BPL 11BE    
04.785656: 11BE : BEQ 11C2    
04.785659: 11C2 : BNE 11C2    
04.785661: 11C4 : BEQ 11C0    
04.785664: 11C0 : BEQ 1218    
04.785668: 1218 : BEQ 11C6    
04.785672: 11C6 : LDX #FF     
04.785674: 11C8 : BMI 11CC    
04.785677: 11CC : BPL 11CC    
04.785679: 11CE : BMI 11CA    
04.785682: 11CA : BMI 121A    
04.785686: 121A : BMI 11D0    
04.785690: 11D0 : BNE 11D4    
04.785693: 11D4 : BEQ 11D4    
04.785695: 11D6 : BNE 11D2    
04.785698: 11D2 : BNE 121C    
04.785702: 121C : BNE 11D8    
04.785706: 11D8 : CLC         
04.785708: 11D9 : BCC 11DD    
04.785711: 11DD : BCS 11DD    
04.785713: 11DF : BCC 11DB    
04.785716: 11DB : BCC 121E    
04.785720: 121E : BCC 11E1    
04.785724: 11E1 : SEC         
04.785726: 11E2 : BCS 11E6    
04.785729: 11E6 : BCC 11E6    
04.785731: 11E8 : BCS 11E4    
04.785734: 11E4 : BCS 1220    
04.785738: 1220 : BCS 11EA    
04.785742: 11EA : CLV         
04.785744: 11EB : BVC 11EF    
04.785747: 11EF : BVS 11EF    
04.785749: 11F1 : BVC 11ED    
04.785752: 11ED : BVC 1222    
04.785756: 1222 : BVC 11F3    
04.785760: 11F3 : LDA #7F     
04.785762: 11F5 : ADC #7F     
04.785764: 11F7 : BVS 11FB    
04.785767: 11FB : BVC 11FB    
04.785769: 11FD : BVS 11F9    
04.785772: 11F9 : BVS 1224    
04.785776: 1224 : BVS 11FF    
04.785780: 11FF : INY         
04.785782: 1200 : DEY         
04.785784: 1201 : INX         
04.785786: 1202 : DEX         
04.785788: 1203 : NOP         
04.785790: 1204 : ASL A       
04.785792: 1205 : ROL A       
04.785794: 1206 : LSR A       
04.785796: 1207 : ROR A       
04.785798: 1208 : JSR 1226    
04.785804: 1226 : JSR 1234    
04.785810: 1234 : PLA         
04.785814: 1235 : CLC         
04.785816: 1236 : ADC #01     
04.785818: 1238 : PHA         
04.785821: 1239 : PHA         
04.785824: 123A : RTI         
04.785830: 1229 : LDA 1233    
04.785834: 122C : LDX #00     
04.785836: 122E : STA (17,X)  
04.785842: 1230 : JMP (0017)  
04.785847: 0101 : RTS         
04.785853: 120B : PHA         
04.785856: 120C : PLA         
04.785860: 120D : PHP         
04.785863: 120E : PLP         
04.785867: 120F : SEI         
04.785869: 1210 : CLI         
04.785871: 1211 : SED         
04.785873: 1212 : CLD         
04.785875: 1213 : JMP 123B    
04.785878: 123B : LDX #00     
04.785880: 123D : BIT 17      
04.785883: 123F : BIT 1717    
04.785887: 1242 : LDY 17      
04.785890: 1244 : STY 17      
04.785893: 1246 : LDY 1717    
04.785897: 1249 : STY 1717    
04.785901: 124C : LDY 17,X    
04.785905: 124E : STY 17,X    
04.785909: 1250 : LDY 1717,X  
04.785913: 1253 : DEX         
04.785915: 1254 : LDY 17,X    
04.785919: 1256 : STY 17,X    
04.785923: 1258 : LDY 1717,X  
04.785928: 125B : CPY #17     
04.785930: 125D : CPY 17      
04.785933: 125F : CPY 1717    
04.785937: 1262 : CPX #17     
04.785939: 1264 : CPX 17      
04.785942: 1266 : CPX 1717    
04.785946: 1269 : JMP FF3F    
>> 

[JimDrew]

I get 1141 with my emulation. I created a ROM image of this code. It's a 16K image (overkill), but it contains the reset vector to jump to $C000 which is where the code is assembled to. There is no self modifying code here (yet), so there is no need move it to RAM. I have a wrapper for that if necessary.

Just incbin the ROM image and reset the CPU. The reset vector jumps to $C000. Start counting cycles from there to the 2nd NOP ($C269).

Attached is the ROM image.

[BigEd]

Thanks Dave - I've updated the table of results at viewtopic.php?p=38590#p38590

[JimDrew]

I can confirm (using the ROM I made above) that WinVICE gets 1141 cycles for the test. Just a note for those testing VICE... you need to use the "sidefx" command to turn off the surrounding hardware so just the CPU runs without interruption.

[BigEd]

That's great, thanks - I've updated the summary table.

[JimDrew]

For testing with a real 6502 CPU I could make a wrapper that moves the code from ROM ($C000) to RAM ($0400) and jumps to the RAM, and when the test is complete it would jump back to ROM. Why? Because doing this, you could attach a LA to A15 and watch the state change high-low-high, and determine how long it took. You would have to subtract the time for the JMP instruction back to ROM.

[BigEd]

It's a good idea - you can validate the timing idea by timing say 32 NOP instructions.

[JimDrew]

Yes, we could also do something like LDA $4000 to cause a toggle on A14 to denote the next test, and check times between these toggles for each different test. That way, if the total result was unexpected you should be able to determine exactly where the problem was.

[BigEd]

Yes, finer grained timings would be a good step forward. I've been thinking, on a system with a VIA, it would be possible to read the low byte of a timer and push it to the stack, and then finally check page 1 against a known set of values. Probably needs a macro to wrap each instruction.

Another thought: to do a really thorough check of page-crossing timings, especially for branches, would be to catch all the near-miss cases, in case of an out-by-one error in an emulator. A branch instruction can even straddle a page boundary: it's important to know which value is important: the PC, or PC+1, or PC+2. An emulator could get this wrong and still pass the current test.