I've had a background project over the past decade of disassembling the original Williams Defender code. I tend to work on this about 3 or 4 half-days a year, when I'm on vacation - between doing other stuff. So although a decade sounds like a long time (I think it might actually be 12 or 13), I've only just begun. It's not a serious thing, just a little fun when I'm absolutely completely bored.
Defender was written by Eugene Jarvis, the same guy who wrote the original Robotron 2084. The main game loop iterates over object lists: linked-lists of data structures which have references to sprite objects, "think" (behavior) functions, timers and other things (TBD!). Sprite objects reference pixel data, and have pointers to draw and erase functions. Then there's the think functions for the on-screen objects, input, overall game logic, etc.
The 6502 version of Robotron is likely going to be structured similarly, even if it's an independent reimplementation by a different programmer. This "object oriented" style is a pretty natural way of writing games with many different "actors" that have their own lifespan, behavior, and graphics/animation.
My approach to the big-picture disassembly was:
- Start with a two-pass disassembly that creates labels for all the trivial branch/jump destinations. This identified the 'direct-call' subroutines (called via JSR) and the intrafunction logic (control flow / loops).
- Make a list of all the JSR destinations. This is the "known function list".
- For each RTS instruction that isn't followed by a known function or an intrafunction label, take the address of the next byte and look through the binary for (LO-BYTE, HI-BYTE) references to that address. This becomes the "possible function list". If there a multiple bytes between the RTS and the next known-function, search for those too. The locations of the (LO, HI) references are "possible function pointers". Keep those.
- For each "possible function", disassemble it and see if it looks like garbage or an actual function. If you know 6502 then it's pretty easy to tell the difference.
One difference between 6809 and 6502 is that 6809 natively handles 16-bit values, so (LO, HI) pairs are often directly stored in contiguous memory addresses. When I used to write small games on the 6502 (back in the early 80s) I would spread object structure fields over arrays indexed by X. So, for example:
Code:
struct object {
struct object *next;
uint8 xpos;
uint8 ypos;
uint16 thinkfunc;
};
In 6809 this would be a 6-byte object (2 bytes for the 'next' pointer), whereas in 6502 I might have structured it like this:
Code:
obj_next: .ds 128 ; support 128 objects
obj_xpos: .ds 128
obj_ypos: .ds 128
obj_thinkhi: .ds 128
obj_thinklo: .ds 128
obj_head: .ds 1 ; head index
obj_runloop:
lda obj_head
obj_loop:
tax
; call the think function
lda obj_thinklo,x
pha
lda obj_thinkhi,x
pha
rts
objloop_next: ; think function JMPs back to here
lda obj_next,x
bne obj_loop
rts
If 6502 Robotron structures things like this then finding these (LO, HI) pairs might be a challenge. You can find loops like the above example to find the arrays, and then adjust your analysis accordingly.
With the known-function-list you can create a call-graph. There will likely be many call graphs - they won't all neatly form one big tree. The call-graph lets to see structure. There will be many shared nodes on the graph. These help identify utility functions.
- Using a graphical utility, locate the sprite patterns for the program. I wrote my own utility for this, by studying how Defender draws its sprites and therefore what order the data had to appear in memory. Different sprites are different widths and heights, so the utility was interactive I could move though memory and adjust the x and y strides. Find all the sprite data and record the start address of each pattern.
- Find all the draw and erase functions. Defender did not use a double-buffered screen, rather it would erase an object from its old position just before drawing it at its new one. The draw and erase functions in 6809 are pretty neat as they use two stack pointers to efficiently move memory in blocks.
- Reference to the draw/erase functions should be in your (LO, HI) function pointer lists, so with that you should be able to find the sprite descriptors (data structures that describe a sprite, what it looks like, how wide and high it is, and where its draw/erase functions are).
- Find the object lists. Objects in Defender come in different sizes and are allocated and free using utlity functions. Want a new alien? Call the allocate function for a "type 1" object, and then fill in its fields. The object comes to life on the next iteration of the main loop.
- Find the hardware manipulations. These are reads/writes to hardware registers, which you find out by studying the Apple II reference manuals. Every time there's a register read, understand what that does. Every time there's a write, know what it does. It is changing a color register? Is it starting a timer? Is it clearing an interrupt? Is the read getting a joystick value? A key value? Find all these and try to understand their enclosing function.
- Find the interrupt handlers. If the Apple II uses interrupts then find the vector and start trying to figure out its big-picture. Interrupt functions end with RTI, but the Apple II ROM may take care of that and the interrupt might be invoked via a software vector. Go read that vector and see where it points to.
Being able to BREAK into an emulator makes a lot of this easier as you can see the state of the machine while the game is actually running. For Defender, the MAME emulator was invaluable in this initial study as the game uses ROM paging, and just figuring out which banks are the game proper vs. the menus and diagnostic pages wouldn't have been easy without prior knowledge.
The hardest thing with reverse-engineering is figuring out intent. Not _what_ something does, mechanically, but _why_. Why is the code comparing this to that? What's the significance of the literal value #$CE in this CMP? With an emulator you can change these values and see if you observe a difference. That's playing on hard mode, for sure... but you can do it.
I hope this gives you some ideas. Some people have a natural talent at reverse-engineering. I'm not one of them, and I'm sure I'll never make significant progress with Defender, but hey... everyone needs a hobby... even if it's only a handful of hours a year.
Good luck!