Everything there actually looks pretty good, voltage wise. It looks like your clock might not be 50% duty cycle (the high and low parts should be equal), but I think we're also running into the limits of your bandwidth on the scope, so it's difficult to tell for certain. If you can reduce the clock to 1MHz (or less), we should be able to get a better picture there. That's probably not your issue, though.
You will want to verify that each line goes both high and low at reasonable levels, similar to these that you've shared with us. This sometimes requires writing and running special programs that exercise the hardware a certain way so the scope can look for a particular signal. For the address lines, that might mean writing a program that reads from $0000 and then $FFFF (perhaps using LDA instructions) in a loop. The address lines will have those addresses (as well as the addresses to run that bit of code) on them over and over, so you will see both highs and lows on every address line. Without the test program, you might have the upper address lines never change at all if the code is all running in the $C000-$FFFF range.
You might then change the software to load A with $00 and then write it into a RAM location, then load it with $FF and write it to the same RAM location. This will have both reads and writes occurring on the databus (there will also be the bytes to for those assembly instructions, as the CPU pulls the code from ROM one instruction at a time to run it). You can look at each data line to make sure they all go fully high and low. Note that the databus usually flats at times, and can have "in-the-middle" voltages on it during those times, but I believe your level shifters have devices that hold the bus at the last value until it is driven the other way, so you probably won't see that on this setup.
You should also probe all the control lines, like RESB (which should just be high near 5V when your program is running). You are looking for any voltages that are "in the middle" of the voltage range. The slight excursions below ground and over the supply voltage aren't exciting unless they are large (and some of them may not be real and are a byproduct of using the alligator ground lead on the probe). The dips on the clock line, for example, are the size I would normally be concerned by (over 1V on the 5V side!), but I'm guessing the ground you used is a ways away from the oscillator's ground. If you move the scope's ground connection to the oscillator's ground, I expect that dip below 0V will be much smaller. Fast moving signals, like clocks, are more likely to exhibit this effect.
Also check your power rails (and your GND connections at each IC). They should be pretty steady, but will have ripples just like you see in your signals. When looking at the power for a particular IC, you should use the IC's GND pin as your ground reference (rather than just clipping on whatever is convenient to clip on), because the GNDs are not all the same and it's the "local" ground that is important for an IC.
If you go around your circuit and all of your logic looks like logic (eg. either high close to the supply voltage or low near ground), then that will greatly reduce the likelihood of your issue being signal integrity issues. Your level shifters appear to be working well in the scope shots you have provided, so I'm less concerned about those at this point.
If you aren't sure how to write the suggested test software, let us know and we can help you out.
hoglet wrote:
Most likely this is executing an infinite series of BRK instructions.
That was my thought as well, meaning the 65816 is getting $00 over and over on the data bus. The ROM has large sections of zeroes in it, so it might be running in the middle of one of those blank sections during that capture. I'm not sure if it's a hardware, software, or FPGA issue, though. That's why I recommended a minimal test program that does not use RAM at all. We're making some progress on the hardware front of things with the scope, even if it's just determining which things probably aren't the issue.