So testing on your other system has verified that your software should be doing the right thing and that you probably have your pinouts correct.
What we don't know is if the software is actually running.
I see multimeter leads in your photo, so I know you have that. Do you have an oscilloscope? That would be the best tool for looking at the signals and making sure they "look correct".
If your working machine has the same memory map, then you can look at each address, data, and control line on that board and compare to this system, just in terms of what the signals generally look like (eg. high most of the time, alternating rapidly, etc) and will also show if you have signal integrity issues (eg signals not going fully high/low).
You need to break the problem down into little pieces and then devise a simple test that will eliminate possibilities. Here is my understanding of your current position:
Code:
1. You have software that should work on this sytem.
2. Your ACIA is constantly spitting out garbage and you suspect the issue might be here.
3. We do not know that your FPGA board is working properly.
4. We do not know that the 65816 is working properly.
5. We do not know if your emulated ROM is working properly.
6. We do not know if your emulated RAM is working properly.
7. We do not know that your emulated ACIA is working properly.
8. It's likely that the connections are correct.
9. We do not know if there are power or signal integrity issues.
It would be good to test a subset of these with the goal of narrowing things down to just #2. We need to assume at least some of this stuff works, but should be open to the possibility that it does not.
If this was my project, because I have had signal integrity issues before, I would likely start by reducing the clock speed. That might make #9 better, but really shouldn't affect anything else. Theoretically, you could go so slow that you could follow along with your multimeter, or even single step. Your FPGA is a very powerful tool as you can add and subtract things from the bus with ease. You can even add a snippet of code that stops the clock at an exact address or bus cycle.
Without an oscilloscope, I would likely write myself an 8-bit latch in the FPGA that just latches when a particular address is written to (can even be an in-use address as this will be a write-only device on the bus). I would then write a small assembly program that does not use any ram (so no JSR - just a single routine) that writes a value to the latch and then you can check it with your multimeter or LEDs+resistors. Put a label at the end and have it jump to itself at that label in order to stop the program while continuing to "run". In that state, only your lowest address bits should be changing and you can check the upper address bits, even with a multimeter, as they shouldn't be changing.
If that works, then you have much more confidence in your FPGA, 65816, emulated ROM, and your connections, and you now have a debugging port that you can write to at various places in your code to see where your program actually runs. This is the hardware equivalent of printf debugging, only you have much more power because you can bring any signals you want to a pin - and even latch them if you want so you can slowly look to see if something happened or not. If this simple test does not work, then you need to focus on the parts that it uses (FPGA, 65816, emulated ROM, connections (even though you think they are all good) and signal integrity (which includes your level shifters).
I am encouraged to see that you're not trying to do this on a breadboard and that your connections are likely short (although I can't see them). To that end, we'd be interested in seeing the back of your board as well. I'm interested to see how your signals and grounds are routed. We'd also love a schematic of where you are right now. It may take some effort to make, if you are working without one, but it will help both you and us determine what you
think should be connected to what.