I have worked on several emulators such as Chip-8 and 6502 for NES emulation in various programming languages. One difficulty I have run into repeatedly is - you write the emulator and all the instructions by looking at a reference (or multiple) and then you have an emulator with, say, 50% of opcodes emulated before you start testing. Once you get into the testing phase, it becomes difficult to determine how well your emulator is handling some more complex processes (BCD, overflow handling, etc.). This is especially hard in languages like Python where you don't have native 8-bit types so you have to simulate unsigned 8 bit numbers with proper overflows.

What invariably happens is I write a small program that tests just the opcodes that are currently implemented, and then I run through a few hundred cycles and print out some logging for each function. Then I stare at it and ponder for days and slowly try to figure out how I misunderstood the explanation from the book.

This is an OK cycle but it ends up getting rather frustrating to the beginner because of the dependency of the opcodes, loader, register handling , etc. on each other.

I have seen Klaus' tests, and they seem like a great test for validating an already working emulator. There is also nestest.nes that tests opcodes for the NES, but this requires a PPU because it renders the results to the screen.

In a TDD development pattern, typically you write tests before you write the actual implementation. I have also written emulators based on TDD, where all tests are written before the emulator development starts.

The difficulty is two-fold. First, there is a dependency on the unit test developer on understanding the way that the opcodes should actually function. If there is a misunderstanding in the mind of the test developer, this misunderstanding will be translated into the test and in the final opcode development of course, so the code then passes an invalid test perfectly. The second is that there is a tight coupling between the development language and the unit test suite, because there is not a predetermined interface that decouples the two (for example test_and() will probably look something like (pseudocode)
def setup():
# initialize 6502, reset registers, set PC to 0x0100
6502 = new 6502()

def test_and():
6502.A = 0x0F;
6502.and(0xF1);
assert_equal(6502.A, 0x01)

You can see the tight coupling to the implementation. Yes, it would be possible to actually load opcodes and data into memory and step the emulator, but there are a lot of side-effects that may not be validated for (eg. you may check that the A register has the proper value, but are you also going to check that there were no other side effects on any other registers, that the PC advances properly, that the increment of the cycle count is proper, etc.? There are a lot of artifacts of each test and most either don't write tests explicitly, or write shortcuts to just test the component that is broken, as writing tests is high effort.

Another hurdle is properly writing a disassembler, step debugger with breakpoints, visualizer, etc. These tools are almost necessary to build the emulator but again have a dependency on understanding and interpreting some of the nature of a machine.

Let's talk about what a CPU is doing. A 6502 in this case is taking some input data (data loaded into memory, registers, etc.), executing some deterministic system (processing opcodes to determine function to be performed) one or more times and resulting in a set of side effects on that same data set.

essentially for each step it is a function, f(memory, registers, cycle_count, interrupts) => (memory, registers, cycle_count, interrupts)

So here's the idea, and please let me know if this already exists in some way or not. I really think this would bring a lot more people into writing emulators for all platforms if it is executed properly.

When writing a new 6502 or chip-8 or z80 or whatever emulator, the first thing you could do is to write a mapper of your memory array, registers, cycle counts, and interrupts to and from a standardized interface provided in a c library (C as most languages can easily wrap a C library with a ffi), give it a callback for a few functions, and then this entire external testing system with a GUI, a step debugger with breakpoints, an entire test suite per-opcode, etc. could all be loaded in a separate process and interfaced with your system. So one testing environment can support many CPUs and many emulators in many different languages.

For example, the C library api might look like this:
void reset(); // the implication here is that all registers, memory, etc. will be completely reset.
void load(memory); // this loads an array into memory
uint8_t *get_data(); // This would return data in a pre-determined byte-array per CPU type. there would be helper methods per CPU type to map a struct to a byte array.
void step();
void start();
void stop();

Then you could have a suite of tests that can isolate each op code and test it and give you expected vs actual results as they run.
The key is the unit test definitions, test runner, assembler/disassembler, file i/o, interrupt emulation, etc. can all be hosted in a separate process, so there can be a single test definition that all emulators could agree upon based on implementing this simple API.

Any thoughts on this approach and whether it's been attempted before?

Just a thought: instead of sharing and checking an image of memory, you could record the sequence of reads and writes. For writes, you need the data as well as the address. (For a very low level emulation, you need also to record the 'dummy' accesses.) If a test isn't sufficiently carefully constructed, it could possibly pass with the right memory contents even with the wrong sequence of accesses.