lib65816 seems to be the best option for cycle-accurate 65816 emulation. The problem I have with it is that it takes over the main loop of the program (something I'm not a fan of in any library except for GUI-central ones), which makes emulating a whole system outside of lib65816's scope more difficult.

For one, I need to simulate wait states from reading certain sections of the address space, and when copying video data from RAM to the display device.

Anyone has any tips for dealing with this?

I think it's common enough for general machine emulators to run some number of instructions and then handover to the emulation of peripherals such as video, and this turns out not to be a good match for highly detailed emulations which need accurate cycle-by-cycle accesses.

Possibly a good start would be to take the leaf routines from lib65816 and rework them into a different framework. But then, these routines won't be taking care to perform cycle by cycle activity, so they might need a fair bit of work.

For hints on approaches, I do recommend the blog posts on the construction of JSBeeb: it's a different language, but a very accurate simulator of a 6502 system.
https://xania.org/201405/jsbeeb-emulati ... javascript

I see. I probably won't do that, though, since lib65816 is very intimidating, and I know little about the 65816's internal behaviour. I'd rather just lose that level of accuracy for getting the system emulator out sooner (and it's my own design, so I have more freedom there).

Sounds like a good idea! Cycle-accurate emulation should only be needed for the most intricate low-level code: graphical demos and copy protection routines spring to mind. And anything which races the beam. On a new system design none of that need apply.