For me, modularity and encapsulation are useful ideas here: a video chip is a particular fixed function, which happens to be quite complex and which happens to have been available back in the day. For me personally, any kind of subassembly which does the job of a video chip, and only that job, is a reasonable substitution, especially if it allows for the project as a whole to be finished sooner, or more easily.
Many projects use a USB adapter chip of some kind, without worrying too much that it has a CPU system inside. It's encapsulated and dedicated. So, for me personally, using a programmed FPGA board, or a programmed microcontroller, for some specific subsystem, is almost always a reasonable thing to do.
Given that, there's an argument to be made that you should have no issue with using a CPLD or an FPGA: those are effectively the modern hobbyist equivalent of the ASICs that commercial designers could call upon to be built for them if they had sufficient financial resources to do so. Effectively, you're just taking on the job of the MOS chip designers who were working for Commodore, as well as the system designers using those chips. Alternatively, you can go the route of programming an FPGA with someone else's design such as a TMS9118 clone, which is the modern equivalent of ordering a TMS9118 from TI.
For my MicroPET I decided to optimize for space, so I used 8bit video bus, separate from main memory (video has prio) run at 12.5 MHz, for a VGA output of 640x480 (400 actually used). For B/W I need two video reads per 8 pixels - one videobdata and one for char 'ROM' if in character mode. 25MHz pixel clock means two pixels per memory access or four memory accesses per 8 pixels. So half the memory bandwith is CPU the other half is Video.
Adding colour means reading an additional byte of color (in this case per 8x8 pixel cell), so only 1/4 of memory bandwith available for the CPU.
Thanks for sharing your insights folks - I'm learning a lot from your experience.
I'm starting to realize that with my current skill level I should put my "purist" ambitions on hold and try doing some simple FPGA work first that will have its own VRAM, accept instructions from 6502 through several memory-mapped registers, and do some composite video rendering.
And most importantly - soft-core approach will be faster in my case, so time-to-completion will be lower than going for a hardware-only solution (not counting the old stock chips, of course, which would be the fastest).
Effectively, you're just taking on the job of the MOS chip designers who were working for Commodore, as well as the system designers using those chips.
Thinking about this, it actually sounds really exciting & encouraging!
Doing some simple math, 320x240 with 8-bit color = 76800 bytes of memory. That's if you don't overscan or anything else. Typical expected memory on a simplified level would be 128K, just for video, yet the 6502 can only officially access 64K. If you have a 6502 running at 1 MHz for low end users, or even something around 4 MHz let's say, and updating each pixel on the screen takes at minimum LDA/STA statements, you are looking at a LOT of time to update a single screen. Imagine running this thing at 60 FPS!
As Yuri said, it may be mitigated by offloading most of poking logic onto the video chip and having it maintain its own VRAM. But yes, having 6502 poke every pixel would be unrealistic.
TL;DR: I'm going to play with my MachXO2 and see what it can do. It's going to be more effective than trying to chase unrealistic "purist" goals, at least for now.