I have investigated these white lines artifacts a bit more. By changing the code I confirmed that they only occur during writes to video memory, and also only when writing variable data - they don't seem to occur when writing the same value over and over again. Making the system sit in a tight loop writing $00 then $FF over and over again causes a lot of artifacts.
There is a fair amount of noise on the power supply lines in general, and especially in the VGA circuit. This has pointed to some issues with the computer circuit that this is attached to at the moment, but they're not what causes this issue so I'll post about those separately when I've investigated them more. It was clear though that there's a very specific dip/ringing in the VCC line during these video RAM write operations that is obviously very suspicious:
Attachment:
20231230_112817.jpg [ 275.47 KiB | Viewed 594 times ]
The top, yellow trace is VCC, measured at an extreme point on the circuit, a long way from the power supply. It is actually even worse in some other parts of the circuit but this happens to be what I was probing when I captured the traces. The trigger is set to 4.36V here; generally I tweaked it until the scope triggered fairly often, but not all the time, so that I could be reasonbly confident I was looking at the same artifact each time.
The red trace that goes up and down a lot is the /COE signal which drives the counters' output-enable pins and the video RAM's output-enable pin. It's at 1V per division, so it's swinging over about 4V. It cycles every four VGA pixels - for four of them these chips are outputting, and for the remaining four they are not; and during those remaining four, sometimes, a write occurs instead. Otherwise the buses float. It is tempting to add a pull-up to this line, to help it get up nearer 5V, as the counters are HC devices not HCT. Although the GALs don't always pull all the way up to 5V, they are quite good at pulling down to 0V, so the pull-up resistor does a good job of fixing the high levels without harming the low levels.
The blue trace that is set to 5V per division shows the opposite output-enable signal - /XOE. This was originally named as the output-enable for some transceivers, but they've been replaced with registers now. These drive the video address and data buses during write operations, whereas during the read half of the cycle the counters drive the video address bus and the RAM drives the video data bus. The start of this signal is delayed slightly after the end of /COE (red trace) and you can see that when it does start, that's where the dip and instability in VCC occurs. There is also some instability in VCC at the opposite end, where /XOE ends and /COE starts again. Unlike the red trace, the blue one is managing a good 5V swing - it is driving far fewer ICs though.
There is not much more I can do in terms of decoupling for these chips - some artifacts like these are inevitable on breadboards. It is possible that using HC registers rather than the current 74AHC574 ones would be a better choice though, as it appears the drop in VCC is due to these registers driving the video buses. Indeed, probing lines on the video data bus shows that they are all rising from low to high at the point of the instabliity - so it seems more related to the video data bus than the video address bus.
I don't have 74HC574s, but I do have some 74HCT374s, so I swapped the data bus register (U9) over to one of those - but it did not seem to reduce the voltage drop. The only thing I've found which does help here is to add pull-up resistors to some of the video data bus lines (pins 12-19 on U9) - adding one low-value resistor like 470 ohms stopped the visible artifacts on screen, and adding three or four higher value resistors (5K6) was also enough to remove the visible artifacts. I think the reason this helps is that after /COE ends, this bus is floating, and the resistors can start to pull it up a bit, meaning that when /XOE begins the data bus line is already nearer its target voltage level.
Attachment:
20231230_121729.jpg [ 260.35 KiB | Viewed 594 times ]
This solution isn't great, but may be the most that can be done on a breadboard without expanding the circuit a lot. I expect these issues will get better when this is on a PCB, as it'll be possible to provide better decoupling and ground return paths between the relevant ICs, and reduce the amount of wiring in the bus itself to reduce its overall capacitance.
Another option that sprang to mind was to insert small series resistors between the tristate register's output pins and the data bus lines, to reduce the transient currents when the register's outputs are enabled. I believe this would have helped, but there's no space for that on the crowded breadboard.
Finally I could change the way the bus multiplexing works, to use multiplexers instead of tristate devices, as Jeff has suggested in other threads. Whether that helps with this issue will depend on the multiplexer, but a big advantage there is that the bus that's being driven would be very short, so easier to drive back and forth.
Any other thoughts on solutions are welcome, but in general I'm happy to leave this as it is, with the pull-up resistors, and when I expand it to support text attributes, or make a graphics-based version, then I can consider either making space for the series resistors, or using multiplexers or another solution instead.