So going off of
this the resolution we'd be looking at would probably be the first.
It comes out to 30.4Mb/s of bandwidth, but that's if we were feeding all frames directly. The VLSI chip has its own RAM and supports operations like block move and scrolling. If we load all the textures we can into the SRAM and move them from there, it gets significantly easier.
I'm going to bullshit some numbers here, just to get some thoughts going
I'm making an educated guess that most textures in DOOM are 32x32 pixels.
let's say we have 10 wall, floor, and ceiling textures per level, for a total of 30 textures for our "3D" objects (2.5d really).
Let's also say we have a large texture for the hud, maybe 128x128
And we need to throw in some character sets for displaying information, if we give each character 16x16, and only do numbers and letters (and we'll say a couple symbols, like %) we end up with about 40 characters
adding all this together, and assuming 16 bits for color (since the memory is organized x8 I believe) we end up with 67,584 bytes for all the textures in a particular level.
The VLSI has about 128k of RAM, so we can fit this into that comfortably, and I believe you can tell the chip to use a certain area of memory as the display section, and other areas are able to store textures or other garbage you may need in it.
This means we may not need to swap textures in and out of the chip at all during game play, which makes this task quite a bit easier, since we'll only need to calculate texture movements and send those as block move commands to the chip.