oooh, interesting project idea! i hope you go through with this one.
i've had a similar idea a while back, using one of these cheap TFT shields designed to directly plug onto an Arduino (most of which use the same controller as the DT028BTFT you mentioned):
Attachment:
tft-lcd-touch-display-shield.png [ 1010.18 KiB | Viewed 6907 times ]
320x240 @ 16-bit or 18-bit color with an 8-bit parallel interface (instead of SPI like they usually have), plus a built in touchscreen and µSD card slot.
so all you need is a CPU, some RAM, ROM, and logic to combine everything and you'd have a complete self contained system that (besides a µSD card and power) doesn't require any external connections like a keyboard or UART to function. though of course that does require some software to take care of user input, the display, and a file system.
then you just need a battery setup with a protection/charging circuit and you would have the worst DIY smartphone in the world!
but i've shelved that project for now because i did a bit more digging about these shields and found that while they boast both 3.3V and 5V operation, they will not function without a 5V supply and only the LCD controller has some level shifters, and while the µSD slot does get it's 3.3V from the onboard regulator, the data lines are directly connected to the pins on the side, so on a regular 5V arduino using a µSD card with this for a long time will likely damage the card.
so overall don't buy these if you plan on using the µSD slot.
but the overall idea is not dead. you could still try something similar by just using
Adafruit's variant of the board. while it's obviously a lot more expensive, it's also much higher quality and larger, leaving more PCB space for the board that it would plug into, plus it has proper level shifting and it does actually work with only 3.3V power and IO. (which i'd recommend anyways if you plan to go battery powered).
running everything at 3.3V would also give you the chance to just go with an FPGA like a MachXO2 which are relatively affordable.
i know you said you wanted to avoid an FPGA, but honestly i don't thnk a 65816 can feed the LCD data fast enough for a usable framerate...
let's just do some math:
If set the display to 16-bit color mode, every pixel needs 2 Bytes of data. so for 320x240 that's 150 kB which need to be send within ~16.6ms to achive 60 FPS. (assuming the worst case where each pixel needs to be updated).
that works out to ~108ns per Byte (or 216ns for 30 FPS), the controller has a write cycle of 60ns so it could in theory be possible, but definitely not if the CPU does it manually (even at +20MHz).
so using an FPGA to implement some hardware acceleration, like a DMA controller/Blitter, will help the CPU out massively.
you could go a step further and have the CPU work with 8-bit color and have the DMAC convert the 8-bit color to 16-bits before sending it to the LCD using an internal look up table or similar. this would obviously lower the total colors from 65k to 256, but would make operations like copying and blitting twice as fast while also reducing the memory footprint.
welcome to hardware design, where every decision is a trade-off!
anyways hope you continue this project and keep us updated!