But as for the cursor, isn't this what XOR cursors were invented for? You don't need to map the damage, you just re-apply.
You're correct in that regard. Using Commodore's VIC as an example, which lacked a hardware cursor, the kernel merely XORed the character under the cursor to produce the illusion of a cursor. Bit 7 of the screen code determined if the character was in foreground or reverse mode, so the IRQ handler could simply toggle bit 7 at whatever rate the cursor was supposed to flash. If a static cursor was to be displayed, bit 7 was set and remained so until the cursor was moved to a different location on the screen.
However, the Commodore kernel was directly manipulating display RAM, which can't be done with the Picaso µVGA-II SGC module. The Picaso has several functions for writing characters to the display but none for later reading back what was written at any given screen coordinate. Hence a display map in RAM would be required that would be a "shadow" of the screen contents. In fact, two such maps would be required, one for the character data and another for character attributes. Nearly 4K of RAM would have to be allocated for this purpose. Complicating matters, a way to write to the module during the IRQ without messing up foreground activity would be required.
It all comes down to just how much time and system resources am I willing to consume just to achieve one function. If I'm going to go through that much trouble just to have an interactive cursor I might as well use a video display controller with dedicated video RAM, where I'd have even more control. However, this is a lot to bring to bear just for a basic console that will always operate in character mode.