6502.org

A bit easier than an RTI detector might be a time-delay fuse. I'm sure I've read of a machine which does this: there's a register to poke at which has a delayed effect, enough time to do something particular after you poked it. For example, if only supervisor mode can poke I/O hardware, then the supervisor can light the fuse then do the RTI. The delay is figured to be just the right amount to have the RTI execute in supervisor mode, but (for example) get its data from the user-mode stack.

Another approach might be to place the special RTI for returning from supervisor mode at a particular address, and detect that address instead of detecting the RTI opcode. Not sure, but there might be advantages.

One advantage is that addresses become valid early in the cycle, whereas memory-read data in a high-performance system is unknown until the cycle is almost over -- too late to test whether a just-arrived opcode is a $40. But if you're willing to invest in a latch to capture a copy of the opcode then you have lots of time in the following cycle to evaluate its "RTI"ness.

Another point to consider: an address test will involve more than just 8 bits (which argues in favor of using the opcode-test option instead).

Before my KK computer was built, its predecessor, my non-microcoded KIM-1, used a time-delay fuse to extend the address space. It wasn't triggered by pokes; instead the "fuse" (actually a nuanced timing spec loaded into a shift register from a lookup ROM) got set by undefined 'c02 opcodes.

Ah, thanks, it might well have been your idea I was thinking of. Although, I think maybe one of the minicomputers did it... yes, at least the PDP-8 has a couple of non-obvious delays:
"ION is used to turn on interrupts. The effect of this instruction is delayed one instruction so that, for example, an ION instruction immediately before a return instruction will not take effect until the return instruction is executed."
"CIF Delay is set whenever there is an instruction executed that could change the Instruction Field. These include CIF and RMF I believe. If you are including extended memory then you need to take this into account as well. The CIF Delay is cleared after any JMP or JMS instruction"

In our case the late availability of the instruction read is not a problem. You only need to know that the instruction just read was a RTI. We need to switch to user space only during the next PHI2 low phase, approx. as fast as the CPU emits the address to read from the stack. We don't have to be faster than the CPU. A delay is a good idea and was also my first idea (see my other thread using 74LS610), make an address the trigger to switch to user space after n PHI2 cycles.

That's assuming you are going to permit nested interrupts. When BRK or COP is executed IRQs are automatically masked, which means your ISR would have to intentionally re-enable IRQs once your front-end code has control—that is, after the MPU has toggled VPB and jumped to the start of the BRK or COP handler. However, you then have to have a way to tell the IRQ service routine (ISR) that you are already in supervisor mode, so it doesn't take you back to user mode when the ISR terminates. Such a function would have to be done in hardware, since the memory map could be changing as you switch from one interrupt level to another.

As for NMIs, they are best reserved for one high priority event that cannot be ignored under any circumstances. Using NMIs for routine processing can give rise to hard-to-diagnose deadlock problems (which I discuss in my 65C816 interrupt article). Many systems don't even pay attention to NMIs.
Atmel's logic doubling feature allows you to "borrow" unused product terms from other MCs, which may help you make a Yorky bark like a Rottweiler. You may want to add the following to the beginning of your WinCUPL code:
I suggest you write the logic without pin assignments (except the reset and clock inputs) and let the fitter assign the rest of the pins. If the design fits and synthesizes then use the pin assignments generated by the fitter, which you can find in the .fit output file. That's how I did the CPLD for POC V2, which is an ATF1504AS with all pins in use. I couldn't get the design to fit with my preferred assignments, so I let the fitter work it out, with only GCLR (connected to /RESET) and GCLK1 (connected to Ø2) specifically assigned to functions.

Assuming memory is gated so that it places the RTI bit pattern on the data bus the instant Ø2 rises ("instant" meaning within the guaranteed performance of the RAM—it is implied that chip select must occur before the rise of Ø2), the CPLD would "see" that RTI before the MPU does and act on it in nanoseconds. Unless the CPLD logic uses physical pins as nodes (as opposed to using pinnodes, explained in the CUPL user manual), the CPLD should be able to reconfigure itself to user mode well before the MPU starts pulling registers from the stack.

Nested interrupts do not need hardware support. As in most OS there is something called "stackdepth" a memory location that is incremented by the kernel for each interrupt level entered and decremented before exiting the level. So the OS knows when to go back to user mode (have a look at the sources of RSX-11S from digital, there it is also used to switch between kernal and user stack).

NMI is required as I want to avoid a user task to hog the CPU when it disables interrupts. I thought about using the NMI for the timer interrupt.

As I said the processing time for the RTI is of no importance at all. First the CPLD, or whatever logic we use, will see the RTI more or less at the same time as the CPU. Latest at the falling edge of PHI2 the CPU and the logic know that a RTI has been fetched. So our logic can switch to user mode on the falling edge of PHI2, even a slow CPLD would have activated user mode early enough for the CPU to continue with the right address space in the next cycle. Even more, the 2 cycles following the fetch of the RTI opcode are internal operations cycles.

As for dealing with Atmel CPLDs. The fitter is a piece of opaque software, you never know what it does. I have been surprised many times how bad this software is written and documented. I have designs that will not fit when the CPDL assigns the Pins and only will do it's job when you "help" him. Then I have designs that do strange things with the OE PT for bidirectional pins (they set int ON, which definitively does not work). And using the ATMEL PROPERTIES is another source of surprises. Even when you think you help the fitter by activating logic-doubling it is in some cases the wrong decision. And the documentation is virtually non-existing. I would really like to have more documentation, but even Atmel seems not to have more . If you by any chance have some more information than available on the internet would be great.