Disabling NMIB (with multiple sources)

[AndrewP]

This is a quick post (mostly for gfoot) kind of documenting how I want to deal with NMIs. It hasn't been spun into hardware yet and so may have issues I haven't seen.

The main goal is to be able to stop interrupts completely as I have a couple of operations that MUST be atomic*.

At its simplest the normal '816 going /NMI is run into the circuit below that captures it in a flip flop before ORing the flip-flops output with a positive mask. If the mask (called NMI Mask) is set then /NMI is held high and no interrupt occurs. If it's not set then /NMI can go low (depending on the state of the flip-flop).

However there are four NMI sources and so four flip-flops; all of those outputs must be ANDed together. If I reduced it to two sources then I'd be able to remove at least two ICs. But. This is for the fun of it so I'm happy to go overboard.

I have a fairly standard schematic layout where inputs are laid out on the left and outputs are on the right. Signals that are only used internally to are not shown on the borders.

The NMI mask can be set both programmatically or by bringing the Set Disable NMI signal high. It can also be cleared programmatically or by bringing the Set Enable NMI signal high**.

Write Enable NMI and Write Disable NMI are decoded from (some) address and bits 4 and 5 of the databus. A one in bit 5 disables NMIs and a one in bit 4 enables NMIs. Zeros are ignored because writes to that address also control clearing of individual interrupts (hopefully that will become clearer further down.

Lastly the state of NMIs can be read from that (same) address - called /NMI State - in data bits 0..3 and once an NMI is serviced it can be cleared by writing to those same bits. When reading if NMIs are enabled then a one is set in bit 4.

/WD and /RD are the fairly standard active low write and read signals derived from PHI2 and RWB.

The complete schematic is attached below and physical board will be a 32pin 22mm wide DIP***.

Off topic but. Kern[a]l (a.k.a. Superviser) mode is entered on any interrupt; software or hardware. Interrupts can be nested and User mode is only re-entered once all nested interrupts have returned. Tracked by an Up/Down counter that is triggered up by the /VP signal and down by an RTI opcode. The two signals that trigger the counter are also fed into Set Disable NMI and Set Enable NMI

Happy deciphering!

* Each User Program has Bank 0 mapped into it's own 64KB of memory which is allows each to have their own Stack Pointer and Direct Page Pointer. Absolutely awesome unless a poorly timed interrupt happens before the stack is properly placed.
** It's important that an RTI turns NMIs back on.
*** I've tried to use standard DIP sizes where possible and whilst the 68000 CPU came in a 64pin 22mm wide DIP package I don't want to waste that much space but also couldn't fit this circuit on a 15mm wide board.

HCP65 MPU NMI.pdf

(257.36 KiB) Downloaded 73 times

[barnacle]

AndrewP wrote:

*** I've tried to use standard DIP sizes where possible and whilst the 68000 CPU came in a 64pin 22mm wide DIP package I don't want to waste that much space but also couldn't fit this circuit on a 15mm wide board.

Hmm... components on both sides? Depends on your soldering technology, I guess.

Neil

[AndrewP]

I've discovered the hard way that putting ICs on both sides of the board ends up using more space than putting them only on one side. I found It makes it way less efficient to route vertical traces on the back when having to dodge pads.

On the board above I've only got decoupling caps on the back, and even those (0805s) take up a lot of trace space.

Previously I've tinned the pads of TSSOPs and then used hot air to solder them but I've finally got a hotplate (that hasn't literally exploded - thanks China) so it's time to experiment with that, solder masks and solder paste.

[gfoot]

AndrewP wrote:

This is a quick post (mostly for gfoot) kind of documenting how I want to deal with NMIs. It hasn't been spun into hardware yet and so may have issues I haven't seen.

Thanks for sharing!

Quote:

At its simplest the normal '816 going /NMI is run into the circuit below that captures it in a flip flop before ORing the flip-flops output with a positive mask. If the mask (called NMI Mask) is set then /NMI is held high and no interrupt occurs. If it's not set then /NMI can go low (depending on the state of the flip-flop).

However there are four NMI sources and so four flip-flops; all of those outputs must be ANDed together. If I reduced it to two sources then I'd be able to remove at least two ICs. But. This is for the fun of it so I'm happy to go overboard.

Where do the interrupts come from - are they from devices that expect an edge sensitive response, or are they the usual active-low ones? If the latter then with schemes like this I think there's a danger that the device's interrupt line does not clear e.g. due to a second interrupt while you're servicing the first one, but you reset your flipflop anyway, and ignore the second interrupt.

It may be better, if they are the usual level sensitive ones, to not use the flipflops. That way the ongoing state of your NMI output indicates whether the same, or another, device is still waiting to be serviced. So long as your ISR masks NMI while it runs and unmask it after servicing the interrupt, you'll get a second NMI at that point if something is still pending.

If the interrupts are edge-based though then it's a different story. But I've not seen any devices like that.

Quote:

Off topic but. Kern[a]l (a.k.a. Superviser) mode is entered on any interrupt; software or hardware. Interrupts can be nested and User mode is only re-entered once all nested interrupts have returned. Tracked by an Up/Down counter that is triggered up by the /VP signal and down by an RTI opcode. The two signals that trigger the counter are also fed into Set Disable NMI and Set Enable NMI

That's interesting. I'd considered using a two-way shift register for this, but have settled on a software workaround I think.

[Proxy]

a software workaround that would work is using a byte in memory to keep track of how many times the ISR was entered and only return to user mode when it's the first one. so it's similar to the hardware idea of using a counter, but manually done in software.
this means you need an atomic operation that checks a memory location and writes to it at the same time, lucky the 6502 has a few of those. specifically i would use DEC.

so imagine it like this:
a memory location is by default set to 1, whenever an interrupt of any kind happens the system decrements it and checks if the result is 0, if it is then the system knows it's the first to enter the ISR and that it should switch to user mode by the end.
so if a NMI happens while inside the ISR it would go back to decrement the same memory location again, down to $FF this time, which is not 0 and therefore the system would know not to enable user mode when returning as it's a nested interrupt.
of course every ISR increments the memory location again before exiting via RTI.

a slight alternative would be to have the memory location start at 0, decrement it at the start (but don't check the flags), and then near the end of the ISR it would increment the memory location again and then check the Z flag and decide right there if it should switch to user mode or not, saving having to remember the result of the decrement through the entire ISR (likely on the stack via PHP).

Using a single byte (which is the only atomic option the 6502 has) allows for ~256 nested interrupts which should be more than enough... because if it ever gets that nested the stack would start overwriting itself anyways.

[BigDumbDinosaur]

Proxy wrote:

...this means you need an atomic operation that checks a memory location and writes to it at the same time, lucky the 6502 has a few of those. specifically i would use DEC.

With the 65C02 and 65C816, TRB and TSB are atomic test-and-write operations.  What is particularly useful about those two is they report the state of the location being accessed prior to changing it.  That allows the code immediately following TRB and TSB to modify its behavior if the location being accessed is actually changed.  I use those two extensively in my programs for processing bit fields.

[Proxy]

True but I like INC/DEC more in this case because they don't require the accumulator to be set up prior to the instruction, meaning you can use them at the very beginning of an ISR to reduce the chance of an NMI hitting before the flag could be set in memory.
i don't know if an NMI can be acknowledged if it hit during an interrupt sequence, if it can't then using INC/DEC should be fully atomic, otherwise there will always be a small chance of stuff breaking.

[AndrewP]

gfoot wrote:

Where do the interrupts come from - are they from devices that expect an edge sensitive response, or are they the usual active-low ones? If the latter then with schemes like this I think there's a danger that the device's interrupt line does not clear e.g. due to a second interrupt while you're servicing the first one...

This is one of those annoying questions that makes me stop and ask "What am I doing?"

The thinking I had when I designed the above board is that I would have four sources of NMI:
1) A timer - to invoke the preemptive supervisor.
2) A signal from the second '816 indicating that it needs attention.
3) A memory access address based break point.
4) A push button for testing.
(All of which are edge signaled)

Expert knowledge.

It's something we humans are really bad at weighting appropriately; something I know is true for myself. BDD has written about NMIs on this forum* and why only one source is a good idea (or more practical). And I have to weight that against my own "but it's so cool to have four NMIs!" thinking. He's right and I'd imagine he has hit all manner of strange corner cases, things I haven't the experience to dream of.

Looking at the complexity in both hardware and software needed to replicate what IRQs are doing already makes it just not worth it. It was a fun learning experience but... nope. Looking at my NMI sources:
1) A timer - this is required
2) Another '816 - it can use an IRQ or be polled from kern[a]l mode. The NMI would ultimately have only set a flag saying that there is a message in the pipe that needs processing.
3) An address break point - most break points I intend to set by using BRK instruction trickery. Do I really need a break point on a non-opcode address? It would be cool but. No.
4) A button - this was just for testing multiple sources on NMI anyway.

With only a timer remaining I don't need to mask out NMIs. It's a software bug if one NMI can't be completely serviced before another occurs. That's not something I need to build hardware around.

So now that I've taken everyone down the garden path I'm rolling the idea back. Hopefully it's triggered thoughts in others; and I'll do another post below on why multiple NMIs cause software issues.

* I can't remember where though. I'll update this post to link back to it if I can find it.

[BigEd]

> It's a software bug if one NMI can't be completely serviced before another occurs. That's not something I need to build hardware around

I think this is really important - a lot of extra ideas pop up in hardware discussions, and it's worth considering what they are coping with - and whether or not they need to. Almost always in the land of hobby 6502, we're not dealing with malicious input, only dealing with programming mistakes. A reset is almost always a reasonable way to keep going, if indeed there's any need for anything. The stakes are, in hobby land, very low indeed.

[AndrewP]

Here's why multiple sources of NMI in my code are problem. And it mostly hinges around changing the stack pointer before an RTI is executed.

I'm using a 65C816 and allocating / remapping User Mode program addresses in 64KB banks*. For example the first user program is given a PID of $FF and has bank remap addresses from $7FFFF (user program bank $FF) to $7FF00 (user program bank $00). The second user program is given a PID of $FE and has bank remap addresses from $7FEFF to $7FE00. If $FF is in the user program latch and memory address $7FF00 contains $05 then when an address of $00ABCD appears on the 65C816's bus the actual address presented to RAM will be $05ABCD.

A user program believes it is running in memory $00xxxx but is really running in memory $05xxxx. And now we start coming to the problem. Each user program has a full bank zero to play with. That allows each user program to have its own Stack Pointer and Direct Page pointer. However the caveat is that when the Kern[a]l switches back to User mode it must set both the Stack Pointer and the Direct Pointer just before the RTI occurs. Specifically the Load-stack-pointer and RTI code must be atomic from the point of view of stack usage. i.e. interrupts.

If an NMI occurs just after the stack is updated but before the RTI triggers an exit from Kernal mode then random Kernal memory will be stomped by the interrupt's stack pushes.

The following demonstrates normal operation:

The teal blocks are the individual interrupt and RTI cycles so the point of change of both NMI being enabled and disabled can be seen.

The problem occurs in the "Disable NMI (in software)" portion - there is no easy way to guarantee that an NMI cannot occur during it. If an NMI occurs during the instruction that disables NMIs then they could be re-enabled and then another NMI could occur after the stack pointer is changed but before the RTI. It's a corner within a corner but random unexplained program crashes are the suck.

(Another good side effect of restricting interrupts that can occur in Kernal mode is that probably none will. There are unlikely to be nested Kernal interrupts and that means I can swap out the 74F269 up/down counter I'm using for an 74HCT193 - an IC that's much easier to source.)

* The address selection is similar enough to gfoot's that I'm happy I seem to be on the right path

[gfoot]

AndrewP wrote:

If an NMI occurs just after the stack is updated but before the RTI triggers an exit from Kernal mode then random Kernal memory will be stomped by the interrupt's stack pushes.

I see what you mean - it's not a problem for me on the 6502 because it will only corrupt page 1, and there won't be anything interesting in there anyway - but as I understand it, on the 65816 the stack pointer can be anywhere in the bank, and it's probably not viable to have the kernel just not use bank zero at all!

So it is then necessary to be able to mask NMIs, and...

Quote:

The problem occurs in the "Disable NMI (in software)" portion - there is no easy way to guarantee that an NMI cannot occur during it. If an NMI occurs during the instruction that disables NMIs then they could be re-enabled and then another NMI could occur after the stack pointer is changed but before the RTI. It's a corner within a corner but random unexplained program crashes are the suck.

As I understand it, the issue is that an NMI could have been latched just prior to the disabling, but not processed yet - this is something I observed in controlled tests recently. I'm not sure that it is specifically related to having multiple sources of NMIs though.

Could you do something like this to work around it?

Code: Select all

    1. disable NMIs
    2. are NMIs disabled?
    3. if not, loop back to 1

I think this would resolve the race condition as if the NMI crosses with the disabling instruction, then the NMI will get processed after it, re-enabling NMIs on its return, but then the test in step 2 will fail and you'll loop back to try again.

Quote:

(Another good side effect of restricting interrupts that can occur in Kernal mode is that probably none will. There are unlikely to be nested Kernal interrupts and that means I can swap out the 74F269 up/down counter I'm using for an 74HCT193 - an IC that's much easier to source.)

I think there can still be situations where you want interrupts to get processed, otherwise the kernel is restricted to very short operations. Imagine something like clearing a page of memory - you don't really want interrupts disabled while you do it, it may work but will probably delay I/O responses more than is acceptable. You could offload that to a user-mode helper process of course, as is quite commonly done in modern operating systems, or find a way to support the nested interrupts reliably.

Regarding IC choice there, I had considered doing something like this using a two-way shift register, functioning as a one-bit-wide stack. At the moment though I think I'm going to be able to manage this in software - though it's early days.

[gfoot]

AndrewP wrote:

Looking at the complexity in both hardware and software needed to replicate what IRQs are doing already makes it just not worth it. It was a fun learning experience but... nope. Looking at my NMI sources:
1) A timer - this is required
2) Another '816 - it can use an IRQ or be polled from kern[a]l mode. The NMI would ultimately have only set a flag saying that there is a message in the pipe that needs processing.
3) An address break point - most break points I intend to set by using BRK instruction trickery. Do I really need a break point on a non-opcode address? It would be cool but. No.
4) A button - this was just for testing multiple sources on NMI anyway.

Going back to this bit briefly - I think for me the razor would be, is it urgent? If the interrupt is not urgent then it can wait until the previous one returns and be handled afterwards. As an example, I/O without a FIFO is urgent, if the data will get lost within a short space of time if it's not consumed. Slow I/O devices aren't really urgent, nor are ones with deep FIFOs. System timer interrupts are not really urgent.

Another way to do that, for something like a timer interrupt that performs general system housekeeping but doesn't access I/O hardware, would be to re-enable interrupts soon after being triggered, allowing other more urgent things to still jump in if they need to - so that's an alternative to NMI for some of these things.

In place of NMI, the ARM architecture has a thing called FIQ - "fast IRQ" - that is used for really urgent I/O devices, and has higher priority so it can interrupt IRQ handlers that are already in progress (but not other FIQ handlers) I believe.

[AndrewP]

gfoot wrote:

I see what you mean - it's not a problem for me on the 6502 because it will only corrupt page 1, and there won't be anything interesting in there anyway - but as I understand it, on the 65816 the stack pointer can be anywhere in the bank, and it's probably not viable to have the kernel just not use bank zero at all!

Yup, havine a full 64K of stack space is so liberating. And to proselytize for the '816 it can also place Direct Page (the 6502's Zero Page) anywhere in bank zero. That is extraordinarily useful because DP can used as the stack for a C++ program with all the usual ZP addressing modes.

gfoot wrote:

Could you do something like this to work around it?

Code: Select all

    1. disable NMIs
    2. are NMIs disabled?
    3. if not, loop back to 1

That would solve it and together with Proxy's software Kern[a]l depth solution above I could get away with an almost entirely software solution. And I was quite excited for that until I remembered that Kernal mode forms a part of my address decoding and has to be available in hardware.

gfoot wrote:

I think there can still be situations where you want interrupts to get processed

Absolutely! But I'm hoping I can get away with 4bits of interrupt depth by polling IRQs (one 16bit read) as part of often called Kernal methods. As you say, early days, and I haven't written any of this code so it's all a bit hand-wavery.

gfoot wrote:

Slow I/O devices aren't really urgent, nor are ones with deep FIFOs. System timer interrupts are not really urgent.

Another way to do that, for something like a timer interrupt that performs general system housekeeping but doesn't access I/O hardware, would be to re-enable interrupts soon after being triggered, allowing other more urgent things to still jump in if they need to - so that's an alternative to NMI for some of these things.

Yeah, I would probably want IRQs to be enabled in *some* Kernal methods but I am trying to steer away from any devices that have tight timing requirements. Only time will tell if that's possible...

The shift register is a cool idea (two together would give me same interrupt depth as a '193) and they're cheaper and faster.

[barnacle]

Proxy wrote:

Using a single byte (which is the only atomic option the 6502 has) allows for ~256 nested interrupts which should be more than enough... because if it ever gets that nested the stack would start overwriting itself anyways.

I think it will already have overwritten itself at 128 interrupts... and probably before, given other stack use.

Neil