6502.org

I'd say that HEX maps to four bits though, not eight.

I was going to say the same thing, but then I thought that since one hexadecimal digit can represent any combination of 4 bits, and 8 is a multiple of 4, it's still technically correct. You just need two hex digits per 8-bit byte.

Sure but common quantities in assembly code are whole bytes, not nybbles.

Even after 30 years of exposure to it I still find myself not quite having hex sit comfortably in my mind. But it is very useful to see hex as a first class base and it will improve any (assembly) programmers productivity immensely if it comes naturally. The obvious application is addresses, where it makes very little sense to specify addresses in decimal. In my own assembly code I reserve the use of decimal for loop counters, record lengths, graphical positioning and similar things. Binary is useful too, especially for hardware register writes and such like, where the datasheets nearly always describe functionality in terms of register bits. But for memory addresses hex is nearly always far more descriptive.

There's a not-quite-obvious consequence of hex being so well-suited for bytes and addresses, which is that with four bits expressed in one character, it's so easy to convert to binary that the binary more or less jumps out at you, once you're used to it. So, you can tell for example $9A has the top bit set, because you know 9 is 1001 in binary. Having a ready conversion between bit patterns and numbers is handy when you're doing bit-twiddling data manipulation or figuring out address decoding.

The great advantage of octal is that you only use digits for digits, but I've never found it natural. That's most probably because I came across hex first. I can imagine if I'd learnt octal first I'd find hex a bit of a strain.

(Another consequence of getting used to hex is that you have a chance to realise that numbers are a kind of thing which is different from the way we write them down. Indeed, much of the time the bytes we deal with are not numbers, but maybe characters, or distances, or angles, or bit patterns. One of the insights of computer science is that computers deal with symbols. Letters and numbers are some of the possible interpretations of those symbols.)

Basically: learn hex, and level up!

If you were working with 9/18/36 bit systems, then I'm sure octal would be just as natural to you as hex is to 8/16/32 bit systems. Some of the 18/36 bit systems even subdivided their words into 3- and 15- bit field pairs, so having 3 bit wide digits in octal was very appropriate to slicing up and bit-masking values.

But regardless of word sizes, everybody should learn binary. And once you do so, you start to see the advantages of digits holding exactly N binary bits, in these base-2ⁿ numbering systems.

Back to his original question. It isn't obvious as to if he intends to do signed or unsigned math.
All negative numbers are less than 16, in signed math, while in unsigned math the msb being set would be larger than 16.
I think he is doing signed math but since the code is wrong, I'm not sure.
Dwight