Something to note, which I've never completely got my head around, is that signed multiply and unsigned multiply do have to be tackled differently. I assume it's the case that if you've got one, you can do the other, but I imagine that's a lot easier one way around than the other way around.

It feels like a good idea! I'm not sure I'd bother with the squaring, since multiplication gives you squaring with only one or two instructions.

Is this an FPGA core? I'm used to FPGAs which have heaps of free multipliers, so single-cycle multiply is really easy. But I think there are some FPGAs without a multiplier. How many clock ticks do you expect your table approach to take?

I echo BigEd's recommendation regarding the use of the muiltipliers built into current FPGA families that you should be considering as hosts for your 6502 core.

If you just want to make a multiplier, I would recommend considering a Booth multiplier. BigEd, myself, and others discussed these types of multipliers on this forum several years ago. A 2 bits at a time Booth multiplier is not particularly large in terms of FPGA resources, and it would yield a full 16-bit signed result in 5 cycles. If you'd rather conserve resources, then you could construct a 1 bit at a time Booth multiplier using the on-chip registers. A one bit at a time Booth Multiplier will require 9 total cycles to yield a 16-bit signed product. For Booth multipliers, you will need a single 8-bit temporary register in addition to CPU registers you intend to use. Since the partial product additions are done using right shifts from lsb to msb, you will only need the 8-bit adder function you should already have in the core.

A small amount of additional logic is required to protect the sign bit of the product. Some modifications to the sign bit protection logic ought to suffice if you also intend to provide an unsigned multiply. For greatest flexibility, I would consider including both a multiply instruction for signed operands, and a another multiply instruction for unsigned operands. Finally, I think that because the X register is more often used that you should consider using the {A,Y} register pair instead of the {A,X} register pair.

I have a repository on github with some (Verilog) code examples for 1-bit, 2-bit, and 4-bit at a time standalone Booth Multipliers that you could review if that's the route you choose to go.

Good luck with your project. There are many ways to skin a cat. Do not let comments on the forum, or previous projects, discourage your efforts. We are always interested in seeing new approaches to the implementation of 6502s and extensions to its basic instruction set.

(You might be interested in the machines being discussed over at Arlet's challenge, which are intended to fit within 128 slices.)