My fast optimized unexpanded VIC game of life fit in 976 bytes (this includes a BASIC stub to start the machine code, but for purposes of the challenge the BASIC code can be ignored and the code directly started with a typed in SYS call.) For purposes of the rules in this challenge, I think an additional 8 bytes of ROM are utilized - the character ROM for "\" is used as a table to calculate 2^X (where X is from 0 to 7).

http://isaackuo.cloudapp.net/stuff/dynalife/

My first version actually fit in 641 bytes, including a pixel editor (both keyboard and joystick input) and code to dynamically allocate custom tiles where needed (the unexpanded VIC doesn't have enough RAM to provide a bitmap to the entire screen...I dynamically allocate and trash collect tiles where 1+ pixels are set).

But the routine for running the Life simulation was way too slow for my tastes. I kept improving things until I got what is perhaps the fastest full screen Life simulator possible on an unexpanded VIC. It does a lot of weird stuff in parallel, and crams various tables and arrays in RAM that the video chip doesn't have access to (including the weird nibble memory of the alternate color page).

It's generally annoying to use [the nibble-wide memory] because the upper 4 bits could be read as anything. I use it as a temporary store of the lowest two bits of a vertical strip of pixels. Writing to it is simply copying the currently processed byte to the nybble store...don't care that the upper 16 bits are discarded.

Then, when it gets read, an AND mask wipes everything except for the lowest two bits anyway. So there's no performance penalty for dealing with the random junk read in the upper nybble.

Basically, the program processes things in vertical strips, top to bottom. But it's not aligned how you might expect. The input is a 10 pixel wide strip including two pixels from the previous column. The output is an 8 pixel wide strip which includes one pixel from the previous column and only including 7 pixels from the current column. You might think that the most natural thing would be to output the 8 pixels aligned with the current column of tiles. But that turns out to be less efficient since it has to draw data from three different columns, and it's also a nightmare keeping track of last turn's pixels vs next turn's pixels, unless you have full double-buffering. On the unexpanded VIC, RAM is way too tight for full double-buffering.

Instead of double-buffering, I store a temporary copy of a 2 pixel wide strip. The 3x4 bytes used to store neighbor totals aren't corrupted by the flipping pixels because the currently written row was already read and summed via a couple table lookups. But that thin vertical strip needs to be saved so the previous turn's version of those pixels can be used for calculations by the time the "paint roller" moves on to the next column.

The most important optimization was the system which tracks "clean" tiles so they can be entirely skipped. But the various weird things done in parallel roughly double performance compared to a more naive optimized implementation.

I was trying to keep memory usage low and frame rate high because I had this idea for using Life as a basis for a trilogy of videogames:

LIFECOMMAND - a bioweapon/zombie apocalypse themed Missile Command variant, which I did actually revisit in HTML5 form: http://isaackuo.cloudapp.net/ijk/

LIFETEROIDS - after evacuating Earth, the generation ship Exodus must survive waves of aliens (Life spaceships)

LIFEBOTRON - Not really sure how this one would have worked, but it would have featured robotron style action defending the colonists from the aliens in their new world

Hmm...I found some of the opening title text I had started writing, designed to fit within the tight confines of the VIC20 screen. It would have ticked onto the screen character by character, like a teletype.

Opening/Ending of LIFECOMMAND
--------------------
IN YEAR 199X A.D.

THE ALIEN ASSAULT
THREATENS THE FEW
REMAINING CITIES.

STRATEGIC DEFENSE
PROTOTYPE MITHRIL
IS THE ONLY HOPE.
--------------------
IN YEAR 199X A.D.

STRATEGIC COMMAND
COMPLETES THE ARK
SPACESHIP EXODUS.

THE EARTH IS LOST
TO THE ALIENS BUT
MANKIND SURVIVES.
--------------------

Opening of LIFETEROIDS
--------------------
IN YEAR 199X A.D.

THE ALIEN ASSAULT
DESTROYED THE FEW
REMAINING CITIES.

THE ORBITAL ALIEN
ARMADA INTERCEPTS
SPACE ARK EXODUS.
--------------------

When I was coding this stuff, I didn't do any research into sophisticated Game of Life algorithms. I was just doing what was straightforward to me. Better to code something you make up yourself and understand, than to try and code based on something you don't understand.

And the unexpanded VIC's tight RAM prevented me from even considering the various algorithms that pop up when you do a cursory search on Game of Life algorithms. (They use huge table lookups.)