Easy data access from an SDcard

[BigEd]

Elsewhere:

Oneironaut wrote:

I have decided to use an SDCard as the external cartridge as a bare SPI Flash memory is just too expensive.
In order to keep the boot code ultra small, I will read the SDCard in SPI mode (Raw data only).
But to be tricky, I have made a PC program that adds a secret "key" to the 6502 memory image.
This will allow the PC to use FAT16, FAT32, or EXFAT and read the SDCard as normal.
The 6502 code will simply hunt through the sectors until it finds the key.
Finding the key will let the 6502 know that what follows is 64K of memory to write to SRAM.
The key will be a sequence of 256 values starting from 0, and ending on 255.
It is unlikely that this sequence could just accidentally exist, and makes sensing code ultra small.
...
Cheers!
Radical Brad

This is an interesting idea, and might be worth discussing in its own thread. So here's a new thread!

I can think of two things which might trip up this plan... I think both of them will be OK if the FAT filesystem is formatted before putting the special 64k+header file onto it. And they might be OK anyway.
- I'm not sure FAT will always put the special file into a contiguous range of sectors.
- I'm not sure FAT will always overwrite an existing special file when it's updated, so you might find a previous version instead of the latest.

Not sure if it matters, but it seems sectors on an SDcard are 512 bytes - that's the unit of read/write you get - and clusters on a FAT filesystem are from 2k bytes to 32k bytes depending on how it's formatted

[Arlet]

Also keep in mind that there may be huge amounts of space between the start of the SD card and the FAT, depending on the manufacturer. This could be a few megabytes. It could take a 6502 a long time to scan that much data.

[BigEd]

Interesting point.

(Brad clarifies that this scheme needs only to load enough 6502 code to be able to interpret the FAT properly.)

[Arlet]

Well, there's always the option of putting a secundary bootloader in the bootsector, which can look for a partition table, and skip directly to the first partition, if present.

[GARTHWILSON]

Quote:

Not sure if it matters, but it seems sectors on an SDcard are 512 bytes

A few have 1K sector sizes.

[jds]

There are probably a number of easier ways of doing this. The simplest is to do away with the FAT FS and use rawrite to write directly to the SD. This is a standard utility on linux, but you can get versions for windows too. You can write a file directly to the SD and then it is easy to read. You also have the advantage that the file is sequentially arranged. The downside to this is that it is not as convenient on the PC side, but it's still quite simple.

An intermediate solution is to write just enough code to navigate the FAT for a simple fixed case. We could set a rule that there is only one file on the SD card, so it may be possible to read the FAT table only and find the blocks for that single file. This would require a bit of research, but sounds possible.

There is a bit of work to set up the SD card interface in the first place though, so the boot code is not going to be tiny. It is a lot easier to use a SPI FLASH than SD card.

[Tor]

Or maybe use a variant of the LiLo (Linux Loader) scheme. A PC program writes the file to disk (SD card), then it figures out the physical sectors of the that file and writes a list of sectors to a secondary file, a 'map' file. That file is constructed in such a way that if it itself needs more sectors, each sector includes a pointer to the next one. Finally LiLo puts a tiny boot loader into (for example) the first sector, with a pointer to where physical map file starts. This makes it completely file system independent, the only tool which needs to know the file system is the PC program (and that one can use operating system features in the case of Linux). (That boot loader wouldn't be code to read an SD card, that takes a bit more, as was pointed out. )

[John West]

It's unlikely that an SD card will erase old versions of a file - they normally try to avoid unnecessary writes. But that's OK! Just use a slightly shorter magic signature, and include a version number and page number after it.

When you see the signature during scanning, check the version number. If it is greater than the one you currently have for this page, load it.

It does complicate the loading code a bit, but hopefully not too much.

[Oneironaut]

John West wrote:

It's unlikely that an SD card will erase old versions of a file - they normally try to avoid unnecessary writes. But that's OK! Just use a slightly shorter magic signature, and include a version number and page number after it.

When you see the signature during scanning, check the version number. If it is greater than the one you currently have for this page, load it.

It does complicate the loading code a bit, but hopefully not too much.

You are correct, I have verified this using a sector editor.
The workaround is to simply full format the card each time.

I will maintain a directory on the PC where my Vulcan-74 toolkit resides, and it is copied to a fresh formatted SD.
If this all works as planned, I will add this functionality right into my Vulcan-74 Toolkit.
Currently it can format sound and graphics data, and make the boot images.

I am looking into the RAW write idea suggested by TOR now.
If I can find a simple Windoze program to make a pure image from a directory of files, then this simplifies everything.
It would just be a matter of making my Toolkit create that image based on the required data.
From there, the Diode Matrix would hold just enough code to get the SD into RAW Read mode for the 6502.

I wonder if some of these imaging programs like TrueImage can do it?
Something to look into while I am waiting for the next free weekend of hacking time.

Actually, why would I need a ready made program? I can code.... doh!
I will be looking for a few Visual Studio examples showing how to RAW write to an SD Card.
Once I find one, I will just add the function to "Make Cartridge Image" right into my toolkit!

.... yeah, this is the new plan.
Thanks for the suggestions and ideas!

Brad

[BigEd]

You can write a file's data to a device using dd - there's a Windows version of dd at
http://www.chrysocome.net/dd
If your files are round-number lengths then you could just concatenate them. Somehow your reading software needs to know how to split them - that's true whatever scheme you adopt.

[Arlet]

The RAW format is the simplest, but the advantage of the LILO idea is that you can still use the rest of the device as a regular FAT formatted disk.

[Oneironaut]

I'm really liking the idea of RAW format with my own basic file table at the start sector.
I will ONLY need to read, and the image will always be created on the PC.
If I write a simple RAW manager (like C64 Commander), this will work well.

I would allocate say the first 1024K to just a file table. Something very simple...
File Name, Start Sector, Length (all fixed number of characters).

My Vulcan-74 Toolkit would then read a PC directory and format the SD Card accordingly.
The File Table would always be at the beginning, and also have a secret key just in case.

The second file on the card would then start just after the File Table, and be 64K long.
This file would be the "KickStart", which is loaded by the very basic program stored in Matrix ROM.

So, a power on sequence would go like this...

1) Boot Logic (counters and latches) read and store the contents of the Diode Matrix to the Program Memory.
2) Control is handed over to the 6502, which can now initialize the SD Card and hunt for the Kickstart at a known location.
3) The Kickstart image (64K) is then loaded into the remaining Program Memory, and the 6502 can continue to execute.
4) At this point, the 6502 is now smart enough to read the File Table and display a Load Screen to the user.
5) The user toggles through the menu using the joystick to load the chosen game or demo from the SD Card.
6) As the program requires more data, it can load it as needed by referencing the File Table. 32GB... no problem!

Seems like a good plan so far....

- Allows cross development on a PC.
- Uses off the shelf and inexpensive SD Cards.
- Allows unlimited storage of sound and graphics.
- Does not overload the 6502 with a FAT library.
- Can be adapted in a few years when SD Cards go extinct.
- Still includes a Diode Matrix, which is now a MUST have item!

Brad

[cbscpe]

I would have done the following

- Define my own partition type (RAW)
- Format the SD having 2 partitions, one of type RAW and the other of type FAT
- Overwrite the Master Boot Record (physical block 0 on the SD card) with my own boot code but maintaining the normal fields (partition table), so other systems can access the FAT partition
- The diode matrix just needs to read block 0 and after that hand over to the code in block 0. You have almost 512bytes this should be enough to analyse the partition table and find your RAW partition
- Read your RAW partition to RAM and then start that program

The RAW partition will be nothing else than your kickstarter of course. When you change your kickstarter you just need to write (dd) to the RAW partition and do not have to change the content of the FAT partition. A FAT library is not that complicated especially when you only read files. You could for example restrict yourself to the root directory. I'd rather build a simple FAT library than a tool-chain only to have your own SD card filesystem.

[Oneironaut]

Many interesting ideas!

My first step is to code a working RAW SD reader in AVR assembly and then port it to 6502.
I have all of the hardware (AVR+ SD) connected now, and only need the magic ingredient... time!

Brad

[jac_goudsmit]

For maximum compatibility, use an OS such as Linux or Windows to put a "standard" MBR on the media, with the FIRST partition FAT or FAT32; (Some versions of?) Windows ignore(s) all but the first partition on most removeable media. Make the SECOND partition a RAW type (it doesn't have to be big) and you can put your 6502 code there.

The way I would (and probably one day will) bootstrap a 6502 system from an SD card:

A program in the ROM reads sector 0, verifies that it's an MBR, then finds the first partition in the partition table that has type RAW, then read the first sector from that partition, check for my own signature bytes and execute the boot sector if they match. The ROM would never need to execute the code in the MBR (it only cares about the partition table) so it's fine if the MBR is stored there by other operating systems that do need to execute the code and use other processors. And those other OSes will generally be careful not to touch the RAW partition and regard it as something else (like a file system). They just won't mount the partition.

The code on the first sector in the RAW partition (our boot sector) can then contain code that's smart enough to load more sectors from the same RAW partition. That code may be an application, or it could be some sort of primitive mini operating system that understands the FAT file system and is smart enough to find and load a file with a special name from (the root directory on) the first partition to continue booting.

With this, the work of getting your system to bootstrap is nicely divided in three portions:

• The ROM code is the most difficult to update, but only needs to know how to read sectors from the SD card, and needs a little bit of code to analyze the MBR and load your boot sector from the RAW partition and check it for a signature.
• The boot sector and mini-operating system are probably a lot more difficult to write, but it will be relatively easy to replace it when you have to fix a bug, by using dd to the raw partition. Also, it will be relatively easy to partition other SD cards and write your OS to it. Once you fix the bugs, you won't have to dd it to any SD again except when you want to make more SD's.
• Once you have an OS that's able to load its initial program from the FAT partition, it will be extremely easy to replace that program from a Windows or Linux machine: just drag and drop it to the SD card.

By the way, I would probably not even use a real ROM, but a microcontroller that emulates ROM or pre-initialized RAM. Depending on how you design your system, you could let the microcontroller do the work of finding the boot sector (or the boot sector and a certain number of following sectors) and map them into 6502 address space.

Also, if you want, you can use the microcontroller to completely keep the SD card reader out of the 6502 system; you can let the 6502 program put some bytes at some "magical addresses" that are caught by the microcontroller, to which the microcontroller responds by reading or writing SD card sectors or files and mapping them into 6502 memory.

My own "Software-Defined Computer" projects use a Propeller as microcontroller, and the Propeller uses a 32K I2C EEPROM as boot media; I have some plans on using the extra space in a larger I2C EEPROM to store software. It won't cost any extra pins (unlike an SD card reader) and I2C is fairly easy to implement.

===Jac