Recommendations for good/small UART file transfer code?

[pdragon]

I got a UART serial/usb interface set up for my 65c02 build (VIA SR <-> MAX3100 <-> FTDI232R USB/serial cable). It was actually much easier than I expected since MAX3100 runs SPI-mode0 so "just works" alongside my existing SD card SPI interface.

This is great as an alternative to keyboard input/lcd output but I'd also like to do bulk file <-> memory transfers across the serial link. Does anyone know of small/portable 65c02 assembly code for that which I could customize? I've been using tio as the client from my macbook which seems nice for hacking around. It supports xmodem (crc and 1k) and ymodem out of the box so 6502 implementations of those might be easiest?

Thanks
Patrick

[Yuri]

I do not have any assembly code for XMODEM on hand. (Planning to write some in the future)

That being said here are some documents you might find helpful. It isn't a very complex protocol to be sure.

If I'm not mistaken others have generally rolled their own file transfer systems.

[pdragon]

thanks for the docs! worst comes to worst I figured i'd implement based on tio's C-code but assume this road has been walked many times before

https://github.com/tio/tio/blob/master/src/xymodem.c

[drogon]

XModem is really quite simple to implement especially if you don't care about "link efficiency" however it's down-side is that it was designed in the days of CP/M where disk files didn't have a size attribute associated with them and were always represented in 128 bye blocks. Text files were easy to terminate - with a Ctrl-Z but binary files less-so.

So if transferring from a PC/Mac/Linux to 6502 RAM then maybe you don't care about the exact size and multiples of 128 bytes is good enough... Which it probably is for binaries to be executed directly.

-Gordon

[pdragon]

I'm not hung up on [x|y]modem specifically, just they seemed already supported by the terminal, and I figured there must be some nice clean code out there for the 6502 side already.

my 6502 kernel is forth, so ultimately i probably just want to wrap up some words that put a file from os x to an address on 6502 side, and vice-versa. i could roll my own protocol or paste b64 data into the terminal but not sure I need to reinvent every wheel .

otoh it would bring back a lot of memories just pasting (... err typing ...) hex dumps like the old days of Compute!

[BigDumbDinosaur]

In a hard-wired serial link running a short distance, i.e., a few meters, you don’t need the monkey-motion of XMODEM or its friends (XMODEM-CRC, YMODEM, etc.).  A serial link running over a short distance on reasonably-good cable is unlikely to experience the sort of errors typical of telephone service back when Ward Christensen devised XMODEM’s predecessor.

I use Motorola S-records to transfer code from my Linux box to my POC units. The S-record format appends a one-byte, easily-computed checksum to each record for basic error-checking.  Although the serial interface itself runs at 115.2 KbpS error-free, the receiver code running in the POC unit makes use of the checksum on the off-chance that an error has occurred (which, to date, has never happened).

Unlike XMODEM, whose design, as noted, is strongly bound to CP/M, the S-record format carries information that tells the receiver how much data to expect in each record.  There is no end-of-record (Ctrl-Z) marker, no pad bytes, etc.  The result is something that is simpler to implement and more efficient in operation.

[pdragon]

Good point. tio has nice support for interactive scripts (ctrl-t r Run lua script; ctrl-t R Execute shell command with I/O redirected to device) so it'd be easy to use whatever protocol for transfer

[tius]

BigDumbDinosaur wrote:

In a hard-wired serial link running a short distance, i.e., a few meters, you don’t need the monkey-motion of XMODEM or its friends (XMODEM-CRC, YMODEM, etc.).  A serial link running over a short distance on reasonably-good cable is unlikely to experience the sort of errors typical of telephone service back when Ward Christensen devised XMODEM’s predecessor.

I agree with that. A simple serial protocol should be completely sufficient in most cases. Dedicated hardware for the serial interface can often even be omitted. In my experience, up to 57600 baud software serial works absolutely reliably with a 1 MHz CPU clock.

In my opinion, XMODEM has the advantage that it is a very established protocol and is supported by many terminal emulations. There are already many XMODEM implementations for 6502 assembler. For bootmon65, however, I reprogrammed again it in order to achieve the maximum possible transfer rate with software serial.

Perhaps tinyload65 could be helpful as an example of a fast and compact block-based transfer.

I don't have a very good overview of the enormous number of programs that already exist. So please excuse me for focusing on my own software here.

[BigDumbDinosaur]

tius wrote:

In my opinion, XMODEM has the advantage that it is a very established protocol and is supported by many terminal emulations.

My opinion of XMODEM is it sucks and is unnecessary on a short-distance serial link.  Just because it has been around since when dinosaurs grazed in the grass doesn’t excuse its crude design and gross inefficiency.

[tius]

BigDumbDinosaur wrote:

My opinion of XMODEM is it sucks and is unnecessary on a short-distance serial link.

Technically speaking, you are certainly absolutely right. But sometimes it's fun to keep old things alive

[pdragon]

I ended up implementing >SREC / SREC> words for my taliforth console which makes it easy to paste compiled code snippets thru the terminal.

just for fun i'm also trying a very simple delta-coding thing so I can generate small binary patches with respect to the code that's already in the eeprom. then I don't need to send across the entire image every time I change anything.

[White Flame]

I missed this thread the first time around, but codebase64 is a good place to search for stuff as well. It has xmodem send/receive, with the comments even saying it's for the 65c02.

https://codebase64.org/doku.php?id=base ... nd_receive

[Broti]

White Flame wrote:

https://codebase64.org/doku.php?id=base ... nd_receive

Which are taken (as stated here: https://codebase64.org/doku.php?id=base:io_programming) from the on-site Source Code Repository: http://6502.org/source/io/xmodem/xmodem.htm

But yes, Codebase64 is a fantastic source!

[BigDumbDinosaur]

tius wrote:

BigDumbDinosaur wrote:

My opinion of XMODEM is it sucks and is unnecessary on a short-distance serial link.

Technically speaking, you are certainly absolutely right. But sometimes it's fun to keep old things alive

That’s what the nice folks at the doctor’s office tell me when I arrive for an appointment. 

Back on topic, some old computer technology is worth maintaining, e.g., the 6502 and TIA-232.  Other old technology, such as IBM’s SNA and XMODEM, should be relegated to the “nice try” museum.

XMODEM was a stop-gap hack that catered to CP/M’s primitive filesystem structure and the desire to somehow get error-free data transfers through the noisy phone lines that were endemic to the Chicago area in the 1970s¹ (Ward Christensen was a long-time Chicago-area resident).  XMODEM permutations, such as XMODEM-CRC, YMODEM and ZMODEM, soon followed as the weaknesses of the original XMODEM became apparent, and the desire to transfer data between non-CP/M systems arose.

The limitations of XMODEM are several-fold:

1. “Packets” are only 128 bytes.  Christensen chose that size because a CP/M data block on disk was 128 bytes.  Unsurprisingly, the protocol didn’t scale well as file-transfer sizes got bigger and non-CP/M filesystems became the target for the transferred data.
    
2. Each packet contains a header consisting of ASCII <SOH> (start-of-header, $01), two eight-bit sequence numbers, the payload—fixed at 128 bytes—and a one-byte, simple checksum.  That arrangement has a number of weaknesses:
   
   1. Error checking is unreliable.  The one-byte checksum will not reflect a case in which bytes in the payload have been interchanged—a distinct probability with the non-error-correcting Bell 103 modems that were in common usage into the 1980s (the Commodore 1600 modem, along with the error-prone 6551 simulation code in the Commodore 64 kernal, was famous for this sort of issue).  In fact, the byte order in the payload can be completely randomized relative to the (uncorrupted) original and as long as the sum of the bytes is the same as the original, the packet will appear to be good.
       
   2. The packet doesn’t indicate how many significant bytes exist in the payload.  XMODEM pads the payload with ASCII <SUB> (substitute, $1A) to bring it to the fixed 128 byte size.  The flaw with that scheme, if the receiver is not a CP/M system, is there is no easy way to distinguish the <SUB> padding from actual data.
       
   3. As the header’s sequence numbers vary from 1-255 and 255-1, a large transfer will repeat sequence numbers, which has the potential to result in out-of-order packet processing on a slow system.
    
3. XMODEM can go into deadlock by repeated <NAK>s from the receiver.  There isn’t a provision in the protocol for determining when too many errors have occurred and it is prudent to discontinue the transfer.

On hard-wired TIA-232 links that span relatively-short distances (five meters or less), communication errors should not be occurring at all.  TIA-232 voltage levels are sufficiently robust to easily overpower most noise.  Back in the days when my company routinely installed serial terminals, printers, etc., we often had links running several hundred feet over CAT5 UTP at data rates as high as 115.2 KbpS...error-free.  If you are getting errors on a five-meter link, you have a hardware problem and need to address it, not mask it with a lame file transfer protocol that was conceived during a snow storm and a fit of desperation. 

That said, it would be foolish to not have some error-detection, especially when debugging new code.  If a transfer error does garble a datum and you have no error-detection, you could find yourself chasing a ghost bug that results from garbage in the program.  In my activities with my POC units, I transfer code and data using Motorola S-records.  An S-record’s error-checking is rather basic (a one-byte LRC-style check per record) but has a much better probability of catching the sort of errors that will fool XMODEM’s error-checking.  The difference is S-records are transferred as ASCII, not binary, as is done with XMODEM.  This arrangement makes it possible for a one-byte checksum to catch a byte transposition error, as well as a bit-level error.

————————————————————
¹Chicago-area telephone service in those days was hamstrung by the widespread use of step-by-step switching in central offices, some of that switchgear dating back to before World War II.  A typical Chicago phone conversation was usually an exercise in competing with static, hiss, buzzing, cross-talk and who-knows-what.  There were no error-correcting modems back then, so transferring data through POTS was often hamstrung by a high error rate.  Even as old step-by-step switching was replaced with newer crossbar switches in the 1980s, the noise problem persisted due to extremely old cable.

[BigDumbDinosaur]

White Flame wrote:

I missed this thread the first time around, but codebase64 is a good place to search for stuff as well. It has xmodem send/receive, with the comments even saying it's for the 65c02.

https://codebase64.org/doku.php?id=base ... nd_receive

Attached is a file with Daryl’s code.  I collected the code into a file because the horrid color scheme at codebase64 makes reading the page impossible for me.

xmodem_crc.asm

Daryl’s XMODEM-CRC Implementation

(17.99 KiB) Downloaded 138 times