Adventures in FAT32 with 65c02

Programming the 6502 microprocessor and its relatives in assembly and other languages.
Post Reply
User avatar
BigDumbDinosaur
Posts: 9425
Joined: 28 May 2009
Location: Midwestern USA (JB Pritzker’s dystopia)
Contact:
Contact BigDumbDinosaur

Re: Adventures in FAT32 with 65c02

Post by BigDumbDinosaur »

jgharston wrote:
The standard principle is "read many, write once", so if a file is open for reading, it can be open again for reading as many times as you have resources, but only for reading.  But any file that is open for writing blocks any further opening of any type.  So, opening for update (read *and* write) blocks that file from being opened again as it is open for writing.
One can avoid having to block an open-for-read operation on a file already opened for writing if a reasonably intelligent caching scheme is used and there are separate read and write pointers associated with each open file descriptor, even with descriptors that point to the same file.  That, in principle, is how it works in most modern operating systems, and should be no problem to implement on a single-tasking system.

An alternative if exclusive access is needed during a write operation would be to implement a file locking method.
Quote:
You make the action dependent on what the caller wants...In general, you should minimise how much of your calls fail, they should just pass a result back to the caller so that the caller can decide what to do.
Exactly.  Don’t make assumptions about what the caller wants or doesn’t want when a call returns with an error.  Just tell him or her what went wrong.

Using UNIX as an example, the original open() kernel function returned with an error if the requested file couldn’t be opened, with an error code describing why.  If the user then wanted to create a non-existent file, he/she would have to call the creat() kernel function to do so—which, of course, might also fail.

In later UNIX kernels, as well as with Linux, open() may be called with the optional O_CREAT flag as an argument so a non-existent file may be created as part of the open process—one-stop shopping, as it were.  Note that despite that added feature, open() never assumes that it should create a non-existent file, which honors the “rule of least surprise.”

In a more expansive sense, low-level system calls of any type should do only the absolute minimum needed to carry out their designated functions.  The more features added to a kernel call, the greater the chance that, at some point, the rule of least surprise will be broken and the programmer will end up chasing bugs caused by kernel behavior that is unwanted.
x86?  We ain't got no x86.  We don't NEED no stinking x86!
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

With regard to open() I am firm that it will only open a file if it already exists. It won't ever create a file that doesn't already exist. That's what f_create() is for. And I spell it with six letters, so apologies to Ken Thompson. :mrgreen:

Of course, what is least surprising to me may not be least surprising to anyone else. For me it's surprising that C's fopen() can open _and truncate_ a file if it already exists, and it's opened with "w"; yes, yes, yes, one should know one's tools, but if I want to start writing to an empty file, it's not a problem to delete and then create again.

Hence I have f_open_r which opens an existing file and positions the file pointer at the start of it, and f_open_w which opens an existing file and positions the file pointer at the end of it - so it appends. But 65c02's lack of a stack frame addressing mode makes passing parameters sometimes problematical, hence two variants. No reason why the other variants couldn't be done in a similar way.

By the way: May I take this opportunity to thank all who have contributed to this thread, guiding both myself and hopefully others? Cheers, guys!

Neil
Top
jgharston
Posts: 181
Joined: 22 Feb 2004

Re: Adventures in FAT32 with 65c02

Post by jgharston »

barnacle wrote:
Hence I have f_open_r which opens an existing file and positions the file pointer at the start of it, and f_open_w which opens an existing file and positions the file pointer at the end of it - so it appends. But 65c02's lack of a stack frame addressing mode makes passing parameters sometimes problematical, hence two variants. No reason why the other variants couldn't be done in a similar way.
Werl, on the BBC you have XY=>filename and A=action, 40=input, 80=output, C0=update, with extensions allowing options in the lower bits, eg 80+xx overwrite/don'toverwrite existing file, C0+xx create/don'tcreate if file absent. 8 bits of open options is quite enough. ;)
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

Yeah, but the people that wrote the BBC OS were smart people, not like me. :mrgreen:

Neil
Top
User avatar
BigDumbDinosaur
Posts: 9425
Joined: 28 May 2009
Location: Midwestern USA (JB Pritzker’s dystopia)
Contact:
Contact BigDumbDinosaur

Re: Adventures in FAT32 with 65c02

Post by BigDumbDinosaur »

barnacle wrote:
With regard to open()...It won't ever create a file that doesn't already exist. That's what f_create() is for. And I spell it with six letters, so apologies to Ken Thompson. :mrgreen:
Supposedly, when Thompson was asked if there was anything he would do over in UNIX, he replied that he would spell creat() with an ‘e’.  :D
Quote:
Of course, what is least surprising to me may not be least surprising to anyone else. For me it's surprising that C's fopen() can open _and truncate_ a file if it already exists, and it's opened with "w"...
At least in UNIX, C does that by selectively calling open() or creat() according to the mode you pass to fopen().  When creat() is called with the name of a file that exists, that file will be truncated, a behavior that has existed since UNIX’s inception.  Viewed in terms of the underlying kernel calls that it makes, my opinion is fopen() is not really doing anything unexpected.
Quote:
...65c02's lack of a stack frame addressing mode makes passing parameters sometimes problematical, hence two variants. No reason why the other variants couldn't be done in a similar way.
Yes, that can be a problem, although some creative programming can ease the pain.  As your system is uni-tasking, use of a parameter block to pass parameters into the kernel might be most efficient when there are too many for the registers.  If you can page-align the block, you only need one register to tell the kernel where to find it.
Quote:
By the way: May I take this opportunity to thank all who have contributed to this thread, guiding both myself and hopefully others? Cheers, guys!
Well, I hope I didn’t muddy the waters too much with my drivel.  :roll:
x86?  We ain't got no x86.  We don't NEED no stinking x86!
Top
fachat
Posts: 1123
Joined: 05 Jul 2005
Location: near Heidelberg, Germany
Contact:
Contact fachat

Re: Adventures in FAT32 with 65c02

Post by fachat »

Just a note. Some of the flags like O_CREATE have been added - as I understand it - to make operations atomic and avoid race conditions that may result in security issues for example.

André
Author of the GeckOS multitasking operating system, the usb65 stack, designer of the Micro-PET and many more 6502 content: http://6502.org/users/andre/
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

That's a fair point, but on a single-user single-threaded 65c02 system, the whole thing is a security disaster... as long as can interchange files with the linux box, I'll be happy.

Neil
Top
User avatar
BigDumbDinosaur
Posts: 9425
Joined: 28 May 2009
Location: Midwestern USA (JB Pritzker’s dystopia)
Contact:
Contact BigDumbDinosaur

Re: Adventures in FAT32 with 65c02

Post by BigDumbDinosaur »

fachat wrote:
Just a note. Some of the flags like O_CREATE have been added - as I understand it - to make operations atomic and avoid race conditions that may result in security issues for example.
Your understanding is correct.

The combination of the O_CREATE and O_EXCL flags as parameters to an open() call guarantees exclusivity.  An open() call with these flags will fail if the file exists, but will create the file if not present.  As internal kernel functions are general immune to preemption (but, of course, may be interrupted), the result is an atomic operation.
x86?  We ain't got no x86.  We don't NEED no stinking x86!
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

Well that's been a long week... I discovered that my str_to_83 routine, (turns a string into an internal MS filename) had a bug with eight-character names. Took a while to fix.

Unfortunately, that _wasn't_ the problem I had...

I had a simple test that initialised the system, dir'd the existing files on the disk, then tried to create a new file. Which worked perfectly right up to the time I ran it :( On the next boot, it was clear that it had in some way trashed the directory on disc.

It took a lot of testing to discover that everything being written appeared to be correct, but was not being written correctly. Turns out that rewriting fc_write to accommodate multiple buffers broke it: it required the correct buffer to be defined before writing (doh) and I'd missed it. And of course, every time I break it, I need to reformat the disk...

Code: Select all

NeoDOS32 v0.0171                                                                
Press any key to boot:                                                          
                                                                                
BIOS.ASM                                                                        
CF.ASM                                                                          
FILE_SYS.ASM                                                                    
MAIN.ASM                                                                        
MAIN.HEX                                                                        
MAIN.LST                                                                        
U32MATHS.ASM                                                                    
VARS.ASM                                                                        
DIRECTOR                                                                        
                                                                                
+
That's version 171 since xmas...

Neil

(edit: the files listed are added on the linux box after formatting except 'DIRECTOR' which is created locally (and will soon be a new directory))
(edit 2: Why so long? Well, there was a little diversion into MS time and date formats, and leap years and suchlike)
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

Okay, I'm in a more productive mood today :mrgreen:

I believe I have resolved the bugs outstanding in f_create, f_cat, f_dir, and f_del. Changing cf_write and cf_read to work with buffers other than transient caused a lot of subtle, sometimes spectacular, errors, most of which required the CF to be reformatted each time.

Here's an attempt to delete then recreate "another.txt" with a directory listing between each attempt:

Code: Select all

NeoDOS32 v0.0180                                                                
Press any key to boot:                                                          
                                                                                
BIOS.ASM                                                                        
CF.ASM                                                                          
DOS.ASM                                                                         
FILE_SYS.ASM                                                                    
MAIN.ASM                                                                        
MAIN.HEX                                                                        
MAIN.LST                                                                        
U32MATHS.ASM                                                                    
VARS.ASM                                                                        
ANOTHER.TXT                                                                     
deleting 'another.txt'                                                          
                                                                                
BIOS.ASM                                                                        
CF.ASM                                                                          
DOS.ASM                                                                         
FILE_SYS.ASM                                                                    
MAIN.ASM                                                                        
MAIN.HEX                                                                        
MAIN.LST                                                                        
U32MATHS.ASM                                                                    
VARS.ASM                                                                        
creating 'another.txt'                                                          
                                                                                
BIOS.ASM                                                                        
CF.ASM                                                                          
DOS.ASM                                                                         
FILE_SYS.ASM                                                                    
MAIN.ASM                                                                        
MAIN.HEX                                                                        
MAIN.LST                                                                        
U32MATHS.ASM                                                                    
VARS.ASM                                                                        
ANOTHER.TXT                                                                     
Finished                                                                        
+
The directory entries for deleted versions of the file remain unchanged, so each attempt makes the directory a little fuller. The card can be read and written in the Linux file manager.

Neil
Top
User avatar
BigDumbDinosaur
Posts: 9425
Joined: 28 May 2009
Location: Midwestern USA (JB Pritzker’s dystopia)
Contact:
Contact BigDumbDinosaur

Re: Adventures in FAT32 with 65c02

Post by BigDumbDinosaur »

barnacle wrote:
Okay, I'm in a more productive mood today :mrgreen:
Yep!  Takes a while for New Year hangovers to dissipate.  :D
Quote:
I believe I have resolved the bugs outstanding in f_create, f_cat, f_dir, and f_del.  Changing cf_write and cf_read to work with buffers other than transient caused a lot of subtle, sometimes spectacular, errors, most of which required the CF to be reformatted each time.
Buffer management is an interesting little exercise in itself.  As I was working through my buffer-management code, I got numerous opportunities to exercise my POC unit’s panic and reset push buttons...especially the latter.  :evil:  So I do understand a little bit of your pain.
Quote:
The directory entries for deleted versions of the file remain unchanged, so each attempt makes the directory a little fuller.  The card can be read and written in the Linux file manager.
Nothing like making some progress.  Any thought on trying to reuse a deleted file’s entry instead of expanding the directory?  It seems as though that shouldn’t be too difficult to achieve.
x86?  We ain't got no x86.  We don't NEED no stinking x86!
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

Not immediately; for my use case I can't see any way to fill half a gigabyte. But I do need to address the allocation anyway.

For the short term I think I'll follow MS's lead: use the on-disk indicator for the next free cluster (actually it's the last allocated block, AFAICS) and search the FAT forwards from there for an empty cluster. That works fine until you run out of disk space.

If you do get to the end of the cluster map, then you need to go back to the beginning of the root directory, find a deleted file (possibly recursively into any child directories), clear its directory entry, follow its FAT chain, erasing FAT entries as you go, so that you can look for them again while allocating. But it's messy: the only options for the directory entry are (1) filename begins with legal character = filename, in use, (2) filename begins with $E5 = file has been deleted, or (3) filename begins with $00 = there are no more directory entries. The sequencing for deleting things and reusing the space might have complications.

The old MS policy was not to reallocate deleted files' clusters immediately so altering the $E5 to an ascii character would magically restore the file from the dead, mostly. I'm not sure what the Linux file manager does; certainly the old filenames remain (with the $E5. While one obvious policy would be to zero the FAT entries for a deleted file as soon as its deleted, that would be unkind to Peter Norton, and would leave a potentially very fragmented filesystem. On the other hand, with no seek time, is fragmentation even an issue?

On a philosophical note: the basic 'user' file operations (dir, cat, mkdir, del, rmdir etc) can either be built into the BDOS rom, or, given the basic abillity to load and execute a file, can be independent files as part of the OS. None of them are particularly large, but I would like eventually to be able to max out the available RAM. Still thinking about that one... it does give a very easy way to change my mind later about how things work.

But first, I have to do some testing as to why f_dir sometimes starts by loading the last sector used, unless I provide a debug line to tell me where it's starting... then, it always works. Hmm.

Neil
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

barnacle wrote:
But first, I have to do some testing as to why f_dir sometimes starts by loading the last sector used, unless I provide a debug line to tell me where it's starting... then, it always works. Hmm.
Well, sorted that out at least: a cf_wait is required before loading the lba into the CF registers. When I inserted debug outputs, they took sufficient time that the CF readied itself before I needed it.

Code: Select all

cf_read:
	jsr cf_wait
	jsr cf_set_lba		; set the desired sector
	jsr cf_wait
	lda # 0x20			; read a sector command
	sta CFREG7
cf_read_info:			; enter here if presetting info block ID: 0xec
						; but only after having set lba
	jsr cf_wait
	lda #1				; one sector please
	sta CFREG2
	jsr cf_wait
cf_r0:
	lda CFREG7
	and #%00001000		; check the DRQ bit
	beq cf_r0			; loop if not set
	ldx #2				; one sector, two pages - 512 bytes
cf_r1:
		ldy #0
cf_r2:
			lda CFREG0
			sta (sector_ptr),y
			iny
			bne cf_r2
		inc sector_ptr + 1	; page finished; inc pointer
		dex
		bne cf_r1			; get next page if required
	; at this point, the buffer should be filled with data
	; FIXME - add error check
cf_rx:
	rts
Onwards and upwards!

Neil
Top
barnacle
Posts: 1831
Joined: 19 Jan 2004
Location: Potsdam, DE
Contact:
Contact barnacle

Re: Adventures in FAT32 with 65c02

Post by barnacle »

There's an offset of two between the entry in the FAT and the associated cluster. The _position_ of the FAT entry is directly related to the 'cluster number' but the first two FAT entries are reserved. The first cluster is therefore indexed by entry 2.

This is confusing.

I think that I am converting from FAT entry to cluster number (and therefore the LBA address of the first sector of that cluster) too late in the process, at least conceptually. The next_cluster value in an entry, if there is one, is in sector entries (which rather annoyingly, MS refer to as 'cluster numbers')

This is what a section of the FAT looks like prior to seeking the next free cluster:

Code: Select all

creating a directory                                                            
7ED8                                                                            
7E00  F8 FF FF 0F FF FF FF 0F F8 FF FF 0F 04 00 00 00 ................          
7E10  05 00 00 00 FF FF FF 0F 07 00 00 00 FF FF FF 0F ................          
7E20  09 00 00 00 0A 00 00 00 FF FF FF 0F 0C 00 00 00 ................          
7E30  0D 00 00 00 0E 00 00 00 FF FF FF 0F 10 00 00 00 ................          
7E40  FF FF FF 0F 12 00 00 00 FF FF FF 0F 14 00 00 00 ................          
7E50  15 00 00 00 16 00 00 00 17 00 00 00 18 00 00 00 ................          
7E60  19 00 00 00 1A 00 00 00 1B 00 00 00 1C 00 00 00 ................          
7E70  1D 00 00 00 1E 00 00 00 1F 00 00 00 20 00 00 00 ............ ...          
7E80  21 00 00 00 22 00 00 00 23 00 00 00 24 00 00 00 !..."...#...$...          
7E90  25 00 00 00 26 00 00 00 27 00 00 00 28 00 00 00 %...&...'...(...          
7EA0  29 00 00 00 2A 00 00 00 2B 00 00 00 2C 00 00 00 )...*...+...,...          
7EB0  2D 00 00 00 2E 00 00 00 2F 00 00 00 30 00 00 00 -......./...0...          
7EC0  FF FF FF 0F FF FF FF 0F 33 00 00 00 34 00 00 00 ........3...4...          
7ED0  FF FF FF 0F 00 00 00 00 00 00 00 00 00 00 00 00 ................          
7EE0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Immediately before it is the calculated offset to the next free cluster - actually, that's the last allocated cluster, but it's where you should start looking for free clusters - 7ED8. That should be 7ED0; it's wrong because I took the value from the disk and assumed the wrong sort of cluster... :roll:

So I think I need to do another rewrite. Rather than saying the cluster that lives at cluster_begin_lba is 0 I need to call it cluster 2 and change things accordingly.

Oh well, here we go again.

Neil
Top
User avatar
BigDumbDinosaur
Posts: 9425
Joined: 28 May 2009
Location: Midwestern USA (JB Pritzker’s dystopia)
Contact:
Contact BigDumbDinosaur

Re: Adventures in FAT32 with 65c02

Post by BigDumbDinosaur »

barnacle wrote:
There's an offset of two between the entry in the FAT and the associated cluster. The _position_ of the FAT entry is directly related to the 'cluster number' but the first two FAT entries are reserved. The first cluster is therefore indexed by entry 2.

This is confusing.
It’s Micro-$oft at its finest...which is why it’s confusing.  :D
Quote:
So I think I need to do another rewrite. Rather than saying the cluster that lives at cluster_begin_lba is 0 I need to call it cluster 2 and change things accordingly.

Oh well, here we go again.
Two steps forward...I know your pain.  I went through something similar in writing the SCSI driver for my POC unit.  Ultimately, a rewrite with some fresh perspective got it right.
x86?  We ain't got no x86.  We don't NEED no stinking x86!
Top
Post Reply

Return to “Programming”