Subject: Mailstation CP/M

From: "FyberOptic" <fyberoptic@...>
Date: Sun, 31 Jan 2010 19:52:33 -0000

Yep, that's Zork 1!

So, how is it working?  Basically, I (mostly) completed my CP/M BIOS
module, finally emulating a floppy disk with the dataflash.  This is the
same BIOS code that I was using for the core of FyOS before, btw, though
just mostly for the console I/O at the time.  Except now I'm using the
real CP/M BDOS v2.2 with it, because I thought that would be the easiest
way to ensure my BIOS module was working properly. I have no command
processor at the moment, but I didn't need one. I just wanted to
directly load this one program to see if it worked.  And it does. 
Kinda.

There's still limitations.  First is the obvious, with the text display
I'm generating only supporting 40 x 16 characters.  Zork likely expects
an 80 character display, and I would bet other CP/M apps would as well. 
Second, my BIOS still can't write to "disk".  This was partially due to
the fact that I wasn't even sure my read routine was going to work
properly.  Which it did, and only needed one little fix, surpisingly. 
I'm hoping it won't be much effort to do the reverse now for writing. 
Originally I wanted to go to the trouble of some kind of smart system
which would cycle the dataflash sectors to increase their lifetime, but
then I decided a.) it was too much work, and b.) if the Mailstation
folks were too lazy to do it, then why should I?

For those who may have not followed some of the other work I've done
around here in the past, then you should know that this is only possible
due to a hardware mod I made to my Mailstation.  I've replicated this
functionality in my current test build of the emulator.  Basically, I
added in a circuit to allow the lower 16KB of address space to be
swapped from codeflash to RAM.  Normally the first 16KB of codeflash is
permanently in that spot.  My toggle lets me put in the second page of
RAM (since the first page is permanently in the uppermost 16KB of
address space).  This is necessary in order to run CP/M and its
applications natively.

As for the "disk", I took a bit of a clever approach.  My #1 requirement
was that it couldn't disrupt the normal functionality of the
Mailstation, and oppositely, the Mailstation's normal functionality
couldn't disrupt it.  I am in fact still running the Mailstation
firmware at power-up, after all.  So, what I did was take advantage of
all the available app space.  Which is apparently 128KB worth.  This
meant that nothing which the Mailstation firmware did should mess up my
disk image, and nothing my disk image did should mess up Mailstation
settings.

The virtual disk has 256-byte sectors (to match the dataflash), 16
sectors per track, and 32 tracks.  That makes each track 4KB.  I
reserved 4 tracks for system use, which is 16KB, which is conveniently
the same size as a dataflash page.  This means the system tracks can be
used as a normal Mailstation app.  And this is exactly how I used them,
placing a loader app to put the BDOS + BIOS into upper memory, which
then executes the BIOS cold boot procedure.

Since I wanted to support any version of BDOS that might be attached to
this BIOS at some point, I decided to handle deblocking internally. 
This means that as far as CP/M knows, the disk actually uses 32 128-byte
sectors per track.  128 was the common size, and even in the later
versions is still the size of things like the DMA and BDOS read routines
use from what I understand.

Regardless of CP/M version, the disk is configured to use 64 directory
entries, using a 1KB block size.

I wrote a utility on the PC to generate this disk as a 128KB file, using
a separate text file to determine what files should be inserted into the
disk (calculating extents, writing the directory entries, all the crap
necessary for CP/M disks).  That disk image file can then just be dumped
into the dataflash.bin that my emulator uses.  Since the first 16KB acts
as a standalone Mailstation app, you can simply launch the OS from the
Mailstation menu like you would any other 3rd-party app.

The fact that you're seeing the game text on the screen shows that it is
indeed properly reading files from the virtual disk, because zork1.com
loads all of that from zork1.dat.  That's part of why I wanted to use it
as a test.  The executable was small, and I knew what to expect from it.

I haven't tried this on the Mailstation hardware yet, mostly due to the
fact that it will erase my loader app.  Before I go writing anything
into my datflash, I'm going to wait until I have both a command
processor and a standalone loader utility which I can run from there. 
But I have no reason to think it wouldn't run, because the disk routine
is the only new part.  All the console I/O functions have been run on
the hardware before.

I'd still like to finish my own original BDOS code, since I think it's
still technically stealing to use this one even though it's 30 years
old.  But then again, it's probably stealing to be using this copy of
Zork too.  And my BIOS module is currently using a ripped out version of
the Mailstation's keyboard routines, as well as the original CGA font
(which I'm sure is copyrighted or something).  Oh well!

Anyhoo, this is turning into a huge post, but my work is finally coming
along into something I can actually show results from, so I thought I'd
share what I have so far!

---

From: Ethan Dicks <ethan.dicks@...>
Date: Mon, 1 Feb 2010 11:51:25 -0500

> Yep, that's Zork 1!

Well done!

> an 80 character display, and I would bet other CP/M apps would as well....

There are versions of the CP/M Z-machine floating around (I found them
in one of the massive repositories a couple of years back) that came
with an Infocom-written customizer front-end.  You might look for CP/M
Zork or Starcross or Planetfall that happen to have a .ASM file
included beyond just the usual .COM and .DAT file.  They mostly let
you select your terminal type (ADM3, VT100, etc), but there might be
something in there for screen size.

Overall, though, Infocom did support 40 column machines (Apple II,
C-64, etc.)  It's up to the Z-machine where to wrap, etc.  The
appearance of the status line depends on matching up the column width,
and there is at least one puzzle clue that I know of that is easy to
spot with an 80-column screen but hard to spot at 40 columns.

> as a test.  The executable was small, and I knew what to expect from it.

Yep.  I've done similar sorts of things with the Z-machine for the
6502.  It's not large and you know how it's going to behave (lots of
disk reads for game text and game data).

In order to get that opening screen, lots of things have to be working
right.  I had a long-standing bug with my port of the 6502 engine to a
new platform - for me, the opening screen was fine, but it got
squirrely with the first line of user input.  At least it was at an
obvious blocking point, so somewhat easy to create test-cases for.

Smashing work.  I can't wait to see a photograph of it running on the
real hardware someday.

-ethan

---

From: "cyranojones_lalp" <cyranojones_lalp@...>
Date: Tue, 02 Feb 2010 02:08:51 -0000

> Yep, that's Zork 1!

Hey, that pretty cool!!!

> Except now I'm using the real CP/M BDOS v2.2 with it, 
> because I thought that would be the easiest
> processor at the moment, but I didn't need one. 

Is the "console command processor" (CCP) a separate prog??? 
Or is it part of the BDOS???  I don't remember.

I do recall that the BIOS was supposed to be the only
part that needs to be customized.

Perhaps you are closer than you think!

Did you cold boot CP/M, or just preload cp/m & Zork, and
start from there (0x0100)?

> properly. 

Might it be useful if the bank-switching was not one way,
and you could call routines from the ms firmware?
Then you could use ms firmware to access the dataflash, and
even use the built in keyboard code.

I guess you would need to give back the ram at C000 to
get that to work, but I don't see that as a real problem.
You get a pretty big stack out of the deal, too!

Most CP/M systems never had full 64K RAM.  Mine
did have 64K, but a big chunk was dedicated to screen 
RAM (I forget the actual size, but it did have a graphic
mode, I want to say it used 24k for screen, but I would
need to look it up somewhere).

> So, what I did was take advantage of
> all the available app space.  Which is apparently 128KB worth.  

There is a lot more space available in the mailstation's
"file" area.  I think there is even one or two "filenames"
available (where filename is just a byte, from 0 to 0x2f.
Using the space that way would let your cp/m files coexist
with the other mailstation files.

> processor and a standalone loader utility which I can run from 

What if you combined your loader, your cpm boot code, & cpm system
tracks into a single app?  Or just use 2 app pages?  For that
matter, if there is any space left in your system page, you
could still use it as part of cpm's non-system tracks.  An
app doesn't need to use the whole 16K, they just need to
start at begining of one.

> I'd still like to finish my own original BDOS code,

Your own, but "original" might not be the right word!  :-)

> since I think it's still technically stealing to use 
> this one even though it's 30 years old.

There goes your "innocent" defense.
"Anything you say may be used..."

How do you interpret this:
http://www.cpm.z80.de/license.html
Does it mean we can just use it now???  Or does it only apply
to that site?????

> But then again, it's probably stealing to be using this copy of
> Zork too.

Gosh, and now you even confessed!  In public!!! 

> And my BIOS module is currently using a ripped out version of
> the Mailstation's keyboard routines, as well as the original 
> CGA font (which I'm sure is copyrighted or something).

I wonder if they have net access in prisons these days?  :-)

I have a putchar that uses the ms font, and a getchar that
calls ms keyboard code.

CJ

---

From: "FyberOptic" <fyberoptic@...>
Date: Tue, 02 Feb 2010 06:13:43 -0000

Couple of updates to point out here.  One is obvious, the other maybe
not so much unless you know what you look for!

Let's start with the font.  It's now 4x6.  The reason I used an 8x8 font
originally is because it fit perfectly into the restraints of the
hardware.  The screen being 320x128 meant you could divide both
dimensions evenly by 8, getting 40x16 characters on the screen.  I had
the font data aligned to a 256-byte page in memory, and formatted the
data so that each character row could be pulled out by simply
incrementing the high byte of the memory address.  The low byte was set
to which ASCII character you wanted (since there were 256 characters in
the font).  No multiplication or anything was necessary to get the
character offset into the font data.  And since each byte of LCD data
represented 8 pixels, it was an easy matter to write an entire row of
character data to the screen at a time, then just increment a couple of
pointers, and repeat 7 more times.  This made drawing characters pretty
speedy.  What took the most time is calculating all the junk necessary
to translate X/Y coordinates into the wacky backwards way the
Mailstation LCD hardware works.

Well 40x16 isn't really enough for many circumstances, particularly in
width.  Not even a CP/M directory listing.  So I did some math on
various sizes.  A 5x7 font would give me a 64x18 display, with a few
pixels left over at the bottom.  Not much more vertically, but 24 more
horizontally was good.  So I looked around for some fonts on the
internet, and ended up on this page
<http://rasher.dk/rockbox/fonts/misc/> .  It has a bunch of Linux fonts,
but best of all, has images available of all the character sets.  There
was a nice 5x7 font, but I also noticed the 4x6 one.  More math showed
that a 4x6 font would give me 80x21!  That's almost a full CRT display's
worth, just a few shy vertically.  And the more I thought about how I
would implement a font which wasn't 8 pixels wide, the more I realized
that using one with an even width would be easiest.  A 4x6 font means a
single byte of LCD data can hold exactly two character columns of row
data.  Imagine the nightmare of trying to calculate your position on the
screen if every character was 5 pixels wide.  Five bits of one character
would be in one byte, then the next character would have 3 bits in the
last byte and 2 bits in the next byte, etc.  It hurts my brain to even
think about it!

The other thing I noticed is that most of those fonts, including the 4x6
one I wanted, are public domain.  Awesome.

So I formatted the image of the font characters into something I could
work with and insert into my assembly.  I didn't go with 256 characters
this time because I'm honestly not sure if all the IBM ASCII characters
that were in the CGA font are even present in this one.  If they are,
they're so small that it's hard to distinguish which are which.  I left
the first 32 characters blank as well.  No more smiley face characters! 
I might still extend it to 256 later, but CP/M was primarily 7-bit ASCII
anyway.  I only included the upper 128 of the CGA font before because it
could be used for drawing lines and borders and things.

It took a bit of figuring things out and code reworking, but I converted
my putchar object file to work with the new font data/size.  And
conveniently, changes were only made to that object file, which means I
can recompile my BIOS to use the 8x8 font again as easy as specifying a
different object file when linking.    The new one probably isn't as
fast due to extra math involved, but the increased display size is worth
it in most cases.

Anyway, I'll stop rambling about fonts.  It looks small in the emulator
unless I 2X, but I think on the real hardware it'll be okay.

The other thing you might not have noticed on the screen there is that I
actually loaded Zork via the command processor.  I got a directory
listing beforehand too, though there's only three files on the virtual
disk.

I've had some quirks out of this.  I can launch stuff, but when the
application exits and the warm boot happens, the CCP never reloads
properly again.  I haven't been able to figure it out yet, unless it
and/or the BDOS are storing values which are messing it up, or
something's getting overwritten when Zork runs.  I dunno yet.  I suppose
I could totally re-read the BDOS/BIOS too.  Maybe that's even what
you're supposed to do.

Part of the problem might be that I didn't use any of the original tools
used to build a CP/M system.  I've manually converted the assembly code
over to work with AS-Z80, and am linking everything together as best I
can figure it should work based on CP/M manuals.  I had to increase the
default tiny BDOS stack too because my BIOS calls were overflowing it
(since I doubt most BIOSes needed as much bloaty hardware translations
as this one).  I think the CCP is normally part of the whole mess that
gets loaded off the system tracks in v2.2, but I was trying to take more
of a CP/M 3 approach.  I'm pulling the .COM off the disk, but just
loading it at an abnormal location (under the BDOS) instead of at 0x100.
I probably should rename it to .SYS just to avoid confusion.

CP/M 3 actually loads its CCP at 0x100, but puts a loader module up
under the BDOS for executing programs (using the memory-resident module
functionality of v3).  The reason I didn't go all-out in supporting the
CP/M 3 BDOS though is because the BIOS would need a bunch more stuff
added to it.  Which I'd still like to do eventually and all, but for the
time being the plan has just been "make it work".  CP/M 2.2 was the
easiest/quickest way to reach that goal.  And I still have some fiddling
to do.

So yeah, a lot of blabbing just to mention a new font and CCP.  But
maybe folks are interested in the technical junk too.  Either way, Zork
fits on the screen fine now!

---

From: "FyberOptic" <fyberoptic@...>
Date: Tue, 02 Feb 2010 06:34:52 -0000

> On 1/31/10, FyberOptic <fyberoptic@...> wrote:

Thanks!

> something in there for screen size.

You know, that'd be pretty handy to have.  One of the things I want to do to all of this is make it work on both modified and unmodified Mailstations.  I'll detect if the hardware mod is present, and if not, the CP/M system area will start at 0x4000 instead of 0x0000, and apps will load at 0x4100 rather than 0x0100.  Of course, this means only apps compiled to run at that starting address will work.  But if one has the original source to something, it makes it a lot easier.  Disassembling something and trying to find all the memory accesses to the system area is a huge chore.  I've tried!

But yeah, I could compile a separate version of the Z-machine to run using the other start address, if I can find what you're talking about.  Maybe for executables that use the higher starting address I can use an .EXE extension, and write a custom command processor (or modify this one) which knows to load them at the different location.  That way unmodified .COM programs can launch at the traditional 0x0100, if one has the hardware for it.

And of course, recompiling would also let me do your original point, to specify the display width depending on which font I want to use.

> spot with an 80-column screen but hard to spot at 40 columns.

Well, who knows where this copy of Zork even came from.  It might have been for a specific machine.  I've found lots of software during my digging around which is computer-dependent, despite CP/M being so open-ended.  All the different displays available was probably the biggest reason.  Especially for computers which had graphical hardware rather than just character output.

I think this copy was probably suited to a 64 character-wide screen.  I would have had that with the 5x7 font I wanted to switch to, but now I'm up to 80 characters with this 4x6 font.  That should be fine for any CP/M app!

> obvious blocking point, so somewhat easy to create test-cases for.

It's funny you had that problem, because my very first problem after getting the opening screen was an issue when I input text.  But I was using a clunky way of dumping the .COM file into memory rather than properly loading it off the disk, and I think I as clipping the end of it off.  Once I upgraded to using BIOS commands to pull sectors off until EOF, it worked okay.

> Smashing work.  I can't wait to see a photograph of it running on the

Thanks again.  I'll be sure to take one!

---

From: "FyberOptic" <fyberoptic@...>
Date: Tue, 02 Feb 2010 07:49:00 -0000

> Or is it part of the BDOS???  I don't remember.

Depends on the CP/M version.  From what I understand, the CCP/BDOS/BIOS were one big clump in the system tracks before version 3, and all loaded into upper memory by the loader also on the system tracks.  With version 3, the CCP became a .COM file.  Though v3 also uses a stripped down BDOS and BIOS in the system tracks, which it uses to then load the full versions off of the data area of the disk.

Even though I'm using v2.2 of the CCP now, I'm loading it off the data area of the disk rather than being part of the system tracks, though still placing it into an area below the BDOS.  I originally decided on this method because my BIOS was getting a bit pudgy from having to have the font data in it, not to mention all the code in general for handling the virtual text display and virtual disk which a normal CP/M system wouldn't have had (I think a normal BIOS was supposed to just be 2KB), and I didn't want to waste too much disk space with system tracks.  I started out with 2 tracks (8KB), realized that wasn't enough, went to 3, then realized it would just make sense to use a full four for 16KB, giving me some growing room.  I think I might could cram it in there now though, especially since the new font I'm using takes less space.    

> part that needs to be customized.

That's true.  The thing is though, there are tools used for actually creating/modifying a CP/M system.  Tools which run on the system itself, already running that version of CP/M you're going to modify.  I obviously didn't have that luxury, so I had to get by with modifying the BDOS and CCP assembly files to work with SDCC's AS-Z80 assembler.  I'm linking the BDOS against the BIOS myself, along with my loader app in front of all of that.  It seems to be fine, surprisingly, aside from the CCP part.  It works at boot, but doesn't work after the warm boot when an app exits, even though the same code is called.  I haven't been able to figure out why yet.

> Perhaps you are closer than you think!

Let's hope!

> start from there (0x0100)?

My original test involved literally dumping the zork.com directly off the dataflash and into memory and executing it.  The BDOS and BIOS were already in place in upper memory though via the OS loader app, and this file dumping was happening in the BIOS as part of the startup, so it was practically the same as loading it off a disk, even if a bit of a kludge.  It was reading zork1.dat off the disk in any case.

After I saw that the disk routines were working, I went on to actually loading zork1.com off of the disk using BDOS commands, reading 128 bytes at a time until I hit EOF, then executing 0x0100.

Now I'm still doing something similar at startup, except with CCP.COM, but loading it into higher memory underneath the BDOS.  It's compiled to base at that address too, so I ought to name it like CCP.SYS now probably so that it can't be executed from the command line.

I did originally make the CCP.COM load at 0x0100, but it was incapable of launching apps at that location, which is what prompted me to shift it upwards where it belongs for this version.  CP/M 3's CCP loads at 0x0100 though, but loads apps via a resident loader module which it loads up high using the RSX functions in the v3 BDOS.  The transient portion of the CCP then gets overwritten by the new app.  I like this approach, and will probably use it if I ever write my own CCP.  Though if I rewrite the BDOS too, I'll just write the app loading capability directly into there via a custom function call.

> You get a pretty big stack out of the deal, too!

My hardware mod can actually switch slot0000 back to codeflash by toggling P28.3 back to 0, but that would put the original interrupt code and everything back into place.  I could still get around it with the interrupt jump trick using the other interrupt mode, but as you mentioned, I'd have to waste some ram to do it.  I only really wanted to have to use this method for people who won't have a hardware mod, since it's such a kludge.

I wrote my own routine though which reads any 256-byte dataflash sector into a buffer.  I probably could have ripped that function out of the Mailstation disassembly too, but I prefer to write as much of my own code as possible.  I started rewriting the keyboard routines before, but debouncing and key repeat were giving me issues, so I ended up just going with the borrowed code so that I could move forward at the time.  Not to mention, making the keyboard work again was fairly boring, to be honest.  They're still separate and compatible object files though, I can just swap them in and out of the BIOS module, so maybe one day I can have all original code in the BIOS.  The font I'm using is public domain now, too!

Speaking of switching banks though, I've considered it for some of the larger functions (and the font, for example) into one of the other RAM pages, and just bank-switch it in when needed during BIOS calls.  CP/M 3 supports bank-switching, even, but I don't have a full understanding of how it works yet.  Not that I need v3 though to take advantage of such a thing if I wanted.  But so far RAM hasn't been a problem for the things I've tried.

> with the other mailstation files.

That's the thing, I knew that the Mailstation uses a file system of some kind in its OS, but I have no idea how it works.  So I just avoided that whole area where it stores emails and things.  

How much do you know about it?  Like, where exactly in dataflash does it start, what format do the files/directory entries use, etc etc?  Anything you've figured out might be useful for getting more usable space!

Eventually I also want to finish the code which would let me have a drive B which works across the parallel port cable to a server app on the PC side.  That would let a person copy and move files onto their Mailstation flash disk via the OS.  At least, once I actually implement disk writing, that is!

> start at begining of one.

I'm actually using a fair amount of the 16KB of system tracks for my OS loader + BDOS + BIOS.  The CCP I compiled is a little less than 2KB though, which is smaller than I originally expected one to be, so I might be able to fit that into the system tracks too, which would save a bit of space on the data area of the disk.

I've thought of different ways of including a traditional loader app.  One is to include it in the OS loader, where maybe you hold a key when you launch the app in the Mailstation menu and it falls back on that mode to let you load things on the parallel port like before.  Or maybe putting a key combo you can press any time the OS is loaded.  And then of course is the other method of just making it a .COM file on the disk.  It's one of the next things on the list to do though so I'll figure something out.

> Does it mean we can just use it now???  Or does it only apply

Well, considering that's where I downloaded most of the stuff from, I guess that means it's okay!  I dunno.  That's one reason I always like writing as much of my own code as possible, like with the BDOS.  It'd perform the same interface functions for CP/M applications, but is still different on the inside.  And be another member in the ocean of OS clones~!

> Gosh, and now you even confessed!  In public!!! 

I'll plead insanity when they drag me away to jail.  They'll surely believe it when they realize I was rewriting a 30 year old operating system for a 10 year old device.