Making OS/2 family programs with Visual C++
18 Sep 2023Back in the day, I loved OS/2. Unfortunately developing for it was difficult due to lack of tools and documentation - in the pre-Internet era, finding books on the Windows API was a lot easier than the OS/2 API. Recently I've also been wanting to run some of my tools on DOS. So with an abundance of optimism, I tried to kill two birds with one stone: write programs for the OS/2 Family API that can run on DOS but also OS/2, while learning a bit about the OS/2 API.
The Family API tooling consists of two parts. First, an API.LIB which is a statically linked piece of code providing implementations of OS/2 functions on DOS. Many of these translations are straightforward. Second, a BIND.EXE program which takes an OS/2 executable, adds a stub MS-DOS program that implements an NE loader, and links the code from API.LIB so the NE loader can resolve OS/2 functions to the DOS translation.
Unlike WLO, both of these components were widely distributed, with C 5.1, C 6.0, MASM 5.1 and MASM 6.0. There may have been other tools distributing these.
16 bit development tools for OS/2: MASM 6.0b and Visual C++ 1.5
For tools, I got a copy of MASM 6.0b from eBay. This had some unexpected good points and unexpected bad points. In hindsight these were all obvious, but they hadn't occurred to me earlier.
The good:
- MASM 6.0b is capable of generating 32 bit code. There's a kb article describing how to use it to generate both 32 bit OS/2 2.0 binaries, as well as Win32 binaries.
- It includes CodeView for DOS as well as OS/2. These are clearly related but also substantially different. OS/2 is a multi-process system, so a debugger process attaches to a child process. DOS requires the debugger to execute the program in its own context.
The bad:
- MASM includes include files for the OS/2 API. It didn't include anything for Presentation Manager, because it seemed unlikely that people would write UI programs in assembler.
- The include files don't specify a return type, because in assembler, return values are in registers. The caller is expected to know which. As luck would have it though, the vast majority of OS/2 APIs return a 16 bit error code and are quite consistent about it.
- Assemblers don't need a C runtime library. Although it's simple enough to link C with assembly, doing anything in C requires a pile of code not included in MASM.
- The kb describing Win32 is incomplete, because it requires a CVTOMF tool which was only included in pre-release SDKs.
Nonetheless, it did have the core capabilities I was looking for: the ability to write 16 bit OS/2 programs and bind those into DOS.
Very quickly I found that the best development environment for writing these programs is Windows NT. That was unexpected, but NT can run both DOS and OS/2 programs in a unified console, while also supporting Win32 development tools. Both the DOS and OS/2 debuggers work on NT, and can even support 80x50 mode. NT was also a good choice due to using Visual C++ 1.5 as a compiler.
Visual C++ 1.5 has some good and bad points as an OS/2 compiler:
- It is capable of generating OS/2 binaries, just won't by default. Add a module definition file with "EXETYPE OS2" and the linker will generate an OS/2 program. For a "real" program, it should also be marked as safe to run in a window, and supporting long file names:
NAME FOO WINDOWCOMPAT LONGNAMES EXETYPE OS2 - Unfortunately it doesn't generate debug information that the OS/2 Codeview can support, which made progress frustrating.
- It is bundled with Phar-Lap, which provides a form of protected mode OS/2 runtime on DOS.
- It crashes on real OS/2 badly. I haven't found a solution for this. Since it requires DPMI, it is a challenging program for OS/2 to support. (Edit: Visual C++ on OS/2 requires DPMI_DOS_API to be set to ENABLED.)
- Nonetheless, it is much more ANSI than the "native" OS/2 versions such as C 5.1.
Unlike Win32, the startup code in OS/2 depends on assembly. A newly launched program is informed of its state in x86 registers. For a non-trivial program, the startup code needs to save these registers before launching C. These registers inform the program of things like command line arguments, which is needed before getting to main.
.MODEL large, pascal, FARSTACK, OS_OS2
__startup PROTO FAR PASCAL
.DATA
public __acrtused
__acrtused = 1234h
_EnvSelector WORD ?
_CmdlineOffset WORD ?
_DataSegSize WORD ?
.STACK 6144
.CODE
.STARTUP
; OS/2 Arguments
; AX Selector of environment
; BX Command line offset within environment selector
; CX Size of data segment
mov [_EnvSelector], AX
mov [_CmdlineOffset], BX
mov [_DataSegSize], CX
call __startup
Edit: The code above is preserved for the record, but note it makes two assumptions. It uses "OS_OS2" to tell MASM which operating environment the code is for, then uses ".STARTUP" to tell MASM to generate startup code for that environment. Unfortunately support for OS/2 was removed in MASM 6.1, and later versions are far more common than 6.0. Fortunately the OS/2 startup code is only a label to tell execution where to start - there are no implied instructions. The way to express this in MASM is cleverly hidden as part of the "END" directive, which was ommitted above. Here's the version for later versions:
.MODEL large, pascal
.DOSSEG
__startup PROTO FAR PASCAL
.DATA
public __acrtused
__acrtused = 1234h
_EnvSelector WORD ?
_CmdlineOffset WORD ?
_DataSegSize WORD ?
.STACK 6144
.CODE
startup:
; OS/2 Arguments
; AX Selector of environment
; BX Command line offset within environment selector
; CX Size of data segment
mov [_EnvSelector], AX
mov [_CmdlineOffset], BX
mov [_DataSegSize], CX
call __startup
END startup
With that, a C function called "__startup" can resume execution. Note at this point it has no argc or argv, just a stack, and access to the OS/2 API.