Up till now, I have been following Ekhart's template for enabling the serial signals: directly calling the ROM functions to toggle SerPortSel. The only issue with this approach is that the OS also toggles these signals when a port is opened. This means you have to configure the signals after the serial port is opened.
Wouldn't it be great if you could tell NewtonOS that you wanted SerPortSel to be "set to internal" when you open Serial0 or Serial3?
The trick is the functions TVotagerPlatform::SetSubsystemPower() and its twin TVotagerPlatform::GetSubsystemPower(). These functions involve a bitmask that tells the Newton which power rails a subsystem requires. It is a 8-bit mask but is set/returned as a ULong. By looking at the disassembly of TVoyagerPlatform::PowerOnSubsystem(), I have found the following flags:
#define kNeedsPowerOnSrc5v 0x01
#define kNeedsPowerOnSrc12v 0x02
#define kNeedsPowerOnIC5v 0x08
#define kNeedsPowerOnDMA 0x10
#define kSetPortSelectExternal 0x20
The constant names are my own invention. The key one is kSetPortSelectExternal. When this bit is set, the NewtonOS conigures the SerPortSel LOW, indicating an external device connected to the interconnect port. When this bit is clear, it configures the SerPortSel HIGH, indicating that the internal serial slot will use the port.
These functions need an index or ID of the subsystem involved. Again, looking at the disassembly, you find the relevant index. There may be more, but the ones involving the serial port are:
#define kSerial0SubsystemIndex 1
#define kSerial3SubsystemIndex 3
Again, the constant names are invented. So some simple pseudocode to tell NewtonOS "when you open 'mdem' set SerPortSel3 to HIGH" might be:
theVotagerPlatform->GetSubsystemPower(3, &flag);
flag &= ~0x20; // clear the bit, setting internal
theVotagerPlatform->SetSubsystemPower(3, flag);
When the serial port is opened in software, the NewtonOS reads the bitmask and enables the proper voltage supplies and sets the SerPortSel apporopriately. When the port is closed, the GPIO line is returned to its default state. Note for Serial0, the default state is "internal" (HIGH) until you open the port, as this disables the LTC1323 (used on the external port), probably to save power.
These settings are reset on reboot, which I think is probably for the best. You could always have a Pacakge that sets the flags on boot.
Currently, this is what I consider the "right way" or at least the "best known way" to use the serial port select signal for the internal slot. It is much better than manually toggling the signal with the SerialPortXLineDriverConfig functions.