AppleCrate: An Apple II-Based Parallel Computer
Michael J. Mahon – July 26, 2004
Revised – January 11, 2008
Introduction
At the outset, when designing NadaNet, I envisioned that it could be used to support parallel computing on Apple II machines. To add more processors and save space, I decided that I would package several Apple //e main boards together, without keyboards or peripheral slot cards. (I didn’t disassemble the Apples myself, but found a box of Apple //e main boards being sold as an auction lot for about a dollar each!) I settled on a wooden cube about one foot on a side which I slotted to hold up to 8 main boards. For whimsical reasons, I called it an "AppleCrate".

Description
The boards are powered by a PC power supply (out of sight, with only its power switch showing in the picture). They are connected only to the network using the RCA connector on the lower right in front, described below. It is also possible to connect one or more monitors to their video ports, as shown in the picture.
I had originally hoped to use the board edge connectors (front edge) that Apple used at burn-in to power the boards, but discovered that different board revisions have different edge connections, and none have all the voltages normally supplied. I therefore decided to use threaded rods as power busses (on the bottom) and signal busses (on the top).
The usual mode of operation is to connect the AppleCrate to a more fully configured "master" Apple //e using NadaNet. The Master boots the eight Apples in the AppleCrate and uses them to do work. Once they have been booted and started, they can run independently of the master—though they are clearly I/O-constrained!
For testing and demonstration purposes, I have written several programs. The first was a parallel work simulator. The simulator puts independent "jobs" into a message queue of work to be done and receives "results" from a message queue of completed work. The parallel simulator is described in The AppleCrate Parallel Work Simulator. Another very useful program is really just the part of the parallel simulator that runs an Applesoft program on all slave machines, called PRUN (Parallel RUN). In addition, a File Server allows programs running on AppleCrate machines to use the server's ProDOS file system, without the need for a local operating system.
Indicators
It has proven useful to have some real-time indication of each board's activity. The stock board contains a red "power" LED (at the bottom) and a red "speaker" LED at the top. Both are easily visible from the back of the boards (the "front" of the AppleCrate). The function of the power LED is fixed, but the speaker LED is usable as an indicator that software running on the board can operate, just by toggling the speaker. For example, printing a "beep"--CHR$(07)--causes the speaker LED to flash for 0.1 second, and can be used to indicate some condition in the software.
Although the Applecrate network interface described below incorporates an LED to show global network activity, it is very useful to be able to see when any particular board is sending on the network. This need is met by using the PDL 3 timer to "stretch" each packet send operation into a visible flash of a green rectangular LED.

These photos show the modification made to the 558 timer chip, in which a 560 Ohm resistor is connected between pins 5 and 8, and the "send" signalling LED is connected between pins 8 and 12, with pin 8 being the anode. A resistor in the 100k-150k range is plugged into game port pins 1 and 11, to set the timing of PDL 3. The rectangular LED is attached to the cassette output jack with a strip of double-stick foam. In this way, the send LED is added without any modifications to the board itself.
Network Boot
Early in the design of the AppleCrate, it became clear that, since the machines would have no I/O capabilities other than the network, they would need to be booted from the network. This required that the ROMs on the boards be replaced with EPROMs containing modified RESET code to perform the network boot.
A study of the required code, and the easily replaceable code in the ROMs, suggested that replacement of the self-test code was the easiest path, since it is self-contained and contiguous. The Enhanced //e Cxxx-Dxxx ROM region contains $300 bytes of self-test code, while the Unenhanced //e Cxxx-Dxxx ROM contains $400 bytes of self-test. Since the desired network boot code would require over $300 bytes, and since I had no need of the Enhanced features in the AppleCrate, I decided to use a modified Unenhanced Cxxx-Dxxx ROM.
The BOOT protocol is a bit unusual, in that the Master machine initiates it as a delayed broadcast response to one or more GETID requests sent by the machines requesting booting.
Since the booting machines issue GETID requests for booting, it is necessary to ensure that they do not arbitrate using the same delay, since that would cause them to receive the same (non-unique) ID. It is harder than you might think to create a non-deterministic number on a bunch of simultaneously reset Apple //e machines. I had hoped that the video generator might start in a "random" state, but that is not the case--all machines start in the same state. As a result, Version 2.0 of the AppleCrate boot ROM now implements two different methods for disambiguating the AppleCrate machines.
The original method capitalized on the fact that AppleCrate boards use the Paddle 3 timer as a pulse stretcher for an attached "send" LED, so each board can be provided with a resistor chosen to produce a unique paddle read result, supplying a unique temporary ID for booting. If this method is used, by tying all board's PB2 inputs low, permanent IDs are assigned in an unpredictable order.
The new method introduced in version 2.0 is a "daisy chain" scheme in which each machine, after a successful GETID, enables the "next" machine to do its GETID. This is done by having each machine initially set its AN2. Then, it waits until its PB2 is low, completes its GETID, and then drops its AN2, enabling the next machine. PB2 of each machine is connected to AN2 of the "previous" machine. The "first" machine is permanently enabled by connecting its PB2 to ground through a 4.7k resistor. This causes the permanent IDs to be assigned in the fixed order of the physical daisy chain, while keeping all ROMs identical.
When a network-booting machine is reset, it checks page 3 to see if it is being cold started or warm reset. If it is a warm reset, it re-enters its Server loop. If it is a cold start, it initializes, raises AN2, does a Paddle 3 read to create a (possibly) unique temporary machine ID, and then waits for PB2 to go low.
When PB2 goes low, it uses its temporary ID to make a GETID request to the Master (machine ID = 1). If the Master does not respond, then the ROM boot code re-sends the GETID request about every 100ms. until it receives a response.
The Master GETID service routine looks at the requesting ID, and if it is temporary (>127) it assigns the next available permanent ID and sends it in an ACK to the requesting machine. (If the sender already has a permanent ID, then the Master simply returns it in the ACK.)
If the GETID is successful, the requesting machine sets its ID to the assigned value and Data ACKs the Master, confirming that the ID has been received and installed. The Master then allocates the ID and notes that a GETID has been processed.
The requesting machine then lowers AN2 (enabling the next daisy chained machine, if any, to do a GETID) and enters a RCVPKT loop looking for a BOOT request control packet, either broadcast or directed to it specifically. If more than 5 seconds elapse without receiving a BOOT request, the machine will re-issue a GETID (this time using its assigned ID code).
Meanwhile, the Master’s SERVER loop is counting the approximate passage of time. After about one second has passed without processing any GETID requests, the Master checks to see if any GETIDs were received earlier. If they were, then the Master sends a broadcast BOOT request containing the target address and length of the boot code to follow. The reason for delaying the boot response is to allow multiple simultaneously reset machines to "check in" with GETIDs and prepare themselves so that they can all be broadcast-booted simultaneously.
The booting machines, upon receipt of a directed or broadcast BOOT request packet, receive the boot code directly into the address, and with the length, specified in the BOOT request. Upon the error-free completion of this transfer, the video display is cleared, a banner showing the machine ID is written, and control is given to the initial address of the boot code. If an error is detected during the transfer, the booting machine will retry the process starting with another GETID.
AppleCrate Network Interface
NadaNet is a TTL-level serial network in which ONE is represented as a logic high (greater than +2 volts) and ZERO is represented as a logic low (less than +0.7 volts). The fanout capability of a TTL annunciator output is sufficient to drive a dozen or so TTL pushbutton inputs if they are not otherwise connected, as in the case of the early Apple II machines.
Although NadaNet began—and was named for—not requiring any extra hardware, some changes Apple made in the Apple //e pushbutton circuits necessitated hardware buffering.
The Open- and Closed-Apple keys on the post-][plus machines are connected to pushbutton inputs 0 and 1. In order to function in the same manner as game controller buttons, the keyboard contains 470-ohm pulldown resistors for each Apple key, and pressing the key connects the corresponding pushbutton input to +5 volts.
In addition, to support board self-test at the factory, the main board contains 11K pullup resistors pulling a keyboard-less pushbutton input to +5v.
The effect is that the pushbutton inputs on later machines are relatively low-impedance inputs, each of which sinks on the order of 10mA when driven high. While an annunciator output could drive one or two such inputs, any attempt to connect more than a small number of machines together directly would overwhelm the drive capability of the annunciator outputs of the machines.
My solution was to add emitter followers to drive the pushbutton inputs and the network. These emitter followers are built onto a 16-pin header (or machined-pin socket) that plugs into the 16-pin game port and provides the network and ground wires to connect to the network bus.
When building the 8-board AppleCrate, I considered using eight of these network adapters, daisy-chaining them from board to board, and then on to the rest of the network. However, I soon realized that since the boards in the AppleCrate do not have keyboards connected, their pushbutton inputs are encumbered only by the 11K pullup resistors, for a total parallel pullup resistance of about 1.4K.
A single emitter follower with a low-value pulldown resistor easily drives this load. To collect the annunciator outputs, I decided on a distributed 8-input OR using 1N914 diodes connecting from the game connectors of the individual boards to a common bus.
The AppleCrate network adapter circuit is very similar to the individual Apple //e adapter:

Unlike the individual machine adapter, a 74LS08 has been used to standardize logic levels (which would otherwise suffer three diode drops) and to drive an LED indicating activity on the net.
Note that the buses and the network are clamped to stay within 0 to +5 volts. This was a precaution against ESD and minor "metallic mishaps" in its vicinity.
The 220 ohm network pulldown resistor in the AppleCrate interface is sufficiently low that no supplemental pulldown should be required if an AppleCrate is connected to the network.

Running Programs on AppleCrate
Programs are started on Applecrate from the master machine (ID=1), which is also responsible for booting. The sample application PRUN (Parallel Run) demonstrates how any Applesoft program can be run on all serving AppleCrate machines. Of course, the program that is run is responsible for any further communication among the slave machines or between the slave machines and the master machine.