Imagine there are no variable names. Imagine working – in 2016 – with registers. Imagine one minute file load times. Imagine that all commands are just numbers. Imagine there’s no usable string processing.
Welcome to Emme 3. During the years that I worked in travel demand forecasting, this was the main tool available to me.
Emme was undoubtedly a trailblazing innovator when it first came out in 1982 and remained a power user’s dream through to the early 90s. But it clearly missed the Windows boat; the software seems to have stagnated until beginning a revival in the late 00s.
Like any early 80s software, the original version is old enough that I’ve never even heard of the hardware: it was implemented on the Pixel 100/AP Supermicro, a 32-bit 68000 based UNIX workstation. The later shift to DOS was in the pre-486 era when CPUs didn’t have floating-point capabilities and required a separate co-processor chip or board, as this 1986 press release hypes:
“The first computational tests show that the [16 MHz co-processor board] DSI-780 beats even our most optimistic expectations. In raw computing power, it is more powerful than a full size VAX 11/780 and is 3 times faster than the already astonishing DSI-32.”
By the early 2000s, the vendor had misguidedly put energy into a basic Unix GUI, just as the final users abandoned Unix. The vendor belatedly added a real interactive GUI with 2007’s Emme 3, offering a Windows interface on top of an unchanged command-line core. The revival gained steam with 2012’s Emme 4 – which began to repackage the core algorithms with both a Python API and GUI, including IPython Notebook. It’s now a pleasure to use – a shiny, modern and easy-to-use front-end with a very powerful core under the hood.
But I’m going to talk today about the earlier intermediate version, Emme 3. This article is about masochistic techno archaeology, not productivity. Emme 3 was the awkward teenage years, when the transformation to full adulthood wasn’t quite complete yet.
Next sections:
The Pain
Menus & Macros
Analysis and database management in Emme 3 was still entirely a 1980s experience, driven by a console menu system. And – the kicker – the programming/macro language is actually just a recording of a menu session. If your interactive session looked like this, with user input shown in red:
3. M A T R I X E D I T O R
3.01 Input / modify / output zone groups
3.11 Input matrices using batch entry
3.12 Input / modify matrices interactively
3.13 Plot matrices
3.14 Output matrices
3.16 Plot histogram of matrices
3.21 Matrix calculations
3.22 Matrix balancing
3.23 Triple-index Operations
Enter: Next module=3.12
3.12 Input / modify matrices interactively
1= initialize a matrix
2= delete a matrix
3= modify a matrix
4= copy a matrix
5= list a matrix
6= list matrix directory
7= initialize matrix directory
8= change matrix protection flags
Enter: next=2
Enter: Matrix=mf13
mf13 diftrt transit time difference (12-09-13 10:31:21)
Delete this matrix?y
Deleted
3.12 Input / modify matrices interactively
1= initialize a matrix
2= delete a matrix
3= modify a matrix
4= copy a matrix
5= list a matrix
6= list matrix directory
7= initialize matrix directory
8= change matrix protection flags
Enter: next= q
Then your “source code” would look like this:
3.12
2
mf13
y
q
Or, if you took advantage of the shorthand notation, you could instead say
~+|3.12|2|mf13|y|q
Perfectly legible – a nice, succinct way to say “I deleted a matrix!”, right? The menus themselves are mostly in the manual, and they’re mostly consistent between software versions. A particular delight is that the sequence of menu questions is state-dependent, with no easy way to predict whether an extra confirmation will pop up in any given situation. If matrix mf13 doesn’t exist, you’ll get a different series of questions.
Variably Painful
The data model is pretty straightforward, mostly:
- a map represented as a graph of nodes and directed edges, with per-node and per-edge custom data
- transit routes
- origin-destination data such as matrices
- global variables
The pain is in working with this data. Most of these were accessed by numerical IDs, not by name – mf123 is an origin-destination matrix with ID 123. The macro language offered no arithmetic operators to let you work with numerical IDs. Matrices and per-node/edge data were the rare exception – you can refer to them using a six-character name. Unfortunately, mf”ttawq6″ wasn’t really enough to clearly define “am weighted travel time for the home-station leg of commuter rail trips”.
The global variables were also painful: a small set of floating-point registers, no scope management when calling between macros, and painful string operations. The whole experience felt a lot like programming DOS batch files, or working in assembly language. You can do it, but roll up your sleeves and write a lot of comments.
The Promise
So why did people put up with this, instead of just using a normal GIS system – say a Python+PostGIS combination? Like with any domain-specific language, you have to consider the trade-offs against a general-purpose system, and I’d describe four main reasons:
- A data model that includes transit lines.
- Ready-made implementations of most of the standard operations you’ll need, like traffic and transit assignment, management of park & ride demand, etc.
- Base of existing trained users
- A domain-specific GUI and graphing system. (While I’ve complained here about the unmodernized parts of Emme 3, the GUI front-end was modernized and was on an equal footing with peers of its era like ArcGIS.)
I have little experience with peer software like TransCAD or CUBE, but I’ve heard that Emme’s advantages over those include:
- High-quality, battle-tested algorithms
- Customizability
While Emme 2 was still an ugly DOS application in 1996, it made good use of the 1980s and 1990s, building a truly powerful database system for the era, with a sound data model and good algorithms that were well-tailored to their problem domain. I’m sure that if you were an Emme expert in the late 90s, you could do some truly mind-blowing things compared to your peers. (For the Emme veterans in the crowd, which 1986 tool would you choose: the Emme network calculator or… uh… ArcINFO and its network tools?)
Even in the mid-2000s, that tradeoff still tipped in favour of Emme and its competitors like TransCAD or CUBE. Recreating the Emme tools in a general-purpose platform would far too big an investment for a Metropolitan Planning Organization or other transportation agency.
My Own Path
Despite the obstacles, I got a lot done in Emme 3, and found some good patterns to work around its pain points. Perhaps the most surprising is that Windows filenames proved the best abstraction around the agonizing internals. If I needed to delete three matrices, I didn’t have to just repeat the earlier 3.12 incantation three times; instead, I had a nice file called matrix-delete.mac that allowed me to just write:
~<matrix-delete mf3 mf4 mf15
The windows filenames aren’t limited by the awful 6-character limits that pervade everything else, and it became possible to make my code vastly more legible. (And no, I never imagined that I’d praise the virtues of Microsoft’s 6~n.3 file naming.)
I’m sure that if I’d chosen some standard auxiliary tool as my second home – like shelling out to Python to generate custom macros on demand – I could have gone a lot further. But… taking on external dependencies is also a challenge, particularly when working in a team environment with a lot of non-software staff who can’t readily install/configure Python themselves. Not to mention having to fight the government IT staff every step of the way.
Good software practices can still emerge in such an environment – some day, ask me about how we implemented version control on our network data.
Conclusion
I’m being deliberately snarky about a legacy piece of software in this article, but Emme was utterly invaluable to me as a modeller, and it has advanced by leaps and bounds in the last decade. I’m writing about it today mostly as an entertaining anecdote: a glance back at computing/data science history, and a snapshot in time as Emme’s vendor took the painful steps necessary to modernize 1980s-era software. I honestly agree with the vendor’s decisions – I’m sure when they planned Emme 3, it was painful to limit the scope and leave the macro language untouched, but it was the right choice. That version kept their software alive, and gave them the breathing room to make Emme 4. I remain a staunch supporter of evolutionary software improvement rather than the “clean slate rewrite.”
References
There’s a great article on the history of Emme from Michael Florian. And there’s clearly some drama in the backstory – it’s clear that long-time contributor Heinz Spiess had a major falling out with the company in 2005, when the Unix GUI “Enif” was abandoned in favour of the Emme 3’s Windows GUI. Speiss also keeps a great archive of the 80s and 90s Emme 2 newsletters.
