On ASCOM Alpaca, INDI, and the Observatory of the Future

This post is my response to a question posed by Ray Gralak to the ap-gto mailing list on 26.04.2019:

Hello all,

Of those users wishing for a non-Windows support for your mount, which platform would you prefer Astro-Physics to devote resources?

INDI or ASCOM Alpaca?

A simple but timely question that deserves a full answer. Grab some coffee and let’s go for a ride.

Recent years have seen an increase in the number and types of astronomy hardware that offers network connectivity as a communications option. For example, one may use an Optec FocusLynx focuser that has 100Mb ethernet, with WiFi as an option. The Astro-Physics CP4 (and CP5) mount controllers offer both flavors of ethernet as well. Cameras from several makers feature 1Gb ethernet as a way to ship bits. As sensor pixel counts increase, cameras that implement IEEE 802.3bz, Wi-Fi 6, or 10Gb ethernet will not be far behind. Clearly ethernet (in both its wired and wireless forms) with IP at layer 4 is slowly becoming the preferred method for the control and communication of astronomy devices. We will surely see it become commonplace as time progresses.

In addition to astronomy devices themselves, performant yet low-power and physically small compute platforms have gained an everyday presence under the observatory dome. We first saw this with simple microcontoller-based Arduino-like project boards. Over the past 10 years, we’ve seen very powerful and cheap SoC (System-on-Chip) platforms arrive. These systems are capable of running full OSes and are powerful enough to operate desktop-level application stacks. They also offer a myriad of connectivity options, all while sipping power and fitting inside the palm of one’s hand. Combined with remote desktop or other remote management means, these systems – some of which can live on the telescope itself – are enabling the astrophotographer to be physically removed from the observatory without the need for custom and expensive bespoke systems.

Open protocols are a must

The combination of these two classes of devices, along with popularity of them due to the general spread of embedded platforms to the consumer class, has revealed gaps in astronomy software stacks. With network-enabled astronomy devices, we have been, in a way, hurled back to the pre-ASCOM and INDI days of closed and proprietary APIs and wire protocols. This often means connectivity to these devices can be accomplished only through a gateway. The gateway is usually software without which it would be impossible to communicate with the remote hardware’s undocumented network API and command language. More often than not, the gateway software is also closed-source and the API it and the hardware speaks has minimal public documentation available, if any is made available at all.

This is a problem for a few reasons. First, I’m a firm believer in open protocols that enables any person to communicate directly with a device without restriction. Closed, and even semi-closed, protocols are going to be a non-starter here. This is certainly the case if legal agreements are required to gain knowledge of them. Second, closed protocols decrease a device’s long term value. Companies go out of existence, product lines concluded, and software authors move on. The small corpus of knowledge around closed protocols and APIs is steadily lost to the sands of time, irretrievably locked away in a pharaonic sarcophagus of NDAs and IP law. Eventually the hardware cannot be utilized in a reasonable manner despite it being in working order. Third, closed protocols are viral in the wrong direction. Creating proprietary protocols encourages other vendors to act similarly. The result is more closed protocols and a worsening situation across the industry.

ASCOM Alpaca – A solution?

The the world of computer-controlled observatory technology saw the recent introduction of ASCOM Alpaca. If you have not heard of this yet, or may be confused as to where this fits in, here is a summary. Alpaca’s goal is two-fold:

  1. Enable network (ethernet, WiFi) communications between the controlling computer and astronomy/observatory devices at the ASCOM layer.
  2. Introduce non-Windows and non-PC platforms directly into the ASCOM ecosystem.

Alpaca attains this by taking the classic ASCOM object classes (Telescope, Focuser, FilterWheel, Camera, etc.) with their methods and properties and exposing them as a RESTful API over HTTP. This brings ASCOM clients to the network, but it also means that the system which implements the Alpaca API service may be running any OS and CPU architecture under the hood. Combined with today’s powerful embedded systems, such a service may be running on a focuser controller, camera, or mount control box. It can be implemented in C, Go, Rust, PHP, Python, or any language that is suitable for the environment. In the end, we get an open protocol which can be used to interact with devices over the network. This sounds fantastic.

That comes with some problems…

However, it’s not all sunshine and butterflies here. There is an issue which the current Alpaca specification outright ignores, as well as some missed opportunities. As I mentioned, Alpaca is a translation of the current ASCOM classes, methods and properties into a REST API. It does not add any additional facilities or features beyond what those that ASCOM already deals with. This means that any device features which fall outside of ASCOM’s (and hence, Alpaca’s) purview are completely omitted from a use and manageability point of view. Devices will need to offer additional out-of-band mechanisms for the user to manipulate such features. This need can bring us back to the introduction of closed or severely under-documented protocols employed by closed-source apps.

Alpaca also does not take the opportunity to strike at a problem that is commonly seen with network-connected devices in general – the mere act of finding them on the local network. Alpaca is a prime example of a protocol and device combination that would benefit from the use of a standard cross-platform discovery service such as DNS-SD or Multicast DNS. It is currently required that a user have pre-existing knowledge of the IP address of the device on their network in order to configure it in an Alpaca client. This can be problematic because these devices generally have no way of conveying to the user the IP address assigned to them via DHCP. There would be great benefit for technical and non-technical users alike if Alpaca demanded implementation of service discovery in both clients and servers. The opportunity to do this was with the first cut, and that chance is now behind us.

Wait, my mount’s DHCP lease changed!? I guess it’s time to scan the network for it. Again.

One small issue is that the spec does not address common network pathologies and how clients and servers should address them. TCP’s Nagle algorithm in particular can adversely impact time-sensitive communications between Alpaca client and device. There is no advice on employing Alpaca client-server communication over high latency or lossy links and how a client should be expected to recover from connection faults. I suppose some experience needs to be gained in these categories, but some insight into the potential impacts of these problems can be had by studying 35 years of TCP protocol use. I think we’re going to see a lot of teething issues here as users try connecting to devices farther afield than localhost where any deleterious effects of buffering, latency, and packet loss are going to be more pronounced.

INDIlib – The lurking alternative…

Due to ASCOM being generally confined to the Microsoft Windows platform, the desire for an analog for non-Windows platforms resulted in the creation of the INDIlib project 16 years ago. The goal of INDIlib is to create a cross-platform application library and network protocol (INDI – Instrument Neutral Distributed Interface). It attains this by demanding strict POSIX adherence and leaves any OS-specific implementation details to lower levels, away from the Public parts of the API. The result is an application and driver API that does not depend on any feature that is specific to a single operating system in order to function, a counter to ASCOM and its complete reliance on Windows’ Common Object Model.

A network-centric client-server model was part of INDIlib’s design from the start. Local connection types, such as USB and serial communications, are also supported. Given that this design was initiated 16 years ago, the network API revolves around XML. While not as clean and as modern as Alpaca’s REST API and JSON, it’s still very serviceable and any contemporary programming language offers a plethora of ways to handle XML.

On the surface, INDIlib has a lot of similarity with ASCOM. It defines broad classes for types of familiar hardware devices, with a few differences here and there. Like ASCOM, each class defines a set of Standard Properties that are both generic and particular to each class. Where INDIlib departs from ASCOM in this area is that INDIlib also has a provision for Custom Properties. A camera, mount, or other driver for a specific bit of hardware might have features that the Standard Properties do not apply to – sensor offset, configuring the GPS of the QHY174M-GPS, or turning on or off PEC on a mount, and so-on. These can be defined as Custom Properties in their respective drivers and those then become accessible to INDI clients. I’m not familiar with any review process, but I would imagine if sufficient camera drivers provide a Custom Property for something like offset, that property may one day find its way into the list of Standard Properties. No matter how you cut it, though, INDI offers a way to get advanced management back to the user through a single protocol and interface. This single feature is sorely missing in ASCOM.

…Which also has some drawbacks

With that, you might ask why we don’t see more INDIlib devices or better support for INDI offered by vendors? The answer to that is two parts.

First, there has never been a full, native Windows, and FOSS INDI client library implementation. The closest one can get to such a thing is INDI Sharp and wINDI. INDI Sharp looks like a promising .Net/Mono client implementation, but it is largely incomplete and development appears to have stopped in 2017. wINDI is actually just a INDI-speaking wrapper around ASCOM and is non-FOSS, being that it is not open source and comes with restrictions on use. If there were a native Windows client library available under an OSI-approved license, I can guarantee that INDI would be a more central part of the general astronomy software offerings today. For whatever reason, however, this seems to have never been a priority of the project. Personally, I think it’s an enormous missed opportunity and has played an outsized role in confining INDI to the fringes of astronomy software.

Developers!
Developers!
Developers!
Developers!
Developers!

The second reason, which amplifies the fault of the first, is that the Windows monoculture in the astronomy software world is very strong. Not only because of ASCOM, but also because computer-based astronomy really got rolling in the mid-late 90s, when Windows was the predominate desktop OS. UNIX workstations were prohibitively expensive, Linux on the desktop was still a CS lab and basement curiosity, and MacOS X was but a twinkle in the eyes of Steve Jobs. Windows 95 and NT 4.0 were good enough, accessible, and familiar. When Windows-only ASCOM arrived, those combinations of factors cemented it, and its massive inertia meant it stayed there. Many applications that are considered core to the discipline got their start around then and are now 2 to 3 decades old. If there’s going to be a popular alternative platform at all, involving any OS, it absolutely has to include Windows.

So, what does the future look like?

Here we have two systems that are different from each other but are very similar in their goals, with capabilities that are largely in parity. All-out technical competition between them is really just a matter of a full Windows port of INDIib away. Long story short, I see the future coalescing around an improved INDI rather than ASCOM, and I predict ASCOM has 10, perhaps 15, more years left in it. Here is why:

I don’t take seriously Alpaca putting a flashy sticker on itself and claiming it makes ASCOM cross-platform. If anything, I think of it more as a side-effect of getting ASCOM on the network. Sure, ok, it’s a REST API and anything short of an abacus can interface with that. By that measure, my IoT doorbell has enough power to throw mount slew commands at it. But I don’t take the hype seriously because ASCOM Org (the organization) has spent 20 years with fingers in its ears when it comes to the needs of non-Windows platforms. Until now, there has been zero interest in anything but Windows, and at times it seemed like the users of non-Windows platforms were regarded as unwashed mongrels who were unwilling to accept their Redmond savior. But with Alpaca, it’s like the habitual miscreant of a brother woke up one morning and found religion. It’s strange and deserves a bit of wary side-eye after two decades of frustratingly watching ASCOM Org deftly avoid, and at times condescend, anything that lacked a C: drive.

ASCOM also has a glaring long-term weakness: it’s not governed like an open source project should be. As far as I can tell, ASCOM Org is comprised of two, maybe three individuals. Peter seems to do most, if not all, of the coding, and Bob runs the website and mailing lists. From the outside, ASCOM Org appears to operate more like a company than a collaborative project. Architectural decisions and ideas are rarely discussed in advance and in the open. Code changes are not reviewed by anyone. There’s no spirit of collaboration with outside projects. No pull requests have ever been accepted (because it seems no one feels like they’re worth submitting.) Even Alpaca was a surprise to most, its name not mentioned on either of the public ASCOM mailing lists until the very day its existence was announced after a year of quiet development.

All in all, ASCOM is very opaque for an open source project and this is always going to turn new people and new ideas away, as it has since I started observing it 12 years ago. This severely reduces the chance of the project attracting new talent and future maintainers as a result. To that end, I do think that if Peter and Bob decided one day to pull up stakes, ASCOM will become pilotless. At this moment, I can’t really pick out a natural successor who has the vision, institutional knowledge, and inclination to keep the lights on and further develop ASCOM. This is not a comfortable outlook for a framework that is depended upon by hardware manufacturers, software producers, amateurs, and professional scientists.

The future is (the truly) open

This leaves us with INDIlib. I feel that, of the two, INDIlib is in a much stronger position in the long term. Its governance as a project is far more familiar with the sort of open source projects that are welcoming to new contributors and ideas. Along with that, its larger corps of committers and maintainers assures a collaborative environment in which to operate. The downsides to INDIlib are primarily technical in nature and are rectifiable if the motivation and commitment could be found. It is severely in need of parity on Windows and better awareness with hardware vendors in general, and knowledge and advocacy on both fronts would benefit it greatly. I have not talked with INDIlib project leaders about their feelings here, but I would certainly welcome their thoughts and input.

I recognize that my thoughts on this subject can easily be construed as adversarial towards ASCOM the technology, as well as ASCOM the organization. Yes, at times I have been frustrated with ASCOM’s stances on how its decisions affect its users and astronomy software as a whole. However I do recognize the extremely high value that ASCOM has had in bringing a common interface to an incredibly large and varied zoo of hardware devices since the late 90s. Application developers such as myself are thankful of that. But time marches on and needs evolve, and I feel ASCOM has not been able to keep up with that. In light of that, it may be time to consider a new direction.