Dale Ghent

I had been a Sun/Solaris fan all of my professional life – I started out on a Tatung SPARCstation 5 clone running SunOS 4.1.4 which served as the main server of the small mom & pop ISP I worked for immediately out of high school. SunOS 4 turned to Solaris 2,  and as I progressed through my career I always found myself in good company with other people who appreciated Solaris. When OpenSolaris eventually came to be, I recall being quite excited to have the mysterious veil around Solaris lifted and to be able to explore and improve it without the requirement of having a Sun Microsystems, Inc. ID badge and paycheck. When Sun became Oracle, and upon Oracle unceremoniously removing OpenSolaris from existence, the illumos project was born and it now serves as a nexus for several distributions, commercial platforms, and individual technologies.

One thing I would like to emphasize in the previous paragraph is the fact that I, while a life-long Solaris and Sun hardware user, was never a Sun employee. When OpenSolaris came about, much of the internal style of process and communicating was laid bare to the world. “The Sun Way” was now in full view, and for the environment OpenSolaris operated in it was still relevant albeit with some tweaks here and there to accommodate non-Sun interests. When Oracle closed OpenSolaris and forced the subsequent creation of illumos, this obviously caused some change. People left Sun/Orcacle, started their own companies and fostered the creation of illumos in ways patterned after the best way they knew how – The Sun Way.

Coming back to the community after the genesis of illumos, I did notice that a lot of these old “Sun-isms” were still in play. People still spoke of “putbacks” and still wanted to see “webrevs” which is gibberish to anyone who lacks their historic context. Beyond these procedural elements, we continue to this day to rely on the Sun Studio compilers and adhere to self-defeating standards observance, some of which was implemented to support now non-existent Sun products. While the illumos code itself is housed on GitHub, we utilize none of the plethora of tooling available for it which quite a number of projects have come to use. GH, in essence, is only used for the repo infrastructure aspect and not much more.

As a maintainer of OmniOS, I often have to deal with these idiosyncrasies – creating webrevs, having to use closed-source, outdated, and unmaintained compilers, posting diffs to mailing lists for other people to deliberate and so on. For all my love of illumos and respect for the incredibly knowledgable people who labor daily with their acumen, insight and knowledge to keep it going, I have to say that there are a few things here that are just too moldy to keep around and, I fear, contributes to dissuading interested persons or parties from fully engaging productively with illumos.

I’m going to pick a few things to go off on. These might be trivial to some, but they’re important to me for some reason or another.

Reliance on a closed-source compiler (Sun Studio). One of the features peculiar to illumos is that it supports what is called a “shadow compilation environment” – where the same C source code can be compiled serially by a compiler that is different from the main one. In this case, this “shadow compiler” is the one provided by the Sun Studio 12.1 suite – the last version of Sun Studio that was available before Oracle slammed the door shut. Running the shadow compiler is optional, but in the case of testing diffs prior to submission, its use is encouraged as the Sun Studio compiler has always been more attentive about code correctness around typecasting and pointers, among other things. This is good. What is (exceedingly) bad about this, however, is that it’s a closed-source compiler which will only bit rot over time, destined to become a living fossil of dodgy province and increasingly dubious utility (C standards move on while it quite obviously won’t.) Now, Sun Studio isn’t all that bad – it does provide the undeniably useful lint utility… however I feel the time is ticking on this relic and a suitable open source replacement should be devised.

Reliance on an aging version of a hacked compiler just to compile illumos (GCC 4.4.4-illumos) This is one I’ve never fully understood. I recall back in the OpenSolaris days the big push to get ON to build using GCC rather than it being the exclusive domain of Sun Studio, and GCC at the time was firmly in the 4.x era. The version used today is 4.4.4, altered for ~reasons~ which are lightly outlined here. Strict adherence to certain Open Group standards and such is the reason I see cited most of the time, however it does bother me that scant few in the community seem comfortably knowledgable about these differences and have the acumen to alter GCC appropriately.

It also begs the basic question: “Why us? Why must we be unique in this way?” As far as I’m aware, FreeBSD doesn’t require an altered GCC to compile; nor does Linux or any other *BSD for that matter. Why must we be a departure from this? Linux actually provides compatibility in the other direction – the kernel source provides the compatibility layer for each compiler it supports (currently Intel, GCC, and clang.) For example here is its compat header for GCC. Is it not possible to do something similar in illumos? Whatever the reason, this arrangement makes me feel really uneasy and one where the institutional knowledge required to maintain it could be easily lost to the sands of time.

Too many false golden cows. Within illumos-gate we still find dusty cobwebs left over from the Sun product ecosystem – a custom Kerberos stack based on a old version of MIT Kerberos was part of the SEAM suite, Sendmail which is based on 8.14.x and includes Sun-specific LDAP alterations, CMU Cyrus SASL which, again, has largely languished untouched since the fork and is mysterious as to where it fits in. Undoubtably there are other obscure corners of usr/src which contain more examples of derelict and customized externally-sourced code. These sub-systems are problems because they’re not well understood, any engineering documentation behind them is lost, and the closest we can get to any in-depth knowledge on them are the recollections of people who might not have had actual first-hand experience developing them in the first place.

I once provided a patch to get illumos-gate’s Sendmail updated to the latest version of 8.14. It was met with a bit of doubt – and the sense I got was that it was regarded as just too spooky to touch. NO, I wasn’t going to sit there and test this new sendmail against weird Sun Directory schemas to see if it behaved the same as the previous version. NO, I wasn’t going to channel John Beck. I made sure it compiled cleanly and it passed the basic out-of-the-box local and remote delivery tests. But no, the topic got the bike shed treatment. The topic of “Why don’t we just remove it?” subsequently came up, which was also met with “but we need a MTA/MDA because parts of illumos-gate depends on mail” which then devolved into another “what should -gate provide vs. demand externally” conversation which, predictably, went nowhere. Seriously? So we’d rather keep a old version of Sendmail which has at least one known CVE lodged against it in our source tree because a newer version might not be tested enough in unarticulated ways and we would be responsible for breaking someone’s ancient Sun Directory-based delivery map?

At any rate, it’s hand-wringy, petrified-in-fear attitudes such as this which completely works against us in several ways. We lose out because we willfully allow bit rot and thusly suffer its effects (security issues, and it just gets more spooky over time as memories fade and people move on), and we give this massive impression to potential interests external to our community that we are unwilling to let things be fixed or be improved. I wouldn’t blame anyone who felt estranged before they even got heavily involved because these are terrible things to be impressed by.

A dated participation system. I saved this one for last because it’s the topic of conversation on the developer@ list which spurred this post. Personally, I have a love-hate relationship with webrevs. I hate making them, and I hate maintaining them. I do like reading them, though, but that’s about the extent of it. I’ve tried Review Board and it’s alright, but sometimes I find myself fighting an invisible force to actually use it. Maybe the server it’s running on is a Raspberry Pi, but it’s just always tortuously slow for me.

A recent maneuver by a community member has ignited this topic, centered around the question of “Why can’t we use GitHub?” In retrospect, I think it’s a good question to bring up although I disagree with the manner in which it was. My only misgiving about branch-and-merge contribution styles in Git (and is the default for GitHub) is that the commit logs become absolutely littered with annoying and low-value merge commit messages. GitHub now allows one to merge without those such as using a rebase-to-branch option on a pull request, but that isn’t the default. Make a mistake when merging a PR and the merge log is indelibly etched into the memory of the project. Yuck. Perhaps someone with more GH-fu can explain this better, or change the default behavior on pull request acceptance. Regarding code review systems, FreeBSD seems to have found a comfortable spot with Phabricator. Could this be a compromise?

In the end, I really wish to see illumos continue to succeed and – most importantly – grow. I’m really worried sometimes that the public’s perception of the project is that it’s a ex-SUNW playground and you need an “in” in several ways to be a contributor to it. While I don’t at all believe this to be the case, I can certainly see where we (“illumos”) can easily come close to coloring impressions that way. I’ve observed several community members welcome and thank new submitters for their work and always engage them constructively – I was on such recipient on my first big contribution after all – but we still need to work on the curb appeal quite a bit, I think, and modernize some things and re-evaluate some overly-rigid stances on others.