I think it's not so much the lack of good research being done, but the lack of real world application of that research. And furthermore the real applicable research wasn't really theoretical research but more lessons learned by engineers in the industry.
It's crazy to me that researchers prefer to be published over having world renowned software to their name. You could figure out the fanciest database research, but there was a period where every fancy new NoSQL database was based on BerkeleyDB despite all that, bolstering the reputation of the University of Berkeley.
Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting.
Imagine if university groups would band together to produce really state of the art production quality reference software. They could use them as base for their research, bolster their reputation by having them be great, and even draw funding from big companies looking to hire the PhD's produced by the efforts.
Instead we have got all these half finished demonstration software that no one industry would touch with a ten foot pole.
It took just a few people to develop the first versions of rust, a research based programming language that in my opinion has been the biggest shock to the industry in decades (if you disregard machine learning).
>It's crazy to me that researchers prefer to be published over having world renowned software to their name. You could figure out the fanciest database research, but there was a period where every fancy new NoSQL database was based on BerkeleyDB despite all that, bolstering the reputation of the University of Berkeley.
Unless the BerkeleyDB was written based on novel research, it didn't move the state of the art forward, it was just a good, used, embeded database.
In which case, your statement can be rewritten as:
"It's crazy to me that researchers prefer to be known for their research over being known as developers of popular programs".
Doing research is the reason they went to an academic career, if they wanted to be developers, they'd just have gone and be developers.
So, why not strike two birds with one stone, and make high quality widely used software that's ALSO state of the art research based?
Well, because creating "high quality widely used software" is a different process, and needs different resources, dedication, and even skills, than doing research.
To showcase some novel research in practice is much much faster, easier, and cheaper to create "half finished demonstration software", that to create a mature, final, implementation that people can use.
So, either they slow their research considerably, and spent most of the time doing software development work - which besides slowing down the pace of research will also count against them in their academic field, they opt to focus on research and just do simple implementation that are not production ready and aren't meant to be.
Seen in this light, it's not a "crazy" choice, it's an essential tradeoff.
There are some points in the middle, but it's rare. I work on one of these [1]. We've been building the system for over ten years, and are starting to see some truly killer apps being built on top of it [2, 3].
While it has some great benefits once you arrive, the upfront costs are enormous. You need to find a funding source (or sources) that will pay for this product while you're building it. Also, in order for the research payoff to be worth it, you need both the product itself, and subsequent innovations it enables, to be research-worthy. Not all areas of research can support this. On top of it all, even when you do this, you'll still spend years of effort in activities that are essentially not research. You're basically responsible for all of your own customer support, sales, marketing, etc.---like a startup, but without the financial upside if you succeed. Yes there is recognition and so on, but the payoffs aren't as dramatic. Most people aren't ready to commit to this path.
Keep in mind that you can't build this in 5 years either. So a single generation of PhD students can't get it done. The only reason we were successful is because the key staff on the project stuck around for 5+ years after our PhDs because we all believed in doing the work.
Given all that, I don't hold it against people at all who just want to build prototypes and then move on to the next thing. It's way less risky and higher reward relative to the costs.
It seems like it's just that the goals are different. Computer science researchers could also make the opposite case, "The underwhelming impact of software development on producing state-of-the-art computer science discoveries."
> It's crazy to me that researchers prefer to be published over having world renowned software to their name.
If you went into research, you have occupational incentives to publish papers, but not to publish software.
As someone who went into applied software, there's a lot of published research that's so disconnected from reality, that it couldn't possibly be useful for anything but publication stats for the named researchers. On the other hand, every once in a while, there's a gem that's useful.
My personal favorite is network research where everything was done with models of TCP that aren't at all similar to actual TCP stacks, production or research.
If you went into research you did so because you want to solve new problems. With no disrespect to software engineering, most (production) software engineering involves solving old problems over and over again on a daily basis, with genuinely new problems tossed in only from time to time. This is even more the case when you’re maintaining a codebase that’s widely used.
There is nothing wrong with solving old problems: that describes most useful human work. It’s just that finding and solving new problems is a very specialized kind of work, and it’s very hard to do that specialist work if you aren’t spending most of your time on it. You need to keep up with the literature and fight to stay at the forefront of your field and you can’t do that as a 20% time position.
A good analogy here is like the interplay between a general contractor and a licensed electrician. The contractor could do their own wiring in between framing and drywall (and take the necessary tests and certifications to learn the code), but it usually makes more sense to have specialized contractors come in for that work. Unfortunately the problem described by this article is that the software developers (contractors) aren’t using the output of the researchers. Insofar as this is true it should be addressed. However the answer probably is not to have the specialized researchers do more general software development, which will just make everyone less efficient.
> Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting.
Software collaborations tend to end poorly in Academia from my experience. This is due to most of the day to day boring software work being viewed as "less than" the sexy publishable research.
What this means is that academic software projects become spaghetti code messes that no one wants to touch at warp speed.
Also, how do you reproduce even the demo software from a paper that’s been out for a year (an eternity in software maintenance time?). Git repo might be gone, some dependency broken, or maybe they used some system that isn’t around anymore.
> Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting.
What new language implementations are using Boehm?
To be fair I hadn't verified it was still true when I wrote the message, and I'll admit all I've seen recently is reference counted languages. It was definitely true when I was exploring C garbage collector options for my projects but that's almost ten years ago. I just checked the wiki and it seems Crystal uses Boehm.
You use Java's GC right? Or is there something custom in there now?
Quite a few "new" languages target the JVM and thus use JVM GCs, which are all state of the art. E.g. Kotlin, Clojure, Scala. And of course then there are the new implementations of other languages, like what Chris works on (Ruby), which also run on the JVM.
Most new languages are either high level languages and running on the JVM, or they're attempts to compete with C/C++ like Rust, Swift, Zig etc and thus don't use GC at all. I don't think many people use Boehm anymore.
It's only the obvious solution in the sense that it's not really a solution but it comes close enough that memory often doesn't leak fast enough for it to matter for short running programs.
A drop in garbage collector (like Boehm) makes your toy language instantly production ready. Basically code generation and garbage collection are the two things that make language implementation tedious. That's why so many languages are on top of Java or .Net. Implementing a language on those platforms is almost trivial because it solves both code generation and garbage collection.
It would be more obvious to have a basic toolkit to let everyone build their own GC with LLVM.
From my experience with building my own GC. There are only two challenges. Stack unwinding to obtain root references and then the GC strategy (generational GC, mark and sweep, etc).
For most people, reimplementing the latter is a waste of their time.
People should code in Haskell more, and we would have massive amounts of reusable code. All the problems stem from premature optimization, and rush to market.
>and we would have massive amounts of reusable code
Haskell even has problems coming up with a single string / text interface. It's not some magic bullet for code reuse, any more "everybody would write Java and we would have had massive amounts of reusable jar libs".
Well, in fairness, the industry has got quite far with "everyone writes JARs". There are a heck of a lot of them out there. And now with Truffle you can mix JARs, Python packages, Ruby packages, JS packages ..
Academia trying to "Make Haskell Happen" is a clear indication of how disconnected it is from the practice of software engineering. Haskell is not going to happen.
My current Haskell job and previous Haskell jobs disagree. Also I'm not academia since I don't even have a bachelors degree and am a professional Haskell developer.
I enjoy watching production blow up while I munch on salty popcorn. Then I come along with my master plan and solutions that management just begs to throw resources into. Then I move onto the next blow up.
It’s much more lucrative to be a firewoman when there’s idiots pouring gasoline and throwing matches everywhere.
IMO most of the practical "research" I've seen comes in the form of best practices from Big Co. who a) hire very intelligent people and b) have extreme financial incentives to write the best code & have the best processes possible.
In a world where the dependent variable isn't something that can easily be measured in a lab, the next best we can do is either 1\ Ask experts or 2\ experiment in an environment where it's financially advisable to experiment & measure (AKA an environment where running an experiment with 100 engineers is <1% of your engineering headcount).
I.e. I don't work for Google, but I do share things like [0, 1] with new engineers who join my team.
This is a point of frustration to me. It seems like there's a significant disconnect between research in academia and research in the real world.
Beyond that, a lot of software engineering research doesn't seem to be easily reachable or discovered unless you're in academia.
Real world software engineering has to ship a real working product, while academic software engineering only has to produce a paper. Maybe that's an unfair representation though.
I told a coworker once that I subscribed to the ACM (which I don’t anymore), and he looked at me like I’d just parted the Red Sea. He was amazed I could understand them. I told him I could only understand about half (closer to 3/4) which mollified him a bit.
There are any number of reasons I have stayed engineering track instead of going into management but the biggest is that talking about things and doing them are vastly different, and we’ve structured management and research around talking. Some days I worry I may know more about project management than I know about cloud or React, but it would kill me to be telling people how to do the job that I’m not also doing, and if I don’t like how they did it, have only words available to get it fixed.
In one of my hobbies, time and the pandemic have slowed me down quite a bit and I’m trying to manage more and do less, but unlike work there are plenty of opportunities to lead by doing. Doing 5-10% of the work and then guiding people to do the rest is socially acceptable.
In software we sometimes call that a lead developer, but that definition changes from situation to situation. But the more you do the less influence you tend to have on what you’re doing and why. Is that human nature or leftover from feudalism?
I feel that in the past there were more products shipped by Academics. Unix BSD from Berkley, Minix from Tanembaum, Kermit from Columbia, Pascal from USCD; Pascal, Modula and Oberon from N. Wirth, etc.
Building these products would be a great teaching tool and contribution to society. Also there could be a lot of research done on how to build complex software where the product is the software itself and it can be applied to other sciences that badly need software. I for one would love to get a graduate degree while building some product where I get to try different software design approaches.
> It seems like there's a significant disconnect between research in academia and research in the real world.
It's funny, back in my undergrad days I would walk past the Google offices that opened near my campus. I'd look in and see a giant wall-sized candy machine, primary colors everywhere, bean bags, pool tables, an air hockey table. I honestly thought it was a daycare of some kind until I asked someone why there's never any kids in there. This is what they call the "real world" apparently, as a contrast to what goes on in academia.
I can definitely say that in all my years in academia, I've never been treated to a candy wall.
I think it’s important to understand what the author means by “software engineering research”. If I understand it correctly, he means research about the discipline of software engineering, not the output of engineering. So research on analyzing agile and waterfall approaches would be “software engineering research”, a programming languages paper would not be. A paper on whether static typing reduces defects is a grey area to me—-I think it’s SE research, but I’m not sure.
It’s clear that there’s been lots of progress in technology: in networking, in databases, in compilers, etc. It’s just that none of that was substantially influenced by research on the process of software engineering.
The problem is obtaining solid data and not falling on cargo cult. If there’s a paper showing that static typing is safer, but is assuming A, B and C, it can be very easy to end up with a lot of people saying that only static typing should be used in every imaginable situation.
Applying the results of research on these kind of practices can be incredibly difficult. Engineering processes are not a hard science…
I don't think the problem is located in academia or research, it's in the market and user obsession of the software industry. For a very long time we didn't produce software primarily for consumers, we produced software for engineers and researchers. Bell Labs and the corporate research lab as an institution with its significant freedoms is an example of this. A lot of innovation was the result, including practical innovation.
Hardware still works this way today basically. The performance optimizations, practices and goals are engineering centric, not consumer centric. Consumer benefits happen downstream as a side effect. The hardware sector is still run by scientists and engineers, one only needs to look at the people running the companies.
There's no shortage of great software research that could be put into practice, plenty of languages with great features are underused. It's the result of the mentality in software that puts costs, users and marketshare over an engineering culture.
I agree. I think a central reason that software, and the process for creating it, have not improved is that no standards for software quality have been demanded of the industry. The creation of no electrical or mechanical device on which people comparably rely is as underregulated.
Few software developers are required to acquire professional certification, or perform in compliance to standard practices intended to assure due diligence in design, implementation, testing or documentation. Very few software systems must undergo compliance certification. It's unsurprising then that software engineering continues to founder and fail to advance as a discipline.
I am glad we did not install standards to freeze the way software was developed in e.g. the 1990s, and I expect that in 25 years I’ll be equally pleased that we did not set standards today.
It hasn't failed to advance as a discipline. I'm nowhere near retirement and even so I've seen massive advances in my lifetime:
1. The widespread adoption of unit testing and associated techniques like continuous integration. A massive boost to software engineering reliability and productivity.
2. Introduction of crash reporters and associated analytics, this enabled huge improvements in software stability. On the server side, good logging and monitoring systems provide an equivalent.
3. Sandboxing going from unheard of to relatively widely used.
4. Design, development and then open sourcing of extremely robust and well engineered runtimes like the JVM, .NET etc. This has huge benefits throughout the entire industry.
5. Open source and the huge code reuse it made possible.
etc etc. All these things have been drastically more impactful than bureaucratic central planning a la certifications.
I often read SWE research that is just kind of common sense to somebody in industry. Of course, it's worth trying to take an empirical approach to quantify merits of certain techniques, but the vast majority of the research that's done does not seem that novel.
I'm fairly certain that you could probably write and publish a decent quality paper yourself. But why would you, when industry is so much more lucrative? There are many competent people in CS Academia, but their motivations tend to be quite different from the average engineer, and they don't have access to the same resources or expertise that tech giants or startups do (outside of internships and fellowships that are commonplace for visiting researchers who spend a season or two working at a lab)
As a software engineer in the industry, what problems am I hoping academic research could investigate.
Compilers that work like devs do. Almost every time I compile, the code is almost exactly the same, but the compiler forgets everything and throws away details from the last compile. Probably all the optimizer's choices are going to be the same this time as last time, so maybe we could cache these?
Better code merging. When my code diverges from the main branch, I typically have to do a bunch of merging work. Would a merging algorithm that understands code semantics be able to do a better job of understanding what I mean during the merge?
Language safety and ergonomics is huge. I very much appreciate Rust's progress, and it's a shining example of academia brought to industry, but can we do more? Jai's idea of making constraints codeable and incremental is pretty cool, and I bet there's a lot of room to improve on it.
Measures of code complexity and modularity. There's a bunch of silly measurements of the "goodness" of code, but maybe it's time for another! Maybe we could describe how "replaceable" components are automatically, and gives a solid idea of the level of real tech debt in the codebase.
How about some project management focused studies? Have a programmer give an estimate for a task, try to force it to be a deadline, and watch as quality of the results drops. Try applying something like scaled agile framework to a real problem and watch as the meeting load snuffs out meaningful development progress.
Or some systems work. Apache airflow is a pretty cool job running agent, but I have yet to see a good error handling workflow for distributed job agents. Can you come up with the "Result" type of job scheduling that's nicely composable?
Good news for you: I'm well on the way to solving the problem of better code merging. Specifically, the algorithms I am developing appear to be able to do a correct merge on both [1] and [2]. They also appear capable of merging binary data.
The tradeoff is that people need to write some code to tell the VCS about the format of each binary file type or semantics of each programming language.
The biggest problem is that, like Rust, a new VCS has to be well-executed to make its innovation stick. We'll see if I succeed.
Cached builds are a thing but actual incremental analysis is a very hard problem. Seemingly unrelated changes a pretty common sources of interesting analyses bailing so you get more spooky action at a distance than you might expect. This gets worse with whole program optimization.
Perhaps a good dependency process would be an interesting research. Knowing what parts change when changing a small bit it’s actually very difficult to know, and it seems like a common problem that could improve things in many areas (for example, compilation, but also knowing what tests to run if some piece of the code changes, for example)
There are hundreds of papers on incremental analysis. You can find them in PL conferences like POPL, PLDI, and OOPSLA. You can also find this kind of topic analyzed from a less theoretical perspective at conferences like FSE and ICSE. It is a hard problem.
I did a PhD in the field. And only then became an actual software engineer. As it turns out, doing software engineering research does not involve a whole lot of engineering software. So, I had a lot to learn.
So, this criticism resonates a bit with me. The problem with software engineering as a scientific discipline is that many engineers probably would not consider it a proper engineering discipline to begin with. Also, many computer scientists look down on it as not really related to that either because it is a bit of a soft science.
So, there's a problem with researchers in this field not really having the right background to do good research:
- they are typically not experienced software engineers. Quite many, like me, start working on a PhD without actually having worked as a software engineer.
- they typically have studied computer science which teaches you a lot of math, algorithms, formal methods, and other typical beta skills but is perhaps a bit light on the type of stuff a sociologist or anthropologist would learn. That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software. Studying that behavior to verify theories you might have about how software engineering works or ought to work requires an empirical approach. That's not something a lot of computer scientists learn how to do.
So there's a lot of software engineering research that gets published that you could accuse of being a combination of naive, wrong/misguided, and poorly executed in terms of its methodology. Mathematicians or computer scientists trying to be empirical is not great typically. Sometimes you get some decent papers but there is a lot of pseudo scientific nonsense that gets published. Doing empirical research properly is hard and many computer scientists have no clue how to do it properly nor any appreciation for doing so.
Just looking at ICSE's program last year you can see from the titles a weird mix of empirical studies of various topics and other papers clearly written by more computer science leaning types working on algorithms, machine learning or tools. Looks like there were some interesting papers in there but also a lot eye brow raising titles like "How Java Programmers Test Exceptional Behavior". I actually skimmed through the pdf of that one and it fits the pattern beautifully. It's an actual empirical study that tries to answer the question in the title. That's it. There's no new theories being formulated. No remarkable findings that were reported. It's just a very dry survey that yielded exactly what you'd expect. The answers to the research question are almost beside the point and should be completely unsurprising to any Java programmer. Great example of a well executed emprical research paper that answers a question any Java developer in need of actually testing some code that throws exceptions would answer with a simple Google query in a few minutes.
"many engineers probably would not consider it a proper engineering discipline to begin with."
Calling software engineering "engineering" is perhaps the most blatant uses of describing a thing by it's exact opposite. Like calling hate love, or war piece. (Which sometimes is done, but mainly for propaganda purposes - for the intent of obfuscation.)
Software engineering is much closer to a craft than anything capable of teaching or predicting anything successfully. The best analogue that comes to my mind are the craftsmen who built cathedrals in medieval Europe. Yes, they built things. But they did not understand with scientific rigor, what and why are they doing it. They did not have proper theory, so they could have built something completely different.
You can learn the activity known as software engineering. For most individuals, the best way is from an informal apprenticeship by working with talented colleagues. Again, this is closer to art or a craft than engineering.
You can learn the basics in structured setting. But based on the stories of long winded recruiting interviews, nobody actually trusts any of that, or at least they don't trust the accreditation mechanism. Which implies you can pass a degree, but only accidentally learn the skills the industry expects a great candidate to have.
Something is institutionally broken here. Imagine if medicine worked this way. Are doctors whiteboarded over complex surgeries before being hired?
Well, it seems to be in line with the general absurdity of naming conventions in anglophone computer science, since SWE is one of the few subfields of CS that actually uses the scientific method. However, I'm not sure where you are going with your tangent on engineering; in my mind, engineering mostly differentiates itself from the other types of academic research in the kind of question it asks: "How do we [build this]?" in contrast to "How do we [prove this]?" or "How to we [establish evidence in one direction or the other]?"
I agree with the lack of rigor in software development, but I don't think that it is much better in other traditional engineering disciplines. I know from my own experience that you can go quite far in electrical and mechanical design without any rigorous engineering. Usually, all the hard math-y engineering things are done by software and its users don't understand how the software works. You have to know a bit more about physics and such to understand the data sheets, but that's not much different from knowing the complexity of elementary algorithms and data structures.
- they typically have studied computer science which teaches you a lot of math, algorithms, formal methods, and other typical beta skills but is perhaps a bit light on the type of stuff a sociologist or anthropologist would learn. That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software. Studying that behavior to verify theories you might have about how software engineering works or ought to work requires an empirical approach. That's not something a lot of computer scientists learn how to do.
One thing that particularly scratches me lately is the popularity of Actor-Network-Theory in some of the humanities: It is a theory that might benefit quite a lot from CS/Maths input, and I think it could maybe be applied quite well in SWE theory as well.
I have no idea if it is actually the case, but one thing I'm quite sure of at this point is that humanities need more formal systems, and SWE research needs to look more towards the humanities and social sciences in terms of frameworks to explain human and machine interaction.
> That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software.
I'd hypothesize that most applicable improvements to a software engineering company are about - collaboration & best practices - than anything highly technical.
Sommerville was my graduate software engineering textbook in the late
1980s. Along with Jackson, Yourdon and a few others it was the bedrock
of my thinking about how to manage software projects over the past 30
years. It's not as if I got stuck in that mind-set, having been a keen,
early adopter (and early critic) of many faddish methodologies over the
years - but I've been astounded by the almost total lack of deep
traction these philosophies have had in cultivating a mature, long
term, responsible engineering attitude in our business.
I am really pleased to hear a heavyweight like Ian Sommerville
pitching in on what I've been seeing as a cultural crisis in software
spanning almost a couple of decades now. It's disappointing to hear
the usual dismissal of research and lionising of "industry" as if a
cowboy "move fast and break things" mind-set constituted a single
coherent commercial ideology. Can you imagine if the pharmaceutical
or aviation industry took a cavalier attitude towards "eggheads with
their cell biology theories" or "crazy academics with their fancy
aerodynamic equations".
Is software the only engineering discipline where the product gets
less efficient each year? Partly I think it's also the public's fault
for accepting such shoddy produce, and of governments for not being
stricter because of a Faustian bargain to enjoy mass surveillance and
cybernetic social control as vicarious side effects of a hands-off
approach.
It must be frustrating to have put a lifetime into carefully
considered formalisms and see so little uptake.
Anyway, what we've ended up with a bunch of non-interoperable,
insecure devices and services built by well paid but mostly unhappy
workers who have almost no grasp of the complexity, history or
societal import of what they create. Sooner or later the chickens are
coming home to roost, as cybersecurity, resilience or interoperability
issues.
Says Sommerville:
"As a community, senior members (and in this I include myself) have
failed to create an inclusive environment where new ideas from outside
of the traditional elite are supported. We do not act together when
dealing with policy makers and funding agencies and, with rare
exceptions, have not spoken out against the futile short-termism that
has hampered the development of our field."
I absolutely salute you Ian. I wrote similarly a year ago in my last
book that I feel a great sense of regret as a computer scientist who
did not speak out sooner and louder. Perhaps it's grandiose/omnipotent
to think we ever could have made a difference, but if not our
generation then I really hope a new, radical generation will emerge
who have the courage to "take back tech".
I wish there would be a publication habit for industry SWE, we could all benefit from the reports from the trenches. I know we have blogs and the occasional books but I would really enjoy an in-between.
This article would be a lot more convincing if it illustrated its points with examples. I have no doubt that there is truth here, but the objections here are fairly general and unsubstantial. For example, take the scaling objection: give some examples of useful techniques that won’t scale and then talk about what you might do differently.
Is that all he has to say? That reads like a cover sheet.
There's been considerable progress in programming languages and tools. It's now possible to write very large programs with extensive concurrency without having them crash frequently.
There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work.
> There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work.
Does this stuff come from academia? That's what's being debated.
It's crazy to me that researchers prefer to be published over having world renowned software to their name. You could figure out the fanciest database research, but there was a period where every fancy new NoSQL database was based on BerkeleyDB despite all that, bolstering the reputation of the University of Berkeley.
Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting.
Imagine if university groups would band together to produce really state of the art production quality reference software. They could use them as base for their research, bolster their reputation by having them be great, and even draw funding from big companies looking to hire the PhD's produced by the efforts.
Instead we have got all these half finished demonstration software that no one industry would touch with a ten foot pole.
It took just a few people to develop the first versions of rust, a research based programming language that in my opinion has been the biggest shock to the industry in decades (if you disregard machine learning).