Log4J Vulnerability

Another Post about the Log4J vulnerability.... I know. I can hear the sighs. Everything that has been said about it probably already has been said right?

Well, I'm going to throw my thoughts into the mix from a slightly different angle... That this incident was inevitable, and its going to repeat itself over and over again...

For those that know me, I maintained a Open Source project called OpenZwave for over 10 years. It was a library (much like Log4J is) that was adopted by a large number of Open Source and Proprietary Home Automation Solutions. At its peak, it was the only open source solution that was able to interface with Z-Wave. Z-Wave is pretty big in the home automation market as a "standard" with hundreds of vendors and thousands of products to do everything from turn your lights on and off, to detect water leaks or lock/unlock your front door.

I started working on this project out of a "personal itch" mainly and found it rewarding. As the project grew in popularity though, it started to become a burden. I was essentially the only one developing the application although I did have a few other people that helped with adding support for new devices.

I did have some pretty generous support from Home Automation vendors as well, but not with the code - They saw that a lot of their users were interfacing with their products via my Library, so they would be sending me their devices hoping I would add support for them in OpenZwave. But only a tiny handful actually sent enhancements, bug fixes or updates.

By 2019, I was burnt out. By my estimates, OpenZwave was deployed in over 100,000 homes, powering peoples home automation. The support burden on me was huge (wives tend to get irritated when the lights wont turn on!) and I had a full time job. OpenZwave was no longer fun, nor rewarding. A small, but vocal minority of users were constantly not happy with the pace of development or how long it took me to fix bugs.

And so - I walked away from the project. I did try to find some other maintainers for the project, but no one really stepped up. So I basically stopped working on OpenZwave despite it being extensively deployed, and used by many other open source projects. Its essentially "dead" now.

So what does this story have to do with Log4J?

I think this xkcd commit perfectly sums up the situation. Despite log4J being a "Apache" project - its still run by volunteers doing it in their own time, without any financial support. No one is paid to support and develop Log4J, despite it being used by >7000 Open Source Applications and probably just as many internal applications. On the flip side, many enterprise and/or proprietary applications utilize Log4J and are sold for for hundreds to 10,000's of dollars. The vendors get the benefit, but have some sort of expectations that someone else will support, audit and maintain those dependencies, so they don't.

And this is where Open Source, Maintainer ship and Security are going to continue to fail. Many companies have (or will now) start thinking more seriously about "Supply Chain" risks and the costs involved with adopting Open Source in their infrastructure and applications.

The problem is though, that risk mitigation will probably do nothing to fix the root cause - that is, ongoing maintenance and support of Open Source Projects they utilize in their products.

This is not a new development either. In 2014 another widespread vulnerability was discovered in OpenSSL which powers encryption for many open source and proprietary software projects. After the dust settled there, some industry players formed the Open Source Security Foundation (OSSF) to provide funding for "critical open source applications" and in 2020, Google did some analytics to identify "Top Open Source Projects" that qualify for help from the OSSF. Log4J was "ranked" by criticality in position 28304 (and OpenZwave was ranked at 6128!!)

Despite the good intentions from Google and the industry players that setup and fund the OSSF, this log4j vulnerability goes to show we actually have not found a solution to fix these ongoing supply chain risks every company now faces if they know it or not!

What we really need to start considering as both a Open Source community as well as a industry- How do we truly contribute back and support all the volunteers that work on Open Source - To avoid the burnout (like I suffered) or real help maintaining and supporting their projects.

And Money is not the solution - The Apache Foundation, which Log4J is under, reported in their 2021 financial report $3M in revenue and $4M in the bank!

I don't really have a idea what the solution is - This is a problem that is much bigger than supply chain, and will have implications into every corner of our lives. As a community and industry, we need to really start thinking about how to make Open Source sustainable from a maintenance perspective. Maintainer burnout is a often discussed topic in Open Source, and a lot of great ideas are offered up, but if a project as large and ubiquitous as Log4J can suffer such a vulnerability it goes to show that, from a Security perspective at the very least, we need to start thinking about other ways to identify and support these projects, otherwise all the Developers and System Admins are going to continue to have sleepless nights!

Don't get me wrong, I'm still a huge proponent of Open Source and its benefits from both a individual and developer (proprietary or not) are not to be dismissed, but unless there is a fundamental change to how we "support" open source, vulnerabilities like log4j and Heartbleed will continue to haunt us well into the future.