Ceki Gülcü recently wrote an interesting post on the Apache community model and its vulnerability in cases where consensus can not be reached with reasonable effort. Also the discussion in the comments is interesting.
Ceki's done some amazing work especially on Java logging libraries, and his design vision shines through the code he's written. He's clearly at the high edge of the talent curve even among a community of highly qualified open source developers, which is why I'm not surprised that he dislikes the conservative nature of the consensus-based development model used at Apache. And the log4j history certainly is a sorry example of conservative forces more or less killing active development. In hindsight Ceki's decision to start the slf4j and logback projects may have been the best way out of the deadlock.
Software development is a complex task where best results are achieved when a clear sense of architecture and design is combined with hard work and attention to details. A consensus-based development model is great for the latter parts, but can easily suffer from the design-by-committee syndrome when dealing with architectural changes or other design issues. From this perspective it's no surprise that the Apache Software Foundation is considered a great place for maintaining stable projects. Even the Apache Incubator is geared towards established codebases.
Even fairly simple refactorings like the one I'm currently proposing for Apache Jackrabbit can require quite a bit of time-consuming consensus-building, which can easily frustrate people who are proposing such changes. In Jackrabbit I'm surrounded by highly talented people so I treat the consensus-building time as a chance to learn more and to challenge my own assumptions, but I can easily envision cases where this would just seem like extra effort and delay.
More extensive design work is almost always best performed mainly by a single person based on reviews and comments by other community members. Most successful open and closed source projects can trace their core architectures back to the work of a single person or a small tightly-knit team of like-minded developers. This is why many projects recognize such a "benevolent dictator" as the person with the final word on matters of project architecture.
The Apache practices for resolving vetos and other conflicts work well when dealing with localized changes where it's possible to objectively review two or more competing solutions to a problem, but in my experience they don't scale that well to larger design issues. The best documented practice for such cases that I've seen is the "Rules for revolutionaries" post, but it doesn't cover the case where there are multiple competing visions for the future. Any ideas on how such situations should best be handled in Apache communities?
If one doesn't approach software development with your attitude that "I’m surrounded by highly talented people so I treat the consensus-building time as a chance to learn more and to challenge my own assumptions", one is probably not a good fit for the Apache Software Foundation. Or working in any other team environment.
ReplyDeleteIt isn't about technology; it is about community. Technology can be supplanted and surpassed. Community lasts. Diversity is a double-edged sword: an individual may be more nimble, being literally single-minded, but a community, once organized, has more power and more stamina. Community can maintain *and* community can leapfrog.
On "multiple competing visions":
ReplyDeleteThe different visions are presented and the project takes a vote on "majority approval" rules.
A community can have its own internal forks, such as the multiple times when Tomcat had its own internal fractures. Eventually the Tomcat community converged, having explored the different visions in actual releases.
ReplyDeleteRe: competing visions / internal forks
ReplyDeleteHandling such cases in a consensus-driven community requires quite a bit of time and effort, much more than in a BDFL-driven project. It's not clear which approach produce better technical results (both models have their strengths and weaknesses). What troubles me is that, as seen in Ceki's case, the consensus model can easily feel like an extra burden to people with the strongest vision and who may have the most to give to a project. How can a community best support the efforts of such people?
The BDFL-model grants too much power to a single individual, the current Apache model distributes power indiscriminately, a committocracy is a another fairer model for power sharing.
ReplyDelete@ceki as one that has been trying really hard for years to emerge hard data out of latent traces of behavior, I can tell you that while your "committocracy' sounds appealing, it is a quantitive measure and not a qualitative one. I do resonate with your concerns that "meritocracy" while great to maintaining the status quo is flawed when it has to deal with innovation (I've been pretty vocal about this even inside the foundation, which is what lead me to the proposal of labs.apache.org as an escape valve). At the same time, whatever metric you invent to automatically evaluate people, at some point in time, you'll find yourself on the other side of this metric and another set of complaining will start (does a commit that changes tabs into spaces count as a commit? a one liner? applying somebody else's patch? a total refactor? how about people committing to the wiki? or writing emails to help reach consensus? or writing on their blogs about the project? or holding meetups to evangelize the notion? or writing privately to distressed leaders to make them feel supported?)
ReplyDelete"committocracy" is appealing but there is no reliable and fair way to measure "being committed" to the project and certainly there is no reliable way to measure merit.
If a single person managed to get enough social momentum to strangle the evolution of a project you once controlled (yes, it happened to you, it happened to me, it happened to many), pointing the finger at the rules is one way to cope. Another is to admit that, maybe, that role was not for you and your energy is better focused somewhere else. It might sound like a shame, but, really, Darwin never claimed evolution to be pretty to watch or fun to experience.
@stefano
ReplyDeleteYes, committocracy is based on a quantitative measure. And, yes, whatever metric you invent to automatically evaluate people, at some point in time, you’ll find yourself on the other side of this metric, which is part of the appeal of a committocracy: it makes for a fair system. In this sense, the Apache way is also fair as everybody has to play by the same rules. However, the Apache-way is unjust because it largely ignores the investment made by contributors.
Counting daily commits might be an imprecise measure of investment in a project. It might be even less precise in measuring quality but at least it attempts to measure the extent of contributions over a period of time.
Note that a committocracy might also drastically reduces the risk of a project going haywire after admitting a new member. As a corollary, a project can safely reduce the wait-and-see period preceding the admission of new committers.
My or your feelings aside, you seem to be assuming that the chaotic nature of the current decision mechanism somehow results in healthier projects. Lousy rules may cause otherwise viable projects to die or otherwise helpful contributors to leave, the organization ending up only with the projects and the contributors who can cope with rules. There is nothing inherently desirable about that.
@ceki
ReplyDeleteI think our disagreement gravitates around the definition of "healthy". The 'chaotic nature of the current decision mechanism' results in projects that are capable of consistently survive their creators.... so much, in fact, that they become harder to steer and influence even for the original creators.
I have to say, a project where your vote counts +1343 and mine counts +1 won't make me stick around: it feels like I'm working for you for free and with no way to gain any power or respect. Delegation of ownership is a big part of what makes the open source model so attractive and sticky.
On the other hand, if a newbie's -1 is enough to strangle your innovation and make you leave the project, the problem might not really be how much weight your vote carries within the rules, but how brittle, dispersed or balkanized is your network of committers inherently was. I really don't think that mechanically inflating vote's weights is a good substitute for the effort required to weave a supporting social network of peers.
Nobody has ever claimed the Apache model to be perfect (and I have voiced my fair share of criticism about it inside and outside the foundation), but it currently unsurpassed as a way to create stable and sustainable volunteer-based development processes. It has achieved an unparalleled consistency in long-term stability of software projects and in reducing risks and costs associated with software inter-dependencies. It used to be that nobody got fired for buying IBM, now nobody gets fired for depending on an Apache project. No matter what side of the fence you find yourself in, that's a huge achievement that felt almost laughable just a decade ago when we first talked, at your house, about you donating log4j to apache.
Unfortunately, all that doesn't come without a price: the Apache governance model makes the system feel inertial, big, convoluted, bureaucratic and dehumanizing.. this is no different than the feeling that governments of modern countries get bigger and bigger and are harder to steer for their citizens (which spawns endless "smaller government" appeals from various parties around the world).
Don't get me wrong: I think these debates are healthy and useful because it is indeed true that the Apache model sacrifices some things for others and when you find yourself in need for those things left on the cutting floor, it hurts. I applaud you for coming out and saying so explicitly.
But it is unfair to consider that because those things are important to you (or me), they must be important (or a priority) to everybody. The Apache model is not about creating the best, it's about creating the most stable. I used to believe that the most stable would inherently become the best under the Apache model but I agree with you that there is pretty strong evidence it is not necessarily the case.
Unfortunately, I don't find your commitocracy governance model appealing, nor stable: it looks to me just like a more politically correct way of ratifying dictatorship; also metricizing leadership completely destroys its appeal and romanticism.
The important question for me is not which model fosters the best innovation but if there is a governance model out there that can yield both long-term stability and consistent convergence to state-of-the-art. I really hope so, but I don't know of any: it's not Apache's meritocracy for sure, but it's not your commitocracy either.
In a committocracy, it is the *sum* of the votes that determines the outcome of a ballot and not the single vote carrying the most weight. Therefore, it is possible for two or more committers to overcome the committer holding the most committer-points. In no way is a committocracy a BDFL-model under disguise. I think a committocracy strikes a nice balance between the BDFL and the Apache model.
ReplyDeleteI know votes gets summed up, the problem is that you're assuming a distribution of commits that is gaussian (meaning, it's centered around a bell curve). In real world, that doesn't happen: it's almost always a power-law as commits exhibit preferential attachment (people that have committed a lot tend to know a lot of the code therefore tend to commit even more).
ReplyDeleteLet's use a real-world example dear to your heart to explain this. According to http://github.com/ceki/slf4j/graphs/impact the list of committers on the SLF4J mainline (as of today and 5 years in the making):
- ceki 729 commits
- ravn@runjva.com 55 commits
- jncharpin@qos.ch 23 commits
- jconlon@verticon.com 21 commits
- spennec 19 commits
- rgoers@apache.org 13 commits
- raphael@qos.ch 4 commits
The sum of the all the other committers is 135. So even if every single individual that ever committed anything to that project is counted in this governance model, you'll be the only one deciding what to do, and for years to come (the same happens in your other recent project logback which is also 4 years old).
I find this model worse than the Apache model for long-term stability (the leader becomes the central point of failure) and worse than the BDFL model for innovation and community engineering (the leader hides his/her absolute power behind a set of mechanical rules causing even potential benevolence to be more hidden).
Again, I applaud you for trying to innovate on governance model: not many people are brave enough to do it and tend to follow the pack no matter what and focus only on coding. But there is also a reason why so few stable governance models for open software development exist: estimating their game-theoretical stability on paper is incredibly hard given the high non-linearity of collective social behavior.
You should not take slf4j as a model of a committocracy because it is not and was never managed as a committocracy. The same goes for logback, although if lilith and logback were combined in a single project, Joern Huhorn and Sebastien Pennec, with approximately 600 commits each, could have a short majority. Again none of this is quite relevant because neither project is run as a committocracy.
ReplyDeleteThe rules in a committocracy are transparent. As such, a BDFL cannot hide his absolute power if indeed he had such powers. Granted, if committer Alice has N commit-point points, and the accumulated number of points held by all the other committers is less than N, Alice's vote will determine the outcome of every vote. But when she wins a vote, the votes have to be counted and Alice's majority-stake will be revealed. Also note that other committers can gain committer-points by working on the project. It is quite possible for the other committers to accumulate a number of commit-points greater than that of Alice. Since Alice can only accumulate at most one commit-point per day, 'n' committers can overtake Alice after N/(n-1) days for any N and n greater or equal to 2.
As for stability, if for one reason or another Alice leaves the project, the other committers can continue to work on the project.
My hunch is that as soon as someone puts together and publishes the tooling necessary to run a committoctacy, the approach will gain wide usage.