28 May 2024
For many software developers, career success comes when they are placed on the "R&D" (research and development) team. It seems like a golden ticket kind of role: Spend time exploring new technology (which lots of developers enjoy doing), weighing in, offering insights, and just... it's like, all play, and no deadlines! And yet, strangely (or perhaps not), those teams never seem to last long--they go through one, maybe two iterations on something, and then the team is broken up and nothing replaces it for a few years until a new VP comes along and says, "Wait, who's tracking our future direction?" and forms one again.... Only to have it dismember again in a year or two. What gives?
Typically, the "failures" of these teams have nothing to do with the team's lead or its members; in fact, it's often counterintuitive that they would fail, since they're often made up of some of the most senior people in the company. In fact, in many cases, participation on this team is itself a reward for success elsewhere in the firm!
We begin our analysis with the classic philosopher's question: "Why?" Or in this case, "Why does this team exist? What's the purpose of the team, and what are they asked to do?"
And, as with many of these philosophical questions, almost right away we run into an interesting problem: Nobody knows.
Most companies know, intuitively, that technology can be a competitive advantage, and almost every company that is over twenty years old has found themselves in a situation where they slept on a technology and suddenly found themselves on the wrong side of history when a competitor suddenly roars past them. Never mind the fact that the technology is only the surface symptom of why the competitor surpassed them, clearly the fault is that "We aren't studying the future enough! We aren't protecting our future! We need a team to do research and development!"
Armed with this "realization", a VP or CTO acts as the sponsor/champion for the founding of an R&D team, a team leader is selected for the role, and the team is told... pretty much nothing solid. "Go evaluate the industry! Find us that competitive advantage that we need! Go 'future-proof' our tech investment!" and other such empty statements are offered up as the vision or goal of the team, and they are turned loose.
NOTE: Keep in mind that "R&D" here could easily be replaced with "innovation"--as in, the "R&D Team" could easily be called the "Innovation Team" and staffed with not just engineers but also product and UX folks, but the goals are still equally vague. It simply highlights that most companies don't know how to Innovate any more than they know how to Research & Development.
The CTO might check in once or twice a quarter, particularly around review time, but for the most part the team is left alone, to their own devices, to char their own way. "These are the best and the brightest in the company," any doubters are told, "And they need time and space in which to work." If the doubters persist, they're told stories about teams at big companies that turned out the most amazing things by being left alone--Smalltalk and the DynaBook at Xerox PARC, or the iPhone at Apple, or any of a dozen other stories (only some of which actually have any bearing in fact, but that never stops the stories from being told). The R&D team itself will often get into the mix, too. "We need to behave like a startup," the R&D team tells anyone who comes knocking. "That means we have to go out there and Innovate. R&D. Discover. Disrupt, even!"
After a while, though, enough questions are asked that the CTO or VP is finally ready/willing/able/persuaded to take a deeper look at what the team is doing. "Can you present some of your findings to the Board/Company/Department sometime next month? We're all eager to hear what you've come up with." And the R&D team, just as eager to show off the fruit of their labor, agree to put together a presentation.
THe appointed hour comes around, and the R&D team proudly stands up to present what they've spent the last six, nine, twelve, sometimes even eighteen months working on. And it's....
"We're really excited to show you how Flutter can be used to completely change the way we build software. We've combined it with Firebase to produce this amazing prototype demonstrating...."
It's one presentation after another about existing technologies out in the world, just outside the mainstream. Elixir. Flutter. Kotlin. Two years ago it would've been "Cloud Native". Five years ago it would've been "Serverless". Today it's probably something in the AI space, such as demonstrating how to build a chatbot or use a vector database.
The CTO is excited. Wants to be excited. Tells everyone else they need to be excited by this. But the rest of the Board or the rest of the company are demonstrably less so. This doesn't really showcase how the company will catch up on their competitor, particularly because the competitor doesn't seem to be using that....? If the CTO or VP are in a strong position, they might double down on the R&D team, but in time, before long, somebody points out that this team is "the most expensive team in the company" and pressure builds to break the team up and put all that brainpower and skill to use on other teams that are sorely lacking in such. Before long, the R&D team is gone, the former team's members are either bitter or sad, and sometimes they start sending out resumes shortly thereafter. The CTO or VP move on at some point, which--if the team isn't broken up by then--almost guarantees the replacement is going to take a look at this huge "investment" and its return, which is pretty close to nil, and break them up at that point.
And all of the findings are essentially lost to the company, but helpfully captured on each members' resume.
Many readers are already jumping out of their seats, screaming, "Of course the team failed, they weren't researching the right thing!" But as other readers will point out, that's not the real problem. The real problems are that:
The team was never given any goals. What problem are they trying to solve? "Innovate" isn't a goal, it's a vague word that could mean anything to anybody at any time. Worse, the term's shared definition and context can easily change over time. Sure, it's entirely possible the team is versed enough in the company's business domain to know intrinsically what the problems are--just as it's entirely possible that the CEO, who doesn't actually program for a living, knows programming enough to know what language and stack the company should use. (Without the regular feedback that comes from doing the job, any "knowledge" about a job should be treated as suspect and untrusted until proven otherwise.)
The team had no business problem guiding it. Technology, as any halfway-trained product owner will be quick to point out, has to solve an actual business problem in order to have any value to the business. What's the business problem that needs to be solved? What does solving that problem look like? The best R&D teams are, in fact, run like a startup, complete with pitch: "This team is going to (blank). Our customer (people inside the company) faces pain in the form of (blank), and we aim to solve for this pain by creating a solution that (blank)." The R&D team has to have a driving goal, a purpose, or it risks wandering around in the technology jungle for a while before finally being declared "missing in action" and rescued (disbanded).
The team has no checkpoints or accountability points. Even in those rare cases where the team has goal, it frequently lacks any sort of checkpoint or accountability. "We're innovating! You can't put a schedule on that!" True, but you can put a periodic checkpoint to see how the team is faring against the goal, and/or force the team to present their findings on a much more regular basis. It's a simple agile problem--without feedback the team cannot know how well they are actually attacking the problem. In a startup, you get those checkpoints via "seed rounds" and "Board meetings" and "revenue targets"; in a larger firm, you'll have to create them. The R&D team needs to show progress on a regular basis towards the goal that was set, or it will be assumed to be lost, regardless of its actual progress or status.
The team had no plans beyond the research. For most of these presentations, the R&D team creates a prototype, a wonderful example of using the thing(s) they've researched, and they show it off during the presentation. "It's amazing, it's wonderful, it cuts our development time by (some wild-eyed number)!" Which is all great, but did anybody on the team stop to think about how this would/could get rolled out within the company? Like, if the research is on Elixir, what's the plan if the CTO says, "Great! We're in!"? As the natural experts within the company, the R&D team is now going to be tasked with figuring out the rollout, starting with which projects will get updated, which teams get training, and which projects will see the perceived benefits of the new thing right away. What are reasonable timeframes? What are the likely obstacles groups will hit? Is the intent to freeze all development across the company and do a rewrite of everything in Elixir? How reasonable is that? How possible is that? It's not enough to just say, "Look there!"; the R&D team also needs to have some kind of plan around how this would actually work.
The team's prototype doesn't actually address the hard problems. The prototype usually bears a slight resemblance to what the company does, but usually it's an idealized version of the problem, with none of the ugly parts. "It's just a prototype," they'll say when challenged on that point. Unfortunately, these prototypes, which often are built to be basic CRUD apps over a data store of one form or another, show off how well the new thing does the easy parts. Folks, any technology can make the easy parts look easy--any prototype that's actually gong to be useufl needs to show how well it handles the Hard Parts. This was one of the major pain points among the Ruby-on-Rails and NodeJS crowds back in the early stages of each of those technologies--yes, they made it absurdly easy to build a CRUD app, but then again, so did everything else. Show me the RoR app that streamlined how I talk to a mainframe, or how the NodeJS app can talk to our legacy COM/DCOM or CORBA objects. And they didn't--or couldn't--and as a result, it took a very long time for either of those two to break into "the enterprise", but were overwhelmingly adopted by startups, who lacked those brownfield obstacles. (Then, of course, when the startups were bought, they had to learn how to do those integrations, but by that point the die was already cast.) If you build a prototype, it has to demonstrate how to handle the hard problems, not the easy ones.
Even then, however, the R&D team can still get lost if they aren't run properly or in the right direction. But starting by addressing some of these points will help at least get the team to a 50/50 shot of succeeding at something.
I have much deeper thoughts about the different kinds of R&D teams and how to think about creating a group that focuses on innovation (to the scale of something like what Facebook created with React or what Netflix did with Chaos Monkey), but that's going to have to wait for a new post.
Last modified 28 May 2024