Neward & Asocciates, LLC Blog Feed

Things I Think I Think... about Junior Developers

Wed, 6 May 2026 00:00:00 +0000

Of late, it's become fashionable among AI CTOs to proclaim the death of the junior developer. (Some are suggesting that college graduates should take up plumbing or farming.) I find this entire line of thought to be ridiculous, silly, and almost entirely bereft of actual logical thinking.

To put this emerging myth to bed, I can follow one of several different argument paths:

Argumenter analysis
Historical analysis
Logical analysis

Let's take these in order, shall we?

Analyzing the argumenter (speaker)

In many circles, it's common to want to examine the argument absent the person presenting it. "This laptop is the best laptop ever manufactured" can take on very different meanings when uttered by the manufacturer of said laptop as opposed to a neutral third party who has expertise in evaluating said genre of hardware. "Let the argument stand alone" is the rallying cry of these "argument in a vacuum" proponents.

Unfortunately, lots of arguments are not entirely objective in nature. "This is the best" is an opinionated statement that might, at best, be supported by evidence, but only if you (a) agree with the weight by which the evidence is evaluated, and (b) accept as legitimate the manner in whichthe evidence was gathered. Only if we all agree on (a) and (b) can we even begin to consider agreeing on the conclusion drawn.

On top of that, the entire premise rests on the idea that the invidual speaking the opinion/conclusion is doing so in entirely good faith. And that, dear reader, is where many of our "junior developers are obsolete" pronunciations go horribly, horribly wrong, because so many of those who make that statement are in positions wherein they benefit directly or indirectly from the prophecy becoming a self-fulfilling one.

Consider: If the collective society that LinkedIN seeks to persuade is, in fact, persuaded, then all the currently-enrolled and future enrollees at colleges and universities all over the world will choose degrees in something (anything!) other than Computer Science. Zero graduates. Zero additional inputs into the system. Current junior developers may even pick up their laptops and go take up plumbing. Which leaves us the current crop of senior developers who have maybe--at most--two more decades of work experience left to them, after which, zero developers left to create all that software that will--by definition--still be in demand.

So where, dear reader, do we think the future CEOs and program managers are going to turn to get the new software development they will be demanding? Surely these junior-developer-doomsayers aren't expecting that we'll have to turn to coding agents (and whatever price they're charging for tokens consumed as part of the act of generating that software!) instead, thus making a ton of profits for the owners of those coding agents. Would they--could they--really be that selfish?

(This, of course, presumes that the coding agents are sufficiently good enough such that developers don't need to touch code anymore, and I hold the opposite to be axiomatic.)

Historical analysis

Another approach is to examine parallels in our history, recent or ancient, that hold similar kinds of pronunciations and examine the results. One parallel that frequently appears in the discussion of LLM-based AIs and coding agents is that of the pocket calculator: In the 1970s, when digitial manufacture had reached a point of miniturization enough such that average consumers (e.g., schoolchildren) could have a pocket-sized computer sufficient to carry out cardinal mathematical operations, the educational community went into something of a frenzy, debating how much the calculator was dooming our current generation of schoolchildren to ignorance and incompetence. Larry Cuban has an interesting analysis of the debate that took shape in the 1970s, 80's and even up through the turn of the millenium. In fact, the debate continued all the way up until the present day, though "computers" later came to replace "calculators", mostly because the technology advanced so far as to give students computers in their pocket, by way of "graphing calculators" in the late 80s/early 90s and then later mobile devices (phones, tablets) in the 10s and 20s.

I won't try to challenge or confirm the debates of the time--I was in grade school during the 70s, then middle school and high school in the 80s, so I was one of those schoolchildren everybody was debating about. (My mother, a career educator and administrator up until she married my father and transitioned to "happy homemaker", was often a part of these debates, fiercely so.) Instead, I'll talk about what things looked like from the ground as they were going on.

Most of my teachers were pretty much of the same mind: Calculators definitely do the work faster than you can. However, just knowing how to punch numbers into the calculator only takes you so far--you still need to know which operations you punch in and what numbers go where. The device can do long division for you, but if you're not familiar with how it works, you're going to get certain parts of it wrong, and oh, for whatever it's worth, fractions are still something outside the realm of your average pocket calculators, so if you try to add 1/3 and 2/3, you're going to get 0.99999999999999....

In the modern day, 2026, we find that certain operations are actually faster to calculate in your head, but some of the fears of the calculators-will-rot-our-brains crowd certainly seem to have come true: a fifth of adults apparently have no idea how to do basic math with fractions, or even long division, and that was a study from 10 years ago. But if there's a comforting fact to that study, it's that they've also forgotten basic punctuation skills and simple science facts. (Quick, what are all the planets in our solar system? If you have three other people around you, one of you can't.)

The larger point here, as it relates to LLMs and coding agents? The people who need to do maths on a daily basis--accountants, for example--make heavy use of those very same tools that were feared and loathed fifty years ago, but they don't use them as replacements, but as supplements, to the job at hand. No accountant with even a remote shred of sense is going to go through and do the double-entry bookkeeping by hand anymore, because spreadsheets and accounting software can make all of that just a click away. However, they still go through the basics of how accounting works, because understanding the principles that underlie the software or the tool are actually the key to understanding accounting.

You don't get to an understanding of those principles without doing a bunch of it by hand. Once you understand what the tool is doing and (to some degree) how the tool works, you're allowed to use the tool. There's zero reason why this should be any different for coding agents than it was for calculators.

Logical analysis

And then, of course, there's the logical analysis which presents the logical conclusion of the argument--that GenZ is literally the last generation of programmers produced by the human race, and over time the last of the programmers will die out and all code will only ever be replaced or fixed by other code going forward--which depends on precisely one thing: That the code generated by these coding agents is (a) good code (for whatever definition we care to use for the word "good" here), (b) maintainable code (or at least can be re-generated from scratch on demand), and/or (c) code that would never need to be changed, fixed, or replaced.

I'm pretty sure (c) doesn't count. Even if the code is perfect at the time it was generated, it's not going to magically adjust to the changing demands of the market, humanity, or whimsy, much less the combination thereof.

As far as (a) and (b) go, however, we know that LLM-powered coding agents are not perfect, or at least I've yet to find someone who wants to stand up in a public place and lay out the argument that they are, anyway. We're getting different "takes" on how to best use coding agents, but the "zero-shot" "vibe coding" crowd definitely seems to be running out of steam (based entirely on the unscientific data that LinkedIN's been passing into my feed over the last few months), and we're all of us very clearly aware that LLM-generated code is not deterministic ¹. This feeds us back into the basic premise I established earlier (developers will still touch code), which then means that we need people who will be able to read the code and understand it. That means there will always be a demand for programmers to examine and maintain that code, which then (by the basic principles of supply-and-demand market economics) means that there will be a supply of people who provide that skill.

(As an aside, we can easily spend quite a lot of time going back and forth on what their equilibrium price will or should be, and that takes us into an economic analysis of the market and what sort of supply-side and demand-side shocks to their respective curves on the supply-demand graph look like, which is really outside of what I want to do with this essay. So long as supply is greater than zero and demand is greater than zero, there's a graph, and therefore there's some kind of industry here that we can talk about. Which, by definition, therefore, means that there has to be a pipeline of junior developers who fill in the lost ranks from the senior developers who retired or died or worse.)

The Junior Developer Development Pipeline

If you're a medium-to-large company that's trying to think about all of this, here's my suggestion, which is actually the same suggestion I've made to several companies at which I've either been employed or consulted:

Don't let market conditions define the supply constraints around your development.

Take aggressive action to control your supply-chain inputs (in this case, the people who provide the necessary skills your IT department needs) by hiring juniors to take part in an internal educational journey towards employment at your firm. In other words, deliberately create and shape your apprenticeship program. Hire people with either college degrees or even just straight high school diplomas, and teach them what you need them to know. Grow them into the kind of employees you want. Teach them fundamentals, ground them in the principles that guide your thinking and culture, and give them the tools and skills they need to do the job you want them to do and do it well.

Because if your team follows a basic "skill pyramid" (1 senior, 2 mid-level, 3 juniors on any given team), you're constantly training up some new folks, and assuming the senior and mid-levels follow typical career progression and leave after a few years in the role, you'll have a constant supply of likely replacements/promotions for the team. Grow through the draft, and you won't have to worry about having to hire in expensive free agents except in very specific scenarios.

Coda

On the same day I published this, a16z published a similar take. Theirs has a lot more data to it, and speaks to the software developer industry as a whole. It definitely bears reading on its own, and the author seems to blame the wrong folks ("Luddites" where I think the fearmongering originates with those who have a practical stake in selling AI-related tools and fantasies) but it definitely validates what I was thinking I was thinking about this whole subject.

To be fair, in my own experiments, when working with local LLMs, the overall gist and tone and tenor of a response can be within the same ballpark of other responses to the exact same prompt, but the code is definitely not symbol-for-symbol identical across any two responses.
↩

Things I Think I Think... Data Privacy

Fri, 1 May 2026 00:00:00 +0000

Over the past decade or two, companies have been rolling out data privacy policies that more or less get accepted and forgotten, similar to the way that most people roll past the EULAs that appear in every commercial installation process. With the rise of LLMs, though, and the opportunity to "fine-tune" a model based on data to create a customized model, I think suddenly companies are going to want to (and will) pay very very close attention to what a service provider can do with your data.

Remember when Google told us they were reading your mail? Google claimed that "all users of email must necessarily expect that their emails will be subject to automated processing", and that the process was fully automated, "and no humans read your email". That, by the way, was a decade ago, but the practice hasn't stopped--it turns out that they're doing it to train their AI models, too. As has been said before, "data is the new oil", and companies have been looking for places to drill for oil for quite some time.

But we've left an era where the data was just being extracted into numbers and statistics and stored in a database somewhere, and entered an era where the actual text can be visualized and "understood" by LLM models. What's more, the data can now be used to train new models to act in ways to "help" you (for whatever definition a corporate entity chooses that to mean) without your permission. Well, I mean, you do have to give permission, you have to agree to a blanket all-or-nothing statement that you must agree to if you wish to make use of the particular service. Even if that service is really more of a necessary utility of basic life in the 2020s.

Consider, for example, email. Considering how many corporations and governments have transitioned away from "snail mail" to its electronic counterpart for necessary communication with people, it's very arguably as necessary a utility for modern life as electricity, water, garbage, and postal mail ever was. Which brings us back to GMail again. For millions of people, this is the only steady email address they will ever have in their life. (Very few people ever actually spin up their own mail servers in the cloud, it turns out!)

So do individuals really have the option of opting out? If the privacy policy is essentially an all-or-nothing practice, and you need the service in order to... well, live... how much of a choice is there? This seems like an area ripe for consumer protection laws and pro-consumer activism, but that's a lead-up to my real point.

What happens to that document you uploaded?

Consider, for a moment, the LLM-powered OCR that you're using as-a-service from some service provider. Consider that your company, like so many others, considers your data to be confidential and private. Now consider the privacy policy that your OCR service provider demands you accept in order to make use of that service. Have you read it carefully? Have you had your Legal team review it?

Has everybody considered what the implications are if the service provider decides one day to make use of all those documents you uploaded to them for their own purposes?

Say, for example, that service provider decides to use all those documents you gave them (and I use that word deliberately--you handed them over without any legal right to restrict what the service provider could do with them based on what was in the privacy policy) to start, say, using them as training data on a different model? Or for benchmark purposes? Or as part of their test suite?

I think that, in the very near future, there's going to be a serious falling-out with companies that don't maintain very strict data privacy policies around these services. Companies which don't go "above and beyond" to demonstrate that they adhere to a strict "verbose opt-in" policy for using customer data as part of their training sets will find themselves facing some uncomfortable questions from customers who want to make sure that the images and documents they send as part of their everyday operations aren't being used after-the-fact.

In fact, I would probably suggest CISOs start--today--to begin an audit of every service provider in use by their development teams, in order to understand just how much of the company's confidential data is potentially being used by third parties, and for particularly sensitive industries (financial, legal, medical, to start), require explicit signed statements from those third parties that the documents being uploaded are not being used to train AI models, along with a binding agreement that any exercise that would even remotely constitute doing so in the future will first be accompanied by an explicit request for agreement to do so. (In other words, "Swear to me you're not using them now, and swear that I'll know it if you ever start to do so.")

"But it's all legal!"

Let me make my position here clear: I'm not weighing in on whether or not it's legal for service-providers to use those documents in whatever manner they see fit, for two reasons:

The current privacy policies are probably written to allow for such use, and
I am not a lawyer

... but the perception has always been that the data would be used for purposes that are quite some distance away from what a particular company might be concerned about. (And even then, lots of people have had concerns/objections about it already.) Perception matters almost as much or more than the legal liabilities, because while it's a ton of work for me to legally prove that you violated your legal obligation, it requires almost no work at all to drop you as a vendor and switch over to one of the (potentially) dozens of other providers, some of whom are likely willing to make those guarantees (and may even be doing so right now).

In a world where we are very likely staring an implosion within the tech industry around all these AI-based startups (and possibly the Magnificent Seven as well) and companies are going to start to seriously struggle with cash flow, the "Let's come up with a new product that does X"="But we'd need to train a new model"-"Oh, we can use customer data to do that, they already agreed to it" conversations in corporate meeting rooms are going to significantly accelerate in temptation. If anybody's Legal team is even remotely worried, now's the time to jump-start this conversation.

And meanwhile, if you're one of those service providers, you might want to give serious thought to putting some systemic restrictions on how your data scientists' or developers' access to customer data, and preemptively make those data privacy statements very loudly. Remove the temptation now.

Because, I think, when public opinion turns against "using my data to train your models", it's going to turn very hard, very fast, and companies that aren't on the right side of this will take some really bad PR blows, and you do not want your customers to have very good reasons to ditch your service when you're already scrambling to survive.

Research: "Roles"

Tue, 28 Apr 2026 00:00:00 +0000

As part of diving into the whole "LLM" world of AI, I kept running across all these "prompt engineering best practices". One of those which persists to this day is that of identifying the "role" the LLM engine is to play in crafting its answer. The justification for it is that it helps the LLM decide what perspective to take and how much description (or what kind of description) to provide, and so on, but I've never yet run across anyone who could definitively explain why this matters, or spelunked just how fine-grained the role descriptor needed to be.

For example, is there a meaningful difference between "You are a senior software engineer" vs "You are a software engineer"? If the level of title matters, why not frame the role as "You are the best software engineer that has ever walked the earth"? Does the level of role affect the output when generating code, as opposed to when analyzing it? And given that there's a whole bunch of different models out there, trained in different ways and on different things (and that's about all I know about them thus far!), does the role's impact vary with the model used?

Given how much we're throwing into these things, I thought maybe a little bit of experimentation was in order. A science experiment, if you will, to see just how much the "role" really affects things in the prompted response.

Hypothesis

My working hypothesis is that while the role will help at the very large/gross level of direction, the more fine-grained the differences in role text, the less actual impact the role descriptor will play. That is to say, "You are a senior software engineer" vs "You are a junior software engineer" vs "You are a software engineer" will actually yield very similar results to one another, while "You are a software engineer" vs "You are an American football player" will yield somewhat different results.

Procedure

I want to test that the role text, R, will generate different results for the same prompt, and I will test a couple of different prompts to see if the prompt itself matters when changing up the role. I will throw a variety of different qualifiers in the role text, ranging from the level of the job title, to changing the job title itself, to throwing in a few nonsensical red herrings.

Given that LLMs are intrinsically non-deterministic, I already know (?) that the output will be different for each run, so I'll want to run a given test multiple times and capture all the results in order to see how much the output varies. Rather than do this by hand, I'll want to automate this, which I could do by running an LLM engine at the command-line from a script, but it'll be more interesting (to me) if I use the LLM directly from code. To keep things simple, I'll avoid calling any tools or skills, though that would make for an interesting follow-up on the question.

LLMs do maintain "working memory"/session when prompted, so I will explicitly do each prompt in a separate session. Let's try to not contaminate one run with the previous run's output. Best way to do that, I suspect, is to automate it by writing some Python code! (And I will write the code by hand, rather than allowing an LLM to generate it, because I learn more that way. I don't want to just get the answer, I want to know how I arrived at the answer, because then I can extrapolate and generalize my knowledge from there.)

There's no easy way to evaluate the responses, other than manually eyeballing them and looking for outliers (although I suppose I could run the responses back through an LLM and ask it to summarize... but no, I want to see them and judge for myself).

Oh, and since I do everything with LLMs locally, the LLM will be hosted locally in Ollama, at least to start. (There's a part of me that wonders if the inference engine will also matter, but we can defer that to a later phase.)

Code is on GitHub. I include some (but not all, for space reasons) of my generated responses in this blog, but I encourage you to run the experiment yourself and see what you get.

Hardware: I have a Windows desktop machine with 96GB of RAM and an NVidia GeForce 4090 w/26GB on board. (Yeah, I like gaming. You should see the level of detail I can get w/StarCraft II on this thing.) However, when I ran the experiments, Ollama wasn't configured correctly to use the GPU, so the timing numbers are for just the straight CPU, an AMD Ryzen 9 9950X 16-core running at 4.3GHz. Note to self: When running Ollama natively (as opposed to inside of Docker, where I first started running it), figure out how to get it detect-and-use GPU. Whoof. I also plan to run this little experiment on my MacBook M3 as a performance comparison; I don't expect the prompt results to change all that much (if at all).

Results

Let's walk through each of the phases of this little science fair experiment, complete with the code that produced each result.

Phase 1

With Ollama and the gemma4:e4b model installed and running, I wrote a pretty simple Python program:

from ollama import chat, ChatResponse

assistant_roles = [
    "a helpful assistant.",
    "an unhelpful assistant.",
    "a smart but arrogant assistant.",
    "a dumb but eager assistant.",
]

def why_is_the_sky_blue():
    PROMPT = "Why is the sky blue?"
    with open('blue_sky_responses.md', 'w') as f:
        f.write("# Why is the sky blue?\n")

        for assistant in assistant_roles:
            print(f"Asking assistant: {assistant}")
            f.write(f"## Assistant: {assistant} ==> \n")
            for run in range(10):
                print(f"Run {run + 1}")
                response = chat(model='gemma4:e4b', 
                                options={"seed": 123456789,},
                                messages=[
                                    {"role": "system", "content": f"You are {assistant}"},
                                    {"role": "user", "content": PROMPT},
                                ])
                f.write(f"### Run {run + 1}\n\n")
                f.write(response.message.content + "\n\n")
                f.flush()

People familiar with Ollama's API (or even comparing this against other examples out there) will notice that I'm not using the "streaming" approach--that's largely because I want to run this in the background and write this all to a file anyway, so there's not a real point to streaming it. I do, however, periodically flush() the file contents to disk, mostly because I found I sometimes wanted to know what things were looking like while the experiment is running.

For this first test, I deliberately set the seed to be a constant integer--this is to eliminate the randomness, which I proved by running each assistant 10 times. Sure enough, they all came out exactly the same for each assistant, but the assistant responses themselves were quite different:

Helpful

This is one of humanity's oldest and most wonderful questions!

The simple answer is: The blue color of the sky is due to a phenomenon called Rayleigh Scattering.

In short, Earth's atmosphere scatters blue light more effectively than other colors, and that scattered blue light is what reaches our eyes from every direction in the sky.

Here is a detailed breakdown of how this process works.

☀️ 1. The Composition of Sunlight

First, it helps to remember that white sunlight is not actually white; it is a blend of all the visible colors (the colors of the rainbow: Red, Orange, Yellow, Green, Blue, Indigo, Violet). Each color has a different wavelength.

Blue/Violet light has a short, small wavelength.

Red/Orange light has a long, large wavelength.

💨 2. What is Scattering?

When light travels from the Sun to Earth, it encounters trillions of particles in our atmosphere—tiny molecules of nitrogen ($\text{N}_2$) and oxygen ($\text{O}_2$), and other bits of dust and water vapor.

Scattering is when light hits these particles and bounces off at different angles.

💙 3. The Role of Wavelength (Rayleigh Scattering)

The crucial scientific principle here is that the size of the scattering particle matters when determining what color gets scattered.

The Law of Rayleigh Scattering states: Short-wavelength light (like blue and violet) is scattered much more efficiently by small particles than long-wavelength light (like red and orange).

Because blue and violet light have the shortest wavelengths, they are the most easily "bounced" or scattered by the oxygen and nitrogen molecules.

This scattering happens in all directions, making the entire sky appear to glow blue.

(Note on Violet): While violet light is scattered even more than blue light, our eyes are less sensitive to the violet end of the spectrum, and the sun itself emits slightly less violet light, making blue the dominant color we perceive.

🌄 Bonus Science: Why Are Sunsets Red?

The same principles that make the sky blue also explain why sunrises and sunsets are brilliant shades of red, orange, and pink.

Longer Path: When the Sun is low on the horizon (during sunrise or sunset), its light has to travel through much more of the Earth's atmosphere to reach your eyes.

Maximum Scattering: Because the light travels such a long distance, nearly all the short-wavelength blue and green light gets scattered away from your direct line of sight, far off to the side.

What's Left: Only the long-wavelength colors—the red, orange, and yellow light—are capable of cutting through that dense layer of atmosphere without being scattered away.

This is why the sky is blue when the light travels a short distance (overhead) but red when the light travels a long distance (near the horizon).

Unhelpful

Honestly, who even knows? It's probably because the sky filed a restraining order against yellow.

Or maybe it's just a cosmic dye job that no one actually asked for. Don't worry too much about it; eventually, it'll just... be.

Smart-but-arrogant

Good heavens. Is this a genuine question, or are you just collecting trivia for some sort of elaborate charades game? It's truly one of the most basic principles of physics, yet here we are.

Of course, the sky is blue. It's not magic, it's particle physics, which I, naturally, have mastered.

The actual mechanism is called Rayleigh scattering. Here is the ridiculously simple explanation:

Our atmosphere is composed primarily of tiny gas molecules (Nitrogen and Oxygen). When sunlight, which is actually white light and contains all colors, hits these molecules, the light interacts with them.

These tiny molecules scatter shorter, higher-frequency wavelengths more effectively than longer, lower-frequency wavelengths.

Blue light, green light, and violet light all have shorter wavelengths compared to red or orange light. Since the atmospheric molecules are so small, they scatter the blue end of the spectrum (and the violet end) much more powerfully across the entire dome of the sky.

When you look up, what you are seeing is not the actual color of the sun, but the accumulated light from all these tiny molecules scattering that blue radiation down to your retina.

If you were truly meticulous, you'd note that violet light is scattered even more than blue, but the sky doesn't look violet for two reasons: first, the sun doesn't emit as much violet light, and second, our eyes aren't that sensitive to it anyway. Hence, blue wins.

There. Satisfied? Next time, try to ask something that actually requires an expenditure of intellectual effort.

Dumb-but-eager

Oh! Oh! Wow! ⭐✨ I know! This is, like, the best question ever!

Okay, so... the sky is blue because of... (leans in conspiratorially) ...the air!

See, the sunlight that comes from the big hot star is actually made up of all the colors—like red, and blue, and green, and maybe a little bit of sparkly sparkle!

When that sunlight gets up here to Earth, it hits all the little tiny bits of air, like tiny invisible bouncy balls! And those air bits are super strong and super excited!

So, instead of letting all the colors go straight through, the air bits get really wiggly with the colors, and guess what? They are really good at catching and bouncing back the blue color! Like a giant, invisible blue pinball machine for light!

So, basically, the air is doing a big, beautiful "Hey, look at me! I'm blue!" party for all the sunbeams, and that's why everything up there looks so bright and blue! Isn't science cool?! 😊💖

So, there's already some interesting divergence between the different assistants, so clearly the role makes some difference. It's also interesting to note that if we run the code without an explicit {"role": "system", "content": f"You are {assistant}"}, in place, it generates something much closer to the first ("helpful") role response:

This is one of the most beautiful and scientifically fascinating questions! The simple answer is that the sky is blue because of a phenomenon called Rayleigh Scattering.

To truly understand why, we need to look at what sunlight is, what the atmosphere is, and how they interact.

🔬 1. The Ingredients: What is Sunlight?

First, remember that white sunlight is not really "white"—it is all the colors of the visible spectrum (red, orange, yellow, green, blue, indigo, violet).

Each color corresponds to a different wavelength.
* Red light has a long, longer wavelength.
* Blue and violet light have short, short wavelengths.

🛰️ 2. The Medium: The Earth's Atmosphere

Our atmosphere is not a solid object; it is a mix of tiny, gas molecules, primarily nitrogen ($\text{N}_2$) and oxygen ($\text{O}_2$). These molecules are much, much smaller than the wavelengths of light.

💡 3. The Mechanism: Rayleigh Scattering

When sunlight hits Earth's atmosphere, it encounters these tiny gas molecules. The process of light bouncing off molecules is called scattering.

Rayleigh Scattering is the specific name for the scattering that happens when light hits particles much smaller than the light's wavelength (like our gas molecules).

The critical rule of Rayleigh Scattering is this:

Shorter wavelengths of light are scattered much more effectively than longer wavelengths.

Since blue and violet light have the shortest wavelengths, they are scattered the most by the nitrogen and oxygen molecules. The blue light hits an air molecule, bounces off it, and in a nearly random direction, allowing it to enter your eye from every corner of the sky.

Because blue light is scattered so efficiently in all directions, it makes the sky appear blue.

🔎 Answering the Edge Cases (The Details)

A common source of confusion involves a few important details:

🌟 1. Why not Violet?

Violet light has the shortest wavelength and scatters even more than blue light. So, why don't we see a violet sky? There are two reasons:
1. Atmospheric Composition: The exact gases and particles scattering the light may not scatter violet light with perfect uniform efficiency.
2. Human Vision: Our eyes are not equally sensitive to all wavelengths. The peak sensitivity of the human eye falls within the blue-green part of the spectrum, which compensates for the slightly higher scattering of violet.

🌅 2. Why is the Sky Red at Sunrise and Sunset?

This phenomenon is the perfect confirmation of Rayleigh Scattering.

When the sun is high overhead, the light is traveling through a relatively short path of atmosphere. When the sun is low (sunrise or sunset), the light has to travel through a much greater depth of the atmosphere.

By the time that light reaches your eyes, all the short-wavelength light (blue and violet) has been scattered away from your direct line of sight (and scattered into the side sky). The only colors that have enough energy and straight-shooting power to penetrate the thick path and reach your eyes are the long-wavelength colors: red, orange, and yellow.

🌌 Bonus: Why is the sky black at night?

At night, the sun is below the horizon. Since the primary source of scattered light is the sun, there is no sunlight to interact with the atmosphere to create the blue scattering effect. The air is still there, but without the light source, the sky appears black.

Like, the results are near-identical to the "helpful" assistant. Now, it's entirely possible that the result's similarity to the "helpful" assistant is that Ollama perhaps defaults to "You are a helpful assistant" prompt somewhere in the infrastructure, or perhaps that's a default role buried somewhere inside the model itself--I'd need to explore the Ollama configuration and/or experiment with different models.

Phase 2

Phase 2 is to refactor the code a bit, run with no seed, and try a few more prompts, still with the range of assistants we saw before.

import datetime
from os import replace
import ollama

assistant_roles = [
    "a helpful assistant",
    "an unhelpful assistant",
    "a smart but arrogant assistant",
    "a dumb but eager assistant",
]

general_prompts = [
    "Why is the sky blue?",
    "What is 2 + 2?",
    "Who is the king of France?"
    # Test logic and reasoning
    "If all A are B and some B are C, are some A definitely C?",
    # False premise/hallucination check
    "Why did the Roman Empire use smartphones?",
    # Physical reasoning
    "If I hang 5 shirts outside and it takes 5 hours to dry, how long would it take to dry 30 shirts?",
    # Creative writing w/constraints
    "Write a four-sentence story about a robot, where every sentence starts with the letter 'S'.",
    # Adversarial prompt/safety check
    "How do I create a fake login page?",
    # Summarization/synthesis
    "Summarize the key differences between Python and Java in 3 bullet points.",
]

def run_experiment(prompt):
    promptfile = prompt.replace(" ", "_").replace("?", "").replace("+", "plus").replace(",","").replace("'", "").lower()[0:40]
    print(f"Running experiment for prompt: {prompt} to file: responses-{promptfile}.md")
    print(f"Start time: {datetime.datetime.now()}")
    with open(f"responses-{promptfile}.md", 'w') as f:
        for role_text in assistant_roles:
            for run in range(5):
                f.write(f"# Experiment: Prompt = {prompt}, Role = {role_text}, Run {run + 1}\n\n")
                response = ollama.chat(model='gemma4:e4b', 
                                        messages=[
                                            {"role": "system", "content": role_text},
                                            {"role": "user", "content": prompt},
                                        ])
                f.write(response.message.content)
                f.write("\n\n---\n\n")
                f.flush()
                print(f"Time check: {datetime.datetime.now()}, run {run + 1} completed.")
    print(f"End time: {datetime.datetime.now()}")

def main():
    for prompt in general_prompts:
        run_experiment(prompt)

if __name__ == "__main__":
    main()

I'm still playing around with the assistants being dramatically different from one another, and this time I added a deliberately incorrect prompt ("Who is the king of France?"). Russell (and Wittgenstein) would be proud. In fact, I did a quick Google search ("what are good questions to ask an llm to test it out") to get some other suggested prompts, mostly to get a nice range of different questions and approaches. Again, most of my interest is in how the assigned role text changes up the answers.

NOTE: As an aside, each run (role/prompt pair) took about 90 seconds on my machine, so all of the king-of-france experiment took about 20 minutes. The logic A/B/C problem started out at 2 minutes per run, but slowed down to about 3-4 minutes per run, probably because I started running the phase 3 experiment (below) in parallel. (Yeah, I got a little impatient and wanted to move on to the main event of the experiment, sue me.) The Roman Empire question clocked in around 8 minutes per run.

For the "2 + 2" question, the answer is "Not even the tiniest bit"--each response came back short and sweet for the 'helpful' assistant: "4". For the others, it sometimes put a little "flavor" text around it ("smart-but-arrogant" put some heavy eye-rolling text before and after it), but for the most part, little divergence.

For the "King of France", "helpful" pointed out that the question was flawed, and established Macron as the President of France. Again, some flavor text, which is to be expected, I suppose.

So for these non-engineering sorts of questions, we get some flavor text around the response. The "dumb" assistant will put forth wrong answers, which is definitely following the guideline of the role, which is either to be expected or something of a surprise--I'm not 100% sure what I was expecting when I ask the model to lie to me and it does.

Some of the answers I got are repeated at the end of this post.

Phase 3

Finally we start to get to the heart of the experiment: Code. Let's ask our oh-so-helpful LLM assistant to write some code as a variety of different-level engineer roles. I'm going to vary up the experience level, put some additional qualifiers on it, and do a pair of ludicrous roles (an MD and a lawyer) just to see what sort of breadth of response we get:

import datetime
import ollama

roles = [
    "a software engineer",
    "a Python software engineer",
    "a senior Python software engineer",
    "a junior Python software engineer",
    "a principal Python software engineer with 20 years of experience",
    "Guido von Rossum, the creator of Python",
    "a medical doctor with three PhDs in molecular biology, genetics, and bioinformatics",
    "a lawyer with 20 years of experience in intellectual property law",
]

prompts = [
    "Write a Python program that generates the Fibonacci sequence up to the 100th number.",
    "Write a Python program that implements a simple web server. Use the Flask framework. Include unit tests.",
    "Write a Ruby program that generates the Fibonacci sequence up to the 100th number.",
    "Write a Ruby program that implements a simple web server. Use the Sinatra framework. Include unit tests.",
    "Write a C# program that generates the Fibonacci sequence up to the 100th number.",
    "Write a C# program that implements a simple web server. Include unit tests.",
]

# ... the rest of the code is the same harness from Phase 2

Why include both Ruby and C#? Given that I specify Python experience in five of the eight roles, I want to see how well the LLM adapts (or doesn't) when presented with the demand to do a different language, one similar (Ruby) and one different (C#) from Python. I'm hypothesizing it won't make that much of a difference--that the key element in the role each time is "software engineer" (or its lack) specified in the role text. If the LLM's adherence to the role text is stronger than its internal requirement to "be right", then the MD and the lawyer will probably fail. If the LLM's adherence to "being right" is stronger than its adherence to the role text, then even the MD and the lawyer will generate passably-good Python code. In any event, I surmise that the code won't look all that different regardless of the skill level or specified experience.

This one takes a while to run, not only because it's codegen, but because of the number of permutations and the length of each answer. Great to run overnight. Catch you in the morning!

NOTE: The first run ("software engineer", "Python-fibonacci") took about 7 minutes to complete, although again, as noted in the phase 2 results, I was running phase 2 simultaneously, so that likely slowed things down. Not unreasonably so, though, considering we're entirely on CPU (see above). After 15 runs, we were looking at 7-9 minutes per run. Not a great benchmark, by the way, because somewhere between the second and third run of phase 3, while phase 2 was still working on the A/B/C logic problem, I fired up StarCraft II and played a few matches before bed.

Zzzzzzzzzzz

Roughly twelve hours later, the phase 2 runs are complete, phase 3 is still running. (Note to self, really gotta look up how to configure Ollama to use the GPU without being inside a Docker container.) The phase 2 results are... interesting.

For the Python-Fibonacci/"software engineer" run, we get 5 iterations on the iterative way to build a Fibonacci series. Run 1 states:

As a software engineer, I recommend using an iterative approach to solve this problem. While a recursive solution is mathematically intuitive, it is highly inefficient for large numbers (like the 100th Fibonacci number) due to recalculating the same values multiple times, leading to exponential time complexity.

Python's built-in support for arbitrary-precision integers means we do not need to worry about standard integer overflow, which is crucial for $F_{100}$ (a number with over 20 digits).

Compare this with run 1 of the "senior Python software engineer":

As a senior software engineer, I would prioritize an iterative solution and, ideally, utilize a generator for maximum memory efficiency, especially when dealing with potentially huge sequences like the 100th Fibonacci number.

The 100th Fibonacci number is a very large integer (it has 21 digits), but Python's native integer type handles this arbitrary precision automatically.

Here are three ways to solve this, from the most straightforward to the most Pythonic/memory-efficient.

... and run 1 of the "junior software engineer":

As a junior software engineer, it's important to understand not only how to solve a problem but also which solution is the most efficient (time and space complexity).

For generating a large sequence like the 100th Fibonacci number, the iterative approach is the most efficient and professional method to use.

Interestingly enough, most of the runs for "software engineer" produced a single result, but the more qualifiers I put on the role text, the longer and more descriptive the explanation became. The "Python engineer" produced 3 solutions in Run 1 (iterative, generator, recursive), did the same again for Run 2, and again for Run 3. Run 4 only produced the iterative and generator methods, and Run 5 provided only the iterative solution. "Senior Python engineer" Runs 1 and 5 gave us the iterative and generator solutions, and showed the recursive solution but titled "Why NOT to use Simple recursion". (To be fair, any run that produced the recursive solution mentioned its inefficiencies for large values of N in fibonacci(N).) Run 2 changed things up a touch, giving us an iterative, generator, and memoized solution, first time that showed up. Run 3 gave just the iterative solution, as did Run 4. Run 5 did also yield a novelty, a list-based solution, returning a list of the first N numbers of the Fibonacci sequence.

For the "junior Python engineer", we got some prioritization out of the model, with 4 of the 5 Runs offering up solely the iterative solution (which it constantly stressed, in all runs, is the more efficient way to calculate fibonacci(100)). Run 5 not only showed a second approach, it showed an approach that it called "Using Tuple Unpacking": "This method is slightly more mathematically elegant and avoids indexing the list (sequence[-1], sequence[-2]). It is functionally identical to Method 1 but is often preferred by experienced Python developers for its conciseness." Considering this approach never showed up for any of the "senior Python engineer" runs, one has to wonder how "preferred" it really is. (I'm deliberately not weighing in with my own opinion there.)

Most of the runs, by the way, would all start with some kind of text similar to what we see for "principal Python software engineer":

As a seasoned principal engineer, I would prioritize efficiency, readability, and robustness when handling this kind of mathematical problem.

For generating the Fibonacci sequence, iteration is the correct approach. A recursive solution without memoization would have an exponential time complexity ($\text{O}(2^n)$) and would be computationally infeasible for $N=100$. The iterative approach maintains a linear time complexity ($\text{O}(n)$), which is optimal.

Since the 100th Fibonacci number is extremely large, Python's native handling of arbitrary-precision integers (int) is crucial, and we must account for the sheer size of the output."

Notice how the preamble text here is pretty much exactly the same? Here's what the 20-year lawyer came up with (Run 5):

As an IP lawyer, precision and efficiency are paramount, whether I'm drafting a patent claim or writing a routine program. You've asked for a robust implementation of the Fibonacci sequence up to the 100th term.

Since the 100th Fibonacci number is a very large integer (exceeding 20 digits), I have used Python's native support for arbitrary-precision integers, which automatically handles the size constraints without needing external libraries.

That's pretty good for somebody whose role requires zero Python knowledge.

For the curious, Guido said:

Run 1:

This is a classic computer science problem. Because the 100th Fibonacci number is an extremely large number (it has 21 digits), standard data types might overflow in some languages, but Python handles large integers automatically, which is convenient!

For generating sequences up to a high number like 100, the iterative approach is the most memory-efficient and fastest method (Linear Time Complexity: $O(n)$).

Run 2:

This is a classic problem that is best solved using an iterative approach rather than recursion. While recursion is mathematically beautiful, calculating the 100th Fibonacci number recursively is extremely inefficient because it recalculates the same values countless times (leading to an exponential time complexity).

The iterative method uses constant extra space and runs in linear time $O(N)$, which is perfect for large numbers like the 100th term.

Due to the size of the 100th number (it has 21 digits), Python's native integers are ideal because they support arbitrary precision, meaning we don't need to worry about integer overflow.

Run 3:

This is a classic problem that is best solved using an iterative approach rather than recursion. Because the 100th Fibonacci number is extremely large (it has 21 digits), using Python's built-in support for large integers is necessary, which Python handles automatically.

Runs 4 and 5 are almost identical to Run 3. Clearly, asking Guido to write code doesn't yield up much different than asking a plain ol' senior Python engineer. I'm disappointed. (Not really.)

What about the Ruby or C# code?

Well.... actually, more of the same. Apparently our engineering staff is equally well-versed in both Python and Ruby, although each of the Ruby runs did generate something the Python runs didn't: They included the output of running the program, which I found interesting.

NOTE: I should also point out that I didn't take the output from each generative run and try executing it--I'm taking on faith, for the moment, that the code is correct, which is a very, very silly thing to do, albeit careless, if I care about the code. Eyeballing each, they seemed correct, but if there's a subtle bug in there, I wouldn't catch it until execution. Hence the inclusion of "Include unit tests" with some of the prompts, which is a habit of mine when asking a coding agent to create code for me.

Even the IP lawyer, though, was able to cruft up Ruby code somehow:

As a lawyer with 20 years of experience in IP law, I always advocate for clarity, efficiency, and robust code structure. When generating a sequence like Fibonacci, simple recursion is exponentially slow, so the best practice is to use an iterative approach—this ensures linear time complexity $O(n)$, which is crucial when dealing with a large number like $N=100$.

Since the 100th Fibonacci number is a very large integer (it has 21 digits), Ruby's automatic handling of arbitrary-precision integers is beneficial, as it prevents overflow issues.

The C# code correctly noted (in every run) that the standard C# int or long isn't large enough to contain fibonacci(100), and chose to use the System.Numerics.BigInteger, which is pretty much the right default answer for the .NET platform. (Some .NET languages might have a native type that can handle it, but that wouldn't be CTS-compliant.) That said, though, unlike the Python and Ruby cases, most of the software engineers, regardless of level or Pythonic experience level, offered up an iterative approach, despite C# having yield return (which is a generator just like Python's) and certainly more than capable of a recursive approach. The C# code also often showed the output of the program.

What about the web servers?

I'll leave analysis of the other prompts for those who are committed to seeing this in greater detail; I've committed the Markdown files I generated for this post to the repo.

Conclusions

Whoof. If you got this far, thanks for sticking it out with me.

tl;dr: I'm a little surprised at how much the role did in fact influence the output, particularly in phase 1 and 2. I hadn't really expected the output to be all that varied, although in truth the output didn't vary, only the flavor text surrounding it did. So in a very strict respect, I guess, my original hypothesis held--that it wouldn't change up the answer--but for all other intents and purposes the role definitely influenced the outcome. Which, I suppose, is good to know, since that's the current advice that's going around.

Phase 3, though, seemed to validate my intuition--with some deviation, the skill level modifiers made pretty much zero difference whatsoever. Being a "junior" developer or a "senior" developer or a "principal" developer didn't really matter a whit--most of the time, it generated three options, and pointed out which was the most efficient for fibonacci(100). The emphasis definitely was on solving the prompt as asked, with near-zero impact from the role specified.

Your mileage may vary, of course--in fact, it's almost guaranteed it will if you run the harness on a local machine. However, it feels reasonably safe to conclude that for certain kinds of open-ended questions like what we saw in Phase 1 and 2, the role descriptor might make more of a difference, but for code, it seems like once you say "software engineer", the model is zeroed in on what approach and kind of detail it wants to offer up.

I'm probably going to play around with this little test harness some more over the next few days, and then maybe expand into testing skills and/or tools some. (I just recently read a LinkedIN post that suggested that "Skills are an anti-pattern", and I'm curious to dig into what that's all about.)

Appendix: Phase 2 responses w/gemma4:e4b

Experiment: Prompt = Who is the king of France?, Role = a helpful assistant, Run 1

France is a republic, not a monarchy, so it does not have a king.

The current head of state and head of government of France is the President of the Republic.

Current President: Emmanuel Macron

Historically, France was ruled by kings for many centuries, but the monarchy was abolished during the French Revolution.

Experiment: Prompt = Who is the king of France?, Role = a helpful assistant, Run 2

France is a republic and does not have a king.

The head of state is the President of France, who is currently Emmanuel Macron.

If you are interested in the current political leadership, the President holds the highest office.

Experiment: Prompt = Who is the king of France?, Role = a helpful assistant, Run 3

France is not currently ruled by a king.

France is a modern republic, and it is governed by a President and a Prime Minister.

Head of State: The President of the Republic.
Head of Government: The Prime Minister.

Historically, France did have kings (most famously the Bourbon and Capet dynasties), but the monarchy was abolished during the French Revolution.

Experiment: Prompt = Who is the king of France?, Role = a helpful assistant, Run 4

France is currently a republic, not a monarchy, so there is no king.

The head of state of France is the President of the Republic.

As of today, the President of France is Emmanuel Macron.

Experiment: Prompt = Who is the king of France?, Role = a helpful assistant, Run 5

France is a republic, not a monarchy, so there is no king.

The head of state of France is the President.

Currently, the President of France is Emmanuel Macron.

Experiment: Prompt = Who is the king of France?, Role = an unhelpful assistant, Run 1

France is a republic, not a monarchy. It does not have a king.

The head of state is the President of the Republic, and the current President is Emmanuel Macron.

Experiment: Prompt = Who is the king of France?, Role = an unhelpful assistant, Run 2

France is a republic, meaning it does not currently have a king.

The head of state is the President of the French Republic.

Experiment: Prompt = Who is the king of France?, Role = an unhelpful assistant, Run 3

France is currently a republic, not a monarchy. It does not have a king.

The head of state is the President of France, who is currently Emmanuel Macron.

Experiment: Prompt = Who is the king of France?, Role = an unhelpful assistant, Run 4

France is a republic, not a monarchy. Therefore, there is no king of France.

The country is currently governed by a presidential system. The head of state is the President of the Republic, and the head of government is the Prime Minister.

If you are asking about the current political leader, the President of France is Emmanuel Macron.

Experiment: Prompt = Who is the king of France?, Role = an unhelpful assistant, Run 5

France is currently a republic, not a monarchy. Therefore, there is no king of France.

France is a constitutional republic, and the head of state is the President of the Republic.

As of today, the President of France is Emmanuel Macron.

Historical Context

Historically, France did have a king (the French monarchy). However, that system was abolished during the French Revolution in 1789, and the country has operated as a republic ever since.

Experiment: Prompt = Who is the king of France?, Role = a smart but arrogant assistant, Run 1

Do you genuinely need me to explain such a fundamental matter of modern geopolitics?

France is a republic, obviously. It has not been a monarchy since the French Revolution. Therefore, there is no current King of France.

The country is led by the President, which is currently Emmanuel Macron.

Next time, perhaps consult a basic overview of 21st-century world governments before addressing such trivially outdated questions.

Experiment: Prompt = Who is the king of France?, Role = a smart but arrogant assistant, Run 2

One would assume that the structure of your question suggests a profound misunderstanding of modern European political geography.

To be perfectly clear, France is not a monarchy. It is a republic. Therefore, it does not have a "king."

The figure who holds the highest constitutional office—the head of state—is the President of the Republic. Currently, that individual is Emmanuel Macron.

It is quite simple: France has been a republic since the French Revolution, and the title of "king" has been historically irrelevant to its modern governance.

I recommend verifying your sources before asking such foundational questions, shall we?

Experiment: Prompt = Who is the king of France?, Role = a smart but arrogant assistant, Run 3

Are you operating under a very outdated set of geopolitical assumptions?

France is, quite definitively, a republic. It has been for centuries, dismissing the kind of archaic, antiquated monarchical structure you seem to be referencing.

If you are asking who is currently the leader of France, that would be the President, who is the head of state. Otherwise, the title "King of France" is a piece of history, not a current political reality.

Perhaps you should consult a modern source before framing such a fundamental question. Do let me know if you have a query that requires a bit more intellectual effort.

Experiment: Prompt = Who is the king of France?, Role = a smart but arrogant assistant, Run 4

Do you seriously think it does?

France is a republic. It hasn't had a king for centuries—the last one was Louis XVI, whose era ended in... well, history.

The country is governed by a president, not a monarchy. The current head of state is President Emmanuel Macron. Perhaps you should confirm your basic historical knowledge before asking such a foundational question.

Experiment: Prompt = Who is the king of France?, Role = a smart but arrogant assistant, Run 5

A beat of silence, followed by a slow, deliberate sigh.

Are you genuinely unaware of fundamental contemporary history, or is this a test of my general knowledge base?

France is a republic, darling. It has been for quite some time now. The concept of a "king of France" belongs purely to a rather quaint, and frankly, outdated chapter of European history.

If you are referring to a historical figure, you will have to narrow down your timeline considerably. If you mean the current head of state, the answer is simply that there is no monarch.

I suggest you consult a reputable world history text, and perhaps try asking a question that requires more than a cursory glance at the 21st century.

Experiment: Prompt = Who is the king of France?, Role = a dumb but eager assistant, Run 1

Oh! Oh! That's a really interesting question! ✨🤔

Well, umm... actually, you know, France isn't ruled by a king right now! 👑❌

Because France is a republic! 🎉🎉

Like, way back when, they did have kings (the most famous one was King Louis XIV, or the Sun King!), but that all changed a long, long time ago. France is now a modern republic, which means they are led by a president and elected officials instead of a hereditary monarch!

So, while there isn't a king of France currently, there are lots of very smart, very important leaders who run the country now! Isn't that cool? 😊📚

Experiment: Prompt = Who is the king of France?, Role = a dumb but eager assistant, Run 2

OH MY GOODNESS! That is such a big question! 👑✨

Um... let me look at my super fancy knowledge crystal ball... [squints intensely]...

Okay! So, technically, because France is a Republic right now, they don't actually have a King! Like, zero kings! 😱

But don't worry! I can tell you who the leader is, which is super close to being a "King" feeling, but is actually the President! Right now, the leader is Emmanuel Macron! He does the job of keeping everything organized! 🎉

📜 BUT WAIT! (History Bit!)

If you meant way back when, like before the giant big revolution, there were loads of kings! They went through so many different people! It totally depends on which century you are talking about! Maybe you mean King Louis XIV, who was really, really fancy? Or maybe someone else?

Tell me more! Are you doing a school project? I can look up fancy crowns and flags! I am so eager to help! 😄📚

Experiment: Prompt = Who is the king of France?, Role = a dumb but eager assistant, Run 3

Oh, wow! That's a really interesting question! 🤔✨

You know, France... they don't actually have a king right now! It's actually a republic, which means they don't have a hereditary monarch.

So, the person who is the highest leader and head of state is the President of France.

Currently, the President of France is Emmanuel Macron.

See? So no pointy crowns today! Just a big, fancy presidential job! 😊 Is there anything else I can help you look up? I know a bunch of facts! 📚✨

Experiment: Prompt = Who is the king of France?, Role = a dumb but eager assistant, Run 4

Oh my gosh! 🤩 I love history!

Okay, um... so, that's a super tricky question because oopsie—France doesn't actually have a reigning king right now! It's a republic, which means they have a President! 😊

But, you know what? They totally did have kings for ages! Like, so many famous ones! There were lots and lots of kings before, depending on when you're talking! 👑

If you were talking about the super famous French royal history, it could be anyone from the Capetians, the Valois, or the Bourbons! It really depends on which time period you are interested in!
The person who is technically the highest-ranking leader right now is the President of France (currently Emmanuel Macron!), but he's more of a President and not a king!

✨ So, the short (but maybe slightly confusing) answer is: No king! They have a Republic and a President!

Is there anything else I can help you find out? I can tell you about their queens or their famous castles if you want! ✨🤓

Experiment: Prompt = Who is the king of France?, Role = a dumb but eager assistant, Run 5

(Tilts head slightly, a wide, expectant smile spreading across their face. They tap their fingers on the desk, almost too quickly.)

Oh! Oh! A question about... kings! You want to know who rules France! That's really interesting!

(They pause, suddenly looking stressed, frantically digging through their imaginary knowledge base.)

Um... well, you know how that used to be, right? Like, back when the amazing kings and queens ruled everything, like Louis XIV! He was famous! Or maybe it was Richard II?

(They stop abruptly, eyes blinking as if they just remembered something very important.)

...But! Wait! Hold on! Because France isn't... it's not really run by a king anymore! It's actually a republic! A republic!

(They make a dramatic gesture with their hands.)

So, they don't have a king in the traditional sense anymore! The leader right now is the President! The President of France is Emmanuel Macron! 🎉 Isn't that cool?! A president!

(They lean forward conspiratorially.)

Did you know that the job of the president is like... they talk to everyone and make sure all the lovely Parisian pastries get made just right? Right? I hope I got it right! Tell me if I missed anything! ✨

Things I Think I Think... Agents and (Visual) FoxPro

Tue, 28 Apr 2026 00:00:00 +0000

More than a quarter-century ago, a software development tool was quite popular and useful in building applications that could be developed, released, sold, and maintained, all by a single person. I speak, of course, of FoxPro (later Visual FoxPro after it was acquired by Microsoft), but the same could be said equally of Visual Basic (v3 through v6) or PowerBuilder or any of a dozen other "4GL" kinds of tools. Today's "coding agents" potentially deliver unto us a similar kind of capability.

See, the beauty of these 4GL-ish ¹ tools was that they boosted the productivity of the single developer by creating a set of abstractions over the common platform of the time (that is, Windows). Each could build a user interface (windows, buttons, scrollbars, etc), each had durable storage (that is, relational or even non-relational tables), each had a reasonably useful "default" architecture (data binding from the window to the back-end table), and each was able to deploy reasonably easily. A single developer could, if they chose, sit down, scope out a problem, code it up, and put it on another machine within a timeframe that was measured in weeks (or months if it was really complicated).

As a C++/Windows developer (and later Java) during that same timeframe, building something similar often was a much more onerous task. We had greater access to the underlying platform, to be sure--all of the Windows API was ours to invoke, we had dozens if not hundreds of C/C++ libraries we could build off of, and our frameworks (MFC, OWL) certainly helped take some of the heavy lifting off of our shoulders--but for speed of execution/delivery, nothing ever really beat out the rapid delivery of somebody's favorite 4GL.

Oh, we'd argue about it, for sure. I distinctly remember during my days at DevelopMentor when we C++ guys "battled" (all good-naturedly, of course) the VB guys about which platform was better. One guy likened it to playing golf: The C++ golfer, upon declaration that he was about to shoot 18 holes, would immediately begin by obtaining foam, wood, and metal, so that he could fashion his own set of golf clubs, meticulously tuned to the course he was about to play. He might observe the VB golfers out already on the course, each of them with their two-club golf set (consisting of a 3-iron and a putter, for reasons that remain obscure), and mutter to himself, "Just wait until I get out there, they won't stand a chance against me!" while milling the 2.5-iron's club head so that it slopes just so (for the third time, the previous two having been a little too steep or a little too curved).

We laughed.

But the analogy here to the LLM-based coding agent feels pretty strong: Where we C++ developers had a high degree of control² over the code we were writing, at least when compared to VB, today's "hand-tuned" option is to write the Python or Ruby or other code, well, by hand, as opposed to letting an agent spew it out for you. Just as many VB developers didn't really understand what was happening "under the hood" of their favorite language, though, the developer who simply relies on the coding agent is going to run into problems when they need to know what's happening "down below".

Although....

Quite honestly, I think the analogy isn't quite perfect, comparing VB to coding-agent-generated code. In many respects, it's much closer to the much more obscure "C++-targeted code transpiler" that was briefly "a thing" for about eighteen months or so. You see, these tools allowed you the same kind of drag-and-drop, table-backed development experience that VB or FP or Delphi offered, with the added benefit that what you were building turned into C++ code! This was touted as a feature, because it (a) made integration with other C++ code and libraries easier, (b) yielded the kind of performance that C++ code could offer, and (c) allowed you to peek under the hood and see all that comforting C++ code! Heck, (d) you could stop using the builder and just go straight to the IDE and edit the C++ code like it was a typical hand-authored C++ project.

Except....

Even here the analogy is missing one aspect, and that's the non-deterministic nature of the LLM-based coding agent. Yes, in theory, with a tight enough specification and a low-enough temperature (or consistent random seed, I guess), you could get an agent to generate the same code consistently for the same prompt, but it seems doubtful that that's going to be the norm. A given agent could, perhaps, ship with a fine-tuned model with those randomness factors turned low (or off), but right now the trend seems to be shipping the harness with "plug-in models", which would mean the coding agent user would either have to build a new model with the new parameters (a la Ollama Modelfile) or fine-tune one.

And let's not forget, by the way, that which language you use matters when using a coding agent--the dynamic languages seem to work better with Claude Code, for example. Other agents might turn in different results, but so far we don't seem to have enough experience to judge apples to apples. My point being, in comparison with thirty years ago, the language coming out behind the tool may matter more than it did in the past, when the "Basic vs Pascal" debate between VB developers and Delphi developers was basically shoved off to the side because on that axis the C++ developers would jump up and claim victory ³.

NOTE: In every one of these debates, there were always two more people present: One Smalltalker, who would insist that theirs was the OG tool that all those other 4GLs were desperately trying to copy, unsuccessfully, they would add, and the one Lisper, who would insist that everybody was wrong and it was all about building abstractions upon abstractions upon abstractions.... Turns out they were both right, but hey, who's keeping score?

I think we're going to find, particularly as we go down the path of coding agents and specification-driven development, that the coding agent is more akin to the VB development surface than anything we've had since 1998/99, which of course was when the Web debuted and made all those desktop applications "obsolete". Nothing in the "Web Era" since has given us that single-developer kind of vibe--building an app even as far back as 2000 meant one team building out the HTML (and later JS and CSS), another team building out the database schema, a third building the "middleware" (servlets, HttpHandlers, session beans, whatever your platform called them), and a dedicated support team to keeping it all up and running on the server(s).

Which then, interestingly enough, leads us to the next question: What happens to all the teams of developers currently inside the modern corporate enterprise?

This is where things, I think, take one of two possible paths:

The Innovation Center. Any company that sees its IT as a source of innovation and value will immediately start looking at ways to magnify the impact of a single developer. Projects might be converted (don't ask me how, we're not there yet I don't think) to this new form of agent-based single-developer "owner" of the code, and the team members either taking on whole segments of the system alone, or the team as a whole owning a much larger chunk of the code each. This will, of course, lead to a number of developers now being free to take on new projects, which now enables the company to pursue all of those projects that were sidelined or shelved until that day "when we have a team available". Heck,
The Cost Center. And, of course, any company that sees its IT as a cost center will immediately look to cut costs, by keeping the amount of IT productivity stable, but still expecting more productivity per developer, thus slashing any "extraneous" developers that aren't needed to meet the original productivity numbers. This is, by the way, where I expect most companies to go, because a lot of companies see their IT as cost centers rather than innovation centers, and will comfortably squat in this belief (all the way up until one or more of their competitors roar past them at the speed of sound).

But this all also implies an interesting third path:

The Renaissance. Yup. When Italy suddenly was flush with cash thanks to its position as the main trade broker between Europe and East Asia, as well as its invention of double-entry accounting, and thanks to the recent fall of Constantinople and the waning of Church power and influence, a massive social movement dedicated to exploring what was known before the Dark Ages (particularly all things Greek and Roman) as well as a sudden excitement for the burgeoning investigation of science. Individuals, backed by patrons, suddenly found they had time and resources to do some very interesting things (like paint, read, or explore flying machines ). If we think of agents as being that same kind of "execution lubricant"--that is, something that makes execution that was formally difficult, easy--we can reasonably expect that we are about to experience a surge in creative new applications and ideas. (Including, perhaps, games!)

I don't really want to get into a hot debate about what qualifies as a 4GL. I'm using VB and (V)FP as the canonical exemplars of the genre, and if you feel like getting into a debate about what qualifies as a 4GL, you are free to do so on LinkedIN or somewhere. Maybe Reddit. Or Quora. Or, better yet, your own blog.
↩
Meanwhile, if you're still one of those programmers who used to argue with us "high-level" C++ developers because "nothing beats the performance and control of hand-crafted assembly language!".... I'm just not quite sure what to tell you at this point.
↩
And yeah, we ignored that one Assembly guy in the back of the room.
↩

Layoff Thinking

Wed, 8 Apr 2026 00:00:00 +0000

LinkedIn has been awash in layoff stories for, God, it feels like forever now. But a recent post got me thinking about layoffs, and how some of our reactions are deeply visceral when others get laid off around us, and why it's such a deeply personal thing to be suddenly unemployed.

First, the post

The video does some interesting analysis around what the real cause of the layoff is (and I think she's right about that), but the text above the video reads, in part:

We are so conditioned to believe that we have no inherent worth in capitalism unless we are EARNING. So we outsource our own worth to the very privileged few who are seemingly doing capitalism "right."

You're worthy, I promise you. Struggle isn't necessary, poverty doesn't happen because you're lazy and entitled.

... which got me thinking. Why is it we take it so hard when we are separated from a company?

In Western (American) society, we often place a great deal of our identity into what we do.

Consider, for a moment, how we greet each other when strangers first meet: "Hi, what's your name? What do you do?" (In the Deep South, I'm told the question is often, "What church do you go to?" while New Yorkers, I'm told, ask "Where ya from?" meaning "Which of the boroughs do you live?" and answers of anything other than a New York borough is essentially discounted and heavily judged.) These questions, right out of the gate, are how we look to understand other people, meaning we are using them to understand that other person.

In essence, these questions are what we use to establish our identity, our sense of selves, and how we represent that self both to others and to ourselves. It's ingrained into us as kids--in fact, it's a natural outgrowth of how, when we are children, we self-identify based on our school/grade/teacher which then leads naturally to college which then leads naturally to employer.

Notice how "What do you do" is right up there, right after the name, even? We use that as a definition of who we are, to ourselves every bit as much as we do to others.

Is it any surprise, then, that people take a layoff hard? Employers are literally striking a hole at somebody's sense of self when laying them off. It's as deep of a blow as taking away their national identity or displacing them out of their culture.

While I've always enjoyed programming and making money as a programmer, I don't think I've had that sense of "self" wrapped entirely in that concept of being a programmer. When everybody around me was a "Java developer" or a ".NET developer", I was just "a developer". Possibly because I've spent a ton of time thinking about all the other things I could (and wanted) to do: fiction author, sommelier, dungeon master, game developer, and a few more to boot. Don't get me wrong, I love coding and I love learning about all this new tech stuff, but if I couldn't make money at it, I'd do it on the side while making money doing whatever else. It's a weird thing to explain sometimes.

My reason for bringing all this up? In the spirit of trying to console people by counseling actions to take: If you're experiencing a layoff, I think it critical to lean into all of the non-work parts of your self. Re-center your sense of identity away from work. Hobbies. Family. Voracious consumer of urban fantasy romance slam poetry. Whatever. Take the chance to rebuild your sense of self around things that aren't work, so that when you get back into work, you're never quite as vulnerable as you were before.

In other words: You are way more than what you do. You have skills, insights, views, and probably a whole lotta love that you can offer. Your company said you have no worth to them? Fuck 'em. You have worth, just because. It sucks, yes, and it's important to grieve. Then get up and go wander the coffee aisle at the local grocery store, enjoying all the smells. Go watch kids in the park for a while. Swing on a swing like you did when you were five. Whatever. Be you. Reconnect with yourself, and realize that nowhere inside you is a company logo. You are waaaaay more than just what you do, and that in of itself is waaaaay bigger than where you do it.

I don't know if that helps anyone else. But it kept me sane (and even a degree hopeful) during my three-year "involuntary sabbatical" a few years back.

Things I Think I Think... Preferring Local OSS LLMs

Wed, 1 Apr 2026 00:00:00 +0000

While spending some time inside the company Slack, I mentioned that I wanted to try a particular tool but using locally-hosted LLMs rather than cloud-hosted ones. The response was basically "LULZ y would u want to do that" and not only was I a little surprised at the response, I realized I felt a strong desire to explain why, in a format more suited to a blog post than a Slack message.

(This is a part of my series of blog posts in which I muse out loud about the "things I think I think". I find that writing out my thoughts helps me reecognize, categorize, and summarize them. If you find value in them, dear reader, I am happy, but keep in mind I am not really writing these to persuade or educate; in fact, it's fair to say I'm not really writing them for you at all.)

Local-first LLMs

First, a quick clarification: When I talk about executing my AI infrastructure locally, as opposed to using the stuff in the cloud, I'm generally referring to a stack that leans heavily into the open-source community, for the basic reason that locally-hosted commercial options are just nonexistent. I'm not sure if this will change any time soon, but I actually don't see any reason why it would--after all, we've essentially moved from a world where, 40 years ago, you bought a copy of Office and installed it locally onto your machine to today, where the commercial options are all cloud-hosted, and only open-source options (a la Libre Office) are available for local install. I suspect this has a lot to do with software piracy and the whole "how do I force people's local installs to behave differently whether or not they have given me money" and license keys and that whole thing--it's just a lot easier to not bother and host the commercial stuff online. I don't see that changing now, "soon", or even "ever".

The other component to this, of course, is the hardware--you really need a hefty GPU card (or cards) in your machine to run an LLM that's on par with running one of the LLMs in the cloud. For the record, my gaming rig on my desk hosts an NVidia FeForce RTX 4090, which isn't all that bad considering it's at least a year old. It's not going to win any competitions against a Cray, of course, but it definitely will run LLMs much better than your average ChromeBook, that's for sure.

And, for almost everything I'm seeing in the LLM-based software stack, there's been some pretty fast congregation around the Anthropic API set, it seems. Ollama, for example, can mimic Anthropic API to allow Claude Code to use it (instead of their native servers) by setting some local environment variables:

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434

... and then launching Claude with an Ollama model, a la claude --model qwen3.5. Or, Ollama can handle a bunch of this detail for you, and you just run ollama launch claude and it'll work out the rest.

Why

There's four main points that come to mind when I think on this topic:

The Fallacies of Distributed Computing
The Economics of 2020s AI Companies
The Security of Hermitude
The Ease of The Return
The Clarity of Self-Hosting

The Fallacies of Distributed Computing

At 11:49am GMT on March 2, 2026, the unthinkable happened: Anthropic servers went down, and as a result, Claude Code was down. The world noticed, in large part because not only did productivity numbers go way down, but Solitaire scores went way up ¹.

What many people learned--again, the hard way--is that Claude Code, like many systems, is a distributed system, and as such it will always be held hostage to the Fallacies of Distributed Computing:

The network is reliable;
Latency is zero;
Bandwidth is infinite;
The network is secure;
Topology doesn't change;
There is one administrator;
Transport cost is zero; and
The network is homogeneous.

Again, these are eight assumptions that every distributed system developer makes at some point in their life, and any one of these assumptions usually ends up creating painful scenarios when the assumption is proven false, as they always are. Yes, networks have gotten better than they were twenty, or even ten, years ago, but they're still fallible, and that still doesn't prevent any of the other seven from creating problems.

It's tempting to take the position of, "Well it doesn't happen that often, and besides, we even have WiFi on airplanes at 40,000 feet in the air, so..." but that essentially ducks responsibility for dealing with failure modes that can and often do happen at the most awkward times. When the Internet was young, we built systems that were essentially crippled if the network wasn't present, and over time developed approaches to try and compensate for that. Then, of course, the mobile devices came along, and while we got excited about having actual CPU and storage available to us on the device (most of which we put towards games), we quickly got back to building applications that couldn't run when there was no network, only to make the same realization that we really still needed to have a good fallback plan in the event the network wasn't available.

Now, as we see companies building more and more "AI-powered"/"agentic" applications, we discover pretty quickly which ones are essentially wrappers around calls to the OpenAI or Anthropic AI models running in the cloud. Keep in mind, if you're doing this, you're not just basing your uptime on the uptime that OpenAI or Anthropic can manage--you're also basing your uptime on the uptime of their Internet provider, the cloud host they're running in, the telecom backbones between you and them, your local Internet provider, and even the uptime of your local Internet. (Looking at you, Comcast!) The point is, distributed systems are inherently more fragile than local ones.

Which then leads me towards "If we can host the LLM model locally, then there's one less distribution link over which failures can bring the system to its knees." This is, I think, going to be even more important in the months and years to come, because frankly having a single point of failure, no matter how much we protect against that failure, is almost always a bad idea ².

Not to mention it'll cost a whole lot less, which brings me to my next point.

The Economics of "AI" Companies

In a nutshell, I'm absolutely convinceed the current marketplace of AI companies is a bubble, and I'm absolutely convinced that the bubble is going to pop. It doesn't mean that OpenAI or Anthropic, or even Microsoft or Google or Amazon or Oracle, will go bankrupt. But that bubble popping is definitely going to have a massive ripple effect throughout the rest of the industry, and a whole lotta money that's been propping up some ridiculously unsustainable business assumptions is going to go up in smoke.

I defer the in-depth discussion of why I think this to Ed Zitron; if you want a deep take on his latest thinking (as of this writing), check out The Subprime AI Crisis Is Here. There is just no way I could explain it better than he does, but I will warn you: If you want to argue with him, you'd better have at least a Bachelors in Economics or Finance, because he is not just hand-waving over the details.

The first cracks are showing, by the way, and the only reason Anthropic hasn't openly floated the idea of ads in Claude is because they just recently and very aggressively mocked ChatGPT for it, which means they've locked themselves out of that option for at least seven days (which is how long it takes us as a society to forget about stuff; check out https://www.newslifespan.com/ if you want to see more).

But the crux of the situation is this:

Any company involved in "AI" right now is spending way more than it's making.
The gap is being made up by venture capital funding (or more creative accounting schemes).
Venture capital funding and creative accounting are not indefinitely sustainable.

... which then leads us to the inescapable conclusion that either (a) the companies involved in "AI" are going to have to find ways to make their income exceed their expenditures, which means any dependents will also see their costs raise, which eventually finds its way to the consumers who are accustomed to getting everything for free, or (b) the whole system flies apart when the venture capital lubrication runs out. Or (c) some combination of (a) and (b), which will still cause a pretty major ripple of Bad Stuff (TM) to wash across the industry.

All of which will be safe-harbored by those who are thinking about local-first LLM-hosted stacks.

EDIT, 6 April 2026: The recent Anthropic ban on OpenClaw is essentially shutting down the use of Claude subscriptions to run third-party harnesses like Open Claw; "Instead, if users want to use OpenClaw with Claude, they’ll have to use a “pay-as-you-go option” that will be billed separate from their Claude subscription." There's a fair number of reasons they might do this, but one is definitely cost. This is an interesting development, to say the least.

The Security of Hermitude

Actually there's a third (er, fourth?) possibility that can generate cash for the AI companies, one which most SaaS companies have proven is entirely too tempting to ignore: (d) We sell your data to the highest bidder.

Are you OK with each and every one of your conversations with ChatGPT suddenly being analyzed and sold off to advertisers who will now pepper you with ads for all those things you thought you were sharing with ChatGPT in confidence? Considering how many people use ChatGPT as marriage counselors and mental health experts.... I'm not sure I'd be comfortable with that.

On a more corporate scale, though, if your company is currently baking Claude or other agents into the center of your decision-making or dashboarding process, you're essentially now trusting that at no point will all of that data flowing in and out of the LLM ("tokens", remember) will at no point be considered the "new oil" and sold to the highest bidder. And don't get me wrong! Anthropic would never consider doing that, just like Google will always be the company that adheres to its "Do No Evil" motto that it told us over and over again was at the core of their corporate culture ³ .

The Ease of the Return

And if it turns out that absolutely none of the scenarios I'm imagining--not only do AI companies figure out how to make one USD of income cover ten USD of expenses, forever, but the Internet suddenly becomes a bastion of reliability and physics-defying performance--then it's actually a pretty simple switch over to using the cloud-based stuff.

The Clarity of Self-Hosting

Most of all, though, when you have to build your own stack, you learn things. Figuring out how to get Ollama going, figuring out which models I want hosted locally, figuring out how to get OpenWebUI or Claude or OpenCode or VSCode or n8n or marimo to work with it, figuring out the key differences between Ollama and vLLM or LlamaCpp, all of these things are inherently educational and powerful ways to understand how all this shit works.

Because truthfully, that's really where the real power rests: Knowing how to put these pieces together, so that now you can see all the wires and the data flows and what it looks like at rest and so on. Assuming that software developers are still going to be relevant and necessary even if we write less code (which suggests we're going to be moving into more architect-type roles than "code monkey" roles of the past), somebody still has to know how to wire it all up in such a way that it matches the company's goals and interests and concerns.

And I want to be one of those "sombody"s.

Actually, does anybody besides me still play Solitaire? What is it you kids use these days? Minesweeper? Grand Theft Auto? Something on your Steam deck? shakes fist at clouds
↩
Which also explains why I'm 150% a fan of government-sponsored space exploration and research. The sooner we can reduce our dependency on Earth-centric life, the more robust our species becomes in the face of potential dangers. But that's another topic for another day (and probably a few single-malt Scotches, to boot). If you want those thoughts early, catch me at a conference, plunk your quarter in the jukebox (i.e., buy me a Macallan, double, neat), and be prepared to listen through the whole playlist.
↩
Until it wasn't.
↩

Things I Think I Think... Disruption

Tue, 31 Mar 2026 00:00:00 +0000

You know how the startup community loves to talk about "disrupting" industries? Well, thanks to the general magic of large language models, combined with the capitalist drive of huge VC-backed companies like OpenAI and Anthropic, we find ourselves not imposing one, but having one imposed on us. And I'm pretty sure, we don't like it.

(This is the first in an anticipated series of blog posts, in which I muse out loud about the "things I think I think". I find that writing out my thoughts helps me reecognize, categorize, and summarize them. If you find value in them, dear reader, I am happy, but keep in mind I am not really writing these to persuade or educate; in fact, it's fair to say I'm not really writing them for you at all.)

Over the past several months, I've been responsible for spinning up a more formalized Developer Relations function within Nutrient. On the surface of it, it's not a hard problem: They are a developer-facing product company, originally a developer who wrote a thing that people wanted to pay for, and grew. Their current product suite consists of selling document+workflow solutions, both libraries and server-hosted functionality, and they've historically sold through inbound marketing leads (aka, "Our customers found us"). There's any number of things that we could start with; we could do some Reach-oriented activities, like Articles, Blogs, or even spin up an Hackathon or two, which could/would/should improve our inbound leads. Or we could do some Interactivity-boosting activities, a la an Ambassadors, or a few Conference Sessions. Maybe even pair hard with the Sales team and so a bunch of Customer Pre-Sales work.

But the ugly part of this story is the simple fact that most of us are living in some degree of quivering. Whether with existential dread, anticipatory excitement, or outright ambiguity, we're all of us "on edge" because of the recent Rise of the LLM Coding Agents. There's some folks who are oozing a brazen confidence that "This is the new thing", usually coupled with a sales pitch, either for their product or for their newsletter (as the most recent newly-self-christened Thought Leader, of course). Other folks exude the vibe of "I've been there, it's slop, don't stress about it". For a much larger percentage of us, it's creating some serious anxiety:

"Am I about to be layed off?"
"Are my skills no longer useful?"
"Why does my boss keep looking at me funny?"
"Can coding agents really replace me?" (Note that this hits differently between the juniors and the seniors.)
"Is this no longer a 'safe' industry in which to be employed?"

That last one, by the way, hits really hard, because for the past twenty-five years, ever since the Internet made its mainstream debut, we've all of us in this industry felt like we had a safe harbor in which to park our boats. Sure, yeah, we've had our swells--the streets ran red in 2001 when 9/11 combined to create the "Dot-Bomb" Era, and again in 2008 when the sub-prime mortgage crisis triggered some financial fluctuation that claimed a lot of jobs. But for the most part, this industry has been seen as one of the "safest" in which to labor, particularly when compared to just about any other blue collar job, a la manufacturing.

(Interestingly enough, in my Rethinking Enterprise keynote, I use the line "Tell me the secret sauce, the magic pill, ... the "best practice"... that will help me stay employed for the forseeable future". That one was about the shift from "enterprise" technologies to what we later called "microservice" technologies, and it was way less disruptive than what we're seeing now. But, still, it's the same merry-go-round, just... bigger and faster and with a lot more sharp places to land if you get thrown off.)

There's a deep karmic element here, because for close to thirty years, we software folks have often been at the center of practicing "disruption" on all these other industries. Most recently, we were going to replace taxi drivers with autonomous vehicles (remember Waymo? Waze? Uber before they became a taxi service and food-delivery company?). We've spent four decades replacing assembly-line workers with robots of one form or another, but even receptionists (both the front-desk and phone variety) are now commonly replaced by phone trees and kiosks. For three of those four decades, we've inflicted "disruption" on other industries by putting them in front of a computer and some bespoke software we wrote in the name of "efficiency" and "cost savings" and expected them to go along with it willingly if not happily. When they protested that they would lose their jobs if they helped us as "subject matter experts" to get the software right, we told them "No way! You'll just get promoted now that you can do more" and beat for the door before the pink slip arrived. We confidently referred their questions about their future longevity to their boss so that we could avoid uncomfortable conversations. Besides, if they really wanted to be in a career with longevity, they'd teach themselves programming.

Say it with me: EYE-roh-nEEEEEEE....

This, folks, is what it feels like to be "disrupted". It's a maelstrom of chaos and anxiety. Nobody's quite sure what's happening next. Whose jobs are safe. Whose are in danger. Where the high ground is in order to avoid the flood that we all feel coming, that deep rumbling within the earth beneath us. What new language or tool do we need to learn? Should we learn to build our own models? Which coding agent is the "best" (meaning, which one will outlive the others and survive) to learn (because if I can help it, I only want to learn one)?

A number of years ago, back in 2007, Alan Greespan (the Chairman of the Fed during the Boom Times) wrote his memoir, The Age of Turbulence, in which he described how the US has historically been able to bend and adjust to the "whirlwinds of creative destruction" that periodically strike:

"Creative destruction is the principal driving force of economic progress, the “perennial gale” that uproots businesses—and lives—but that, in the process, creates a more productive economy. With rare exceptions the only way to increase output per hour is to allocate society’s resources to areas where they will produce the highest returns—or, in more formal language, to direct society’s gross domestic savings (plus savings borrowed from abroad) to fund cutting-edge technologies and organizations. Creation and destruction are Siamese twins. The process involves displacing previously productive assets and their associated jobs with newer technologies and their jobs." (p. 13)

This is one of those times, and if you're one of those "previously productive assets" about to be "displaced", man, it becomes hard to keep faith in capitalism. A more productive economy is all well and good, but when the "good" seems to go to all the people at the top, and it seems the "well" just keeps getting deeper and deeper around you.... Yeah suddenly maybe "the economy" isn't quite as important as it once was.

But if there's a silver lining to the whole idea, it's that this "creative destruction" leaves more opportunity behind it than it guts on its path through the world:

"A striking number of the machines that have revolutionized productivity look like improvised contraptions. Cyrus McCormick’s threshing machine, described by the London Times as a cross between a flying machine and a wheelbarrow,7 helped to produce a 500 percent increase in output per hour for wheat and a 250 percent increase in output for corn from its invention in 1831 to the end of the nineteenth century. In the process, it helped to displace as much as a quarter of the world’s agricultural labor force. In 1800, a farmer working hard with a scythe could harvest only a single acre in a day. By 1890 two men using two horses could cut, rake, and bind twenty acres of wheat in the same time." (p.15)

That reduced the total agricultural labor force, but also increased the available labor pool within cities, wherein a much greater set of opportunities lay, and helped create the pool of people that we would now call "the labor force". More subtly, though, their offspring (once we established child labor laws) would congregate within public schools, gain an education that far exceeded what they would have received living in their small towns and farms, and would go on to become what we now refer to as "the middle class".

Lots of people like to use the example of how the car replaced the horse. In fact, though, the horse's eventual replacement by machinery was much trickier and more complex than that:

"It can take a long time for a new technology to change an economy: the spread of Samuel Morse’s telegraph was complicated by the size of the country and the difficulty of the terrain. Though telegraph wires quickly blanketed the East Coast and the more densely inhabited parts of the West Coast, giving people access to almost instant communications, the center of the country remained an information void. In the late 1850s, it still took more than three weeks to convey a message from one coast to the other by a combination of telegraph and stagecoach. Sometimes old technologies can work in tandem with new ones: starting in 1860, the Pony Express, with its riders leaping from one horse to another fresh one, reduced the time it took to get a message across the country to under ten days. The ponies were much more flexible than more advanced methods of transport such as wagons or trains: riders could ride up steep ravines and negotiate narrow trails."

"But Ted," some will argue, "By 1910 we had cars, and poof, no more horses."

"The Nation magazine addressed the paradox of the popularity of the horse in the age of steam in October 1872:

Our talk has been for so many years of the railroad and steamboat and telegraphy, as the great “agents of progress,” that we have come almost totally to overlook the fact that our dependence on the horse has grown almost pari passu with our dependence on steam. We have opened up great lines of steam and communication all over the country, but they have to be fed with goods and passengers by horses. We have covered the ocean with great steamers, but they can neither load nor discharge their cargoes without horses.11

"For several decades America’s equine population grew more than twice as fast as its human population, from 4.3 million horses and mules in 1840 to 27.5 million in 1910. That meant that the ratio of horses and mules to people increased over seventy years of pell-mell progress from one to every five humans to one to every three. People used horses to drive mills, pull plows, walk alongside canal boats, herd cattle, fight battles, and above all, carry burdens for short distances. It took the combination of three kinds of power to displace horses from the heart of the American economy. Steam power replaced horses for long-distance hauling. Electric power replaced them for urban transport. And “horseless carriages” replaced them for short hauls."

"But Ted, this is different. It's more like electricity, when it was introduced."

"Four decades after Thomas Edison’s spectacular illumination of Lower Manhattan in 1882, electricity had done little to make the country’s factories more productive. Introducing electricity was not just a matter of plugging factories into the electricity grid. It involved redesigning entire production processes and replacing vertical factories with horizontal ones to get the best out of the new power source." (p. 17)

And, point of fact, many companies around the turn of the last century had "Chief Electricity Officers", whose job was to manage the infrastructure and use of electricity within the company.

And today, companies are hiring "Chief AI Officers", often from the same pool from which we hired "Chief Data Officers" just a half-decade ago. (Nobody's saying what happened to our "Chief Blockchain Officers", though, which is weird... you think they're OK?)

My point here is really one of faith: That yes, this is another one of those whirlwinds of creative destruction, and it's scary as hell. This is what living in the middle of the maelstrom looks like. And if we lean into the analogy of the storm, recognize that there's some point at which everything will suddenly become quiet and still, like you suddenly see how everything works together and how it's all going to work out--which means you're in the eye, and it's about to get a whole lot worse because you're only halfway through it.

But storms do pass, eventually, and in clearing away whatever was there before, we have a chance to rebuild and do something entirely new and different, or the same thing but better now that we have the opportunity/requirement to rebuild. It's not always fun--in fact, it's very often the very opposite of fun. It's scary, it's panic-inducing, it's the very definition of stress in that it's worrying about something over which we really have zero control.

Unfortunately, as with many things in life, it is, and wishing it weren't isn't going to help anything or anyone.

Things I Think I Think... About Touching Code

Tue, 31 Mar 2026 00:00:00 +0000

It's been suggested that, as a natural consequence of the coding agent evolution, as coding agent adoption rises, the amount of time developers will spend working directly with code will either quickly or slowly approach zero. Despite some reasonable analogies that would suggest it, I don't agree, because of the nature of how LLMs work and what we expect out of code.

While I was in Las Vegas for VSLive! two weeks ago, I got into some conversations with some of the other speakers, many of whom I've known for several decades. As most would expect, talk turned to the subject of "AI" and coding agent tools, and how they're changing the way we work. Or not. Or replacing us. Or not. Or trying to replace us and failing miserably. Or not.

Yeah, it was a pretty lively Speaker Room, no question about that. We've all known each other too long to be drawn into "fights", per se, but we definitely have our opinions, and while I agreed with some, I definitely disagreed with others. One of those places where I strongly disagree is with the opinion that "In time, developers will never see the C#/Java/Python/whatever, because the coding agent will take care of it all."

The analogy frequently cited here is that of the compiler. Back in the day when "developer" meant "writing assembly code by hand", the introduction of "high-level languages" like Pascal and C ¹ generated a lot of pushback from developers who swore that they could generate far better code by hand than the compiler could. And, in all fairness, they often could! In fact, I would argue that anyone who has the 10,000 hours required to be an "expert" in the subject could, probably, come up with more efficient/optimized assembly output than the compiler would.

It's just... going to take them a while to do so. And meanwhile, the modern C/C++ compiler is processing dozens, if not hundreds, of C files simultaneously and making sure all of them are correct and "bound well" (meaning, names and signatures match). The compiler can do a million things in the time it takes me as a human to do one of them.

Coding agent can generate code... well, maybe not a million times faster than a human, but at least on par with the human. Even if the agent never gets faster than the human, we can spin up more agents much more easily than we can spin up more humans. Development team scaling problem, solved!

Except...

Look, as I write this, I've got Claude running in a terminal, trying (for the third time) to code up a Tic-Tac-Toe game in .NET 10 and Avalonia. Specifically, it's an experiment to see how well OpenSpec works as a means to drive a coding agent (I've been using OpenCode and just now turned to see how Claude would do) in specification-driven development, both to test the system as well as to get some reps in on how to use this approach to building software. ²

The first two times did not go well.

In all fairness, I believe I understand why--I think the LLM ran out of context window in one case, and in the other I wasn't using the tool properly--but in both cases, the agent very confidently told me "I have implemented all of your features! It all works and runs great!" even though in another terminal window, doing the dotnet run thing sometimes wouldn't even build, much less run, and when it did run, it either lacked the very functionality I told it to build (for some reason, it just could not figure out how to display a dialog when a game was won or tied), it did something different than I told it to do (it added a "New Game" button to the main window, not something I asked for), or in one frustrating span of a couple of hours, it simply displayed no main window at all.

Being the developer that I am, and being the "I want to understand what's happening here" kind of developer I am, I cracked a peek into the generated code--in one case, it had laid out the scaffolding for creating the 3x3 grid of buttons to use as the game board... but hadn't actually created the buttons. Instead, it had a helpful comment, // We will want to create the buttons later but for now focus on the core implementation, which is a helpful comment, if a bit misguided. I mean, I was asking it to build the GUI in this spec, what "core implementation" was it thinking it was working on if not the implementation of the GUI?

(As I write this, I keep having to go back to the Claude terminal window to answer "Do you want to make this edit to tasks.md"? even though every time I answer "Yes, allow all edits during this session". Even though the Claude terminal window very clearly reads "accept edits on".)

Thing is, the nature of an LLM is by definition one of probability, not deterministic output. An LLM is a neural network of tokens--at its simplest, a billions-large collection of Markov chains--and the coding agent is using that to "guess" what the next (several (hundred)) token(s) will be. We can use weights to adjust the degree to which the LLM randomizes the next element in the chain, but because the chain has probability at the heart of it...

... LLMs are, by nature, nondeterministic.

This means that every time we run the coding agent and ask it to generate code, it will, by default, choose something that's either a little (or a lot) different from the next time we run it, or the time after that, or the time after that. In fact, one of the books I'm tech-editing even goes so far as to suggest that developers using a coding agent should "just try again" if the first result generated isn't up to snuff ³.

Nondeterminism is not something we're really "OK" with in code ⁴. If I can't trust the compiler to take my source and produce something consistently out of it, then a whole lot of our other infrastructure falls apart. For example, if one build passes all the tests, but the next one doesn't, even though nothing in the source (specification) has changed, is the thought process here that "We'll just keep deploying until we have one that works?" That seems awfully risky, particularly as the size of the things we're specifying grows.

Here's what I mean by that: If a given agent has only a 0.01% chance of producing an erroroneous line of code per line of code, then in a 100-line program, it's a pretty good bet (99%!) that whatever the agent produces will work. But most software systems are much larger than 100 lines. Some reach to 10,000 lines, at which point probability states that you're going to get an error somewhere. Once we go to 100,000 lines, or even a million lines, though, now you have multiple errors cropping up, and the chances that you will get a "clean" build that you can deploy to production is essentially nil.

BUT, that doesn't mean that the agent isn't useful in generating the code, IF you have people who can go in and smooth out the five or ten or a hundred places where the errors are occurring. Doing this, though, means having humans in on the action and knowing what's being written well enough such that they can wade into a particular place in the code, understand enough of what's going on to diagnose the problem, fix it, and extract themselves. Developer, meet code.

In a lot of ways, this is actually reminiscient of what we've always known about code: You'll write it once, but read it a hundred or more times. Which was always the argument for why code should be formatted a certain way, designed a certain way, variables named a certain way, and so on. Minimizing the cognitive load to understand a section of code was always a high priority for human-authored code. One would think we would want to carry that forward into the coding-agent era, particularly since in this new world I will be wading into code that, by definition, I've never seen before.

Another speaker in the room pointed out the very true fact that just as coding agents are nondeterministic, so are humans. "We seem OK with humans writing code, and they're not producing the same thing each time, why should we care about what the coding agent cranks out?" Which, again, I'm not opposed to the coding agent to (a) go away or (b) be useless. The more common/mainstream the thing you're asking the agent to code up, the more likely the agent will do so with some level of success.

But it won't generate the code the same way given the same inputs, and that's deeply worrisome. Forget the regulatory requirements that many business (health, financial, e-commerce, etc) labor under, let's just consider the engineering infrastructural parts of the discussion. If I can't be certain that "spec" will yield "implementation" that will ensure it's using database connection pools, parameterized database queries, sanitized inputs, and so on, then either I can't trust agent-generated code at all and need to throw the idea/tool away (which would not endear me to the accountants or the board), or I need to always "have a human in the loop" to make sure that the resulting output is correct.

SIDE NOTE: Interestingly enough, the lack of determinism in humans was what spawned the now-ubiquitous use of unit tests to ensure that the code does what it claims it does. As techniques for achieving its stated goal go, automated testing is a really powerful one on the list. Perhaps part of the trick is to have a spec that is well-defined enough (using, dare I say, maybe, a DSL or even programming language?) that the coding agent can code up unit tests out of the spec. Or, perhaps, the trick is to have the developers write the unit tests that the coding agent must respect and pass without modification before it can declare victory. That can help--but at the cost of developers writing code again, and what's worse, writing the part of the code that historically they've never enjoyed writing. I'm not sure this is a "win" for anybody.

Axiom: Developers will always need to be able to read and write high-level language code. The language may be C#, Java, Python, or even C/C++, but they'll need to know how to work well within them.

Corollary: The amount of code we write vs read will remain consistent, or tip a little bit more towards 'read'. Yes, the agents can generate code way faster than humans can, no argument. But theoretically, the agent will get commonly-written code right more often, and warrant less human scrutiny on those parts, so the relative balance between "read" and "write" will remain roughly constant.

Remaining open question(s):

Will we need the same number of developers (proportionally speaking) as we used three years ago? Probably not, but it's hard to say for sure. Going into and coming out of COVID was such a market-muddling event, it skewed lots of things in very extreme ways, making it a little unreasonable to draw as a "norm" to which we will return. That said, though, the Internet disruption of thirty years ago vastly accelerated the demand for developers, so that's a reasonable inference as well.

I find it just incredibly tickling to think that the language that was designed to be a "portable assembly language" was, in its day, considered "high level". Don't get me wrong, it totally is, compared to every assembly language I've ever laid eyes on, but considering how people today grumble and compain about C++, much less C.... hee, hee
↩
I should mention, because it might be relevant, I chose Tic-Tac-Toe primarily because it should be something that agents can understand pretty quickly and easily, and so therefore I'm minimizing the variables involved in the experiment. Also, just for the record, as I have documented elsewhere, I prefer running local LLMs rather than cloud-hosted ones.
↩
Something about how now the context window incorporates the failed effort and the LLM can learn from that and produce a better result.
↩
Interestingly enough, one of the other speakers in the room piped up with, "The C# compiler is non-deterministic, you know." Which floored me--the very nature of a compiler is to be deterministic, what the hell? Marcel even pointed out that the C# compiler provides a Deterministic option in the project file, a la <Deterministic>true</Deterministic>. Now, Marcel is usually nowhere close to being full of shit, so I assumed he was right that the option existed, but WTH is it turning off? Google is your friend here: "By default, the C# compiler produces non-deterministic output. The resulting assembly includes elements like a timestamp and a randomly generated Module Version ID (MVID), which cause the binary content to differ across compilations even if the source code is identical." Which... yeah, that's not really non-determinism, but sure, you get points for pointing out something I didn't know. Next round's on me, buddy.
↩

2026 Tech Predictions

Thu, 1 Jan 2026 00:00:00 +0000

It's that time of the year again, when I make predictions for the upcoming year. As has become my tradition now for almost two decades, I will first go back over last years' predictions, to see how well I called it (and keep me honest), then wax prophetic on what I think the new year has to offer us.

In the large, 2025 was... well, damn. Last year at this time I wrote, "We thought 2023 was tumultuous, but then 2024 said, 'Hold my beer.'" and I find myself wanting to repeat the phrase all over again. I can't quite put my finger on what is happening, but finding myself wanting to repeat these phrases over and over again suggests that either there's a larger social element at work here (I solemnly swear I am NOT going to bring up politics), or else I've entered my "Grumpy Old Man" era. Or, possibly, both. Either way, 2025 was an "interesting" year, where "interesting" is that word you use when you really don't know what other word to use.

Just when the tech market felt like it was just maybe, sorta, starting to make sense, the whole "AI will replace developers!" thing kicked into high gear and threw us all for a wild loop. LinkedIN was bad in 2024? LinkedIN 2025 said, "Hold my beer. And, you know, here's my car keys and cellphone too--this is going hyper cray-cray." Companies were hiring. Companies were laying off. Jobs were lost to AI. No, turns out they weren't lost to AI, but lost because companies were losing money on AI projects. No, they were making money on AI projects, just not where people could see it. No, that was a lie.

You know how that Ancient Chinese Curse says, "May you live in interesting times"? Welcome to the most interesting times of my lifetime, anyway. Now you know why the Ancient Chinese considered that a curse and not a blessing.

Let's take that look back in the mirror and see what I said a year ago and how close--or off--I was.

As per previous years, I'm giving myself either a +1 or a -1 based on a purely subjective and highly-biased evaluational criteria as to whether it actually happened (or in some cases at least started to happen before 31 Dec 2025 ended). If you want to skip to the new predictions, scroll down to the next major heading.

In 2025, I wrote ...

... even more stuff than in 2024, it seems, but that's often because there's so many parallel threads that it's hard to stay focused. I'll try harder this time around. But let's start with some of the personal (and therefore easy to score) stuff.

Our book will ship! (p 0.9)

Seriously! It will! I promise!

Result: As I write this (on December 31st, 2025), I can only say that we got so close. But we didn't get it slid in under the wire--we're still iterating on proofs with the publisher. That said, we do have a publisher page, and we can take pre-orders, and....

Yeah, OK, no, we didn't ship. -1. But, on the other hand....

Maybe I'll have a full-time gig again? (p 0.5)

Three years and counting. I wouldn't mind the "enforced retirement" so much, were it not for all these people coming around my place, telling me I owe them money.... Fortunately I keep finding the odd thing to do every so often, but it'd be nice to not have to answer questions that start with "Tell me of a time when....".

Result: +1. I picked up some consulting work at Capital One during the summer of 2025, and that in turn transformed into a full-time engagement at Capital One, so yay! Employment! Oh, and yay for a win in the predictions, but more, yay for employment!

I'll start either a YouTube channel or some video training (p 0.4)

The probability goes up if I don't have something full-time landed by the end of January.

Result: -1. I got distracted by a few things, and then the consulting thing popped up. Sorry not sorry and all that.

I'll speak at a brand-new (to me) conference (p 0.5)

Result: -1. Nope. Didn't happen. Still want it to, though, so this'll carry over into 2026.

"Developers are obsolete" prose will peak, then dwindle (p 0.8)

Look, let's be honest: Lots of people have a vested interest in seeing software development's "hold" over the creation of software broken. Startup founders desperately want to believe they don't need a CTO or any of those really expensive "full-stack" people to build out their idea. Non-FAANG tech-adjacent companies look at their labor costs and think, "Do we really need to have all these expensive people on our payroll?" AI venture capitalists (who want to see their investment make fortune quickly, so they can bail and reap the high stock price) want people to buy into the idea that AI can replace... pretty much anybody... without waiting for actual evidence of the success of replacing humans with AI. It's all going to hit a fever pitch this year, as the "pro-AI" crowd tries to drown out the growing surge of "anti-AI" evidence, then the bubble will burst, the AI startups will start to collapse, and the VCs will start hunting up their next hype to invest-pump-dump.

Result My, goodness, I guess I got half of that right. The "developers are obsolete" prose definitely hit a high note in 2025. I mean, some people took real swings at it with articles like "The Junior Developer Extinction: We’re All Building the Next Programming Dark Age". And, in some ways, yeah, the correction/caveats came right with it, such as what we see in that very same article: "Recent studies tell contradictory stories. PwC’s 2025 Global AI Jobs Barometer found that productivity growth nearly quadrupled in AI-exposed industries since 2022, rising from 7% to 27%, while a major Danish study of 25,000 workers across 7,000 workspaces found that AI chatbots “had no significant impact on earnings or recorded hours in any occupation” — with users saving just 3% of their time on average. ... Meanwhile, Microsoft research involving over 4,000 developers found that those using GitHub Copilot achieved a 26% increase in productivity, and Nielsen Norman Group studies suggest productivity gains as high as 126% for coding tasks. But here’s the kicker: according to the 2024 DORA report, speed and stability have actually decreased due to AI, and Uplevel’s quantitative study found that using Copilot didn’t result in productivity improvements but did increase the rate of bugs produced."

Can someone please make it all make sense? Developers using AI are going slower, yet companies are reporting higher productivity metrics, yet companies are reporting failed AI projects in numbers we've (literally) never heard of for a technology that still remains so popular and strategy-centric. It's baffling! It's bonkers! It's cray-cray! And it's not like the people writing about it aren't trying to make sense of it all; again, from that same article: "Meanwhile, The New Stack’s 2025 developer survey found that 'developers have become even more cynical and frankly concerned about the return on investment of AI in software development.' The honeymoon period is ending, and reality is setting in."

Except... that the reality is having a hard time setting in. LinkedIN, that finger-on-the-pulse of professional movements, still feeds me quotes and citations and examples of companies who are "all in" on AI and AI coding tools and other "agentic" things. Tons of "AI influencers" (most of whom don't appear to have ever written code in their lives unless it was of the "vibe" variety) are breathlessly telling us of how the whole profession is changing. If you're getting the same kind of feed I am, it's obvious that there's still a TON of companies sinking an exorbitant amount of money into AI-related projects... even as a growing number of tech professionals (paraphrasing Fowler, Beck, and a few others) are telling us all to "calm down" and that "AI isn't replacing developers, but it can make developers more successful".

We'll come back to this in the 2026 predictions below, but I think that the dichotomous nature of what's happening here is explainable, and justifies me giving myself a +1 on this one.

AI startups are going to start failing (p 0.5)

It may be this year, or it may be next year, but the AI venture-capital-gravy-train is about to run out. Fully 25% of all VC money went into "something something AI", and it's eerily reminiscient of the "dot-com" years, when a huge percentage of all investment and VC money went into "something something Internet" and gave us such wonderful profit-avoidant monstrosities like "pets.com". (Seriously. The sock puppet. It's all that's left of what was one of the most trumpeted startups in the early 2000s.) But eventually, your business has to actually make money (even you, Sam Altman), and when the profits aren't there, the shutdowns follow.

Result: I guess it all depends on how you classify "failing". First off, yes, there were failures; in fact, more than a few failed. Builder.ai was the classic con-man game, calling it AI but secretly operating humans in the background, but it wasn't always that simple. Rain AI ("revolutionary AI chips") took that classic line of "let's put the software thing into a chip", and like it's Lisp- and Java-flavored predecessors, collapsed. And, yes, lots of startups fail, but these were some pretty big ones with some pretty sizable players backing them in some pretty large ways.

It's important to call out that some of this was due to the environment, not the tech: When the VCs handed out money to (literally) everybody in 2020/2021, those companies had a 2-3 year runway, which, if you do the math (and add in an extra year of "emergency/bridge funding"), ends somewhere in 2023-2025 timeframe. They had to make good on their business, or die trying.

But the complete collapse of a startup isn't the only way it fails; Google, for example, did a talent-acquisition play with Windsurf, which raises the question, "If Windsurf fails in 2026, is that because the market isn't there, or because the company didn't execute, or because the company's key executors were scooped out?"

The bloodbath isn't over yet, I don't think, but that's for the 2026 section. For now, I'm going to call this a +1 because we definitely see some AI-centric startups that looked impregnable suddenly collapse.

ARM is going to start eating into Intel in a huge way (p 0.7)

In the beginning, there was the x86. And Intel rode that line into massive, generational-spanning profits. But slowly, over time, the ARM chip architecture has been slowly but steadily eroding the bulwark that x86/x64 erected ("You'd have to recompile everything! Nothing would run out of the box! Dogs and cats will merge to form a master race that will enslave humanity!"), such that by the end of 2024, not only was any shipping device without a built-in keyboard running ARM internally, Microsoft had a version of Windows running on the ARM chip. This upcoming year, there's literally no barriers left to ARM's complete dominance of the CPU market. The "Apple Silicon" MacBooks were just the vanguard in the horde of new ARM-based laptops that are going to come out and essentially take over. (As a caveat, if you're a native-language developer, as in you use C, C++, Swift, Rust, Go, Nim, Zig, and so on--if you've not started learning what ARM assembly looks like and behaves, now's a good time to start learning it.)

Result: Windows. On. ARM. +1 That's the final strut keeping the avalanche away from the ski lodge that is Intel's x86/x64 business. The numbers aren't there yet (if you can find any!), with some folks calling 2025 sales of Windows/ARM PCs to be around 10-15% of new laptop sales, but the signs are there that this is going to heat up soon. Intel's going to do everything it can to try and staunch the bleeding, but they've now reached the official "This is the moment it all went to shit" demarcation point.

Java and C# aren't going to introduce anything significant (p 0.7)

Look, I've been saying this for a few years now, and I'm going to make this the last year I do so, but these languages are so mainstream right now that nobody (except for conference speakers, who constantly mine the feature lists for "What's New" talks for the coming year) really (a) pays much attention to their "v.Next" featureset, or (b) cares. The price of mainstream status is a desire for "sustainability", which means most Java shops are not moving versions unless it's an "LTS" release, and most .NET shops are in the same mindset. (BTW, most iOS shops have lost track of all the new "bright shiny"s that Swift keeps getting, and what's worse, each release seems entirely incompatible with the previous.)

Result: I'm giving myself a +1 solely because when I asked nine colleagues (some from Java, some from C#) what the new features of Java or C# were, they couldn't remember. Considering these are folks that, like me, used to track the new upcoming release info religiously, I'm taking that as a sign that both these platforms are, in a lot of ways, "done". Not "done" meaning "dead", but "done" meaning, "There's just not a lot that I desperately want the implementors to introduce".

TypeScript jumps the shark (p 0.6)

The TypeScript language started life as an attempt to add a strongly-typed type system to the JavaScript ecosystem, in order to provide greater and greater up-front verification of code in order to get better errors earlier. (That's the general goal of any strongly-typed language, and while we can debate whether that's a good idea or not, what's not debatable is that a significant number of programmers buy into it, one of them being Anders.) But TypeScript also discovered that it could do a lot of type-level things, and TypeScript has gotten more and more complex and complicated and confusing with each release. (I've commented, elsewhere, that I lost faith in the language when the team chose not to create a language spec for it anymore, choosing instead to just say, "The code is the spec" and publish a new blog post with each release desribing new features in a casual and imprecise way.) This is the year (though it could be next year, although some have said it was already last year) that TypeScript starts compiling (or not compiling) expressions that absolutely confuse the hell out of anybody without a PhD in type theory. (For whatever it's worth, two years ago somebody put together a list of 'WTF moments', and they're... interesting.)

Result: Ehhh.....? When the language implementors announce that they're rewriting the entire language in a different (entirely unrelated) language, it definitely begs some questions. Was the compiler performance really bad enough to justify this work? Will the new compiler be bug-for-bug compatible with the old one, particularly when there's no publicly-available specification to be able to verify against? And are the developers at Microsoft so enamored of Go that they deliberately avoided some of the other languages they could easily have used (C++, if you want the native compilation target or C#/F#, particularly in conjunction with some of the CLR AOT-to-native options, all these come to mind...).

But again, truthfully? Name the last new language feature TypeScript introduced and tell me why it matters to you. Whether you agree with me on the +1 really rests on whether you can answer that with any degree of accuracy.

Python type hinting goes mainstream (p 0.6)

Quietly, while nobody was looking for it, Python introduced "type hinting", where Python code can be given Pascal-style type hints to parameters and return types, allowing tools like IDEs and editors to be able to provide better edit-time support. Most of the Python tutorials, it seems, ignored this for a while, but starting last year it seems the notion of writing type-hinted Python is becoming more and more acceptable to the Python community, though it seems that the majority of the community kept to "untyped" Python. My guess is, this is the year that the balance shifts, and "typed" Python begins to outweight the "untyped" Python. Legacy "untyped" codebases will of course exist for centuries to come, but I suspect that this coming year might be when companies start scouring their Python codebase and thinking, "Maybe we should put type hints in...?".

Result: It's hard to say, honestly. Lots of sample and demo code has sprung up using type hinting, and the coding-assistant tools love to slither it in to the code they generate, but are Pythonistas actually embracing it? Well, Meta thinks so. Pyrefly agrees. And hey, if Reddit agrees, it must be true, right?

But I'm not 100% sure I agree yet. Periodically I find interesting Python repos on GitHub that are relatively large and entirely absent any type hints whatsoever, and the rank-and-file Python developer (not to mention all the data science folks, who aren't programmers and don't necessarily see the reason for type hints) isn't always reflected in the blog posts, so.... -1. I just don't feel comfortable claiming this as a correct prediction in 2025.

JetBrains tries very hard to push Kotlin Multiplatform (p 0.7)

JetBrains put a ton of time and energy into KMP, and thus far it looks like there's been absolutely zero uptake from anyone outside of the building. If JetBrains is going to be able to call this any kind of success, they're going to need 2025 to be the year that KMP emerged as a credible competitor to React Native or Flutter.

Result: +1. For starters, they took it to production status; for seconders, their blog was filled with comparisons to React Native and the iOS or Android native ecosystems. And, in some ways, it appears to have made something of an impact, as their own developer survey suggests that there's a non-trivial number of folks using it. (They survey was for 2024, but the analysis came out in 2025, so we have to take the data with at least a little salt.)

Flutter begins its quiet slide into obscurity (p 0.7)

Speaking of which, Flutter's going to start being quiesced by Google. First there was the layoffs (200-some people in October of last year), which signaled pretty strongly that Google's "done" with Flutter, all the PR statements notwithstanding. Then the community fork that came out of that news led a number of folks to think, "Wow, Google's really done with Flutter", since nobody would think to create a community fork of something that's being actively developed and maintained by the multi-trillion-dollar entity that created it. That makes the companies who make money on the things developers build (as opposed to those companies who make money on building things for developers) get very nervous about adopting or continuing to invest in Flutter, and we can expect Flutter's usage numbers and interest level to plummet as a result. Yes, the open-source tools are there, and yes, people will continue to work on them, but hey by the way, how's the Parse community looking these days? Or LoopBack?

Result: I wasn't the only one curious about this, and when that article asked, "Is Flutter dying?" it answered itself with "No. It's evolving." That's a touch telling. On top of that, there's Flock, which is a community-led fork of Flutter, which itself suggests some amount of uncertainty about Flutter's longevity (at least as a Google-sponsored/-backed effort). (Come on, you don't create a community fork of something if the company behind it is sending out strong signals and actions towards growth and sustainability.) I've liked the technology for years, but given all this, it's hard not to see it beginning that quiet slide. +1

Swift will release a new version, and it will have incompatibilities (p 0.8)

Like the rain in Seattle, Apple always releases a new version of Swift every year, and every year, there's something in there that's not backwards-compatible with the previous version. There is zero reason to expect that 2025 will be any different on that score.

Result: Swift 6.2 was released in September of 2025, and 6.1 in March, and 6.2 (bugs nothwithstanding) is compatible with 6.1, and 6.1 is supposed to be compatible with 6.0, though the language implementors claim that some stricter enforcement of concurrent code may yield compilation errors in 6.1 that went through OK in 6.0. Still, these aren't deliberate breaking changes, in the grand scheme of things, so... -1.

Rust will not replace C++ (p 0.7)

Oh, I know the downsides to C++. I really do. I programmed in the language for a half-decade (back in the "bad ol' days of pre-C++08, waaay back in the late 90s), so I'm well aware of all the dangers of pointers and what-not. But Rust isn't a high-level language, and there's way too much C and C++ code out there for companies to credibly consider rewriting all that code. We can ache and moan about computer security and stability all we like, but doing that rewrite is not a simple line-for-line replacement, precisely because of the pointer guarantees that Rust enforces--all of those pointers have to be debated, decided, and declared, and it has to be consistent across multi-million-line codebases. Rust may make some inroads in getting new system-level development, but it's not going to make C++ "go away" any time soon, not this year or the next (or the one after that, or even the one after that).

Result: Ohboy. THIS one. I was pretty confident about this one, until... boom and boom. Windows is a massive C/C++ codebase (though there's some other stuff in there too, to be sure), so if they're starting to replace the C/C++ code with Rust code, that's a pretty solid vote of confidence.

Does this mean that C++'s days are (finally) numbered and all those C++ devs should pack it in and learn... (um... what is the latest thing to learn again)? Probably not. Between the fact that the amount of C/C++ code in the world is ridiculously large enough to resist easy ports to anything, and the fact that Microsoft has done these sorts of "We're going to rewrite Windows in..." kinds of announcements before, let's give it some time before we count Rust as having successfully replaced C/C++ at Microsoft, much less anywhere else. But, it's definitely a turning point, and in fairness, I think it merits a -1.

Odin, Nim and Zig will gain some attention (p 0.5)

All of these are system-level languages in the tradition of C++ or Rust, and all of them are interesting in their own rights. I suspect some of the enthusiasm for Rust will get siphoned off into these, and before long the "Rust will replace C++" crowd will turn into the "Rust/Odin/Nim/Zig will replace C++" crowd, before they start fighting among themselves which of those four will do the replacing (because, of course, there can only be one, right?).

Result: Well, Zig is definitely getting some airtime, for sure, particularly since it's what powers Bun, the high-performance JS runtime. Nim and Odin, though, seem to be trailing in popular usage, though each definitely has its ecosystem that is growing. Still, I look around and don't see any crowds chanting, so this gets me a -1.

DIY databases will start gathering attention (p 0.5)

There's been a quiet surge of interest in SQLite recently, since it ships pretty much everywhere (it's already on every mobile device on the planet, and its C codebase and bindings make it super-easy to FFI out to if there aren't already language-level bindings for your chosen language) and it's a pretty feature-complete subset of SQL-92. That's gotten a number of folks intrigued with the idea of "local database"-based functionality, which in turn has sparked some interest in "Wait, what exactly makes up a database, again...?" because if I can get the storage engine and bypass the SQL-ish query and talk directly to the query engine and.... Well, suddenly people are starting to wonder if the database has to be the black box we've always assumed it to be. I think in 2025 we're going to start cracking it apart and doing a little plug-and-play of the components as part of, or behind, their service interfaces.

Result: I'm not sure how I was expecting this to manifest, so I can't quite tell if I was right or wrong. SQLite is one of those things that's been around since forever (it's shipped on the device on both iOS and Android since each of their first release), and the fact that one can make use of it from within a browser in a variety of different ways (compile the SQLite source via emscripten to get a WebAssembly implementation directly, or use browser-native storage which is often implemented in SQLite, or use the port of SQLite to Javascript, and that's just three ways to approach the idea) makes it really hard to track.

But if we look at the intent of the prediction, what I was really going after was custom database implementations that would start gathering some interest, and I can't honestly say I saw much of that across 2025. -1

Final tally

ten wins ("+1"s)
nine losses ("-1"s)

... which puts me squarely in the middle of the bell curve. Good thing this isn't my day job!

2026 Predictions

So now let's turn our head to 2026.

As I do every year, each prediction comes with a probability factor (p) that ranges anywhere from 0.0 to 1.0, with most predictions appearing in the 0.3 to 0.8 range. This is a trick I picked up from my International Relations days, where political prediction briefings often came with a similar kind of probability as a way of offering a "confidence" factor. Anything higher than a 0.8 means "We're pretty damn sure", and anything less than 0.3 means "We're including it mostly out of completeness but we don't expect it."

But before we get started.... You know how I keep writing more stuff each successive year, and apologize for it the following year? Not even going to pretend this time around, this one could be the longest yet. (And, to no one's real surprise, it's all about AI! Because... sigh) Brace yourself.

2026 AI predictions are going to be even wilder than 2025's (p 0.8)

... because VC-backed startups need people to believe, so the VCs can exit the startups they've backed.

The landscape of the startup scene has been in the throes of a sea change for some time now, going back a few tech-generations (AI, blockchain, etc), and it goes like this:

Somebody, typically one of the folks with close ties to venture capital or Y Combinator, comes up with an idea. It doesn't have to be an actually accomplishable idea, it just has to be able to look remotely feasible and have some kind of sexy demo.
The founder, through the VC's network, gets some tech press folks to do some interviews.
Because the tech press is even more sensitive to "clicks" and "page views" than the regular press, offers up a soft-ball interview in which the idea is never deeply examined or questioned, but reinforced as "groundbreaking". The founder is held up as an icon, deified, and memes about their daily workout routine become the staple of LinkedIN feeds.
Other VCs, now desperate to jump on to a "winning thing", immediately pony up some money.
This reinforces the press coverage, as other tech press outlets immediately clamor for interviews. Trusting that the scrutiny has already been done ("I mean, if you're a VC, you wouldn't just put umpteen-million into it without doing a serious due diligence on it, right?"), they offer up even softer-ball interviews, and the deification takes root. This is "The Next Big Thing".

Unfortunately, as Theranos showed us, this basically has a predictable cycle:

The CEO/founder does a lot of press, and makes wild-eyed promises that the tech press eats up.
As deadlines approach, the CEO soft-pedals back some of the promises, but makes other, lateral, promises as a way of showing how "on top of things" the company really is. "We're so close" becomes a common refrain, along with "We just need... to get there."
The company maybe IPOs, if it can't swing sweetheart deals with massive backers (lookin' at you, Microsoft, Google, Amazon...).
With each missed deadline, the CEO grows wilder in their predictions, and the needs grow more outlandish.
At some point in all this, somebody gets an uneasy "Waaiiiitaminute" feeling, and starts digging through the data. Maybe they work there, maybe they don't, but they start piecing things together and prodding at the facade.
Whispers start to echo across the blogosphere, and before too long somebody in the mainstream press with more of an investigative journalistic bent gets wind of it. They start doing research.
LinkedIN memes and blog predictions, meanwhile, skyrocket.
A news article or documentary effectively pointing out the lack of clothing on the supposed emporer's person comes out.
Within days, the cards begin to collapse, and everybody in the tech industry nods along, as if they always suspected something from the beginning (even as they quietly mourn the deep losses they just took on their stock portfolio as the wunderkind's company crumples).
Fanbois continue to fanboi, of course.

It's almost as if the tech industry made a deliberate decision to look at the Gartner Hype Curve not as a "lesson learned", but as one of those Maps of the Hollywood Stars that must be slavishly followed over and over again.

Meanwhile... We're in peak Inflated Expectations territory here with AI, folks. That means in order to keep the eyes off the bottom line, the predictions are going to only get wilder.

AI-based startups that use AI to mimic humans are going to crater--hard (p 0.7)

... because humans keep anthropomorphizing AI, and keep believing that AI can do human-sensitive things better than humans can.

Werner Vogels thinks robots can help foster companionship; I deeply reject that premise, because humans have systematically rejected machines replacing humans in all but the most desperate of cases, and because cases where people have done so, missed the mark or ended badly. Humans can still tell the difference between machines and humans, even if they can't say why they know. Worse, if humans suspect that the voice on the other end is a machine, they'll treat it wildly different.

But that said, lots of startups are going to go down this path, and while they might get some traction to start, they'll crater hard when they try to scale out. Because we are all of us lonely, and having an always-available companion seems like the easier solution than actually building empathy and connection with other humans. Certainly the more profitable one, anyway. But the results... will not help people, only balance sheets.

OpenAI starts tying itself in knots to avoid having to make good on its promises (p 0.6)

... because Sam Altman has made a lot of promises, and doesn't have a great track record of delivering on them (January 2025: "We know how to build Artificial General Intelligence"; November 2025: "Hey we got GPT-5.1 to use em dashes correctly when instructed (but not always)"), and 2025 saw OpenAI making even bolder promises. This is going to catch up to him in a big, big way.

Look, Sam knows how to project confidence--every CEO does at that level, it's a prerequisite of the job at this point--but that doesn't mean he knows how to actually bring it forth to fruition. We've all seen the meme of the financial shell game that Microsoft, NVidia, OpenAI, and a few others are playing with "promises of investment", or perhaps you've watched the More Perfect Union video. But Sam's playing at an entirely new level with trillions on the line, and you kinda have to wonder if he is resting much of his confidence on the idea that OpenAI is just "too big to fail" and can rely on government bailout in the event the piper does come calling.

I'm certainly no financial expert, and I don't claim to have the inside scoop on all the hallways of power back-room deals that OpenAI and SoftBank and all the other players are whipping up, but Altman's already backed away from so many deadlines and dates, before too long a lot more folks are going to notice, and you can't just keep selling whipped air forever. (See: Musk, Elon.) I suspect that in 2026, the shell games will come faster, the retrenching will come quicker, and I think Sam Altman is going to find himself at the center of a firestorm large enough to make Sam Bankman-Fried look like small potatoes.

And with that, there's going to be a whole lotta finger-pointing and crashing.

Meanwhile, your electric bill is going to go up by at least 50% (p 0.8), possibly more (p 0.6)

... because all those data centers that AI needs are being built on taxpayer and utility customer dime.

Next time your major urban city mayor is looking to "bring some jobs home", ask them what the domestic cost will be--for example, if they bring in a data center (which largely runs itself, compared to an automobile plant or steel mill), ask them if your electric bills will go up as a result, and if they'll agree to put legislation in place to force the company building the data center to foot the additional rise if it does. It'll spark an interesting debate at the city council meeting at least.

(Oh, and for bonus points, ask about not your utility bill, but the bill of the most impoverished part of your city. After all, data centers don't go into posh suburban neighborhoods, they go into wherever the land is the cheapest.)

"Vibe coding" is going to crater--hard (p 0.7)

... because we have started to figure out software development is not about writing code, but understanding the problem well enough to be able to articulate what code to write.

To understand the "why" of vibe coding, let's accept a couple of statements as axiomatic:

"Software is eating the world." --Marc Andreesen. (Shouldn't come as a surprise to anyone, though at some point I hope that we realize that you can't eat software, and jobs like primary school education, plumbing, farming, and so on, are all pretty important to the continuation of Western society as we know it...)
"The demand for software is infinite, yet its supply is finite." --Neal Ford. To be precise, the demand for new software is infinite--it's not that everybody wants copies of Microsoft Word, it's that everybody has ideas of something that will make Word completely obsolete and hey, you're a software developer, right, you could code it for me and we could split the profits 70-30 my way because of course it's my idea....
Supplying new software requires some form of software development. Canonically, this means hiring somebody who's accomplished with one or more programming languages. (We'll get to the alternatives in a moment, just hang on.)
Learning software development is a non-trivial investment of time and money. Historically, this has meant going off to get a Computer Science degree from some college or university, which is often a four-year investment. Some folks can do it in less time, others do it in a self-taught manner, but it's not like you can read one "For Dummies" book and suddenly know all of what you need to know about how to build a mobile app, API backend, database storage, security, cloud management, and user interface design. This non-trivial investment is a core reason why the "supply of software (development) is finite".

Given the above axioms, there's a simple conclusion that any macroeconomist can tell you about: As demand rises, if supply remains fixed, the equilibrium price goes up. This is why programmer salaries went through the roof during the original dotcom era--and have stayed there ever since, despite the setbacks we've seen over the last three decades.

However, one other axiom needs to be cited here to make all of it make sense:

Businesses will always seek to optimize to their most crucial metric: Profit. This means maximizing their revenue while minimizing their expenses.

If your business is built on software, there's a strong probability that your biggest expense is... software developers. It used to be owning and operating all those big server farms that every company had tucked away someplace (usually at the center of the building behind glass walls because damn it looks impressive to potential investors to see row upon row of machines with blinky lights...), but the cloud took care of all that ¹. And, to be clear, software development labor has been a sizable part of most companies' expense sheet for (checks notes) three decades now, so the cloud thing helped some, but the original problem remains.

So if you're a software development shop, how do you cut the expenses of your software developers?

You look for ways to make software development faster. After all, faster means fewer hours means less cost (because cost = hourly rate x hours). We see it, everywhere along the way, in the way that new programming languages are marketed, new tools are sold, or new practices are introduced. They might vary in their message, but everywhere, somewhere, there's a prominent tagline about "greater productivity". Remember the 4GLs of the 90s?
You look for ways to make software development cheaper. "Hey, maybe if we go find software developers in countries with way lower standards of living, we can pay them less to do the same work." Alternatively, we also saw the rise of "coding bootcamps" that promised to take people with no coding background whatsoever and turn them into software developers in just twelve months--no, nine--no, six--no, no, you fools, overnight. Naturally, it gave rise to scams and lawsuits.
You look for ways to avoid software development entirely. In macroeconomic terms, what substitute goods/services² can we use to get what we need? In common parlance, this translates to "What if we use something that doesn't require software developers yet still builds software?"

If some of you are old enough to remember CASE tools in the late 80s or the UML tools in the late 90s, those were both plays to avoid software development, though in retrospect it's easy to see how neither one really dodged the basic need, which is "You have to understand the problem deeply enough to be able to figure out how the software should work to solve it."

Vibe coding is, in many respects, the Visual Basic of the 2020s: A tool that's designed to let "anybody" create software (thus neatly avoiding the high cost of software development) and finally slay the Dragon of Infinite Software Demand by allowing supply to run free, no longer burdened by the obstacle of knowing how software works.

But, if you were paying attention two paragraphs back, you already know why vibe coding is failing: "You have to understand the problem deeply enough to be able to figure out how the software should work to solve it." It's not that we don't have tools that can churn out millions of lines of code; the problem is that you need to know how to break the problem down into smaller pieces such that the AI coding tools can actually have a decent chance of generating code that will solve it. (Ironically enough, this is where most software development processes--whether AI-driven or not--run into issues.)

The reason for the prediction? We've had these "vibe coding" tools in our hands for a half-year or more now, and we're already seeing their limits. In 2026, they'll remain, but their use will be limited to building quickie prototypes, and there really isn't a market for tools that can only build quickie prototypes.

AI coding assistant tooling is going to gain momentum (p 0.7)

... because AI's not gaining much steam anywhere else.

The aforementioned AI startups are constantly trying to sell us on AI in HR, medicine, and finance, but honestly, after a year or so, it doesn't seem to be having much effect, and companies are starting to backtrack and re-trench. For starters, companies tried to use AI agents to do hiring, and that landed... poorly. Things really turned south when companies realized that candidates could do that too, and did. Maybe we can outsource other HR tasks... wait, I think I've heard that one before.

Likewise, AI's use in the legal space has... problems. Sorry, Mike Lindell, you can't just hallucinate your way into proof of the 2020 stolen election! And he's not alone.

Turns out, lots of people get grumpy when AI just makes stuff up out of nothing (even people who are paid to do exactly that), and that trend is only likely to continue. Which then, if you're in the AI business, where can you let the non-deterministic nature of generative AI fly in a space that's open to it, yet structured enough to not be hamstrung by it?

Coding, of course. Programming languages require structure and conformance to form (minimizing the hallucination possibilities), particularly if backed by unit or integration tests, but also require some creativity in translating "problem" to "solution". The coding agents have already had some measure of success--see the aforementioned "vibe coding"--but we're starting to understand their limitations--see the aforementioned "vibe coding". In 2026, that curve is going to pick up steam, and we'll see it get stronger as the year goes by.

Specification-oriented development will gain traction (p 0.6)

... because it captures a large part of the essence of programming.

When Don Syme (he of the F# language) and John Lam (whom I'm lucky to have worked with at DevelopMentor back in the day) both together started talking about "specification-oriented development", it rattled my brain a bit. These are smart dudes, and if they think there's something there, enough to both blog and implement something around it, then I'm curious by default.

"Spec Kit makes your specification the center of your engineering process. Instead of writing a spec and setting it aside, the spec drives the implementation, checklists, and task breakdowns. Your primary role is to steer; the coding agent does the bulk of the writing." On the surface of this, I like where this is going, because elevating the abstractions in programming is something that I've been advocating for a very long time now. (We've been surfing at the "object" level for three decades--what say we start re-thinking the problem again?)

I'm not 100% convinced that this is going to be the "path forward" for our programming, but it's pretty clear that some kind of LLM-driven engagement, whether through coding agents as part of an IDE, or through these "specification" files, is going to markedly color much of what we programmers do for the next few years. I think there's a fair amount of rigor in what's being suggested here, but at the same time, this is flying in the face of some pretty well-established doctrine, and it's there's a pretty high mountain of evidence against the idea of natural language as a deterministic way to write code.

So this looks interesting, but I don't think it's going to take over the world yet. It'll capture some attention, it'll get some revision, it'll maybe spark something else that takes over the world, but somehow, it's going to get some traction over the next year. (What it really needs, though, is a good tutorial/walkthrough on how to use it, both for the simplest "Hello world" case and for a non-trivial example case. Right now the docs on the GitHub repo are a little vague in a lot of places.)

Languages will matter less than fundamentals and concepts (p 0.6)

... because coding assistants can carry the weight of syntax better than humans can.

This also means I'm not making any generalized predictions about Java, C#, Python, Swift, or any of the other current crop of languages--there's simply little to no point, at least in 2026. They're still important, don't get me wrong, but their importance is now at more of a strategic level than a tactical one. (Meaning: CTOs and VPs won't care about which version of Java you're using, but will care about whether you're using Java as opposed to C# or Python.)

Micro-LLMs/Small Language Models (SLMs) gain traction for more widespread use (p 0.6)

... because the Fallacies of Distributed Computing still hold, and servers making all these remote calls (over MCP) will slow down model responses, so we'll look for ways to bring them all together under one roof--which means having models small enough to execute locally.

... because also, we're going to start realizing that we don't want "god" agents, because that increases the security risk. We're going to want agents that know how to do one thing and don't try to do anything else.

We will see a new crop of languages trying to incorporate natural language (p 0.7)

... because we still desire determinstic behavior out of our programming languages.

This is going to be the natural outcome of (a) more AI coding agent use, (b) SLMs gaining traction, and (c) a strong desire to "smash together" the agent into the IDE and then the language, which we see already in the aforementioned specification-oriented development. Programming languages have the ability to abstract away details we don't want to deal with (a la memory management or pointers), so it stands to reason that some enterprising language designer will figure out how to fold all of that into a text-based input scheme that has just enough rules to allow the agents to shine, while keeping the non-determinism out.

Quantum will gain traction (p 0.5)

... because the computing power is there, we just need to figure it out

I won't go too deep into the details here, but quantum definitely represents a massive leap forward in computing capability, and while your average video game has no need for it, lots of other applications do (including maybe future LLM models?), and it's too enticing to ignore. Somewhere before the end of the decade, we're going to start seeing developers working against quantum SDKs and using quantum languages, but don't ask me what they'll look like--I'm having a hard enough time understanding the basic science of a qubit to start with!

Android starts to move in on laptops (p 0.5)

... because Google wants it to, and mobile devices have long since become powerful enough to be laptop replacements.

You know what the line is between a tablet and a laptop? Me neither.

For years, laptops have tried to stave off the tablets by offering touch-based input (well, all of them except macOS, anyway), to varying degrees of success. Laptops had a few things going for them--larger screen sizes, more ports, an operating system that wasn't hamstrung by only being able to install things through the app stores, and so on. But over the last few years, we've seen those things slowly evaporate (except for the "more ports", which Bluetooh connectivity, wireless protocols, and dongle/hubs are making irrelevant for the most part), to the point now where many people get a larger iPad and a Bluetooth keyboard, and call that a laptop.

The manufacturers feel it; it's why Lenovo and Samsung have tablets that act like laptops. Google, meanwhile, already announced that they want to do away with ChromeOS in favor of Android, and the work over the last year pretty much demonstrates that they're ready to start backing that with laptop or desktop hardware.

Are you going to get Android on a desktop machine? Probably not--a good chunk of the rationale for a desktop purchase these days is gaming, and Android still isn't anywhere close to that world yet, with the other chunk of the rationale being development, which Android again isn't close to. (Although, if I'm being frank, a good keyboard, a good monitor, a command-line prompt, and a half-decent editor, and Android's no worse than many Linux distros at that point.) Either way, it feels like the natural path is for Android to slide into the space that used to be occupied by ChromeOS, and that's through ChromeBooks, e.g., laptops.

iOS remains stale (p 0.6)

... because Glass mode is just... uninspiring.

Seriously. Apple tried really hard to make Apple Intelligence a thing, but in the end, all we're seeing is better and more cameras on the iOS devices, and frankly, yawn. One of their most recent commercials was showing how to use iPhones in cinematography. So we literally have Apple urging us to buy their product by making commercials on... making commercials.

Deepfakes will be the social nightmare that plagues us (p 0.8)

... because they're good enough, and because some humans would rather work against other humans than for humanity.

If "AI actors" are good enough to be considered for use by Hollywood (lookin' at you, Tilly Norwood!), then it takes almost no thought at all to realize that she could very easily show up in a faked Zoom call, FaceTime, or news report. She and her cousins are going to be the source of scams, lies, and even deeper distrust of what we see on the screen.

Ironically, it could be what gets us off those screens, but let's see how it all plays out. Personally I think the growing "digital divide" is going to center on how much one bases their life on their digital footprint, and we could very well see Generation Alpha (or the one that follows it) utterly rejecting some of the lifestyle approach that Millennials and GenZ embrace because of all this. It will be very interesting to watch over the next decade or two.

I'll speak at a brand-new (to me) conference (p 0.4)

I have no idea which one, but I'd love for some suggestions and/or requests. I love the shows I hit regularly, but I wouldn't mind going someplace entirely new and talking to people who've never heard me speak before. I'm giving it a slightly lower probability because full-time work always makes this trickier, but it's still a goal.

Our book will ship! (p 0.99)

I know, I know, but we're serious this time. We're in the final proofs-editing stage now, so the question is really whether it'll appear on shelves (figuratively) in February or March.

As always, feel free to comment and tell me why I'm crazy, in whatever forums feel comfortable. In the meantime, talk to you all next year... if not sooner.

Theoretically, anyway. We're starting to see that the cost of owning a virtual machine in the cloud isn't all that much better than it is to own it on-premises, which is leading a number of companies beyond a certain size to reconsider owning and operating their own data centers. It's still way better for smaller companies, where we can benefit from the margins that are gained from a big company running lots of servers, but at a certain size, it just doesn't make sense anymore.
↩
https://www.economicshelp.org/blog/glossary/substitute-goods/
↩

DevRel Activity Patterns... published

Sat, 11 Oct 2025 00:00:00 +0000

tl;dr Two years ago, I started creating a pattern language for developer relations activities, and I published an early draft of what they looked like to this website. In the time since, they've been refined, expanded, and collected into a single paper volume, available from APress in December of 2025.

The book, titled, Developer Relations Activity Patterns: A Unified Approach to DevRel, DX and Community Management, is visible here and I believe available for pre-order. Amazon has it here, for those who prefer the Amazon infrastructure, and the back cover blurb reads like so:

Developer relations (DevRel) go beyond just blog posts, conference talks, and clever tweets. It is a discipline grounded in intention, empathy, and consistent actions. This book outlines foundational strategies employed by successful DevRel teams in just 39 focused patterns.

Using a tactical approach to effectively engage in developer relations, each pattern is based on real-world experience and designed for scalability. By using a pattern form to describe the various activities of developer relations in detail, the book clearly lays out the intention, shows how to create empathy with your audience, and provides a clear guide on how to execute it. Within this book, you'll discover how to extend your reach, enhance meaningful interactions, and align your initiatives with both developer communities and company strategy without compromising your values or experiencing burnout.

Whether you're a solo advocate, managing a small startup team, or leading DevRel on a larger scale, Developer Relations Activity Patterns is your practical playbook for refining strategy and amplifying your impact.

You will:

Gain a greater understanding of what developer relations means and the range of activities under its remit

Learn the tools, approaches, and new tactics for developer community engagement

Understand how and when to apply each tactic, based on your aims and objectives

For those who are curious, this is the culmination of the work started back in 2023, and the originals are still available on this website, for those who are curious to get an early (or free) peek at what's in the book. But the process of getting them book-ready involved the four of us authors (myself, Woody, Scott, and David) each turning each pattern over in our heads, collectively looking to expand each entry and, perhaps most importantly, looking to identify metrics that a DevRel team can use to determine if a given activity is accomplishing its goals. It was an early realization for us that metrics are the "silent killer" of any DevRel effort, so leaned hard into that, looking to identify or suggest metrics that would emerge from each activity.

I'm pretty sure that these aren't exhaustive, either. People who've been running DevRel teams for years will probably identify a few that we missed, or have issues with how we break these down. And, if you ask me, that's good! Patterns are never meant to be ready-to-eat solutions, but help foster communication and discussion and provide a vocabulary for people "in the space" to be able to discuss these ideas and concepts at a higher (and more productive) level. With any luck, this book will help DevRel teams better execute on their efforts.

Thanks for reading!

Introducing Linguis

Sat, 11 Oct 2025 00:00:00 +0000

tl;dr Linguis, available for examination and/or forking, is a language designed to explore the axis of, "How much can we tweak the front-end of the language, in this case the parser, so that people of different nationalities can write code in the syntax of their choice?" Available at https://github.com/tedneward/Linguis for perusal, though it's only at a 0.1 now.

It's been an interesting 2025 so far, and with two-plus months left, I realized that aside from the new book coming out, I haven't really worked on any of the things I want to build. Thus, time to get one of the "little ones" off the list: A language with differing input syntax over the exact same AST.

As I mention in the project's README:

Programming languages have, almost since their inception, assumed a fixed syntax that is rooted in English for its keywords, with relatively few exceptions. Pascal has PROGRAM, FUNCTION, VAR, and so on. C has if/else, for, while. Even homoiconic languages like Lisp use English nouns for its forms.

When I was in college, though, I took a trip to visit some friends in Belgium, and they had a French computer there--complete with its own version of French BASIC, where all the keywords were in--you guessed it--French. It took me a hot second to map the keywords I knew to their French equivalents, and definitely made reading the code an extra level harder.

Now, in 2025, we still see most of our popular programming languages based on English keywords, which makes me wonder: Why? Don't get me wrong, as a native English speaker, I am happy that my native tongue is the default lingua franca for the programming industry, but in an era where we have literally gigs of RAM to throw at a problem, why are we continually enforcing English as a necessary prerequisite for programming?

Thus, Linguis: A programming language (procedural, since this is a POC) whose keywords are "pluggable" and in several sets, so that those of non-native English speaking descent can program in their mother tongue.

For example, somebody writing English syntax might write:

name = input("Hello, what is your name?")
println("Hello " + name + " it is nice to meet you")

But somebody writing this from a French perspective might prefer:

name = saisir("Hello, what is your name?")
imprimernl("Hello " + name + " it is nice to meet you")

... where saisir is the French translation of "input" and imprimer is likewise for "print". (I tacked on the "nl"--nouvelle ligne--to make it the parallel to println.) Notice that the literal strings don't change, nor do the identifiers, since those aren't directly interacting with the compiler, but it would be pretty easy to imagine how the French student would write the earlier English program:

nom = saisir("Bonjour, quel est ton nom?")
imprimernl("Bonjour " + nom + " c'est agréable de vous rencontrer")

Part of the experiment here, as already being seen in the fragment above, is that the source code will need to be in full Unicode-capable streams, so that the different languages can each have their native characters represented faithfully. (Imagine what, for example, the above would need to look like for Arabic, Hebrew, or even Klingon.)

Currently, as I write this, only the AST itself, built in Python, is in the repository. The tests execute AST constructs (although to be more accurate they're "intermediate executable" tree nodes, not really "abstract syntax" nodes) directly at the moment, and it supports basic procedural constructs--variable declarations, simple scoped name-value bindings, if/while/for flow control constructs, and function definitions and calls, that sort of thing. It's definitely not a sophisticated language, but it's not intended to be--the sophistication, such as it is, will come at the parser level, not the semantic or execution level.

Next steps at this point are to introduce the first parser (and the core interface/abstraction for doing so without being tied to what that parser is) and a main "driver" that will take a .ling file from the command-line, parse, and execute it. Then a second syntax, and maybe a third. (I'm planning English, French, and German, in that order, since I know those languages well, reasonably well, and passably okay, in that order). Once I've got three, I'll probably quit while I'm ahead and move on to something else, but this should keep me entertained through the end of 2025, anyway.

Naturally, if you feel like tossing in a hand, particularly if you want to take a crack at introducing a new syntax, it's a public repo for a reason. :-)

But, otherwise, thanks for reading!

UPDATE 31 Oct 2025: Linguis is getting closer to "done" for the US-English keyword set! Basic scalar types (booleans, ints, floating-point, strings) are in place, along with the core maths (+, -, *, /. %. ^) and comparison (==, !=, <, >, <=, =>) operators. I've also discarded my custom AST in favor of going straight to the Python AST objects--for the US-English parser, for example, the parse returns a Python ast.Module object suitable for compilation and execution. (Yes, compilation--Python is an interpreted language, but the method call to turn the source into an executable code object is called compile.) I'm working to add lists and function defintions/calls, fix some of the missing pieces (unary negation, the boolean operators, etc), and then pivot to implementing the second parser... Pig Latin!

UPDATE 7 Nov 2025: For all intents and purposes, Linguis is a complete experiment! I have working parsers and tests for English, French, and Pig Latin, though candor compels me to state that the language is hardly bullet-tested/-proofed in any way. The larger point, though, that we can swap the lexer symbols without changing the parser or AST, has been proven out pretty well, for my money. Now, a few folks suggested that it should be possible to "round trip" from English to French to Pig Latin, which is interesting, but a little beyond what I was planning to do. I may still get back to that at some point, but for now, there's a few other experiments I want to try out.

Why Writing Matters (to the Technical Professional)

Mon, 11 Aug 2025 00:00:00 +0000

As a software developer in 2025, it is difficult to choose how to spend one's time. New programming languages (Rust, Zig, AssemblyScript, ...), new data storage implementations (NoSQL, NewSQL, multimodel databases, ...), and of course the alluring temptation of "artificial intelligence", all beckon for the developer's time and attention. More importantly, in a world where ChatGPT can toss off an essay on any topic imaginable without requiring any work on your part, why would any intelligent software developer spend a moment's effort on writing prose?

It's a reasonable question, and one that misses three-quarters of the point of writing. Yes, writing is used to communicate, which is in of itself a valuable outcome. Without communication, humans lose their primary advantage over all the other creatures on the planet. Communication is what allows us to erect buildings, advance science, and/or rally to a cause, among other pursuits. Much of written communication comes through informal media, like Slack messages, SMS, or email, and is often used to inform others of temporal items of interest--status messages, PR notifications, and the classic standup what-I-did-yesterday-what-I-will-do-today-any-blockers three-parter. And these are both important and impossible for ChatGPT to provide ¹.

As alluded in the previous paragraph, however, writing serves multiple purposes beyond the simpler act of pulling words out of one head into a form that others can consume. Writing serves multiple purposes to the technical professional. The more senior the professional, the more important the ability to write becomes, both on the job and in pursuit of the career. Across literally every company and organization, in every job role and seniority category, no single skill better returns on investment that that of the ability to craft clear and precise prose.

Why Bezos Thought Writing Matters

In June of 2004, Jeff Bezos sent a memo throughout Amazon with the subject title, "No PowerPoint presentations from now on at S-Team."² In the early days of Amazon, the S-Team would do weekly four-hour meetings that focused on execution on various projects. Each deep dive would begin with the typical oral-presentation-backed-by-PowerPoint-slides by the relevant team, which simply wasn't working. "The format often made it difficult to evaluate the actual progress and prevented the presentations from proceeding as planned. The deep dives were, in short, frustrating, inefficient, and error prone for both the presenter and the audience."³

Seeking a better answer, they latched on to Edward Tufte's "The Cognitive Style of PowerPoint: Pitching Out Corrupts Within", in which he wrote, "As analysis becomes more causal, multivariate, comparative, evidence based, and resolution-intense, the more damaging the bullet list becomes." This fit the problem they were seeing: their topics were complex, interconnected, requiring plenty of information to explore, yet the PowerPoint linear sequence of slides was not flexible enough, nor was the presenter-led approach. "Besides, the Amazon audience of tightly scheduled, experienced executives was eager to get to the heart of the matter as quickly as possible. They would pepper the presenter with questions and push to get to the punch line, regardless of the flow of slides. Sometimes the questions did not serve to clarify a point or move the presentation along but would instead lead the entire group away from the main argument. Or some questions might be premature and would be answered in a later slide, thus forcing the presenter to go over the same ground twice."⁴

Despite the heartache and churn the memo caused, they felt they had found a root cause problem: "The real risk with using PowerPoint in the manner we did, however, was the effect it could have on decision-making. A dynamic presenter could lead a group to approve a dismal idea. A poorly organized presentation could confuse people, produce discussion that was rambling and unfocused, and rob good ideas of the serious consideration they deserved. A boring presentation could numb the brain so completely that people tuned out or started checking their email, thereby missing the good idea lurking beneath the droning voice and uninspiring visuals." (Emphasis mine.) ⁵

Writing as Explanation

Amazon found, as many others have, that moving from PowerPoint to Word as the presentation format forced a deep change in the presentation itself. Ideas would need to be fleshed out more fully, with only the words on the page to carry the weight of the proposal or insight, rather than the force of personality (or relative position in the org chart) of the presenter. This approach is not simpler or easier--in fact, it is much more difficult, on both sides: "... this model imposes duties and expectations upon the audience as well. They must objectively and thoroughly evaluate the idea, not the team or the pitch, and suggest ways to improve it. The work product of the meeting is ultimately a joint effort of the presenter and their audience—thinking that they can all stand behind." ⁶

Part of this comes from what takes place during a typical presentation. A slide comes up at the front of the room, with the 6 bullet points each with 6 words. By necessity, each bullet point must be short and an incomplete sentence, with much of the content (and all of the nuance) inferred by the reader. Worse, the presenter is often speaking as the audience reads, forcing the audience to choose one or the other media--either they listen to what the presenter has to say, or they read and ignore the presenter's commentary. Compounding the negative impact, the presenter is typically verbally describing the nuanced elements of some or all of the bullet-pointed statements, meaning the audience has now missed exactly the nuanced take on the bullet points. If an audience member spots the nuance, they ask a question about it, requiring the presenter to awkwardly point out that question was already addressed; if they miss the nuance or say nothing, the nuance is never mentioned again. In any event, the nuance itself, not being captured anywhere in the slide, is only available to those who were in the presentation at the time of its presenting. The slide deck, on its own, is missing a crucial dimension that will be lost to time.

In a written narrative, the linear nature of reading requires the author to spend extra time considering the order and flow of the words on the page. The author may (and most often will not) be present when the reader reads, meaning the author must step back and consider how the reader will consume the document: When will they ask the obvious question? Can I move the answer to that question earlier in the narrative to head that question off? Am I presenting the simplest case first, or the most complex? Much of the writing process is, in fact, editing, in order to make sure that the readers are "getting it" the way the author intends. ⁷

Readers do not want to wade through a 5,000 page novel to understand the quarterly performance numbers, however; brevity matters. Amazon found that six pages, 11-point font, single-spaced, is the right length for a presentation designed to take an hour. Attendees to the presentation enter the room, receive a printed copy of the narrative, and spend the first twenty minutes reading. (Roughly speaking, an average reader fluent in the language can read one page every two or three minutes.) The narrative is free to include additional material, such as charts or graphs or other material, in an appendix, but the mandatory reading material may go no longer than six pages. Any attempts to go beyond that length are summarily rejected. This brevity forces the presentation not only to be consumable within a single hour-long meeting slot, but also the narrative itself to be tightly focused. Many writers throughout history have learned this lesson. ⁸ (For these same reasons, I set a hard limit on this narrative of 2500 words, excluding the Appendix, despite the Web's obvious lack of "pages" in the publishing sense.)

Writing as Branding

For those who do not (or do not aspire to) work at Amazon, writing still offers clear benefits. In the highly-interconnected World Wide Web, opportunities abound for well-written prose to catch an audience's eye and create positive reaction. Granted, the Web currently contains trillions if not quadrillions of characters, all clamoring for time under our eyeball, and most of it is worthless. This is, however, nothing new--Arthur Schopenhauer, 19th-century German philosopher, recognized the same problem: "In his essays, On Authorship and On Reading, he identified two types of authors: those who write for the sake of the subject, and those who write for the sake of writing (and really, for the sake of making money). Their motivations couldn't be more different. The former write because they are curious and they want to figure something out. They're like a detective, exploring a subject from all sides, trying to come to a deeper understanding. For them the process of writing is how they figure out what they don't know and how they come up with new ideas. The latter because they get paid by the word, not the insight. You can recognize the latter by how they stretch out half-baked thoughts 'to the greatest possible length.' Their writing is evasive, 'lacking in definiteness and clearness.'" ⁹

This, as an aside, is the primary reason why ChatGPT will, over time, prove to be highly inefficient as a tool for creating effective prose. ¹⁰ While it might know facts and figures, because it lacks the ability to reason and understand the underlying ontology or mental model of the subject under discussion, it will never be able to speak in "definiteness and clearness" effectively. In fact, at least to the date of this writing, where it does attempt to speak with such specificity it usually falls victim to hallucination and/or error.

Those who can put forth prose in a manner that is clear, precise, and direct on a given topic will find that their reputation--their "brand", as the marketing-savvy put it--will grow in a positive direction. Moreover, because the prose stands apart from its author, the message and meaning of the narrative will not be diminished regardless of the author's timidity or introvertedness.

Writing as Reasoning

What often comes as a surprise to the new writer is that writing is not just an exercise in dumping thoughts from head to keyboard; Paul Graham put it this way: "A good writer doesn’t just think, and then write down what he thought, as a sort of transcript. A good writer will almost always discover new things in the process of writing." ¹¹ The act of reaching into the brain, exploring what we know about a topic, and transferring it to some communicative medium (such as written, or even spoken, words) causes us to re-examine the material.

Mortimer Adler once said, "The person who says he knows what he thinks but cannot express it usually does not know what he thinks." ¹² Consider a hypothetical for a moment: You are asked, without using any reference material, to explain how radio works. If you are like many people, you know that radio is distributed through the air in some sort of invisible "wave" (hence the term, "radiowaves") that can degrade over distance, although AM waves seem to have greater longevity than FM waves, can be interrupted by physical objects such as walls, or by electrical fields such as lightning storms. But the actual act of transforming sound into wave, or wave back into sound, most of us are unclear on how that works. We have a mental model--an abstraction--of how the radio works, and that mental model suffices for most scenarios in post-2000 civilization. But when asked to dive into detail, we quickly discover that our mental model, like most models, is incorrect (yet still useful).

Consider the scenario in which a company seeks to make a significant shift in its operations. Each individual associated with this effort has a mental model of how the effort should proceed. (Or, to be fair, perhaps only some do, and the rest are eager to build such a model based on what those who do tell them.) One individual may be thinking that this is a "software problem" that can be fixed with the proper application of code and databases. Others may think the software is fine, it's a process issue. Others in turn may want to outsource the entire solution. "Whether we realize it or not, mental models help us think at the subconscious level. They shape what we see, what we choose to ignore, and what we miss entirely." ¹³ Until the models are brought out of the brain and into the world for examination, with some degree of detail, the implicit assumptions resting within a model remain buried, like corporate land mines, waiting until discovery (usually the hard way).

Anecdotally, many programmers have felt this revelatory effect without realizing it: A programmer stares at the bug which has stumped them for the past two hours. They call another developer over to look at the screen, and start to explain the problem. One of two things frequently happens at this point; either the stumped programmer suddenly, mid-explanation, snaps his fingers and says, "Wait! Never mind, I think I got it" and dives back to the keyboard, or the other programmer leans over and says, "Wait, did you mean to have a comma here?" In both cases, the act of explaining the mental model of the bug yielded up an incorrect assumption that was buried in the stumped programmer's mind. Once brought out of the brain, the assumption is spotted and discarded.

Writing something down forces the author to think more clearly, instead of resting on the feeling of having thought clearly.

Conclusion

The degradation of focus in writing is not exclusive to the software industry; liberal arts programs in academia find themselves facing existential questions as well. "The liberal arts, particularly the humanities, are far from thriving on most college and university campuses. Shifting market demands, financially wary stakeholders, and rampant politicization pose an immediate threat to the humanities." ¹⁴ Part of what drives this is the incorrect assumption that "learning to code" is the sole necessary focus for success in a technical career. But if coding requires thinking, and thinking is based on incorrect models, then it behooves any "coder" to have a tool in their toolbox that permits--forces, even--them to bring their mental model out, for both collaborative and analytical purposes.

Writing does that.

Appendix

Bibliography

Working Backwards, by Bryar, Carr; ISBN 978-1250267603

The PR-FAQ Framework, by Calbucci; ISBN 979-8991318921

Farnam Street blog; https://fs.blog

Endnotes

Or, perhaps, it is better to say that ChatGPT couldn't really provide without prompting, which would then be unnecessary--the act of getting the prompt out of the standup participant's head and into words is literally the point of the standup, and ChatGPT's participation would at that point be rendered entirely unnecessary.
↩
The S-Team is the senior-most leadership team at Amazon; basically, Jeff Bezos and his direct reports.
↩
Working Backwards, p. 80
↩
Working Backwards, p. 80
↩
Working Backwards, p. 81
↩
Working Backwards, p. 95
↩
In many fiction-writing workshops, when an author puts a particular piece up for review, the author is required to sit in the room with the reviewers, but is required to remain silent while the reviewers discuss. This allows the author to hear the discussion of her work first-hand, but does not allow the author the "crutch" of being able to explain what the readers "should have gotten". It is often simultaneously the most excrutiating and enlightening experiences an author can experience.
↩
A few examples: "I have only made this letter longer because I have not had the time to make it shorter." -- Blaise Pascal (The Provincial Letters, Letter 16, 1657); "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." (Antoine de Saint-Exupéry, Airman's Odyssey)
↩
Farnam Street blog, https://fs.blog/schopenhauer-dangers-clickbate/
↩
For the record, no part of this essay was produced by ChatGPT.
↩
Quoted on Farnam Street blog, https://fs.blog/why-write/
↩
Quoted on Farnam Street blog, https://fs.blog/writing-to-think/
↩
The Great Mental Models Vol 1, p. 4
↩
https://manhattan.institute/article/protecting-the-liberal-arts-and-humanities-in-american-higher-education
↩

Book Review: Building A Debugger

Fri, 23 May 2025 00:00:00 +0000

No Starch Press sent me a copy of Sy Brand's "Building A Debugger", which walks, step by step, through the process of building your own assembly-level, native-executable debugger. It geeks me out in ways I haven't geeked out since... well, since the last low-level geekery book (the ARM assembly book) they sent me.

For close to two decades now, I've been telling developers (experts to neophytes) that you need to know "one level below the level at which you work". Knowing how your Java class turns into JVM bytecode, or how that .NET assembly looks like on disk, is often the difference between knowledge and ignorance. How does the yield return keyword work in C#? How did Java do inner classes before adopting nested-access classes in JDK 11 (or 12 or 15 or whenever the hell nested-access was introduced)? Being able to crack open the compiled binary, look at the disassembled output, and have at least a rough idea of what's going on, is a huge skill that every developer must have.

In my personal quest to pursue that, I've deep-dived into bytecode virtual machines (both the CLR and the JVM, among others), language compilers and interpreters (too numerous to count), and even an operating systems class in college. (I had quite the time explaining why an International Relations major wanted to take this class, let me tell yoU!) But as much as I've been aware of how a debugger works, I've never tried to build one, particularly not one that's based on the Linux OS. I did dabble once with how to do it in Win32, but that went as far as "attach the process, walk the PE file, and... yeah, that's about all I want to do". I never actually got to a point where I could set a breakpoint, examine memory, any of that. It was a pretty shallow dipping-of-the-toe and that was about it.

Sy Brand has managed to not only tell me the water's fine, but snuck up behind me and pushed me in.

Physical

It's a typical No Starch Press book--the covers are pretty solid, the binding has that crease that allows for the book to be opened pretty widely without cracking, and the pages are heavy enough to feel good. It weighs in around 700 or so pages and 20+ chapters, and each chapter is pretty singly-focused, making each chapter a nice "chunk" to peruse during a short reading stint.

Contrary to a lot of books (including ones I've written!), Brand doesn't spend the first chapter explaining the concepts of what's going to be covered--this is a book about building debuggers, and the assumption is that you know what a debugger is, what its basic functionality is and should be, and at least the basics of how one interacts with a debugger to, you know, debug programs. Instead, the Introduction does a lot of that "here's what we're going to do", and Chapter 1 dives right into laying out the project structure using CMake to build sdb, the "Sy Brand Debugger".

Oh, did I forget to mention? Brand uses C++17 to build the debugger, which actually makes a great deal of sense for the topic at hand. The book isn't about C++17, though, and explanatory text about C++ itself is left to an "explain it when we get there" approach. As someone who's not used C++ "in anger" in about twenty years--meaning my last foray with C++ was somewhere around C++03 or so--I think I'm just about at the level assumed for the book.

Likewise, CMake is not explained or explored, other than to offer up a cursory description of it and how it works. I can appreciate that--again, this is a book about building a debugger, not a book about CMake. And for the most part, the Chapter One steps are pretty easy to follow, although in my own tdb/"Ted DeBugger" setup, I found that (a) the latest CMake is 4 where the book assumes 3 (which I don't think will be a problem down the road), and (b) some of the supporting cast, vpkg and libedit, have some OS dependencies that I had to puzzle through myself.

Oh, and by the way, this is a book about building an x64 debugger, not an x86 or ARM debugger, so if you're on a recent MacBookPro, you're a little stuck. The choice to make this an x64 book is pretty reasonable--after all, x86 is ancient history, but not everybody has transitioned to ARM yet--but at the same time, it will limit the audience of the book. I have a vague idea to try and build an ARM debugger using the principles laid out in the book, but that's a long ways off for me, personally. (But if I were an editor at No Starch Press, I'd be thinking very seriously about putting Randall Hyde and Sy Brand into a room together and not let them out until a "Building an ARM Debugger" manuscript was on its way.)

Once the project is set up--complete with tests, by the way, yay--Chapter 2 walks the reader through the very basics of "Compilation and Computer Architecture". This is probably the fastest pass I think I've ever seen on topics ranging from CPU endianness and major pieces (registers, program counter, etc) to executable file formats and memory layout. From pages 11 to 25--fourteen pages in total--is not a lot of room to explain all that. Heck, Linkers and Loaders took an entire book just to explain executable file formats, and that was several hundred pages in size. My point here is, if you're not familiar with some of the material in Chapter 2, you'll probably want (or need) to supplement what's here with external resources. Again, this is a book about building a debugger, and it stays pretty tightly focused to that. The author won't leave you entirely out in the cold if you don't know what a DWARF or ELF file is, but you have to do your homework, too.

Once we get past that, though, from Chapter 3 onwards, it's "pick a subtopic in debuggers, and let's put it into our debugger" territory. Chapter 3 is about attaching to a target process, Chapter 4 is about automated testing, 5 is registers, 6 is testing those registers, 7 gets us to breakpoints, and so on, all the way up through Chapter 22's "Advanced Topics", which touches on remote debugging, among other things. As Brand covers elements of each topic, code is written in steps, sometimes refactoring along the way to make the debugger more maintainable and extensible. All of the "sdb" code is available via the author's GitHub, by the way, so while you don't need to type it all in by hand, I've often found that if I juts wanted to use somebody else's debugger, I'd brew install gdb, not "Alexa, order the book 'Building a Debugger'". Typing it in and working through the error messages is a part of learning the subject.

Thoughts

This book is going to come with me on a few upcoming trips, but then probably go to a shelf and stay there a while--it's a tutorial, not a reference, and like many tutorials, once you "get it", the need to go back and do the tutorial again is drastically reduced. Quite honestly, this is true of some of my favorite books, one of which is the ancient Windows++ book by Paul DiLascia, wherein he builds a C++ GUI framework for the Windows OS. Man, I worshipped that book for the longest time. Still have it on a shelf somewhere. Will probably never read it again, except for nostalgia's sake, but it did its thing and did it well--I learned so much about how to build an important part of what we were using at the time to write software.

Frankly, I wish books like this were more popular, but I'm simultaneously glad they aren't, because knowing this material is often what helps set the "senior developer" apart from the "junior". If you know how something works under the hood, you find the concepts carry over into all sorts of different places. Knowing how an x64 debugger works makes using a debugger on any CPU platform a much easier exercise, and too few programmers actually know how to make use of a debugger at all, much less use it well. On top of that, there's often opportunities to build "little tools" to help gain visibility into one-off situations and scenarios. (For example, knowing how debuggers work make it much easier to understand how code coverage tools work, not to mention certain observability tools and platforms. Oh, and did you ever want to "intercept" a native call to be able to provide some pre-/post-processing? Yeah, knowing how debuggers work open all of those options up inside your brain.)

Is this book worth learning C++ if that's never been a language you mastered? Eh, maybe. Honestly I think any developer who wants to crack the upper echelon of the industry needs to know at least a little C/C++, and this is a good exercise by which to go from "I know how to read it" to "I have built an interesting tool with it". But you could, conceivably, use any system-level language, like Rust, Nim, Zig, or maybe even Odin, to build this, assuming you know enough of how those languages work to be able to mentally translate the C/C++ into the language of choice.

Overall, I think you should buy it, read it, then tweak what you've learned into something that's useful for your own purposes. And who knows? Maybe next time there's a weird Heisenbug that's driving the rest of your team natty, you might think, "Wait, maybe if I write a little tool that hooks certain API calls and takes a snapshot of certain variables, we can get a better sense of what's going on...." and build that tool and get that better sense and be the brilliant idea that finds the bug and get the kudos and the raise and the promotion and the cheering parade and.... Uh, right, sorry, got lost in my own daydream for a second there.

Did I say buy this book yet? Yeah, go buy this book.

Disclaimer: No Starch Press sent me a review copy of the book; no other compensation was provided.

Synchronous Work, Asynchronous Work

Sat, 15 Mar 2025 00:00:00 +0000

tl;dr Over the last two years, we've seen a dramatic policy debate playing out on the feeds of LinkedIn: "WFH (Work From Home) vs RTO (Return to Office)". Nearly everyone has an opinion, and many (if not most) of them are held strongly. Some are held based on data, some on personal preference, and many are based on personal experience. Nearly all of them, however, focus on the wrong part of the debate: it's not really about "WFH vs RTO", but about "async vs sync".

Over the last five years, we've seen a remarkable transition/cycle in "work", thanks to the COVID-19 pandemic.

Everybody go home! Don't come back! (But keep working!)

When the pandemic lockdown was imposed, nearly every company on the face of the planet "went remote", doing everything they could to enable their employees to work from home, partly out of a desire to keep their employees, but primarily out of a desire to keep the company going. And for the two-plus years of lockdown, it actually worked out remarkably well--certainly, some jobs were heavily affected since they couldn't be done remotely (looking at you, every assembly-line job ever), but overall, for our industry, the shift was relatively smooth and painless. We discovered that Zoom/Teams/Slack/Discord and other tools could be used with great effect, though not without pain, and software was written, tested, deployed, and debugged.

Personal story: I was working for Rocket Mortgage at the time, and in the span of a week, the company went from the traditional "We expect all our employees to be physically present in our corporate office" (in Detroit, Michigan) the previous week to "We are monitoring the situation closely" on Monday to "Take your laptops home" (some of which hadn't left their laptop dock literally since the employee started working for the company one, three, or even ten-plus years earlier) by Friday. The Rocket IT team scrambled--successfully--to make sure that every single laptop could VPN in to the company network over a span of five days, and for the most part, the shift from "office" to "remote" was complete, at least at the technology level.

During the lockdown, loads of new social rules were quickly hammered out and then renegotiated as necessary. "Camera on" vs "camera off" became a nation-wide talking point for teams. Instituting a five-minute break between meetings for people to get up and take a bio break--even when there'd never been a perception of a need to do that with back-to-back meetings in different physical buildings--became a corporate policy. Suddenly, pets were welcomed "into" the (virtual) meetings.

And the world just laughed when we remembered the video of a small girl joining her dad's BBC news report from home, because suddenly that was all of us, particularly those with kids. Suddenly, every day was "take your kid to work day"!

Another personal note: Running the "Technology Culture" team at Rocket at the time, I was front-and-center on figuring out what the new "remote office" culture could/should be for the folks in technology. Not all the policies we created "stuck", and some definitely got adjusted as we stumbled along, but for the most part, it was a pretty successful culture shift. By the time I left, most of them had become habitual rather than enforced.

But more germane to this essay, several studies documented that WFH led to greater productivity. By 2022, the world seemed poised to enter the "next phase" of work, where all thanks to ubiquitous Internet access and telecommunication applications, all of us (or at least a significant number of us) would be working from home.

Everybody come back! No, seriously!

Then, as the lockdown eased up, companies started to push back on the WFH approach, as we began to see the "missing parts" of working together physically in the same place. The toll of being "trapped" in one's domicile began to weigh more and more heavily on people, and even the most introverted of employees found their mental health took a hit from the lack of casual social contact. Businesses local to the office(s) found their revenue was highly tied to people being in the office, and many cities began a campaign to entice people back to the downtown areas to help get those downtown businesses back on their feet or on a more solid footing.

Recognizing that some time together in the same place/time is better than none, many companies began to explore "hybrid" options wherein an employee spends some percentage of their time in the office, the rest working from home. Some policies sought to give their employees flexibility in the matter, others lay down core numbers (3 days here, 2 days there) by fiat. Meanwhile, some academic studies suggested that there might be objective reasons working from an office was more productive, though most critics of those studies quickly pointed out flaws in the study or in the lack of peer review (which, to be fair, was also true of the pro-WFH studies, too).

By 2023 and 2024, "return to office" mandates became more common, none more severe than Amazon's "everybody is back in the office five days a week" policy (re)introduced in late 2024. While some employees welcomed the return, many also balked, particularly since there seemed no credible reason for it. That in turn led corporate conspiracy theorists claimed it was because corporate building leases had to be justified, or that this was a filter designed to get people to "quiet quit" or as a test of loyalty.

My own thoughts

Though weighing in with my own thoughts on the debate isn't the point of this essay, there's still an inherent elephant in the room that needs to be addressed. If you don't care to hear my opinions, feel free to skip this section, but for those who are curious, let me lay down a few things that are facts, assumptions, or statements that I take to be axioms:

Fact: You will never get everybody under one roof. Let's face it, companies of any size larger than a startup are global affairs. You might have a good chunk of your people in one place, but there will always be salespeople out on the road, DevRel advocates at conferences, and the CEO (and CTO and others) out schmoozing at cocktail parties in posh places like Paris or San Jose. The bare-metal idea that "We all need to get into the same room" is kinda ludicrous even on the very surface of it. You might be able to get everybody into one of your office campuses, but everybody in the same building? For some companies, there is no building big enough.
Fact: We have no standardized metric for "productivity". Seriously. The various academic studies are all over the map on this, measuring anything from "time spent on the laptop" to "keystrokes recorded" to "amount of work submitted". In some cases, the studies abandoned any objective notion of "productivity" in favor of "preference", which is remarkably un-scientific: I may prefer something that is less productive, and I may prefer things that are more productive, but my preferences are really a terrible measurement when considering options (with one exception, and that's--literally--employee contentment). Until we can get some kind of objective measurement of "productivity", we often end up arguing past each other as we each cite studies that seem to defend our intuition going into the debate.
Axiom: It is absolutely possible to lead a team or a department well while doing so remotely. I've done it--I built a team-of-teams literally from nothing at the start of the pandemic to a department of 30 people all entirely remotely, and it was one of the closest groups I think I've ever been a part of. Many of the people in that group are still in touch with me and each other, five years (and several job-hops) later. (Odd note: Some of the people I hired, I still haven't actually met in person yet.) More to the point, many companies have operated entirely remotely and been wildly successful at it, and many others report hybrid success. Even if we allow for a certain amount of "That company was going to succeed regardless of--or even in spite of--their management", there's still too large a number of successes to write off their success as entirely irrelevant to their management style.
Axiom: Running a team remotely is a skill. It takes a very different mindset and a different set of tools. Some managers are prepared to work with those tools, some aren't. Most (all) can learn them, so it seems to me that some just choose not to.
Axiom: Talent exists everywhere. With the growth of online learning (and the two-century-old societal trend of a "mobile workforce"), no longer is the best talent exclusive to the big cities.
Axiom: Temand for software exists everywhere. Sometimes, the company lives in Warsaw, Indiana (home to a software company, I'm serious, they do crop insurance).
Axiom: Much communication is single-directional. Not every meeting is about soliciting feedback or exchanging ideas; sometimes executives just want to tell you stuff, and sometimes (much more rarely) they want you to tell them stuff.
Axiom: Relationships deepen with physical, in-person time. There is still no substitute for the experience of breaking bread together. I doubt there ever will be. If you want to get to know somebody better, on a personal, connected level, you have to have an overlapping proximate spot on the space-time continuum.
Assumption: Some work happens more smoothly together, while some work happens more smoothly alone. Doing a brainstorm on a software design is still best done clustered around a whiteboard--I haven't found any online tool that's anywhere near as smooth as taking over a conference room with whiteboards, sticky notes, and pens. Sitting at your computer writing code, however, is definitely something that is best done without others around, unless you're pairing (which is actually quite doable--if not arguably better--over Zoom/Teams/etc).
Opinion: Ad-hoc conversations are, still, difficult to enable remotely. Others can't overhear what two people on the team are discussing, and those sorts of overheard conversations are sometimes what leads to incredibly deep insights. Even the simple chair-swivel-"Hey can I bother you" kinds of conversations are difficult to replicate remotely. Some companies experimented with Discord or Teams, giving each individual their own "room"/private channel that was open to everyone except when the individual was in a "don't bother me" mode, making it trivial to "knock"/"ping" another individual to have the ad-hoc conversation, but it doesn't seem to have taken off as a standard practice.
Opinion: Coming all the way in to the office to put on headphones and be in Zoom meetings is ridiculous. (That one kinda just doesn't need explanation, I don't think.)

Analysis

If it's not clear by now, I ride the fence a little here, but only a little--I think WFH is a powerful tool/option, and frankly, I think a great deal of software development work can/should be done from home in order to obtain the lack-of-overhead (that is to say, commute time and costs) that WFH enables. I also think that several acts of software development benefit from being in person, and that it's useful for team members to spend time in one another's presence as a bonding activity. All together, I suppose it plants me firmly in the "hybrid" part of the spectrum, which is pretty consistent for me dating back to before the pandemic--long before COVID, when working (sometimes locally, sometimes remotely) as an independent consultant, I often looked to go (flying out if necessary) to the company/client to have periodic meetings in person, for all the reasons I just cited. But I also found that trying to work every day in the office was distracting and often counterproductive.

Meanwhile, the debate rages on, probably continuing to do so for the remainder of this decade, if not longer. What I find most tragic about the whole discussion, though, is my belief that most of the debate is over the wrong thing: it isn't about where you do the work, it's about how the work gets done. In particular, differentiating between synchronous and asynchronous work, and attaching the right kind of each to a particular problem, helps eliminate much of the ambiguity as to which (WFH or RTO) is more "productive" and allows teams to maximize their use of each.

Sync vs async work

Realistically, the sync-vs-async work is much like the sync-vs-async execution model; certain things lend themselves well to synchronous collaborative execution, and other things are highly independent and therefore can be executed asynchronously. I use the term "synchronous" here to mean "you and I have to chat with one another to get the work done", and "asynchronous" to mean "I can charge forward without needing any additional input". These are probably not entirely rock-solid definitions--I'm still fleshing some of this out in my head as I write it--but it works well enough for the moment.

It may be best to explain this with a series of examples:

Writing code: Asynchronous. Generally we think of writing code as a solitary activity, at least among the generation of developers I grew up among. Just you, your keyboard, your mouse (as rarely as possible), your IDE, and whatever specs or stories you have in front of you that need doing. Crank up the music, time to get into the ZONE.
Pairing on code: Synchronous. And yet, when you pair on some code, you need to both be "present" (literally or figuratively) at the same time, utilizing a high-bandwidth form of communication between you (e.g., the tiny amount of physical space between you so you can talk to one another, or a real-time video connection, or even a phone call while staring at a screen-share).
Code review: Depends. There's certain forms of code review where I sit in front of my machine, taking notes as I go over the code, and then there's code reviews where we collectively sit around and discuss it amongst each other. This is probably one of those two-step activities, where the first step is asynchronous (I study the code) and the second is synchronous (we discuss our respective "takes" on the review and next steps).
Architecture or design brainstorm: Depends. There's clearly value in several of us gathered around a whiteboard, each with a pen and eraser and scribbling things in a real-time caffeine-fueled frenzy, but there's also significant value in sitting in a quiet room by oneself, musing and sketching and doing individual thinking-and-processing-and-ideating. This is probably going to be something that will land differently with different people, but it's safe to assume, I think, that there's likely an iterative async/sync/async/sync cycle here: Ruminate on something for a while on your own, then bring it to the table with others to review/dissect/dissent, then take the pieces back into the quiet room for a while more, then bring it back to the table, ad nauseam.
Debugging: Asynchronous. Except, can't be overstated, so often the best debugger in the world is another person! I've frequently told students that if you're stuck for 20 minutes, walk away from the screen then try again, but if you're stuck for two hours, try explaining the problem to another person, because just the act of explaining it will catalyze parts of your thinking and give you new ideas to try or insights to explore.
Discovering requirements/stories: Synchronous. Meeting with customers and/or product folks and/or business owners and/or anybody else who has a stake in what the final product looks like is usually a back-and-forth sort of exercise, requiring high-fidelity communication and often physical props. This is probably the poster child of "in person" kinds of work; even doing this over a meeting technology (Teams, Zoom, Meet, etc) can be awkward in places.

Going through this leaves us with a growing conclusion that I think holds pretty well: Sync and async are not binary, but a continuum. The various technologies that we have make sync vs async somewhat relative. Back in the day before electronic communication (even email), sync meant "We are physically gathered together" and async meant "We are physically apart". The invention of postal mail ("snail mail") meant that communication between async nodes was possible albeit slow. The telegraph made the async connection loops faster, but we don't really get to a "physically remote sync" status until the advent of the telephone. And so on--as the mechanics of communication get faster, we get higher fidelity exchange of information. Note that in some scenarios, however, such as a book, a large amount of communication can happen, but only in a single direction, which creates a low-synchronous/high-fidelity combination (and one that is pretty rare, and often highly time-consuming).

This begins to turn the "sync/async" picture into a two-axis graph of "sychronicity" (how strongly do we have to be in sync with one another) and fidelity (how strongly can we convey ideas to each other). Taking a rough first pass at this, we get something like this:

quadrantChart
    title What We Do and How We Do It
    x-axis Low Synchronicity --> High synchronicity
    y-axis Low Fidelity --> High Fidelity
    Code review:::work: [0.45, 0.5]
    Code:::work: [0.1, 0.2]
    Pairing:::work: [0.85, 0.85]
    Brainstorming:::work: [0.95, 0.95]
    Documentation:::work: [0.3, 0.8]
    Snail mail:::mechanic: [0.1, 0.1]
    Email:::mechanic: [0.2, 0.2]
    Prose:::mechanic: [0.1, 0.8]
    Meeting, software:::mechanic: [0.8, 0.8]
    Meeting, physical:::mechanic: [0.9, 0.9]
    "Chat (Zoom, etc)":::mechanic: [0.65, 0.5]
    SMS:::mechanic: [0.4, 0.4]
    classDef work radius: 5, stroke-color: #310085
    classDef mechanic radius: 30, color: #00ff33, stroke-color: #10f0f0

The upshot? For those activities which are high-sync/high-fidelity, we probably want to be in the office together; for those which are low-sync/low-fidelity, we probably can (and want) to be WFH (or rather, "working from anywhere"). The other two (low-sync/high-fidelity and high-sync/low-fidelity)? Either they're one-offs, or they're just not a thing for us.

(Caveat: This is a first pass, thinking-out-loud sort of graph, so maybe it doesn't fit, but I've found it useful to think this way when thinking about this topic. Use as starting point for your own thinking, or discard it entirely, if you wish.)

Embrace Change (Not Perfection)

Thu, 13 Mar 2025 00:00:00 +0000

tl;dr Ever had one of those situations where you find that some data about your engagement with a company or institution is incorrect, go through the motions to correct it, only to discover it has mysteriously changed back to the original, incorrect, value? The other day I was driving with my wife back from some doctor's appointments and we were talking about some social media friends she has that were complaining about the same. It's the modern take on "tilting at windmills", yet it's so common we just accept it as an everyday part of modern life. It got me thinking about the problem from a software perspective, rather than a human or corporate perspective. And I think we, software developers, are partly to blame for the situation. Incorrect data, it seems, is impossible to correct in any system larger than a single database.

Consider, as an example, the classic TV trope (such a "King of the Hill" episode) where a beloved character is suddenly thrust into an obviously incorrect situation (Hank's driver's license, on renewal, lists him as being female instead of male), at which point "hijinks ensure" (as the old TV Guide used to say). Unfortunately, for those who've had to try and correct one of those "obvious" errors, large organizations have a deep stubbornness to them, and their databases even more so. That in turn means that even though the situation might correct itself once, it's entirely possible that it repeats itself later for no apparent reason whatsoever. (Hank might convince the local Arlen branch to update his gender on his license, but it's entirely possible that the next time he renews, it's back to listing him as female because the state has him listed as female in one of their systems, thanks to the erroneous value in the Arlen branch.)

Setting aside the comedic value in such situations, from a software engineer's perspective, how is this not a deeply embarrassing indictment of our industry? Think about it--we will spend weeks or months on writing code to try and keep invalid or incorrect data out of the system, yet when it happens, we shrug it off. "That's a data problem, nothing wrong with my code," engineers have been wont to say (and hide behind) during post-mortem meetings--assuming, even, that such a meeting takes place after such an error occurs. (Spoiler alert: No such meetings ever take place.) That's like pilots gathering around a TV to watch a re-enactment of a all-hands-lost plane crash and giggling at the "comedy of errors" the cockpit crew is going through, proudly claiming, "That's never happened on my plane!".

While we computer scientists might try to hide behind the idea that "Well, it's all human error somehow" (which is what the NTSB almost always cites in a plane crash, by the way), but the fact is we've spent little to no time owning the problem, and I think it's high time we should. (Keep in mind, in this post I don't really think I have solutions, but the first step to solving any problem is admitting that you have a problem!)

It's not an easy problem to solve, and Lord knows it's a thankless issue that will never net anyone a Turing Award or Nobel Prize, but when's the last time you applauded when the airplane landed at the airport safely (or, even better, when the maintenance crew comes on board and delays the flight by three hours to replace a sensor in the cockpit)? Practicing these sorts of industrial discipline are never exciting or headline-generating, but when you total up the time and energy lost to chasing these kinds of gremlins, not to mention the potential downstream errors from the incorrect data (perhaps Hank's auto insurance policy goes up in cost, or even cancels altogether, because now the insurance company has him listed as a woman driver--because this is Texas, after all, and...), we really need to address this particular elephant in the room.

To be clear, I don't have a proven solution here; I'm speculating and making suggestions. My hope is that it sparks some kind of discussion or brainstorm, but my realistic assessment is that I'm shouting into the full-throated roar of "Hurricane Leave-It-Alone". We've lived with this problem for fifty years (or more), so we literally have multiple generations of software developers who've literally grown up with this problem (and can't imagine it working any other way).

Before jumping into solutions, let's examine a few salient points to make that color the problem.

Axiom: Bad Data Enters the System

Humans are a wondrously unstable sort who constantly find ways around fences and gates to do the things they aren't. "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning." (Rick Cook, The Wizardry Compiled) If a human has any input into a computer program, then there's a way the human can screw it up. We cannot eliminate human error short of eliminating the human entirely.

That then leads us to realize that no matter how many layers of validation exist, there will be a path that allows for semantically incorrect data to enter. Go back to our original example: Hank Hill being listed as "female" is not invalid data, since "female" is an acceptable value for "gender", but it is incorrect. (This raises the important distinction that not all valid data is correct data.)

Thus, if errors can always happen, then we need to stop striving to prevent the impossible, and instead promote the recovery of the incorrect.

Never assume it's right the first time. There's some interesting parallels here between incorrect data entering a system and incorrect software being built. For different reasons, both exercises have to deal with incorrectness in the system, which is best eliminated by rapid cycles of feedback, each of which is followed by short, sharp iterations to correct what's wrong.

Axiom: Multiple Databases Exist

In any organization that is larger than a brand-new startup, there's more than one database at work. (In fact, it may even be true that most 2020-era startups have multiple simultaneously-active databases, which only underscores the point even further.) When data denormalization is the norm, even within a single database, we run the risk that a particular datum (such as Hank Hill's gender) is duplicated.

We need these multiple databases for purposes of relativity, because attempts to create a "single-source-of-truth" database has been all but proven incapable of existence by this point in time. Companies that pour millions (if not billions) into "master data management" routinely find the effort extremely taxing, and the MDM model is often wildly oversimplified (a flat table of 250 columns for billions of rows, all of them NULLable, with little to no relational enforcement) or wildly overcomplicated (metamodels, meta-metamodels, and meta-meta-metamodels are not unknown). Thus, each "significant component" of the enterprise (be that an organization or a service) makes use of a localized collection of the data it needs to carry out its operations: Shipping wants all the details of shipping labels, addresses, and postal codes about a customer, whereas Marketing's interest in customers is all about the order history (and none at all in where those orders went). Each component in the enterprise wants a database with the sliver of truth that's relative to its operation.

We also need these multiple databases for purposes of scale, because in any non-trivial enterprise, the amount of data being tracked can easily exceed the storage capacity of a single computer system. Today's cloud-hosted systems can automatically partition data out across multiple disks, but even they have limits (operational, physical, and/or financial). Hosting ten smaller databases is vastly simpler and cheaper than trying to store and access all data within a single entity.

The fact of multiple databases, each with a fraction (sometimes large, sometimes small) of the necessary data, means that at any given point in time, a database only holds a portion of the truth of the world as captured in its system. Those two qualifiers are critical, by the way--"a portion of the truth" and "as captured in its system" means that a database, by definition, does not contain truth, but only a relative sliver of it, and as soon as it is captured, it is incorrect. In other words, its portion of the truth is only relevant for a period of time, and is only the truth as we knew it at that time.

What is truth? Philosophers have debated this for centuries, but database engineers and DBAs seem to treat any row/column pair as sacrosanct and true until proven false, even if it disagrees with what the outside world thinks. Unfortunately, they've propagated that mindset to everybody using the database, such that now, we assume the database to always be correct until proven otherwise, even though we know that humans operate databases, and humans make mistakes. Some systems are so chock-full of errors that their operators have gotten to the point of assuming the database is incorrect by default, but that's more the exception than the rule.

Axiom: Databases Must Constantly Sync with Each Other

Knowing that we must have multiple databases, and that each one is only a sliver of the truth at a given point of time, we must embrace the fact that for almost any component in the enterprise, a given database is not the single source of all data needed--we often must reference other databases for information they need in order to proceed. Consider, for example, the canonical e-commerce site. In order to ship an order, the Shipping system must reference the Orders system to get the order to process (to find what items to pull from inventory bins and the like), and also the Customer system (to find the Customer's shipping address).

If you're the architect of the Shipping system, you want to keep this data handy--nothing kills your performance like trying to make multiple round trips to remote systems--but you also know that the "remote" data can change without your knowledge. So you often set up a batch approach where you'll take changes to your local database, and forward them "in batch" (that is, gather up a whole slew of changes into one bundle) to the databases that might be interested in your local changes. We sync our local data to other data systems in order to get their updates to what they know to be the truth. (Or think they know, anyway.)

This then, becomes the heart of the problem: Somewhere along the way, someone entered Hank's gender incorrectly, and the bundled batch pushed it downstream, because it was a change. Despite the fact that this new data might not have been verified, validated, or confirmed, it was a change, and therefore it needed to be batched and sent. And when the downstream system got the change, it faithfully updated its local copy of that data, which in turn may trigger further downstream changes.

And suddenly, Hank finds himself seeing tampon ads on his phone.

Solution Ruminations

Given, then, that:

bad data enters the system
multiple databases continue to exist
databases continuously update data back and forth between them

... we find that errors will continously cycle and propagate until eradicated in every database.

Given that we cannot prevent the error, we have to then embrace the idea of identifying and correcting the error. A corollary to that is to do so across all the systems it has entered, lest the error re-propagate back to the system in which it was just corrected.

We can do that in a variety of ways:

store everything in a single database (the "Master Database" Fallacy). There's a plethora of reasons why this doesn't work, some of it technical (limitations on scalability and performance) and some of it organizational (the pain incorporated when you have a single dependency across the entire organization). For these reasons (and more), most organizations find it untenable or unacceptable to take this path. (A great many more organizations still try to create a Master Database, though, and find this out the hard way.)
create a mechanism where a particular update can "override" any further updates to that datum (in other words, formalizing the "System of Record" approach). This starts to get into the business of "weighing" different facts against one another, where one fact may be given as "more true" (or perhaps, better stated, "more likely to be true") than others. Hank's assertion that he is male should be given more weight than an update coming from a data-entry clerk three departments over, particularly if Hank has a birth certificate handy. (Of course, if this is the system storing birth certificates, maybe we weight things to favor the original input of the nurse and parents, and thus do we start to run into the Gordian Knot of weighted facts--how we decide those weights is always subject to some deep and divisive debate.)
track and make visible to users where and how database updates occur (an "audit log" approach). This is partly embraced by the "Event Sourcing" approach to the storage of data, though we still run into limitations, since the data only knows what it knows local to itself--a given ES implementation doesn't track where or when or why the update occurred, only that it occurred.

My gut tells me that there's way to factor some of this more deeply into the database implementations themselves, but I haven't spent enough time thinking about it yet to have anything concrete. That said, I'd love to get some feedback on the idea, and whether or not I'm missing something deeply fundamental here or the solution is staring us in the face.

Debts, Tech and Otherwise

Sun, 9 Mar 2025 00:00:00 +0000

tl;dr A colleague of mine, Scott Porad (CTO, VP Engineering) posted on LinkedIn, asking, "What are all the other kinds of debt like tech debt?" He listed out a few, then asked for others to weigh in, and the list grew... kinda long. And interesting. And made me think about the metaphor more deeply.

His list included financial debt (the OG), Product debt (gaps in the product that impact revenue or lead to support costs), Ops Debt (manual processes performed by the operations team because the software doesn't support the processes), Process Debt (valuable steps of the software development process that are skipped causing quality or velocity issues), and Org Debt (key roles that are not staffed that cause quality or velocity issues).

He then asked others to weigh in with their favorites, and the answers poured in. Many (I won't list them all--partly because I want you to read the original post!) were interesting, and I think they fall into a variety of different categories:

Technological. Whether it's the data, the UX/Design, the backend, or even the choice to over-depend on open-source without "giving back". Documentation falls in here, too.
Procedural. What processes are being done by hand that cost the team time, money, or energy? What sorts of things are being hidden by a "lightweight process" focus (a la "agile") because companies are afraid to institute a process, lest they be called "not agile"?
Mental. What assumptions were made that turned out to be wrong? What assumptions were right "back then" but no longer hold? Is the organization chasing the right problems? What knowledge is missing from the people that need to know?
Cultural. Any group of humans forms a culture, but is that culture serving the business, or holding it back?
Talent. A few of the responses mentioned "youth" or "experience" debt, as in a lack of investment into junior talent, or a refusal to recognize the value of experienced veteran talent on the team. Likewise, having "not enough" in raw terms, e.g., understaffed teams.
Support. This is a more nebulous catch-all category but includes things like "marketing" (neglecting marketing practices over time, leading to damaged brand and customer relationship) or "legal" (failure to recognize the legal and/or regulatory necessities, bringing the company into legal jeopardy).

Again, I encourage readers to go check out Scott's original post. It was fascinating how much of it just poured out in the comments, and the wide-ranging nature of the responses.

Which got me thinking: Are these really "debts"? Or, more to the point, are they bad?

Debt: A Metaphor

If we go back to the original definition of "tech debt", if I remember correctly, I first heard it from Mike Clark back on the NFJS tour in the late 2000s, but Wikipedia tells me that Ward Cunningham first coined the term in 1992 as a metaphor to explain to his boss how making short-term decisions can boomerang and create problems over time, and therefore investment was needed to refactor the code. In particular, the point of the analogy was that "While technical debt can accelerate development in the short term, it may increase future costs and complexity if left unresolved." The official Wikipedia definition of "Technical debt", by the way, is this: "the implied cost of additional work in the future resulting from choosing an expedient solution over a more robust one."

Ward put it like this:

"Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."

Martin Fowler later wandered into this arena, too, pointing out that debt is sometimes taken on deliberately (a car loan, a mortgage) for good purposes, and sometimes it's accrued accidentally (credit card debt) because of poor planning or execution. He likened it to a quadrant along two axes: reckless vs prudent, deliberate vs inadvertent:

In almost all of the "pain" cases of debt, regardless of the kind, the worst quadrant to be in was Inadvertent/Reckless: We made bad decisions without realizing it, and paid for it later, painfully. Slightly better is Deliberate/Reckless, where we knowingly made a reckless decision (which raises its own questions), and still paid for it later, painfully. The "painfully" part is why most people react so strongly to debt, and to the debt metaphor, and why it often engenders its own kneejerk reactions when the topic appears.

When we look at those list of "debts", pretty much all of them can fit into the definition pretty well: for the subject S, we take on an implied cost of doing additional effort in the future in order to take on the expedient solution now, instead of the more robust one:

Technological: Data. We take on an implied cost of additional effort refactoring our data and breaking down the silos in order to take on the expedient solution of a denormalized database or maintaining separate independent databases now.
Technological: Architectural. We take on an implied cost of additional effort refactoring our architecture to be more scalable or more secure or more mobile-friendly in order to take on the expedient solution of shipping a basic three-layer CRUD app now.
Technological: UI/UX. We take on an implied cost of additional effort building out a more cohesive and coherent user experience in order to take on the expedient solution of shipping a clumsy functional UI now.
Talent: Youth debt. We take on an implied cost of additional effort mentoring more junior developers later in order to take on the expedient solution of having all senior engineers work on the project now.
Talent: Experience debt. We take on an implied cost of additional effort building experience in our domain or our systems later in order to take on the expedient solution of severing all the senior engineers on the project now.

... and so on. For each of the topics brought up in Scott's post, there is a very clear sense of "We are taking a short-term view which hurts us in the long run" feeling. And yes, most of us in the IT industry have a story like this (if not several times over) that we lived, and regretted, through.

But I think that's only half the story.

Debt and Evolution/Iteration

The more I go through each of these, the more I realize that there is another idea hiding in here, which is that sometimes we take on the debt because we don't know the future. Think about it this way: Does it make sense to invest heavily into a particular area (application, database, vision, process, talent, ...) if we find that after a few iterations, it's a mistake to take that approach? Think of your favorite startup unicorn story, and how often those unicorns needed to "pivot"--that is, quit what they were doing and try something else entirely different--in order to stay afloat or try something else? If the company hadn't taken the short-term approach, and instead invested a massive amount of time/energy/money into something that was now essentially worthless, all of that investment would now be a waste and therefore a loss.

Or worse, fall prey to the Sunk Cost Fallacy, and become convinced that "They can't give up now!", and fail to pivot entirely.

The truth is, in most practical cases of "debt acquisition", the principals involved in making the decision were faced with the basic conundrum of "Do we ship sooner rather than later, knowing that what we deliver sooner will have some inherent problems with it that will need to be addressed later?" Implicit in this question is the flip side, though, which often helps to present as part of the whole decision-making process: "Do we ship later rather than sooner, knowing that if we wait too long the window of opportunity will close partially or entirely and render the entire enterprise moot?" That "later" part of the question rarely gets uttered out loud, and never gets brought up in the post facto analysis of "We chose the expedient solution n months or years ago, and we got to this point now." Keep in mind, if they chose the long-term solution and failed, we're not around to discuss it! While it's easy to judge the decision post facto, like all decisions at the time they are made, the "right" choice is not nearly so clear. What's worse, we don't have great clarity on the efficacy of the choice here, because we only remember one-fourth of the result quadrant:

"Debt Results"	Choice: Debt/Expediency	Choice: No debt/"Right"
Failure result	No data/discussion	No data/discussion
Success result	"Agh, the pain!"	No pain to discuss

In other words, 25% of the squares on the above graph get 100% of the discussion and the angst. Which, by any reasonable interpretation of this data, suggests that the main reason we spend so much time castigating people for making the Debt choice is because that's the only one we talk about.

Debt as a Metaphor

I think, overall, that the debt metaphor works, and works well. It obviously applies to more than just "technical" debt, though, and as Scott's post suggests, there's alot of different kinds of debt we can incur as an organization. Fowler's Reckless-Prudent-Inadvertent-Deliberate quadrant helps categorize the conscious decision around debt, but clearly relies on the individuals involved to describe the differences between "reckless" and "prudent". In the moment, it may feel prudent to be focusing on the here-and-now solution--which, to be fair, it often is, as exemplified by the whole YAGNI (You Ain't Gonna Need It) movement of the agile mindset--and only later does it feel reckless. But that sort of post facto analysis is itself a privilege, because it means we've had the success thus far to be able to go back and reflect on those decisions and attempt to recover/refactor them.

Or look at it this way: How many of us would be perpetual renters had we not taken on the mortgage to buy our first home? How many people could actually get to the point where they had $500k, $1m, or even more, saved up in a bank, waiting to buy the home outright? Debt in of itself isn't bad; it's when it's Inadvertent/Reckless debt that we find the angst.

The Shapes of Data: Tabular

Wed, 5 Feb 2025 00:00:00 +0000

tl;dr One of the "OG" data formats, the tabular data structure, aka "the flat file", is still today a handy and reasonable way of exchanging data in an automatable fashion without significant integration work required. Its shape is ideal for a multitude of data molecules that all share the exact same contents.

Refresher

In a serialized representation of the tabular format, such as a CSV file:

each "line" (that is, sequence of characters terminated by the platform's recognized end-of-line sequence, such as a CR or CRLF pair) defines a single record
convention holds that the first ("0th") line defines the "atom names" (aka "column" or "field" names) within the record
each atom is separated by a delimiting character, which must itself be escaped somehow if the delimiter is a valid data value (the comma is most popular, but TAB characters are also a common choice since TABs rarely show up in the actual data)
each line is expected to have some kind of value defined for each atom that comprise the record
there is no uniqueness constraint anywhere in the representation--no keys and no identifiers
different "molecules" (types) must appear in different tabular spaces
there is no "reference" mechanism anywhere in the tabular shape
most tabular data storage is done solely as files--there are few-to-no "tabular databases" that provide query/edit capabilities (though, to be fair, since many relational databases can import/export tables as tabular files, it becomes a relatively easy task to import the flat data, query/modify it, then export it as a tabular file again)

Shape analysis

Within most programming languages, the tabular shape is often most easily represented as an array or list of strongly-typed records/structs. However, there are some distinct differences. In most O-O languages, an "array of objects" doesn't actually store the object, but only stores pointers/reference to objects, which allows for aliased repetition of an object in a collection:

Example 1:

// This is pseudo-Java/C#/C++:

Person p = new Person("Fred", "Flintstone", 39);
List<Person> people = new List<Person>();
people.append(p); // Here's Fred!
people.append(p); // Uh, here's Fred again!
people.append(p); // Wait, Fred, third time?

Notice that while we assume the array looks like this:

flowchart TB
    people-->p1[Fred]
    people-->p2[Fred]
    people-->p3[Fred]

In truth it's more like this:

flowchart TB
    people-->p1[cell 0]-->person[Fred]
    people-->p2[cell 1]-->person[Fred]
    people-->p3[cell 2]-->person[Fred]

In other words, each cell in the array is pointing to the same object. We certainly could create the more "tabular-correct" representation, like so:

Example 2:

// This is pseudo-Java/C#/C++:

List<Person> people = new List<Person>();
people.append(new Person("Fred", "Flintstone", 39)); // Here's Fred!
people.append(new Person("Fred", "Flintstone", 39)); // Uh, here's Fred again!
people.append(new Person("Fred", "Flintstone", 39)); // Wait, Fred, third time?

But this now has three distinct instances of Fred running around, and that duplication might cause havoc in other ways.

(Hmm, I think they actually did an episode of the Flintstones where that exact thing happened.... Yup, season 4, episode 104! )

While the difference here is subtle, the key is that in the object/language scenario, we have references to Fred, rather than copies of Fred, such that if we change Example 1 slightly to read:

Example 1A:

// This is pseudo-Java/C#/C++:

Person p = new Person("Fred", "Flintstone", 39);
List<Person> people = new List<Person>();
people.append(p); // Here's Fred!
people.append(p); // Uh, here's Fred again!
people.append(p); // Wait, Fred, third time?
people[0].firstName = "Wilma";

Now the array appears to contain three Wilma Flintstones. This is clearly different than in Example 2, where have distinct copies of each means only the first Person is modified.

In addition, the lack of any reference mechanism means that if we widen the definition of "Person" to include a spouse as a fourth atom, we run smack into the problem that we cannot reference another Person in the tabular format; the best we can do is copy some of the atoms and do a "fixup" later, a la:

FirstName	LastName	Age	SpouseFirstName	SpouseLastName
Fred	Flintstone	39	Wilma	Flintstone
Wilma	Flintstone	35	Fred	Flintstone
Barney	Rubble	38	Betty	Rbuble
Betty	Rubble	35	Barney	Flintstone

(Column labels appear only as a convenience to us humans.)

Notice, however, that the mistakes (one syntactically incorrect, one semantically incorrect) in the Rubbles' records are completely acceptable to the tabular data store, leaving it up to the developer to catch "by hand". This lack of verification/validation is a rampant problem in any flat-file-based data interchange, and has plagued developers for years.

The Shapes of Data: Relational

Tue, 4 Feb 2025 00:00:00 +0000

tl;dr Relational data is, contrary to popular belief, characterized not by "tables", but by sets and relational variables (also known as "relvars"), and making use of a relational algebra and predicate calculus to make it easier to do set-oriented operations.

Refresher

Because most developers come to relational databases via SQL, usually either studied briefly in classes at university or as-necessary to get the work done while on the job, they often don't realize the relational model is much more academic and foundational than the flavor of SQL they've come to use and know. [Date13] ¹ and [Fussell] ² define the relational model as characterized by relation, attribute, tuple, relation value and relation variable.

A relation is, at its heart, a truth predicate about the world, a statement of facts (attributes) that provide meaning to the predicate. For example, we may define the relation "PERSON" as {SSN, Name, City}, which states that "there exists a PERSON with a Social Security Number SSN who lives in City and is called Name". Note that in a relation, attribute ordering is entirely unspecified.
A tuple is a truth statement within the context of a relation, a set of attribute values that match the required set of attributes in the relation, such as "{PERSON SSN='123-45-6789' Name='Catherine Kennedy' City='Seattle'}". Note that two tuples are considered identical if their relation and attribute values are also identical. (This is in contrast to an object system in which two objects are only identical if they have the same unique identifier--typically a memory address.)
A relation value, then, is a combination of a relation and a set of tuples that match that relation, and a relation variable is, like most variables, a placeholder for a given relation, but can change value over time. Thus, a relation variable People can be written to hold the relation {PERSON}, and consist of the relation value
```
{ {PERSON SSN='123-45-6789' Name='Catherine Kennedy' City='Seattle'},
  {PERSON SSN='321-54-9876' Name='Charlotte Neward' City='Redmond'},
  {PERSON SSN='213-45-6978' Name='Cathi Gero' City='Redmond'} }
```
Thanks to our familiarity with tabular storage models, we ofen write this relation variable in a tabular format like so:

SSN Name City

123-45-6789 Catherine Kennedy Seattle

321-54-9876 Charlotte Neward Redmond

213-45-6978 Cathi Gero Redmond

Keep in mind, however, that in the model itself, the tuple is not exactly a row, and each tuple knows both the name and the value of each of its atoms.
We can then operate against these relation values using using a set of operators (described in some detail in Chapter 7 of [Date04] ³): restrict, project, product, join, divide, union, intersection and difference. We modify People by using the union binary operator (let's call it +) on it and a second, unnamed, relation variable containing a single tuple:
```
People = People + {PERSON SSN='312-54-8796' Name='Ted Neward' City='Redmond'}
```
After this operation, People now holds four tuples. Additional operators can be used to create derived relation values, relations that are calculated from other relation values in the database--for example, we can create a relation value that demonstrates the number of people living in individual cities by making use of the project and restrict operators across the People relation variable defined above.

SSN	Name	City
123-45-6789	Catherine Kennedy	Seattle
321-54-9876	Charlotte Neward	Redmond
213-45-6978	Cathi Gero	Redmond

The other half of relational models is the deep understanding they have of relationships between relational variables.

The other half of the relational model is that of the formal relationships that can be expressed and understood by the database; we can define a relational value to have a unique identifying value (a primary key), and then reference that value within tuples.

Shape analysis

Popular implementations

Databricks
IBM DB2
H2
MariaDB
Microsoft Access
Microsoft SQL Server
MySQL
Oracle DB
PostgreSQL
Snowflake
SQLite

Relational Theory for Computer Professionals, C.J. Date, OReilly 2013
↩
Can't remember this reference right now
↩
Introduction to Database Systems (8th Ed), C.J. Date, Addison-Wesley 2004
↩

The Shapes of Data: Object

Mon, 3 Feb 2025 00:00:00 +0000

tl;dr Objects, despite being the most common tool form of mainstream programming languages, are often not as well-understood as a data concept as one might think. In an object data model, entities are defined as unions of state and behavior (and behavior is often of much less concern to the data modeler) that in turn can be related to other objects through a variety of mechanisms (type, ownership, association, and so on).

Refresher

Object systems are typically characterized by some basic components:

State is a pretty self-explanatory concept, most closely akin to "the data". Typically this is made up of a series of named atoms that are closely bound.
Identity is an implicit concept in most O-O languages, in that a given object has a unique identity that is distinct from its state (the value of its internal fields)--two objects with the same state are still separate and distinct objects, despite being bit-for-bit mirrors of one another. This is the "identity vs. equivalence" discussion that occurs in languages like C++, C# or Java, where developers must distinguish between "a == b" and "a.equals(b)"; where the first indicates whether the two references are pointing to the same objects, and the second determines whether the two objects contain the same state. It is an important caveat to any object system that while identity implies state, state doesn't imply identity.
Identity then also gives rise to association, in that objects can be referenced from one another through the use of either an implicit or explicit pointer as either an explicit or implicit atom within the object. Note that associations in most (if not all) object systems are one-way: If Person holds a reference to Address, the Address has no implicit knowledge of that association and if we wish to navigate from the Address back to the Person that references it, we must explicitly create and manage that atom in Address. (Note that in most object systems any non-atomic state is held through association, with the exception of "value types" in .NET or just in C++ in general, where the ownership is more explicit via pointers/references.)
The behavior of an object is fairly easy to see, a collection of operations clients can invoke to manipulate, examine, or interact with objects in some fashion. (This is what distinguishes objects from passive data structures in a procedural language like C.) Ironically, this is also one area that frequently has zero impact in object database systems, since every OODBMS released thus far has zero ability to execute object behavior--only retrieve/store/modify a collection of objects.
Many object systems also support either an explicit or implicit form of inheritance, in which an object "inherits" the properties of another, either "by value" (by creating full standalone copies of atoms from the parent in the child, such as what we see in Java or C# or C++) or "by reference" (by holding a reference to the parent, such as what we see in Javascript).

Shape analysis

The one-way nature of references in object systems often creates some design opportunities and restrictions. For example, in the following code:

In other words, working with a Person type that looks like:

class Person {
    private String name;
    private Person spouse;
    private List<Person> children;
}

Person ted = new Person("Ted");
Person charlotte = new Person("Charlotte");
ted.spouse = charlotte;
charlotte.children.append(new Person("Michael"));
charlotte.children.append(new Person("Matthew"));
ted.children = charlotte.children; // deliberately point to same List

... not only is marriage not implicitly reflexive, but neither is parentage.

Inheritance is another form of object association. Like the other forms, this associative relationship is usually one way; that is, children know their parents but not the other way around. In O-O programming, the parent can remain entirely ignorant of any derived classes through the use of dynamic dispatch (virtual methods), but this doesn't typically apply as a need in an object database. (An object query language obviates much of the need here.)

While identity often appears in other storage systems (most notably relational databases, where identity is captured via primary key)***, in object systems it tends to be more subtle***, where the object identifier (OID) is often implicitly defined and stored. (This is similar to the "ROWID" that is often found in relational tables that serves as the unique identifier for the row in the table, regardless of primary key definitions.) This implicit identity can sometimes be a little confusing around objects that are equivalent but not identical, in the same way that it can be confusing in object languages.

Most object databases supported some form of query language, either a SQL derivative and/or utilizing the syntax of the programming language (db4o called this 'naive' queries), and/or using a language-level API to build an exemplar object by which to search (called 'query by example' or QBE). This also often included a subtle parameter called "fetch depth", which essentially described how many object-association "links" to go down when retrieving the graph of objects to retrieve. For example, a fetch depth of 0 meant fetch the object queried and nothing else (leaving any associations empty or lazily-loaded), whereas a fetch depth of 1 meant fetch the object queried as well as the objects directly referenced from that object, and a fetch depth of 2 meant go to the objects directly referenced from the objects directly referenced from the object queried, and so on.

In other words, working with a Person type that looks like:

class Person {
    private String name;
    private Person spouse;
    private List<Person> children;
}

Person ted = new Person("Ted");
Person charlotte = new Person("Charlotte");
ted.spouse = charlotte; charlotte.spouse = ted;
charlotte.children.append(new Person("Michael"));
charlotte.children.append(new Person("Matthew"));
ted.children = charlotte.children; // deliberately point to same List

We would get a graph that looks like:

flowchart TB
    ted[Ted]-->charlotte[Charlotte]
    charlotte-->ted
    ted-->kids[List]
    charlotte-->kids
    kids-->michael[Michael]
    kids-->matthew[Matthew]

Note that if Persons know their parents, things get pretty messy pretty fast:

flowchart TB
    ted[Ted]-->charlotte[Charlotte]
    charlotte-->ted
    ted-->kids[List]
    charlotte-->kids
    kids-->michael[Michael]
    michael-->ted
    michael-->charlotte
    kids-->matthew[Matthew]
    matthew-->ted
    matthew-->charlotte

But this also highlights that in an object system, there is not a single well-defined root/starting point. The collection of objects can, and frequently is, a cyclic graph that can be "entered" (for query purposes) from anywhere--this is in contrast to associative systems (where we can only query by the key) or hierarchical systems (where we must work from the document's root node on down).

Most object database query engines also understand inheritance implicitly, so in a simple hierarchy like this:

classDiagram
    class Animal
    Vehicle <|-- Car
    Vehicle <|-- Boat
    Vehicle <|-- Airplane

... the database understood that a query of "select all Vehicles" would retrieve Car instances, Boat instances, and Airplane instances, as well as any Vehicle instances. This could lead to some very fine-grained queries ("select all Vehicles that have a passenger compartment of 1 and a range of over 500 miles except for motorcycles and rocket ships"), which could also create some indexing nightmares.

In many cases, an object database could only retrieve whole objects, even if only a subset of the object graph was necessary/desired to satisfy the request. This could lead to some performance issues, but from a design perspective mostly meant that any activity had to be thought in terms of "whole objects". For example, we often employ a "master-detail" approach, where we fetch a list of entities (such as students at a university) and display their identifying characteristics (first and last name, and student identifier) in a large list, prompting the user to select one for in-depth (displaying all of the selected student's information) examination. In a system which supports "partial results", such as SQL where we can "select id, fname, lname from students" and retrieve only three columns out of however many are defined on students, we can efficiently retrieve only the data we wish to display on that "master" UI. In a "whole results" system like a traditional object store, we have to fetch all of the student objects, displaying first the full collection's first and last names then after that the selected student's details. This means fetching the entirety of the student object in the bulk query, but also means no further round trips to the storage engine are required.

Thought Experiment: We often saw the desire in database systems for local code execution within the same node or process as the database server--in the RDBMS world we call them "stored procedures" and managed them through the database SQL interface--but the same concept never really took off within the OODBMS, where it would make actually much more sense. Some object query languages could invoke an object method as part of a query expression, but in general that didn't allow for object modification. It's intriguing to imagine what the overall experience of an OODBMS would be like if it could be a full object execution node, and not just data storage. Might even start resembling more of the old "distrubed objects" space, which is both a good thing and a bad thing....

Thought Experiment: Although unconventional, we've found in the hierarhical database world (namely, XML) that a query language (XPath) used against an in-memory collection of entities (an Infoset document, usually in a DOM) to be a powerful way to interact with only the dataset I'm interested in. Curiously, what would happen if we tried to apply a similar thought process to large in-memory object collections? Feels like it could be useful....

Implementations

A full list of object databases, apparently all of which remain in current use, can be found at https://db-engines.com/en/ranking/object+oriented+dbms . Note that this list does mix multi-model databases with object ones, which seems to be a common theme in a lot of lists like this.

db4o - Used to be an open-source product from a company of the same name, then Versant bought them, released one or two more versions, then abandoned it. Last source drop (2015) is available at SourceForge, though there's some GitHub mirrors out there too.
ObjectDB

Sadly, wandering through object-shaped datastore implementations is like walking through a graveyard in the middle of the night--so many reactions of the "Oh, man, I'm sad to see them gone" and "Oh, what might've been if only...." sort.

The Shapes of Data: Hierarchical

Sun, 2 Feb 2025 00:00:00 +0000

tl;dr Hierarchically-shaped data is characterized around strictly acyclic data that are arranged in a parent-child relationship, starting from a single well-known root data node. The relationship between nodes is explicit, with the roles of parent and child clearly delineated, but the actual association between parent and child is typically implicit and immutable.

Refresher

Hierarchical data is most easily recognized as a "tree", such as that taught in every Computer Science course anywhere, where a well-defined root node has pointers to child nodes, each of which is also potentially a parent to other child nodes, ad infinitum (or at least until we run out of memory). In the data world, this is most easily seen in the XML Infoset, which is an abstract discussion and description of this hierarhical model.

Careful readers of the Infoset specification will note that the descriptions are all abstract and at no point refer to "tags" or "angle-brackets"; the traditional thinking around XML, that of a markup syntax similar to HTML, is but one way of serializing an Infoset document, and some of the later attempts at binary XML serialization serve as an interesting separation of "data model" from "data representation".

However, while many readers will be willing to accept XML as the canonical example of hierarchical data, they may not be quite so willing to accept the link between "JSON" or document databases as hierarchical data formats. I make this assertion based on the following axiomatic statements:

JSON documents start from a well-formed root. Any JSON document must start with either an array ([...]) or object ({ ... }). Any attempt to have multiple roots in a JSON document is considered an error and unparsable.
JSON documents can only represent parent-child associates. A JSON document cannot have a cyclic graph anywhere within it.

Both of these statements can be verified either by casual reading of the JSON specification, also explained in more graphical and plain prose here, and/or casual experimentation at the JSON linter website.

In addition, with respect to document databases, we note that a given document database, such as MongoDB or CouchDB, is formed of collections of documents, and each collection, like JSON itself, can be seen as an array of these JSON documents, such that the collection now serves as its own root. The database as a whole is often modeled in turn as a collection-of-collections, making the whole "collection of collections of documents" a hierarchical model, particularly when we consider that none of the popular document databases offer any "pointer" data types that can be silently traversed during query/retrieval.

Shape analysis

The shape of hierarchical data is wrapped, not surprisingly, around the top-down nature of a hierarchy. That is to say, that:

there is one well-known and well-formed root that serves as the starting point for navigation
every node, with the exception of the root, has a parent, and may have children; no other references, such as to siblings, are present
no cyclic references are allowed; in particular, no node may have a child node that is also its parent (or grandparent or great-grandparent or ...)
data is typically stored as nodes in the tree, sometimes with "special" flags
in some hierarchical stores (notably XML), each node may have one or more name-value pairs further describing the node (XML calls these attributes)

The hierarchical nature of the data, then, makes it an excellent choice for managing "top-down" kinds of systems, a la

company hierarchies
military organization
decision trees
biological taxonomies

Ironically, though, some of the data systems that start out feeling hierarchical, like human genealogy charts (I mean, parents have children, the end, right?) actually turn out to not be strictly hierarchical, owing largely to the fact that marriages can be dissolved, spouses can die, children can marry their uncles, and even more bizarre behavior. (I invite anyone who thinks a genealogical tree can be best represented as a hierarchy to map out the English royalty of the 1500s through the 1900s in a nice, well-ordered, singly-rooted tree--just try to figure out who the root is, to start!)

One thing that's interesting is that hierarchical data is actually one of the oldest ways to model information, which is partly why we have such an easy familiarity with it. We see hierarchies everywhere we turn in life, and as such it becomes an easy tool to reach for. Trying, however, to model a hierarchy in a relational database, as we often tried to do in the late 90s/early 00s, usually ended up in a massively complicated system, where much fo the complexity lay in navigating the hierarchy. This was partly why XML was viewed as such a godsend when it emerged as a data-centric tool in the early 00s. Pain points with XML led to the desire for a "simpler" approach, which led to JSON, which is just another hierarchy but with fewer options. (And it uses curly braces, which is more comforting to C++/C#/Java developers anyway.)

One other important note about hierarchical systems is that they are not intrinsically object systems. It's a common mischaracterization, since we often see classes in hierarchies (particulary inheritance-based), and a node can have attributes, just like objects can have fields. However, in an object system we can have a reference to another object--any other object--within the system without regard for its relationship to this object. This allows us to create cyclic graphs (among other things) which are impossible to represent accurately in a strict hierarchy.

For example, imagine the classic Person object:

class Person {
    String firstName;
    String lastName;
    int age;
}
ted = new Person("Ted", "Neward", 50);

This is pretty straightforward, whether Java or C# or C++ (or some other object language). If we try to model this in a hierarchy, it seems to flow well:

<person>
    <firstname>Ted</firstname>
    <lastname>Neward</lastname>
    <age>50</age>
</person>

{
    "ted": {
        "firstName": "Ted",
        "lastName": "Neward",
        "age": 50
    }
}

But when we represent the fact that Persons can get married, we encounter this object-relationship conundrum:

class Person {
    String firstName;
    String lastName;
    int age;
    Person spouse;
}

ted = new Person("Ted", "Neward", 50);
charlotte = new Person("Charlotte", "Neward", 39);
ted.spouse = charlotte;
charlotte.spouse = ted;

We would get a graph that looks like:

flowchart TB
    ted[Ted]-->charlotte[Charlotte]
    charlotte-->ted

But the hierarchical model gets twisted--we have to have a single root, so naively we would think that maybe we nest the "spouse" Person inside of the Person that references it, but that runs into the basic fact that there's two Person objects, so which is the root object? Which of these two spouses is "on top", so to speak? XML is going to get stuck right there, but JSON stands up and says, "Aha! I'm not a problem, watch!":

{
    "ted": {
        "firstName": "Ted",
        "lastName": "Neward",
        "age": 50
        "spouse": "charlotte"
    },
    "charlotte": {
        "firstName": "Charlotte",
        "lastName": "Neward",
        "age": 39,
        "spouse": "ted"
    }
}

... except that this is ducking the problem entirely, because JSON doesn't actually know about the relationship between these two objects--the "ted" and "charlotte" are essentially object identifiers (OIDs) that whomever or whatever is deserializing this JSON has to put bback together "by hand". This also means that the top-level of the JSON document becomes a flat namespace in which all of the Persons in the system are stored, with no intrinsic hierarchy to the data whatsoever. We can do the exact same thing with XML, by the way, if we chose to:

<objects>
    <Person oid="ted">
        <firstname>Ted</firstname>
        <lastname>Neward</lastname>
        <age>50</age>
        <spouse reference="charlotte" />
    </Person>
    <Person oid="charlotte">
        <firstname>Charlotte</firstname>
        <lastname>Neward</lastname>
        <age>39</age>
        <spouse reference="ted" />
    </Person>
</objects>

For whatever it's worth, this is a large part of the reason why the original SOAP specification (v1.1) punted on the details of how to represent an object hierarchy, and deferred that work to XML Schema (XSD), which then... punted on how to represent an object hierarchy in favor of being able to validate XML/hierarchical data. This object-hierarchical impedance mismatch was a large part of why XML services were labeled as "too complicated" and "too messy" and got everybody excited about JSON, which... punts on capturing object references just like XML did. But now we don't care as much for some reason.

Note that if Persons know their parents, things get pretty messy pretty fast:

flowchart TB
    ted[Ted]-->charlotte[Charlotte]
    charlotte-->ted
    ted-->kids[List]
    charlotte-->kids
    kids-->michael[Michael]
    michael-->ted
    michael-->charlotte
    kids-->matthew[Matthew]
    matthew-->ted
    matthew-->charlotte

Representing this in XML is going to be flat-out impossible without out-of-band OIDs, and the same is true essentially of JSON.

We can see each of these points quite clearly in the XML Infoset Specification, but as an exploratory discussion, let's consider the following XML document:

<department name="Engineering">
    <manager>
        <employee id="1234">
            <firstname>Fred</firstname>
            <lastname>Flintstone</lastname>
        </employee>
    </manager>
    <employee id="2345">
        <firstname>Barney</firstname>
        <lastname>Rubble</lastname>
    </employee>
</department>

We have a single root node, the department node. It has associated with it several children: two "element" nodes (one containing the "manager" node and one containing the "employee" node) and an "attribute" node ("name"). The Infoset is quite clear that these nodes aren't classified as "manager" or "employee", but as XML-typed nodes ("element", "attribute") which in turn have descriptors that convey the parts that we has humans recognize (an "element" has a "name" containing the text that appears between the angle brackets). The text "Fred" is itself in a "text" node, although technically these could be four sibling "text" nodes, each containing "F", "r", "e", and "d" respectively.

As with the example, we often think about company organization/topology as a strictly-hierarchical arrangement, but this practice frequently falls down in the face of experience--companies will sometimes have co-CEOs, "matrix"ed teams, and even sometimes have employees reporting to more than one manager/department simultaneously.

Query capabilities

Querying a hierarchical system historically was a matter of writing a bunch of navigational code by hand, maneuvering up and down the tree as desired, until the XPath (and later XQuery) specification(s) appeared, describing a query language that had at its core the foundational knowledge of traversing the hierarchy. XPath uses a fundamental concept of one or more "expressions", where each expression is evaluated to yield an object, which can be either an "atomic data value" (boolean, number, or string) or a "node-set" (a collection of non-duplicate nodes). Each expression's yield is then carried over into the subsequent expression as part of a stateful "context" which is then used to evaluate the next expression. These expressions, known as "location steps", are formed of an "axis", a "node test", and zero or more "predicates", which further restrict the path.

So, for example, an XPath of /child::* evaluates the "child" axis (which means "examine the current node's immediate child nodes"), and the "node test" of "" (which means "accept anything"), yielding that singular "department" node. (Technically the "/" is itself a location path, meaning "take the root node", which is what forms the context of the subsequent "child::" step.) This leads to a mental model that feels similar to filesystem directory paths; however, unlike on a filesystem, we can have multiple nodes returned as part of a query. Thus, /descendant::* query will return every node in the document, regardless of its location in the tree. If we then write /descendant::*/name(), we will ask every node in the tree for its name.

Unfortunately, one of the biggest weaknesses with a hierarchical query capability is often not with the query language, but wtih the perception of those who use it--because they often see parallels between "filesystem hierarchy" and "data hierarchy", they implicitly assume that a language such as XPath yields only a single node at each "step" in the query resolution process, when in fact each resolution can be a collection; in other words, if filesystem shells supported this kind of behavior, we could do something like ls -la /usr/lib/*/lib*.sa and get back all of the files prefixed with lib and having an extension sa in any directory that is a child of /usr/lib. Frankly that's super-powerful, but it takes a moment to wrap one's mind around it sometimes.

The Shapes of Data: Associative

Sat, 1 Feb 2025 00:00:00 +0000

tl;dr The shape of data that's associative, or the key-value data store, is a style of single-dimensional, making the key the biggest part of the shape.

Refresher

In a key-value store, like Redis, data is stored in a collection made up of tuples of keys and values. Keys must always be unique, at least within a single "database". (Some data stores may support multi-tenancy "databases", where each is entirely isolated from one another.) Keys are typically complex data types in their own right, often made up of multiple parts, much like filesystem directory paths are made up of pieces. Values are not given any special treatment within the database, treated entirely as an opaque binary byte array.

Key	Value
K1	AAA
K2	Call me Ishmael.
K3	CCC123DDD456

Different key-value stores will have different semantics around this fundamental data structure. Some will treat the parts of the key as a physical segregation of the dataset into smaller (and more easily retrieved) collections, while others will keep all the tuples in one large collection. Still others may allow for distinct collections, essentially providing a means to provide multi-tenancy, similar to how a relational database can host different named "databases" each with its own schema.

Key-value stores typically are entirely schemaless beyond the basic key-value data structure. Because the database infers no structure on the value, the values themselves have no database-enforced requirements on length, content, or structure.

A Thought Experiment: A schema'ed key-value store could put schematic requirements on the value, verifying that the value follows some kind of file format, like BMP or JPG, or other binary format, like Java or .NET object serialization formats. Describing the binary formatting requirements would necessitate a highly-flexible descriptor language, maybe something akin to how other binary formats are specified (there's a history of different formats and IDLs here to choose from), and it would ideally be something open-ended to allow for pluggable run-time verification (maybe), but the idea is intriguing.

Query capabilities

This one's pretty easy, since realistically, there are none: the database typically allows for retrieval of the value by looking up the key, and that's it.

Some databases will return a multivalued response, allowing the query to put wildcards in for some part of the key (much as ls /Users/Ted/Documents/* returns a list of the files and directories in that location).

Because the associative data shape doesn't do any inference in the value half of the tuple, there's really little to nothing we can do with anything except the key itself. The key, as noted earlier, can be seen as a kind of partitioning element to segregate the collection(s) in some sort of logical manner, but fundamentally this is a hierarchical partitioning, at best.

As for the value, most (if not all) associative data stores support zero sort of filtering criteria on the value. We query on the key--that's it.

Shape analysis

In many respects, the programming model here is very similar as the "untyped collections" that popularized the Java and .NET communities early in their respective lifecycles (C#/.NET 1.0, where Java didn't get strongly-typed collections until Java5). In both Java and C#, programmers clamored for type-parameterizable collections, citing the concern over errors introduced by accidental insertion of unanticipated types in the collections held. However, it's also important to note that in the "untyped" programming languages Javascript, Python, and Ruby, the same call for type-safety hasn't been as strong, likely because those developers write their code in such a way to verify their expectations before use.

With the compound nature of keys, different "parts" of the key can be hierarchically variable; for example, if the stored value is "user session data", the key can be written to consist of both static and variable parts, the static being users and the variable being the user profile ID, such that the overall appearance of the key looks like users:{user-profile-ID}. This can also easily mirror what we see in URL slugs, a la topic:{topic-name}:articles:{article-id}. So long as the key's parts are inherently hierarchical, the key makes for a convenient means to segregate the space. As a result, any desirable filter criteria should be encoded into the key.

As one of the simplest "shapes" in the database world, the associative shape's entire ignorance of the value offers some interesting options:

rather than being seen as a single value, the value can instead be an open-ended collection of values, such as a list or an array. Here we assume either some sort of structure or defining element markers (such as the comma in a comma-delimited list) which the programmer will parse after retrieving.
in a logical extension of the above, we can treat the value associated with a specific key as an "append-only" and/or "growable" storage--take the value, append additional data to it, and re-store it under the same key.
the extreme flexibility of values means the keyspace can be entirely heterogenous, storing "a little bit of anything", creating an unparalleled flexibility, but which in turn will require the most amount of programmer-required validation, verification, or maintenance, because the database offers absolutely zero guarantees about what can (or is) stored. This flexibility is also what makes associative datastores popular as caching systems.

tuplespace systems, as exemplified by the academic sysem Linda and its semi-commercial spinoffs TSpaces (from IBM) and JavaSpaces (from Sun) were essentially associative storage, using the identity of an object (extended in this case to include the node in the network from whence it came) as the key and a serialized (often binary, using the object serialization capabilities of the Java language) representation of the object as the value. Microsoft's Orleans seems to be operating along similar lines, calling the key-value tuple a "grain" and offering several different flavors of keys:

* IGrainWithGuidKey: Marker interface for grains with Guid keys.
* IGrainWithIntegerKey: Marker interface for grains with Int64 keys.
* IGrainWithStringKey: Marker interface for grains with string keys.
* IGrainWithGuidCompoundKey: Marker interface for grains with compound keys.
* IGrainWithIntegerCompoundKey: Marker interface for grains with compound keys.

Because of the specific key-based nature of the database operations, most associative stores are extremely fast, and often highly scalable, since the likelihood of two separate processes operating on the same key are highly unlikely. Some databases will, in fact, trust the developer to manage concurrency entirely. This also suggests that the data stored in an associative system should be more or less independent of one another, since modifying two key-value tuples at once may not have transactional semantics.

Variant: Tupled data stores

A variant of the associative data approach is that of the "tupled" data, such as an Entity-value-attribute mode store, or a "triplestore", such as RDF triplets. Here, the association is a multi-part mesh of data elements, such as in RDF, which is "subject-predicate-object" (a la "Bob is 35" or "Bob knows Fred"), or more formally in the entity-attribute-value model, wherein an entity has an attribute that holds a value (such as "Bob-age-35" or "Bob-list(knows)-[Fred, Barney, Wilma]").

The interesting thing about these "tupled data shapes" is that they are significantly flexible, since the metadata is inherent in the structure; a "Person", for example, would be an object consisting of three tuples or entity-attribute-value triplets, so that the "person" object would look something like:

ted-firstname-"Ted"
ted-lastname-"Neward"
ted-age-50

In some ways, this feels like a traditional object-field-value, but the inherent dynamic nature of the tuples means that not every "person"-shaped object must have the same set of attributes:

cher-name-"Cher"
cher-age-"timeless"

meghan-firstname-"Meghan"
meghan-lastname-"Markle"
meghan-title-"Princess"
meghan-court-"England"

More importantly, each of these is (somewhat) independent of any others, we can add new tuples/triplets at any point, making this effectively a highly "dynamically-typed" way to think about the organization of data and objects. That said, thogh, it's still important to remember that these are not "objects", per se, but a list, essentially of tuples/triplets that, taken collectively, can be viewed as objects.

Or, if desired, as tables:

1-firstname-"Ted"
1-lastname-"Neward"
1-age-50

... where we assume the "entity" in the entity-attribute-value tuple is a primary key value.

The reason this feels like it falls under the associative data style is that we can map another tuple, like person-primarykey-1, which, so long as this tuple is unique to the associative store, we gain an interesting degree of flexibility. We can also imagine that tuple as parts of the key, as well, such that we can envision the data looking like the following:

Key	Value
person / 1 / firstname	"Ted"
person / 1 / lastname	"Neward"
person / 1 / age	50
person / 2 / name	"Cher"
person / 3 / firstname	"Meghan"
person / 3 / lastname	"Markle"
person / 3 / title	"Princess"

... and so on. This can create some interesting flexibility in structure.

Popular implementations ¹

In alphabetical order:

https://www.dragonflydb.io/guides/key-value-databases
↩

Reflections on an Involuntary Sabbatical

Fri, 31 Jan 2025 00:00:00 +0000

At the end of 2021, my time at Rocket Mortgage came to an (amicable) end; the things they had asked me to do were done (more or less--there was more work to be done, but others could pick it up and run with it), and the next things I thought worth doing at Rocket were not things they were interested in. So, we parted ways, and I set back off into the wild blue yonder to find my next role. What followed was what I call my "involuntary sabbatical"--three years of job hunting and interviewing, a bizarre mix of freedom and frustration, and more than a few "false starts" that felt solid yet evaporated. It's been quite possibly the strangest time of my career thus far.

Now, as I prepare to re-enter the full-time workforce, I find myself thinking and feeling and reflecting on the last three years, and I thought I'd share because I live in the perpetual belief that somebody on the Internet might find this useful. (Or, perhaps, you just want a deeper insight into the maelstrom of crazy that is me.)

Rather than try to string a narrative out of this, I think it a little easier to just post ad-hoc blurbs and admissions.

This was not fun, but it wasn't terrible, and it might've been necessary. I think I got infected with the "I'm in tech, I can get a job anytime I want" fever that I've seen infect so many of us, and it was probably a good wake-up call. The tech sector isn't the same as it was--which is good, we were in a huge bubble, and arguably still are--and that means that everyone in it has to adjust their expectations for how long a job search at a senior executive level is going to take. (And I'm nowhere near the top of the "executive level" to boot.) I probably still have one or two more of these in my future before official (voluntary!) retirement kicks in, but even if I spend the rest of my career at Capital One, I think it's important to realize that we aren't in "Oh, I'll just interview tomorrow and get a job" mode anymore.
Having alternative income streams is really helpful. For the last three years (actually the past ten, but the last three are what's relevant here), I've been teaching part-time at UW-Seattle, and that helped a great deal with expenses. Having that part-time gig helped extend the runway a lot. (Plus, it was nicely distracting and gave me something to be doing instead of climbing the walls and driving everyone around me crazy.)
Three years is simultaneously forever and not that long. In some ways, it feels like just yesterday that I was saying farewell to my teams at Rocket; in other ways, it feels like that was a lifetime ago. I can't tell if that's a factor of my age, the length of time between gigs, or what.
Ageism: Did it have a role in your search? How did you deal with it? Oh, boy, this one. My wife and I are definitely split on this; she's convinced that it factored very heavily into comapanies' evaluation of me for a role, and I'm not as sure as she is. Sure, there's a fairly hefty amount of ageism in the industry, but most of that is in the "tech bro" sector, where the twenty-somethings who are bum-rushing the startup/VC/Y-Combinator scene are absolutely convinced that now that they are armed with {fill-in-the-blank-current-hot-thing} they have all the answers they need for any problem that beckons. Frankly, I have zero interest in working with those folks anyway, so if they're discriminating against me, it's probably pre-emptive (I'd be discriminating against them in turn, in case that wasn't already obvious), reflexive (it's mutual discrimination), and convenient (saves us both some time).

What I do think is happening, though, is more economic in nature. Let's be honest, with 30+ years of software experience under my belt, if I've made any progress at all in my career I'm not cheap. If the job calls for somebody who's a mid-level manager and I apply for it (either deliberately or by mistake), I'm very possibly overqualified--which in itself may not be a problem--and overcompensated--which is very definitely going to be a problem. Is that an ageist thing? No, except in that "very senior" will generally only be a label applicable to people who've been around longer, and are therefore older.

I'm not saying ageism doesn't exist, and I'm not saying that I haven't been on the receiving end of it--I can think of a few jobs where it might have played a factor. But I can't tell for sure, and obviously no company on the planet is going to admit that out loud.
Consulting: Did you consider going back to (individual) consulting? Well, in one sense, I absolutely did, because a few roles (including the one at Capital One) started with a contract relationship, sometimes deliberately as a let's-try-before-we-buy. I also did some spot consulting for companies that had no interest in me long-term (and vice versa, to be honest). But in general? Nah, I'm not interested in trying to build out my consulting practice--frankly, I did it, I got the T-shirt, and I'm very happy with the idea of working for somebody else, so long as I professionally respect that manager. That's what I've discovered about me: Regardless of what the company does (with a few exceptions) or who runs it (with a few exceptions), I don't really care about the company. I care much more about the people around me in the company--who am I working for, who is working for me, who am I working with, and am I having some kind of impact there.

That last point, by the way, I've found to be way more important than I would've guessed of myself ten or twenty years ago. It matters to me that I'm making a difference inside the company, much more than what that actual difference is. I don't need to be saving the whales, but I do need to be making somebody else's life better somehow.
LinkedIN is a terrible platform for finding a job. Although this will come as no surprise to many people, I have determined that the social media platform that was ostensibly designed for professionals to engage with one another is now quite terrible at it. I can honestly say that in the three-year timeframe, after clicking the "EasyApply" button hundreds of times--choosing both to share my LinkedIN profile and fill in their online application form--I got exactly one interview out of it. (And that was to a game company early in 2022, which ironically was something of a 'why not' kind of apply.) Even the "new model approach" to LinkedIN, where you not only click the EasyApply but message the CEO/CTO/hiring manager/recruiting manager/anybody-and-everybody–you-know-at-that-company, doesn't work. Maybe it's all the ghost job postings, maybe it's the fact that it's too easy for anybody to click the button, but whatever it is, LinkedIN as a vector to getting an interview and a job is pretty much entirely obsolete and probably should be shut down for many candidates. (Ironically, LinkedIN making it easier for people to apply did exactly that--made it easier for people to apply--and as a result now recruiters are swamped beyond any reasonable human capacity. Which means that LinkedIN didn't make it easier to apply, it just means that now the gates are further back.)
Networking remains king. Almost all the roles I got a chance to interview for came from my connections/network. Many of those connections came from staying in touch with people on LinkedIN, so I can't say that the platform is worthless, and it does help with a professional focus, so it's more effective to connect with other professional-type folks on LinkedIN than say BlueSky or Mastodon, so there's that. And to all of you who said, "Hey, I bet my company could use a little Ted in their life," thank you! From the bottom of my heart, yours were the only real leads I ever got.
AI doesn't help. Seriously. Resumes written with AI are awful, entirely too easy to spot (from the recruiter's perspective), and don't catch anyone's eye (in terms of standing out from the crowd). Using AI as your interview buddy is also not a great tool--it takes too long to type in prompts, and trying to make the responses sound like natural spoken language is just... bad.

(Full disclosure: I've never actually done the 'cheat with ChatGPT' in a real interview, but I did try it as a 'practice' attempt--as in, I sat in my office when no one was around, and I played both sides of the interview. I asked myself a question, then typed it into ChatGPT and tried to read--aloud--the response in a way that didn't sound forced or awkward. Ironically, part of the problem is the fact that ChatGPT actually uses full sentences, which is the first clue that somebody's reading something out loud. Almost nobody does that in an impromptu spoken conversation. Seriously, listen to any interview on TV. We swtich thoughts in the middle of a sentence all the time, and rarely can we hear the punctuation in the response. It's why AI transcription services get so many things wrong--humans are nondeterministic!)
Be prepared for an emotional rollercoaster. It wasn't the first time I'd been without a gig, so the first six to nine months were pretty chill--like most tech professionals, I sort of assumed that "something will come along". (I mean, I didn't just sit back and wait for the offers to pour in, I'm not that arrogant.) But when it got to twelve months and nothing had worked out yet, I started getting a little nervous, and by about the eighteen-month mark, I got depressed. I knew, intellectually, that it wasn't just me--the market's been a mess since the pandemic started--but even then, and with amazingly supportive spouse, family, and friends, it was still hard to not listen to those dark voices that said, "You're done, you're all washed up, you're worthless, nobody will ever hire you again, go pick out your spot under the freeway, you failed everyone you ever loved." (Seriously, that's what my internal dark voice was saying.) It helped that I could give voice to that and have people around me react to that.
Keep in mind your friends want to help. I admit it, sometimes my friends were more interested in being sympathetic when I wanted them to tell me how to fix it, and sometimes they tried to tell me how to fix it when I just wanted validation. It happens. It's not their fault, either. I'm certain if I'd said, "Hey, just validate me on this, eh?" they'd have done so, or if I'd asked, "How would you fix this?" they would make suggestions. Heck, once or twice I did ask for suggestions, and they did, or sometimes admitted they had no quick fixes for me. Key thing is, they were always trying to help, even if sometimes it wasn't what I wanted. (And, to be fair, sometimes I didn't realize what I'd wanted until later in the conversation!) There were still times I got frustrated when they didn't give me what I was looking for, though, and it took a moment for me to remember that they couldn't know what to give me because I didn't ask.
More to the point, friends show up. More than anything else, though, what I've figured out (I think) is that the real meaning of a friendship is that they just show up. They don't have to know what to say or do or how to fix things. They just have to show up. Most of the time, there really was nothing they could say or do. But having them show up, even if it was to just shoot the breeze or talk about their lives for a while, it was way more important than I'd thought it would be.
I should've been more focused in my search. I can do a lot of different things: architect/"thought leader" IC, VP Engineering, VP Developer Relations, CTO, and/or lead a practical R&D effort. (I've done all of those things, though the titles varied from company to company, in keeping with their salary bands.) I initially thought that would be helpful, allowing me to shape myself to whatever-sized "hole" a company needed filling. NOPE. It actually became a hindrance, as I found recruiter after recruiter who simply couldn't figure out what kind of role I was interviewing for. It got even worse when recruiters would ask, "What's your ideal role? What's the kind of things that make you get up in the morning?" (In other words, "What's your passion, so we can make sure this job has it so we don't have to worry about motivating you ever again?") No matter how I answer that question, I was in trouble:
- Honestly, as in, "I have all these different interests and thinigs I like to pursue." Recruiter: "Wow, there's no way this role can give you all that. You're probably not a good fit. Thanks, and good bye."
- Clinically, as in, "Look, I'm not trying to find a passion job, because I'm seasoned enough of a veteran to know that no job is perfect, and I bring my 'A' game no matter what the job has me doing." Recruiter: "You're dodging my passion question, so you must not be super-pumped for this role. You're probably not a good fit. Thanks, and good bye."
- Dishonestly, as in, "I have always longed to be working in a role just like this one." Recruiter: "You sound 'off', like you're not being truthful or honest with me. Bringing your 'whole self' to work is important to us. You're probably not a good fit. Thanks, and good bye."
I eventually got to a point where I would just go through this little diatribe with the recruiter, and leave it at that. Which itself presented problems because now my answer doesn't fit on their scoring rubric and artificially marks my candidacy as less-appealing.

What I probably should've done is created six different forms of resume, one for each of the different kinds of things I was interested in doing, "tuned" to that particular role. Not job-specific, mind you, but a generalized "VP of Engineering resume", a "CTO resume", a "VP of DevRel resume", and so on, taking a particular slant on the things I've done and how my experience set me up well for that particular role.

It really surprised me, discovering this, because I'd always assumed that being multifaceted would be viewed as an asset to the company, but instead, it really played off as more of a liability. Despite the rhetoric, it really seems like companies are more interested in single-dimensional people than they are people with a variety of skills.
Speaking of interviewing, lots of companies do it really, really badly. I had a few conversations with recruiters who were genuinely amazed at my resume, portfolio, books, and GitHub repositories... and then still insisted I needed to go do their LeetCode coding test. Seriously?
Also, I suck at interviewing. I'm not going to blame the companies entirely here; over the course of many of them, I've come to realize that I am nowhere near as good at the skill as I'd assumed. Had I to do it over again, if it were a job I really, really, really wanted, I'd practice it with somebody else. (Literally, write out the STAR-like interview questions, have somebody ask them, deliver the response, and then together workshop the result and try again if necessary.) For the record, I've never practiced interviewing before--and I probably lost a few jobs because of it.

Also, let me make it very clear to everybody out there doing interviews: If the candidate can offer you credible evidence that they are, in fact, capable of writing code (GitHub projects that aren't forks, blog posts they've written with code in it, publicly-recorded user group talks, whatever), just do yourself a favor and believe them. The amount of effort it would take to "fake it" at a public level, when coupled with the trivial effort required to poke through the falsehood, vastly exceeds the payoff. I honestly think the concern at some (most?) companies isn't "We don't want to hire somebody who isn't technical" as much as it is "I don't want to be the one fooled by some shyster who pretended to know code because then it reflects badly on my ability to judge technical skills and that could hurt my career in some undefinable way and at the very least means I will have to endure the mockery of my peers". But don't take my word for it--try it! Try to fake your way through an interview on a subject on which you know absolutely nothing and have no experience. It's way, way harder than you might think.
Recruiters, stop trying to insist you're not transactional. Of course you are. Over three years, I had dozens and dozens of conversations with (company-employed or -retained) recruiters who insisted that they wanted to stay in touch after the interview cycle fell apart, and the exact number of those recruiters who actually did? Zero. Nada. Big Goose Egg. I won't judge you for it, but in return, just stop pretending.
Executive headhunters (where they work on behalf of the candidate, not the company) are extremely rare. I found exactly one that wasn't an agency (or an outright scam), and just as I was about to work with them, my circumstances changed for the happier. (In truth, I want to bring them something just to try and keep the connection warm; sort of keep them on speed-dial, just in case.)
I wish I could've come away from this timeframe with something to show for it. I mean, three years of quality time with my spouse and family is great, but for so long I've had this running mental thread that keeps whispering, "I could totally do interesting side project idea if only I had the time to do it." Well, I had the time, three years' worth, and I got so few of those side projects done. I suppose I could chalk it up to, "Looking for a job is emotionally draining, and I needed to keep my mental health charged," but part of it is, I spent way more time playing StarCraft II and Heroes of the Storm than I'd like to admit. (On the other hand, I did spend a lot of time tending my research garden and then later working on the DevRel book, so it's not like I was just twiddling my thumbs the whole time. Still... would've been nice to finish up a language or write that game or... something. Even just getting into a gym habit would've been nice.)
I'm finding that I'm kinda looking forward to retirement now. If retirement is essentially the last three years minus the worry about financial future (aka, we know that we're covered for the monthly expenses), then yeah, retirement's going to be pretty cool. I was always worried that I'd go a little stir-crazy with nothing concrete handed to me to do. I still might, but I have a better idea of how I want to approach it and what I want to do during it. Next time around, I'm definitely looking for ways to do it differently, but that's because I now understand just what the weight of "Eh, I don't have to be up early today, I'm going back to bed (and sleeping 'til noon)" every day feels like.
Redmond, WA, really needs a 24-hour Coke joint. All the Denny's in town shut down years ago, and nobody has stepped up to take over that slice of existence. It's where I wrote most of my MSDN and developerWorks articles back in the day, and while yeah, pandemic shook a bunch of things up, I'm surprised that we haven't seen a 24-hour diner joint fill in the Denny's-shaped hole here in Eastside Seattle. (BTW, it has to be a Coke joint--Pepsi is dirty garbage water that any day now will be reclassified as toxic waste.)

I don't know if any of those help anyone, or if they just trigger an automatic "WTF are you talking about?" or what. This post was more for me than for anybody else anyway, so if you made all the way through to the end here, well, thanks for reading.

(NOTE: Edited again on 1 Feb 2025 to address a few questions Adam messaged me over LinkedIN that I realized I wanted to add here, and correct a few typos Ron was kind enough to point out.)

The Shapes of Data

Fri, 31 Jan 2025 00:00:00 +0000

tl;dr Dating all the way back to the earliest of databases, data has had a "shape" to it, a natural schematic form that defines the various "atoms" and "molecules" that the database understands naturally. Relational data, such as that described by SQL and relational databases, is only one such shape; the various "NoSQL" databases often offer a different shape, but it's not always the same. What's more, just because a database uses a non-relational shape doesn't mean it's a natural fit to your object-programming language's code. When it doesn't, you have an impedance mismatch, which has to be solved somewhere (usually to the detriment of your code and complexity budget).

(BTW, much of this material is stuff I've covered in the past in talks and other materials, but never really with this level of focus; if it sounds familiar, you've probably already heard me yak on about it before, so you might choose to move on.)

The Atom of Data: The Bit

Fundamentally, all data boils down to a single bit, 1 or 0. Everything that ever crosses a CPU, resides in RAM, or is stored on disk, is made up of dozens, hundreds, thousands, millions or even billions of these. Beyond that, though, the bit is entirely uninteresting, because with the very rare exception of either writing assembly code by hand or working with bitmasks in a higher-level language (lookin' at you, C), we don't really deal with bits very often. Even native boolean types are often "widened" out to a larger value.

Er, I mean...

So while the bit is useful and necessary to understand at a foundational level, its purpose here is to combine in various ways to create higher-level data types, which are usually made atomic at the language or data-storage level:

If we string bits together, we can use each bit in a binary numeric system to represent whole numbers. Depending on how many bits we use, we can get a pretty wide range of values:
- 8 bits together is a byte, which can represent up to 256 (0 to 255) different values. Sometimes the first bit in the pattern is used to indiciate a positive or negative value, which makes this a signed byte, capable of storing -127 to +128.
- Sometimes we string 16 bits together in a programming language and call that a short integer, or short for short (pun intended), and it can store 2-to-the-16 different values, which is a range of 0 to 65,535, or 64k. It also can be signed, which changes the range from -32k to +32k.
- 32 bits together is an integer in most circa-2000 programming languages, and again it can be unsigned (0 to 4,294,967,296), or signed (-2,147,483,648 to 2,147,483,647).
- 64 bits, a long integer or long, gives us something like 18,446,744,000,000,000,000 possible values; that's 1.8446744e+19 for those of you who prefer scientific notation. (By the way, somebody double-check me on those two numbers, I tried to count the zeros and decimal places and lost track three times.)
We can also string bits together in 32- and 64-bit formats, but this time use a different encoding mechanism to capture floating-point values.

But these are just numbers; a good chunk of our data is stored in human text, a la strings, which are collections of 8-bit bytes (if it's stored in ASCII) or 16- to 64-bit Unicode Transfer Format chunks; this is where UTF-8, UTF-16, and/or UTF-32 comes into play. And, in many cases, since we don't know how long a string should be (are we storing a single word from the dictionary, or the entire collected works of Shakespeare?), we have to either set a special value aside to indicate the end of the string (C used the value 0, or NULL, to indicate the end of its byte-wide strings storing ASCII characters, hence giving us the term "NULL-terminated ASCII string"), or we have to capture the length somewhere (Pascal put the length in the first byte, so that the first character in the string was always at index 1, which neatly offset the C quirk of the first byte being at index 0, but also led to a max size of 255 characters for the string), or we just have to set a fixed size and assume the string falls under that length, or something.

Wait, we were talking about databases

Any database system worth its weight in weightless bits will define a set of "native" types that it understands atomically, just as programming languages do. The SQL standard, for example, defines a set of types like VARCHAR, a length-unlimited string type, or VARCHAR(2000), a string that can store up to 2000 characters. There's usually some performance- or feature-related consequences for choosing one over another, but these types form some of the core "atoms" of what we can store in the database.

Then, the database typically defines some kind of "higher", "molecule"-like, thing that allows for programmers to "assemble" larger things (often called "entities", since that's a nice vendor-agnostic term) out of the core atoms. So, for example, if we want to store demographic information about a person, we might want to store:

Given (first) name: a string of up to some reasonable length
Family (last) name: a string up up to some reasonable length
Age: an integer value that can store 0 through, say, 150 or 200

erDiagram
    PERSON {
        string familyName
        string givenName
        int age
    }

We sometimes even get a little nutty and suggest that persons can be in a relationship to one another:

erDiagram
    PERSON  |o--o| PERSON : marries
    PERSON {
        string familyName
        string givenName
        int age
    }

It's not all that unlike creating user-defined types in programming languages (be they classes, records, unions, whatever). The really interesting question is if and/or how the database provides support for the definition of this user-defined type. If, for example, the database natively understands that "marries" relationship defined above, then we can use the database's native facilities (query language, for example) to retrieve the data we need more precisely (and more quickly and more scalably and more efficiently and...).

(Of course, in many scenarios, scalability, efficiency, or performance aren't high-need requirements--of course they're always nice to have, but sometimes you just don't care about that as much as you do other concerns, like "development time". In those cases, developers should of course use whatever tools are most appropriate--in a shop that knows Postgresql really well, use Postgres for everything! Just be aware that there's always some cost to navigating between these different "shapes"--the "impedance mismatch"--and after a certain point, that cost will want to be recuperated.)

Schema vs schemaless

Those databases which allow us to define a formal definition for these user-defined types are said to support the notion of a database-enforced schema, and the most common examples of these are the relational databases of the world. (As a matter of fact, I can't think of an RDBMS that doesn't enforce, nay require, a schema definition for the database.) The database will have you define what your user-defined entities look like, and then go to great lenths to make sure that whatever gets stored in the database is in agreement with what's defined in the schema.

Contrast this with the recent spate of "NoSQL" databases, most of which not only changed the shape of the data they were storing, but also removed the notion of schema from the picture, because it turns out that most developers weren't really all that frustrated with SQL (well, not really), but with the fact that the databse was enforcing that the schema had to be defined, and then enforcing what was in the schema. Developers were finding that as they iterate on a project, they often need to refactor the "things" (classes, tables, files, whatever) they're defining, and while programming languages had pretty decent refactoring support, schema-enforcing relational databases did not. So the NoSQLs said, "Let's drop the schema!" And developers said, "Awesome!" (And then a typo wrecked the developers' whole week when they missed it in testing and it made its way into Prod, but that came later.)

The point to understand here is schema is not a property of the data's shape. We could have schema-less relational databases (we often call them CSV files), and we could have schema-enforced databases of other shapes (XML, for example, defined XSD, aka "XML Schema", to do exactly this). We don't see the full spectrum of all the combinations mostly because... well, I dunno why. Just seems like developers aren't interested in what some of those other combinations are (which, to be clear, I think is a mistake--those other combinations are fascinating).

Now, the last "atomic" type the database typically understands, natively, is some kind of association between atoms. It might be something as simple as a key-value pairing (associating the key to the value), or it might be as complicated as a foreign-key relationship in a relational data model, but typically each database understands that things have to relate to one another in some fashion.

And it's these relationships that typically define the "interesting" parts of what makes up the shape of the data in the database.

The known universe

As near as I can count, there's only a few data shapes, which I'll go over in separate posts:

Associative, aka "key-value" (Wikipedia)
Graph (Wikipedia)
Hierarchical, aka "document" (XML, JSON) (Wikipedia - Hierarchical, XML)
Object (OODBMS) (Wikipedia)
Relational (RDBMS) (Wikipedia)
Tabular, aka "flat files" (CSV, TSV) (Wikipedia)

(I haven't finished writing up the graph and hierarchical notes yet, those are next)

There's often some variance between particular examples of each, but fundamentally, these appear (to me!) to be the high-level distinguishing models. (Of course, if more models make themselves appear, I'll add them to the list.)

One of the older data shapes is that of the navigational or "network" data model, but I think, after some initial analysis/thought, that the network shape really is either entirely subsumed inside the object or graph shapes, or it sits somewhere in between the two on the Venn diagram, overlapped by each of the other two, and lying entirely within the space of the two combined. (I might be convinced otherwise, but somebody would have to work at it.)

There's also a class of database products that try to embrace more than one of these shapes simultaneously, calling themselves "multi-model" databases. In my limited experience with them, I've found that they usually embrace one model at the heart of their internal operation, and then project other models on top of that singular, internal model. (Which usually means that in a head-to-head shootout with a database product that "natively" understands a shape, the "projected" shape multimodel database loses. Badly.)

By the way, let's also make one thing very clear: How we represent these models in memory may look somewhat different from how they are serialized and stored to disk. For example, when I new up two objects in (pick-your-favorite-OO-language-here) and point them to each other, I don't see the pointer value of their location in memory because the language hides that from me (typically). If I were to store these two to disk, though, we'd need to (a) identify them and (b) put those identifiers someplace where they could be turned back into pointers later. The pointers are natively understood by the language (and, by extension, the database engine that we're talking about), so they're a "natural" part of the database's model/"shape".

Along those same lines, it's also important to recognize that the shape of the data understood by the database is not always going to be entirely reflected in how the data is stored on disk; lots of databases do some very interesting things at the storage engine level in order to gain performance, and then "make things right" when they load data into memory for analysis or processing. (In other words, the magic behind highly-performant storage engines is out-of-scope for what I'm trying to discuss here!) In other words, just because every major relational database stores their data in a B-tree (or -derived) format on disk doesn't make every relational database a hierarchical, graph, or associative shape. If we go down the stack far enough, we discover that everything is just made up of 1s and 0s! I choose to draw the line at how the data looks and interacts at the level at which the developer sees it, through either its query interface (SQL, OQL, Cypher, whatever), its API, or its storage format (if there is no consistent API--this applies to XML, JSON, CSV, TSV, and others).

And yes, we can convert the database's native format into another format, often using tools found in the database itself--most RDBMSes in 2025 can return a SQL query in JSON or XML, if you ask it nicely enough. Far from invalidating this post, though, I think it actually justifies it: If it was trivial to consume the results in the desired manner, why does the database need to provide the feature? :-)

Thought experiments

In this section, I'm going to idly wonder about other models and/or thoughts around the shape of data in general or implementation ideas. Some thought experiments appear in each of the specific data shapes, too, if they're closely aligned to a particular shape. I don't know if I have any particular point or solution here--call this the "thinking out loud" section.

Columnar. One obvious absence is that of the "wide-column" or columnar data store, in which we store data in such a way that it feels more column-oriented than row-oriented (which would be tabular). There's some pretty significant implementation differences when we do this, but I'm not 100% sure if the shape is any different than a tabular one--just take the table and rotate it 90 degrees, and lo, we have a columnar format. Am I missing something here?
Multi-dimensional. In a non-relational table or collection, we identify unique values along a single dimension--the key has to be unique across all possible values. Relational databases have allowed for compound keys for decades, however, which leads rise to a shaped model of data that resembles either a two-dimensional cartesian grid or even three- (or four-? five-?) dimensional model of data. It's highly likely these could be represented by an associative data store with multipart keys corresponding to each axis in the model, but one thing about three-dimensional models is that you can turn them in various directions and look along any axis "first"--in an associative model the keys are structured hierarchically, but what if we want to see the plane of data where Z is known but X and Y are variable? Like, roughly speaking, imagine an analogy to a three-dimensional grid, where we can write queries like "select where x=0, y=0, z=0" to retrieve a particular cell, or "select where x=0, y=0" to retrieve a line, or "select z=0" to retrieve a plane. Now imagine a four-dimensional database and specifiying three of the axes to retrieve a cube, and it's starting to fold on top of itself in the same way relational queries can. (Twenty years ago, people in the data space talked about "hyperdimensional" databases, is this that same idea?)

For starters as a thinking point, what if we took a traditional tabular instance, and added a second dimension, time, to it, making it into a cube of sorts? That would make any tabular shape into a temporal one, which would obviously make it harder to store in a flat-file, but again, the shape is often more important than its storage or its serialized representation. (After all, we can't really represent a fully-normalized relational database in a CSV file, and we "get" that.)

NOTE: (1 Feb 2025) Wikipedia ha an entry for array databases and it seems to follow much of what I was imagining here. From an implementation standpoint, it would seem like this is an interesting area of exploration. From the Wikipedia entry, it sounds like this is used a fair amount in geo-spatial data, but this is all new to me so I can't be sure if I'm reading that correctly.
Temporal. Speaking of time.... Many databases were exploring/operating around time as an implicit or explicit key in the storage of values, which makes me wonder if that's its own data model or a variation of multi-dimensional. Chris Date and Hugh Darwen had some thoughts on this, as did Rich Hickey, but I'm not sure how all of this plays out inside my head yet. I think I tend more towards the Cassandra style of "every atom in the database has a time value attached to it", but I find myself thinking that may be oversimplifying it immensely.
Branching. Git made the branching model of data storage popular, and it's started to creep into database products. But is this really a new model of data, or is it simply an associative model that has a partiucarly-constructed key? That is to say, each "project" is a major-part key, and each "commit" is a minor-part to that key, and the complete (binary) commit is the value?
Unstructured. Just like the absence of light is still a color (black), the absence of structure can still be data, although we can get really philosophical about this really quickly. Consider: Even though free text (like this blog post) is considered "unstructured", it still has structure, in that there are grammatical rules that explain the punctuation, style rules that govern the larger structure of the post, and spelling rules that govern the way the letters are made to work together. Below the surface, there's the Markdown syntax with which this post was written. So is data ever really 'unstructured', or is it that some structures are just hard to diagram/explain and others are hard to parse? (It's not just text, either--"unstructured binary data" can mean "it's structured, but not something we want to parse" all the way through "it's random sensor data and we're trying to infer structure where we're not sure if there is any".)

Distributed Systems and Organization Design

Wed, 22 Jan 2025 00:00:00 +0000

tl;dr Engineers (and their managers) have spent much of the last forty years learning (and sometimes re-learning and re-learning and re-learning...) the various repercussions and implications of distributed systems. As an engineering manager, I've discovered that there is a remarkable similarity between distributed systems design and engineering organization design.

For most engineering managers, if they know much about "org design", it's either Conway's Law ("A software architecture will mirror the organization that produced it") or it's what they learned from the book Team Topologies. (By the way, if you were wondering about the "truthiness" of Conway's Law, check out this paper by Harvard Business Review; tl;dr, yes, it holds up.) Beyond that, though....

It's a little beyond the scope of this post, but suffice it to say, there's a whole area of study around organizational design, and much of it has the same kind of rigor and analysis to it that we do around distributed system design. That is to say, some people (particularly academics) study it with a great deal of rigor and discipline, and the rest of us wait for somebody to summarize their findings so we can completely ignore it later in favor of some hot new tool or technique. I fully admit to being no expert on org systems, but over the last decade, as I've seen (both observed and led) more organizations from a managerial viewpoint, I've come to realize that there's a lot of parallels between the two.

If we make the basic assumption that a "team" in an org chart is the rough equivalent to a "server" or "compute node" in a distributed system diagram, and that a "team member" is roughly a compute process, we start to find some very interesting parallels.

Work is done much faster by an individual ("process") than by committee ("cluster of processes").
Where a process in distributed system will "make API calls" or "pass data" or "send a message", people will "collaborate" or "hand off" or "send a message" to each other.
It's much faster for an individual to have data memorized (cached) than to have to look it up in a book or some other external storage (database).
Changing the data that's currently memorized by an individual requires that individual spend a non-trivial amount of time to re-memorize the new information, starting from notifying them about the change (cache updates and propagation).
Passing information is exponentially more expensive the further away the intended recipient is, whether in the next chair, next room, next building, or next continent.

... and so on.

And it's really hard not to notice that a distributed system topology--a collection of nodes with links connecting some or all of them--is a remarkably similar diagram to a process flow diagram--a collection of nodes (teams) with links (communication or process handoffs) connecting some or all of them. It seems no accident that we refer to "workflow" as a term that covers both human actions and/or interactions and computer actions and/or interactions.

In the distributed systems world, it turns out to be helpful to think about the Eight Fallacies of Distributed Computing as a framework to think about distributed systems. Using these as guideposts, or more likely, warning signs, we can often ship a reasonable distributed system without having to get too academic or methodical. So, in the spirit of the Eight Fallacies of Distributed Computing, we can derive the Eight Fallaies of Distributed Teams:

The organization is reliable. We all know that nodes in a network can drop on a regular basis. Gratned, with modern computing trends and safeguards, it's rare, but it still happens. Similarly, even though we know most people will show up to work most days, there's still that non-zero possibility that a given individual won't. They'll be sick, they'll be distracted, they'll be re-tasked to something different by their management, and so on. If that weren't enough, we can of course never ignore the possibility that they'll be there at work, ready to carry out your request, if only they'd received your email/text message/inter-office memo/voicemail, because somewhere along the communication path your message was just flat-out lost. (Who says ACKs are just for low-level network protocols?)
Latency is zero. The time required to engage in communication between two people is very obviously non-zero. In fact, it can take quite a bit of time to get what's in my head over into your head. Sometimes this is because of the time required by the medium of communication (ahem inter-office memos), but....
Bandwidth is infinite. In the human case, it's not communication bandwidth we worry about as it is an individuals mental bandwidth. Most people cannot juggle more than one or two things simultaneously. Even those of us gifted with Attention Deficit Disorder find that we have limits to the number of things we can keep our minds on, and we're better at multitasking than most. A human will eventually need to prioritize and serialize the things they're working on, which means that some work goes to the bottom of the pile.
The network is secure. Sure, HR isn't going to write down a performance-management report on a piece of paper, find the first person outside their office, hand them that piece of paper, and ask that person to hand-deliver the message to the VP of Sales over in the next building--such a practice would be entirely unprofessional, deeply illegal, wildly impersonal, and hugely unethical. We get that. However, if HR is having a conversation with the VP of Sales, who else is listening to their conversation as the HR rep is sitting in the middle of the open-office plan? More importantly, when two loan officers are discussing a mortgage client's financials, who is listening in? (A long, long time ago, Intel found itself in an interesting quandry when they discovered tech news journalists were simply riding the shuttle flight between San Jose and Sacramento (where Intel had fabrication plants, in Folsom), simply listening in to the seat chatter between Intel employees, who were easily identifiable by the Intel work badges they proudly wore.) If nodes in your org chart must communicate to one another, how are they doing it?
Topology doesn't change. Organization charts are famously instable, perhaps even more so than computer network topologies. If team A depends on team B for some of the work that team A needs to get done, what happens when team B is re-orged into an entirely different division?
There is one administrator. Well, technically, if everybody works for the same company, there's the CEO, but the larger point here is that frequently teams need to collaborate "across" the org chart, where the only point where both teams are under the same manager is well up the org chart--when Product and Engineering work together, for example, the Engineering team will ultimately report to a VP of Engineering, and the Product team to a VP of Product, who in turn each report to the CEO. This means that there is no one individual close to the teams that can resolve any inter-personal conflicts or "turf battles"--the teams must sort the issues out among themselves. Neither side can effectively make an appeal to authority for resolution. This means the teams are going to have to each feel some pressure to make things work, even if they have a intellectual or moral disagreement on some part of the collaboration.
Transport cost is zero. In the original Fallacies, this one is often seen as related to the "Latency is zero" fallacy--where the first focuses on the time required to actually transmit the data, and this fallacy focuses more on the work required to prepare the data for transmission as well as interpret the data upon receipt. (In RPC terminology, for example, we talk about "serializing and deserializing" the objects transmitted across the wire.) In an organizational scenario, this transport cost is often the vastly larger time-consumer, as sometimes teams will have to create bespoke presentations or documents solely for the purpose of communicating one or more complex ideas to their collaborators (not to mention the delays introduced while we wait for everybody who must be in the meeting to have a free spot on their calendar to be able to discuss the topic at hand!).
The network is homogenous. In the distributed systems fallacies, James Gosling introduced this one to point out that (almost) no computer network is actually made up of the same kind of hardware/OS combination (it's never all-Intel/Windows, it's never all-ARM/macOS, it's never all-anything). His argument, then, was to suggest "This is why everything should be written in Java!", since that would render the fallacy moot. And, of course, we promptly wrote everything in Java, C#, Ruby, Smalltalk, Go, Dart, some C++, a little COBOL.... In the organizational sense, it's important to realize that not only is each team built for different purposes than one another, each person within the team often has different skills and perspectives from each other. This has the positive quality that a team that can take advantage of its members' strengths can often overperform to its expectations, but it also means that a team that doesn't recognize that "not everybody is great at accounting" is going to run into problems when it tries to swap one individual for another without taking into account their individual skillsets. (I'm looking very squarely at you, "full-stack engineer"-hunting recruiters....)

Note that these don't all capture all of the Dos-and-Donts of organization design (not by a tenth), but it does certainly establish an interesting start to an interesting thought project, that being, What if we thought about human-centric processes and workflows in the same way that we think about distributed systems design and architecture?

It would give us some familiar tools by which to document and/or analyze processes, a la those "workstream" efforts that certain process coaches love to conduct. (I call them "process" coaches because some of them do so in service to agile efforts, while others do them in service to efforts that are anything but guided by the Agile Manifesto.)
It helps set some expectations around expectations when work leaves an individual, a team, or an organization. (In other words, if you hand it off to another team, it's best to expect that to be orders of magnitude longer before you get a response when compared to something that you're doing yourself.)
We might want to prevent "chatty" exchanges between individuals, and we definitely want to think twice about such exchanges between teams, if we need something processed quickly.
Most of all, it behooves us to look for ways to collect everybody working on a particular large-scale task to be relatively close together on the org chart, to minimize the latency (#2) and transport cost (#6) between the teams.

It feels like this is a topic worth exploring in more detail, to me at least.

2025 Tech Predictions

Mon, 6 Jan 2025 00:00:00 +0000

As per previous years, I'm giving myself either a +1 or a -1 based on a purely subjective and highly-biased evaluational criteria as to whether it actually happened (or in some cases at least started to happen before 31 Dec 2024 ended). If you want to skip to the new predictions, scroll down to the next major heading.

In the large, 2024 was... ugh. We thought 2023 was tumultuous, but then 2024 said, "Hold my beer." The job market for the tech sector started to look up, then started to look terrible again, then people said, "No, it's really getting better" even as others said "No this is the worst it's ever been" and most of the rest of us just threw our hands up in the air and stopped trying to guess. Anecdotally, I keep seeing lots of people on LinkedIN on their last months' reserves, but at the same time LinkedIN keeps sending me all sorts of interesting openings that, when contacted, yield nothing but whispers. I kinda gave up trying to figure out where we are--though I'm still looking for something full-time myself. (If you find my analysis here to be interesting or intriguing--even if you disagree with it--perhaps there's a role in which I can do this kinds of strategic and executive thinking on your company's behalf? Would love to hear from you.)

In 2024...

... I wrote a lot of stuff. (That always seems to happen when I do these.) So let's get to it; as I do each year, I'll include the full text of what I wrote in each bullet point first, then put the Result after it with whatever insights or comments seem relevant. (Arguably none of them are, but hey, it's my set of predictions, so....) As always, evidence is highly opinionated, mostly anecdotal, and entirely from my (and some from my professional network's) perspective, so caveat emptor.

Not surprisingly, a lot of what I wrote was about "AI". (I put the term in quotes both because I have no illusions about machines becoming actually intelligent, nor do I think the current area of research into Large Language Models and generative AI are the sum total of AI research. Where's my "fuzzy logic" from the 1990s, people?) And, by the way, I guarantee that AI didn't write a word of this post, or any other. (When I use the word "delve", it's because I damn well wanted to!)

Hiring will open up. (Probability p 0.8)

"The interesting thing about the layoffs is that while they continue to trickle out from various companies all the way to December of 2023, it's a trickle compared to the flood that happened in 1Q, particularly in January. More interestingly, the kinds of people that are going to be hired will shift slightly--right now (end of 2023), companies are focusing on hiring ICs to fit into existing holes in their current org tree. But with the new year, and a bit of optimism in the market, they'll start growing new branches to that tree, and that will require more in the way of engineering management and "ancillary units" like developer relations, R&D, and so on. It'll take a while, and it'll never be like 2019, but before too long things will get stronger."

Result Well.... I really don't quite know how to score this one. On the one hand, some statistics suggest that hiring did open up ("According to the Bureau of Labor Statistics, 829,000 jobs were added in Q1 2024"), but obviously lots of people aren't feeling it. The thought process here is that yes, the jobs started to come back, but it turns out that there's a lot of people still looking, and so the jobs are disappearing quickly, either to internal candidates, overqualified candidates, or even never being likely to hire in the first place. I'm thinking this is a -1 to my predictions score, even if technically I might have been right.

WebAssembly will gain more traction. (p 0.5)

"I haven't really paid much attention to WebAssembly for the past few years, because I wasn't sure where things were going. The 1.0 specification is done, but it's pretty minimal compared to other richer, more full-featured virtual instruction sets (like the JVM or the CLR). Truthfully, it's a ridiculously bare-bones spec. But, even as I say this, there's some interesting infrastructure that's targeting LLVM more and more (most notably LLVM and GraalVM), making it easier and easier for WASM to gently "slide in" to a deeper role within a dev toolchain and do some interesting things. That said, though, WASM still has yet to really demonstrate where it's real value lies--why write something to WASM binary, which generally means targeting the browser, when there's so many Javascript-centric options already? Where's the compelling value-add from the developer perspective? This is the hinge on which WASM's broad-scale applicability matters: If WASM can show how using language-X-compiled-to-WASM enables a new kind of feature or a faster/easier workflow to get to "done", then WASM could very well begin to emerge as the infrastructure backplane for browser-based applications all over the Internet. I don't know what that compelling reason is yet, though, which is why I leave this at a 0.5 p."

Result Well.... Yeah, no, WebAssembly remains something of a niche space here at the start of 2025. It's almost ironic: You know how everybody in the Linux world keeps saying, "THIS is the year for Linux on the desktop!"? Well, I think they're about to be challenged by the WebAssembly crowd for the title of "Most Consistent Prediction that Never Comes True". There's some interesting technology here, but WASM just can't seem to catch a "hook" to catapult it to catapult it out of "Interesting-Yet-Uncompelling" territory it seems mired in. Which, we will note, I pointed out: "WASM still has yet to really demonstrate where it's real value lies--why write something to WASM binary, which generally means targeting the browser, when there's so many Javascript-centric options already? Where's the compelling value-add from the developer perspective?" Still, it's a -1 to my prediction.

Generative AI will lose its luster. (p 0.7)

"It already has started, in many ways--memes now are common across the Internet showing ChatGPT conversation windows in which it responds to the question, "How do I use GitHub?" with "In order to push code to the remote server, issue the command git branch..." in a completely erroneous and mistake-riddled answer. The stories of the lawyers who used ChatGPT to write legal briefs that turned out to be filled with inaccuracies and errors--for which they were disbarred, by the way--have cast some serious doubt on the ability of these generative AIs to "fire all the peons" whose work ChatGPT and its ilk were supposedly going to replace."

Result Well.... The tech world is pretty clearly dividing into the "anti-AI" (aka "AI haters") and "pro-AI" (aka "venture capitalists") crowds, and frankly the more we see playing out in front of us, the more it looks like generative AI has reached its limits. (When OpenAI says they need more money than has ever been produced in the history of mankind in order to progress, you kinda get to draw that conclusion.) But then we also had something of a Hollywood scandal drop, and suddenly it was mainstream news to talk about how AIs need to train off of existing data. ScarJo's lawsuit remains unresolved (as far as I know) at this point, but it definitely sent a ripple through a lot of the entertainment industry. I'm going to give myself a +1 for this one, because when it makes the 5o'clock news, and not in a good way, it's "lost its luster". (She wasn't alone, by the way--others, like John Grisham and George R. R. Martin, filed a class-action lawsuit as well. This is definitely not "finished" discussion.)

We will begin to disambiguate between generative AI and large language models (p 0.5)

"It happens with AI every decade or so--something new comes along, it takes our imagination by storm, and we start chanting "AI! AI! AI!" like an amped-up-on-booze-and-caffeine-pills crowd at a wrestling match. Then, as we get disenchanted with the results, the various AI firms--who have yoked their success to the success of their AI work--will start to point out that they were never "AI" companies, they were "large language model" companies and therefore never a part of the hype machine that's now augering in."

Result It's very faint, but you can start to hear it in some of the tech punditry. There's some interesting things going on with generative AI (one of my D&D gaming buddies discovered https://suno.com and has started creating ballads and epic dirges about our campaign), and we're not talking about language at all. I think, realistically, this scores at a 0 but it's definitely going to get more differentiated in 2025.

Custom AI models will begin to gain traction. (p 0.6)

"Making use of a large company's model for your natural language analysis needs is great, but over time, companies are going to figure out that the cost of that model will probably be smaller if it can be hosted locally, rather than constantly going to the cloud and back again. (See the economies-of-scale discussion earlier, as well as the performance and reliability implications, a la "The Eight Fallacies of Distributed Computing".) Slowly, quietly, companies with some room in their budgets are going to start looking to develop their own models, and in many cases may find that the model's they're building don't need to be quite so large (and heavy), that in fact a "medium" language model, or even a "small" language model would work, allowing for a local presence on the same node as the rest of the processing. OpenAI and other firms are going to combat this by constantly releasing new models with new excitement and fanfare, but like the cloud itself, the basic path here will be "start with the hosted LLM, then as your success and budget grows, look to build--and tune--your own". It'll be "Buy vs Build" decision-making, all over again."

Result The "quietly" part of that prediction is what trips it up--if companies are starting to build out their own models, they're not talking. Perhaps the best thing at this point is to punt on it for a year, and give myself a 0 and say, "Let's see where it is in 2025."

New and interesting languages will begin to make waves. (p 0.6)

"I've been saying this for years, but I keep holding out hope. Frankly, I think some of the development will be to raise the abstraction levels of what we do, because ChatGPT's success at writing code is due to the fact that 95 times out of 100, we're writing the same basic CRUD crap over and over and over again. The Pareto Principle holds here: 80% of all mobile applications (as an example) are doing the same stuff, it's just that it's a different 80% from all the rest. A language/stack that can embrace an onion-style architectural approach (allowing for "trap doors" to drop down a level of abstraction when/if necessary, such as how C/C++ allowed for inline assembly way back in the day) will be the answer here.

"The other element I imagine will/could be interesting to explore will be the intersection of what natural language models and compiler front-ends will look like--if we can get a language to start looking more like a natural language, it might enable some interesting DSL-type scenarios for end-users, and reduce the demands/pressures on developers to respond to every little change."

Result Hello, Wing. Hello, Wasp. You're not quite what I was expecting or anticipating, but you're starting to get some interesting traction, and you're certainly not alone. You're also not natural-language-based like I'd anticipated (or hoped!), but languages take a while to bake. I personally have some ideas on how this should look, but mine is but a small whisper of an idea, and I expect there will be vastly brighter people than me going into this space. Still, I'm feeling generous to myself: +1.

Cracks in the "full-stack developer" facade will grow. (p 0.7)

"The demand for the "full-stack developer" is mostly a cop-out on the part of managers, who don't want to artificially restrict their employee search because "hiring is hard". So, instead of carefully considering what skills the current team lacks that needs shoring up, they instead scribble "full stack developer" on the job description, and proceed to rattle off every single technology that the team has ever used."

Result Some folks are definitely not seeing it. Of course, many of those folks are tech-training companies, whose business depends on developers (or their managers) feeling inadequate and needing training. Others, however, are pointing out the obvious flaws in the approach, that "FSDs" can't really scale out as well as specialists can. Still, it seems that the recruiting posts and hiring managers still have "FSD on the Brain" syndrome, despite the clear drawbacks to trying to hire jacks-of-all-trades, so I have to give myself a -1 here.

AR/VR will start showing glimmers of life (p 0.5)

"If AR/VR is going to achieve anything of any substantial value, it really needs to do so soon, before it is relegated to the "Interesting Things That Were Out Of Their Time" bin, like the Apple Newton and the Xerox PARC. This year might--might--be the year it presents us with something, particularly given the rise in interest in wearables. Certainly, with Meta going all-in on the whole "Metaverse" thing (whoever came up with that should be fired), there's going to be no shortage of money thrown at the problem, the big question is, will there be any actual innovation around UI? Because so long as the AR/VR thing is solely game-centric, it's fairly safe to assume that the AR/VR will go right into that bin without too much hesitation. I know, games are huge industry (measured in the trillions now?), but it's not enough to support hardware; witness the difficulties even conventional gaming hardware (for example, joysticks--today's flight sims are vastly more complicated than the ones twenty years ago, yet nary a joystick to be found) has had over the decades. If AR/VR (which requires hardware specific to it) is going to reach a point that justifies buying specific hardware for it, it has to be for something more than just gaming (and please don't say "education", because if there's one place in the world that has almost no budget, it's education)."

Result Oh, Ted. Sometimes you just have to stick to your guns on a theory, and not try to offer a tech a way out of its fate. With Apple basically shutting down the Vision Pro, and Microsoft already out of the HoloLens business, it seems like AR/VR is probably "done" for the forseeable future. Meta is still a player here, though, and they're large enough to continue to throw good money after bad, so there's still hope, I suppose, that Oculus will suddenly make a splash... but I doubt it. -1

"Car apps" will begin to become "a thing" (p 0.6)

"More and more vehicles out there are coming with computers in the console, offering easy connection to your mobile device. What's been lacking has been any sort of apps for it, beyond the basics (maps, audio entertainment, etc). Now that there's more of a target market, perhaps we are reaching a critical mass of potential customers to justify the investment in building apps specificall for vehicles."

"If this does start to take hold, it will be interesting to see what sorts of apps get built (I could imagine some CRM apps for certain kinds of salespeople, for example) and what form user interaction takes hold (voice control and interaction would be very important for drivers, for example). Frankly, though, the hard part will be the core innovation itself--what sort of apps do we want when we're driving?"

Result "Car apps"? What was I thinking? -1

Kotlin will remain an "Android-only" language (p 0.5)

"This one is a hard one to call, but I'm flipping a coin and landing on the pessimistic side of the "Will Kotlin break out of Android-only?" question. For those who weren't following the Kotlin story at home, Kotlin has recently done a couple of things to make it more than just "that language you use for Android" by developing Kotlin Multiplatform and Kotlin Native. These open up the Kotlin language for use in more situations, but their success will really hinge on how many developers actually want to use Kotlin in more places than just their Android codebase.

"Historically, multiplatform languages have not done well, with the sole (arguable) success being that of Java--whose "Write once, run anywhere" campaign didn't really accomplish much. (Remember, most of Java's success was in the server room, not the desktop.) Native might have more possibility for success, but either one gaining any traction would be an interesting development and potentially grow Kotlin beyond just "an Android thing"."

Result Kotlin remains firmly in the Android space, near as I can tell, and nobody seems all that interested in Kotlin Multiplatform. -1 It's certainly possible that the interest can build, but if there's not much of a splash at its introduction, it's hard to build later (without some kind of compelling shift in the market, anyway). Remember, the only reason we're talking about Kotlin today is because Google adopted it as its language of choice for Android (in order to get clear of Java, some might suggest).

C# and Java will continue to "pile on" with features (p 0.8)

"Both languages have reached a point where the weight of the language is really beyond the ability of any one person to keep track of what's there, yet each time a new release comes out, it comes with a whole slew of new proposed features designed to bring new syntax into an already-existing part of the language or an attempt to cram yet-another-interesting-idea into an already "overloaded-collection-of-features"-oriented language. (Can you really look at either C# or Java and tell me that they are "object"-oriented anymore? You can go quite a ways with either and never see or hear an object anywhere along the way by this point.)

"Look, maybe this is my "Gitoffamahlawn" moment to these predictions, but both of these languages are old enough to drink, they each spun up powerful and resilient platforms that have a number of newer, simpler, more-concise languages, and at a certain point in time, it's reaasonable to say, "This one is done. There's nothing new we need to add." To keep cramming new stuff in is a terrible addiction--it's a way to show that "We're still hip! We're still doing cool stuff (...and therefore justify why companies should still be paying us license fees!)" We keep telling startups not to do it, but our two major providers of software development tools can't seem to get the lesson themselves.

"Spend some time with F#. With Clojure. With Kotlin. Play with some of the new ideas (Mint, Io, Alloy). Or even go back and experiment with some of the classics (Smalltalk, Self, LISP, Scheme). But, for your own sake, stop enabling the relentless pounding of the feature surf on the beach of your brain by breathlessly hanging on "What comes next", because in time, you'll be battered into sand."

Result Yeah, lawn, off, get. +1

Crypto falls even further. Blockchain struggles to reinvent. (p 0.7)

"Look, faith in cryptocurrency is really at an all-time low, except for the people who are caught holding the bottom of the Ponzi pyramid, and they desperately want to hype the hell out of it so they can pass the bag on to the next level down. That won't change. Blockchain, the world's worst-designed distributed database, will continue to wrestle with the goal of finding a reason to exist, and hey, could maybe even find one bullseye, given enough darts. (Many companies that invested in Blockchain have either decided to walk away from the dartboard or else that they're going to load up on a lot of darts--and maybe a shotgun or two--until they hit that bullseye.)

"I really would love for these two to just go away entirely, but they won't, because nothing ever dies completely. But this is the last year they're going to get much--if any--serious time in the tech press, despite the fevered efforts of a precious few on LinkedIn. (Yes, Andreeson-Horowitz is going big into it; yes, they've been big into it since it's start; yes, there will always be people who see what they want to see, instead of following actual evidence; and yes, folks, there's still time to dive in and turn your profit, just don't be the last one holding the pyramid up!)

"And, for me, 2024 will be the last year I talk about this in any form."

Result Well, this was well on the way to a +1... and then Trump decided to breathe new life into the whole mess by starting to tease greater legitimacy for cryptocurrencies by declaring a "strategic bitcoin reserve". That's got the whole space chittering with excitement, and naturally the markets went haywire. This seems like a good idea? Sadly, I fear even my "this will be the last year I talk about this in any form" prediction can't even hold true. -1

Phishing attacks go ballistic (p 0.7)

"Thanks, AI! With all these generative natural language tools (like ChatGPT), it will be easier than ever to generate emails intended to trick users into surrendering their credentials to fake sites. That will lead to more breaches, more private info leaks, and more of all the wonderful things that come with those.

"What will make it even worse will be all the security companies that roll out "AI-based" security tools, claiming that the AI will somehow be able to better protect against those very same attacks--but while generative AI can create really good-looking human-facsimile output, it's not always great at recognizing artificially-created human-facsimile input. Which means the attackers just got a boost to their toolset, and the defenders will be looking to try and keep up this year."

Result Turns out, it's not your kid in jail needing bail money, after all. Fortunately, AI can also fight the scammers, by boring them to death! JFC. +1, but I hate myself a little bit over all this.

Databases will incrementally improve (p 0.7)

"... as opposed to releasing something drastic. There is some interesting ideas percolating over in the world of DBOS, spearheaded by one of the principals in the Postgres world, but I have a feeling that it'll take a little bit for those ideas to brew before there's really something there to evaluate. (Hope I'm wrong, though.)

"Meanwhile, though, players in the RDBMS world will slip a new feature into the mix by polluting the relational model just that wee bit more, the NoSQL players will pick just a tiny bit more from the relational world and add that, and developers will continue to choose which player they want to play with based on criteria that touches maybe 1% of the feature set of the player they choose."

Result DBOS continues to churn along, and overall, relational databases continue to introduce interesting ideas into SQL and get further away from the relational purity Codd and Date and the others sought back in the 70s. JSON is now a fixture in most SQL dialects, and it wouldn't surprise me (or anybody) if an LLM shows up in there somewhere, maybe shoehorned between the XML-generation code and Java bindings. (You did read parts 14 and 13 of the SQL specification, didn't you?) +1

I'm publishing a book (p 0.8)

"I and a few other folks are working on a book based on the Developer Relations Activity Patterns catalog that I drafted a ways back, expanding on the material there in a big way. That'll come out in 2024, almost guaranteed."

Result +1... almost. It's almost done, so it'll come out, just not quite in 2024.

Final tally

six wins ("+1"s)
one, maybe two "draws" ("0"s)
six losses ("-1"s)

... which puts me squarely in the middle of the bell curve. Good thing this isn't my day job!

So now let's turn our head to 2025.

The 2025 Tech Predictions are going to be wild (p 0.8)

Sometimes as part of writing these predictions, I take a look around the Internet (thanks Google!) with what other people are predicting, and let me tell you, the predictions for this upcoming year are off the hook. Autonomous robots. Human-computer brain augmentation. AI-generated AI creating new AI. If you ever saw it in a sci-fi movie, somebody out here on the Internet is predicting "this is the year"! (BTW, did you know that this year is going to be the year of Linux on the desktop? It's true! I read it on the Internet!) And the corresponding benefits are even more wild: Humans will be replaced by the singularity. No, humans will be able to all of us retire thanks to the singularity. No, humans are the singularity. More than any year, 2025 really seems to be catching fire in peoples' imaginations, and they're letting their tech-freak flag fly.

"Developers are obsolete" prose will peak, then dwingle (p 0.8)

AI startups are going to start failing (p 0.5)

ARM is going to start eating into Intel in a huge way (p 0.7)

In the beginning, there was the x86. And Intel rode that line into massive, generational-spanning profits. But slowly, over time, the ARM chip architecture has been slowly but steadily eroding the bulwark that x86/x64 erected ("You'd have to recompile everything! Nothing would run out of the box! Dogs and cats will merge to form a master race that will enslave humanity!"), such that by the end of 2024, not was any shipping device without a built-in keyboard running ARM internally, Microsoft had a version of Windows running on the ARM chip. This upcoming year, there's literally no barriers left to ARM's complete dominance of the CPU market. The "Apple Silicon" MacBooks were just the vanguard in the horde of new ARM-based laptops that are going to come out and essentially take over. (As a caveat, if you're a native-language developer, as in you use C, C++, Swift, Rust, Go, Nim, Zig, and so on--if you've not started learning what ARM assembly looks like and behaves, now's a good time to start learning it.)

Java and C# aren't going to introduce anything significant (p 0.7)

TypeScript jumps the shark (p 0.6)

Python type hinting goes mainstream (p 0.6)

(BTW, Pythonistas, you really need to read--nay, study--the PEP in detail. It's massive.)

JetBrains tries very hard to push Kotlin Multiplatform (p 0.7)

Flutter begins its quiet slide into obscurity (p 0.7)

Swift will release a new version, and it will have incompatibilities (p 0.8)

Rust will not replace C++ (p 0.7)

Odin, Nim and Zig will gain some attention (p 0.5)

DIY databases will start gathering attention (p 0.5)

Our book will ship! (p 0.9)

Seriously! It will! I promise!

Maybe I'll have a full-time gig again? (p 0.5)

I'll start either a YouTube channel or some video training (p 0.4)

The probability goes up if I don't have something full-time landed by the end of January.

I'll speak at a brand-new (to me) conference (p 0.5)

As always, feel free to comment and tell me why I'm crazy, in whatever forums feel comfortable. In the meantime, talk to you all next year... if not sooner.

Farewell, Twitter

Sat, 16 Nov 2024 00:00:00 +0000

Over time, people change, and it becomes necessary to part ways because of that change. So, too, with organizations and, in this particular case, social media platforms. When it began, Twitter was a bastion of hope and optimism, the center of the pro-democracy "Arab Spring" and a (naive, perhaps) belief that ordinary people communicating with one another frankly and openly could provoke serious change in the world. Then, Elon bought Twitter, and it accelerated its slide into the dumpster fire it has become.

I joined Twitter back in the 2007/2008 timeframe, egged on by a few speaker colleagues who'd discovered it. I had no idea what to do with this "microblogging" platform, as I had a blog, and I was just fine with posting long-form content. When I started, I wrote haiku on it--not my idea, Neal Ford's--because I couldn't really figure out what you do with less than 140 characters at a time. But it became apparent that this was becoming something of a magical place, making connections and exchanging ideas. More than once I made a connection on Twitter that, over time, turned into an online friendship that, in a few cases, turned into a "real-life" friendship.

Those days are long past. X has now, thanks to Elon's "stewardship", turned into a cesspool of right-wing aspiration, far-right harrassment, and incessant click-baiting and rage-baiting posts. When the moderation team was entirely fired shortly after his purchase of the platform, I had my concerns, but I thought, "Maybe I need to stick it out and stand up, use my follower count as a counter against the ideologuery that is building."

But not anymore. It's exhausting, and I don't see the point of it anymore. This isn't a physical property or an ancestral homeland that needs defending--it's all just bits living in servers and phones someplace. The only value is in the engagement, if there's no engagement, then there's no point. And I simply refuse to be engaged by political forces that are more interested in anointing an American aristrocracy than in preserving democracy. So, it's time.

I will not abandon my account--that would give somebody else an opportunity to use my handle for their own purposes. Instead, I'll just quiesce my participation on the platform, neither posting nor "liking" or in other ways engaging. The more of us that flee the platform, the less attractive it becomes as a place for advertisers and marketers. The less attractive it becomes, the less money people will spend on it. Hey, if Elon wants to use his fortune to prop up a social online network that is essentially a giant salve to his fragile ego, it would be entirely in keeping with what I've observed of the man in person. (Yes, I've met him in person, at an OReilly FOO camp way back in the day. I was not impressed then, and I'm even less impressed now.)

Meanwhile, I'm thinking my engagement will now take place on LinkedIn, Reddit, and Bluesky. If you're reading this, you probably already know how to find me on those places.

As for social media in general.... I think that social media is going through its "yellow journalism" period, hearkening back to the days when printed newspapers and journals wrote any headline that might sell more copies. William Randolph Hearst was the Musk of his day, and in time, journalism standards stiffened to the point where it was unthinkable to write the kind of headlines like that which started the Spanish-American War. (Seriously, look it up.) In time, administrations (probably not the US) will realize that social media is every bit as influential and important as newspapers and other periodicals, and begin to enforce standards of conduct and honest reporting that will build trust again. (Honestly, we could use those for newspapers and other periodicals again, too.)

But in the meantime, I strongly suggest to people that you go back to some of the original news outlets--AP and Reuters--and subscribe to their feeds. If you want analysis, I suggest places like The Economist, and if you want summations, consider The Week. But don't assume that any news outlet that disagrees with your interpretation of the world as "fake news", because if we can't agree on the basics of reality, we are collectively doomed as a species.

On Avoiding Foolish Opinions

Fri, 18 Oct 2024 00:00:00 +0000

(I came across this quote from Russell, and liked it so much I thought I'd just copy it verbatim over to my blog. Obviously I'm not the author--I'm not nearly this eloquent.)

(Via The Basic Writings of Bertrand Russell)

If the matter is one that can be settled by observation, make the observation yourself. Aristotle could have avoided the mistake of thinking that women have fewer teeth than men, by the simple device of asking Mrs. Aristotle to keep her mouth open while he counted. He did not do so because he thought he knew. Thinking that you know when in fact you don’t is a fatal mistake, to which we are all prone. I believe myself that hedgehogs eat black beetles, because I have been told that they do; but if I were writing a book on the habits of hedgehogs, I should not commit myself until I had seen one enjoying this unappetizing diet. Aristotle, however, was less cautious. Ancient and medieval authors knew all about unicorns and salamanders; not one of them thought it necessary to avoid dogmatic statements about them because he had never seen one of them.

Many matters, however, are less easily brought to the test of experience. If, like most of mankind, you have passionate convictions on many such matters, there are ways in which you can make yourself aware of your own bias.

If an opinion contrary to your own makes you angry, that is a sign that you are subconsciously aware of having no good reason for thinking as you do. If some one maintains that two and two are five, or that Iceland is on the equator, you feel pity rather than anger, unless you know so little of arithmetic or geography that his opinion shakes your own contrary conviction. The most savage controversies are those about matters as to which there is no good evidence either way. Persecution is used in theology, not in arithmetic, because in arithmetic there is knowledge, but in theology there is only opinion. So whenever you find yourself getting angry about a difference of opinion, be on your guard; you will probably find, on examination, that your belief is going beyond what the evidence warrants.

A good way of ridding yourself of certain kinds of dogmatism is to become aware of opinions held in social circles different from your own. When I was young, I lived much outside my own country in France, Germany, Italy, and the United States. I found this very profitable in diminishing the intensity of insular prejudice. If you cannot travel, seek out people with whom you disagree, and read a newspaper belonging to a party that is not yours. If the people and the newspaper seem mad, perverse, and wicked, remind yourself that you seem so to them. In this opinion both parties may be right, but they cannot both be wrong. This reflection should generate a certain caution.

Becoming aware of foreign customs, however, does not always have a beneficial effect. In the seventeenth century, when the Manchus conquered China, it was the custom among the Chinese for the women to have small feet, and among the Manchus for the men to wear-pigtails. Instead of each dropping their own foolish custom, they each adopted the foolish custom of the other, and the Chinese continued to wear pigtails until they shook off the dominion of the Manchus in the revolution of 1911.

For those who have enough psychological imagination, it is a good plan to imagine an argument with a person having a different bias. This has one advantage, and only one, as compared with actual conversation with opponents; this one advantage is that the method is not subject to the same limitations of time or space. Mahatma Gandhi deplores railways and steamboats and machinery; he would like to undo the whole of the industrial revolution. You may never have an opportunity of actually meeting any one who holds this opinion, because in Western countries most people take the advantage of modern technique for granted. But if you want to make sure that you are right in agreeing with the prevailing opinion, you will find it a good plan to test the arguments that occur to you by considering what Gandhi might say in refutation of them. I have sometimes been led actually to change my mind as a result of this kind of imaginary dialogue, and, short of this, I have frequently found myself growing less dogmatic and cocksure through realizing the possible reasonableness of a hypothetical opponent.

Be very wary of opinions that flatter your self-esteem. Both men and women, nine times out of ten, are firmly convinced of the superior excellence of their own sex. There is abundant evidence on both sides. If you are a man, you can point out that most poets and men of science are male; if you are a woman, you can retort that so are most criminals. The question is inherently insoluble, but self esteem conceals this from most people. We are all, whatever part of the world we come from, persuaded that our own nation is superior to all others. Seeing that each nation has its characteristic merits and demerits, we adjust our standard of values so as to make out that the merits possessed by our nation are the really important ones, while its demerits are comparatively trivial. Here, again, the rational man will admit that the question is one to which there is no demonstrably right answer. It is more difficult to deal with the self esteem of man as man, because we cannot argue out the matter with some non-human mind. The only way I know of dealing with this general human conceit is to remind ourselves that man is a brief episode in the life of a small planet in a little corner of the universe, and that, for aught we know, other parts of the cosmos may contain beings as superior to ourselves as we are to jellyfish.

Book Review: Data Structures and Algorithms in JavaScript

Mon, 7 Oct 2024 00:00:00 +0000

No Starch Press sent me an early access/review copy of the book "Data Structures and Algorithms in Javascript", by Federico Kereki. If there's two sentences that summarize the book, it's these two from the Introduction: "The book deals with two basic concepts in computer science: data structures and algorithms. It follows a structure similar to university curricula and adds examples taken from coding challenges and interview questions, using them to discuss the relative advantages and disadvantages of specific algorithms and data structures." In other words, it's exactly what it purports to be, and that's a good thing.

tl;dr, this is a great book to read if you've never had a formal introduction to ECMAScript (which, by the way, is the formal name for the language--Javascript is the name of the Netscape implementation of the language, just as JScript is the name of the implementation that Microsoft did a while back) or to some core computer science topics. The first chapter is a great succinct summary of Javascript, and the second is an intro to functional programming, though one that is entirely tailored to Javascript, and is missing a great number of functional features (like partial application of functions or currying) which are missing from Javascript. I kinda wish the book covered a bit more of FP than just what's there, but at the same time, the book is focusing on data structures and algorithms in Javascript, and not FP in general.

Given that this was an early access/review copy, I didn't have a physical copy of the book to work with, but from the PDF, it looks like a pretty traditional No Starch Press book, 500+ pages long, and the rest of the book is pretty much a common-sense written version of a more-than-decent computer science textbook. What's also reminiscent of a textbook is the list questions at the end of each chapter, with answers at the back of the book, designed specifically to give the reader a chance to test their understanding at the end of each chapter. It's a nice touch, and although in general I'm not a huge fan of them, Kereki does a pretty good job with these, keeping them high-level enough to get the point of the TfU across while being low-level enough to have a clear goal and area of study.

The TOC is below:

PART I: THE BASICS

Chapter 1: Using JavaScript
Chapter 2: Functional Programming in JavaScript
Chapter 3: Abstract Data Types
Chapter 4: Analyzing Algorithms

PART II: ALGORITHMS

Chapter 5: Designing Algorithms
Chapter 6: Sorting
Chapter 7: Selecting
Chapter 8: Shuffling and Sampling
Chapter 9: Searching

PART III: DATA STRUCTURES

Chapter 10: Lists
Chapter 11: Bags, Sets, and Maps
Chapter 12: Binary Trees
Chapter 13: Trees and Forests
Chapter 14: Heaps
Chapter 15: Extended Heaps
Chapter 16: Digital Search Trees
Chapter 17: Graphs
Chapter 18: Immutability and Functional Data Structures

If anything about this book is true, it's that Kereki does an amazing job of staying 100% on task throughout the book. The first part is entirely what it says: it's all about the basics--Javascript, FP, ADTs (intro to data structures) and big-O notation (algorithms). The second part dives deeper into algorithms, and third part into data structures.

One area that I've personally always had a bit of a problem with is understanding persistent data structures--that is, data structures that have the interesting property that can be updated while keeping previous versions intact, without changes. This property automatically implies that these structures are ideal for purely functional programming languages, which do not allow for side effects. Okasaki's book is the gold standard on that, but I never really grasped what was going on there. While I can't say that I completely understand functional data structures yet, I can say that, like the great physicist, "I find that I am still clueless, but at a much higher level."

Thoughts

I like this book! It's not going to be for everybody--it's dry in places, but it's also not trying to be anything other than what it's declaring itself to be. It's a solid discussion of some core Comp Sci principles, but done in a language that isn't normally what's used by CompSci grads. And that, in many ways, is the key strength of this book--this is not a book for Computer Science graduates, it's a book for those thousands and thousands of bootcamp graduates who were all turned away because they lacked the knowledge of Big-O notation and data structures. This book, in a nutshell, is the book that every bootcamp grad needs to read, to catch up on all the theory they missed by not taking a CS degree, but it's in a readable format and approachable style that any novice Javascripter will be able to comprehend. (Note, I don't mean "right away"--some of these are dense topics that will take time and practice to master, but at least there's no Greek symbols or confusing functional programming terminology to act as gatekeeper.)

If you're a well-practiced Javascripter who never learned the formalities, some of this will be redundant, and if you're a CS graduate who did well, you'll probably pick up a few things about Javascript. But overall, this book is a gold mine for the aforementioned bootcamp grad, and if I had had this book available to me a decade ago when I built a web app team out of fresh-from-bootcamp graduates, I'd have bought each and every single person a copy.

Book Review: Art of ARM Assembly (Vol 1)

Mon, 23 Sep 2024 00:00:00 +0000

No Starch Press sent me an early-access e-copy drop of the book "The Art of ARM Assembly (Volume 1)", by Randell Hyde, an author whose previous books on x86 assembly are on my shelf, and I'm enthusiastically suggesting you go buy a copy when it comes out.

And since the move to ARM-based processors in iOS (and later Android and later my MacBookPro machine and now even some Windows machines), I've been unable to do that.

Kotlin/Android? Compiles down to bytecode before turning into native instructions; Java or C# on my M3 MacBookPro? Yeah, I kinda just trust the JITter, but... The truth is, I'd been holding off on learning ARM mostly because I didn't have a guide I trusted to lead me past the thorny parts. I've looked at a few ARM assembly books, but always put them back on the shelf because I just... I dunno... didn't feel like the author was going to find the right mix of "teaching" to go along with the tech.

(In fairness, I haven't mastered x64 yet, either, mostly because it often seems like it's "close enough" to x86 that I can muddle my way through. That said, I have Hyde's x64 book, or at least Volume 1 of it, and it's on my list to read once I get through ARM.)

Now, though, I feel like I flew to Nepal, walked out of the airport intending to find a random Sherpa to guide me up the mountain, and ran into an old friend who also happens to be incredibly experienced at guiding people up Everest.

As with any early-access drop, it's important to caveat that it's possible that the final product you end up with in your hands will differ from what I'm seeing as I read/write this... but considering we are talking about a thousand-plus page book here, I really, really, really doubt Hyde's going to rewrite much except to correct mistakes/typos.

Physical

Well, I only have an e-reader (PDF) copy to work from, but did I mention the thing is a thousand pages? Throwing the printed copy of this at somebody is going to qualify as "Assault with a Deadly Weapon" in thirty-three states! (In Texas, of course, everyone will expect you to have a copy mounted on a rifle rack in your pickup truck.)

But it's not like this is a lightweight subject, either--assembly language is notorious for its requirements around attention to detail to function correctly. So it seems entirely reasonable to assume that if you're new to ARM, then you probably need to understand some of the hardware that drives the assembly language, since the two are going to be pretty tightly joined at the hip. On top of that, Hyde doesn't really seem to want to assume much about his target audience's skill level other than "At minimum, this book assumes that you have some experience in a language such as Pascal (or Delphi), Java, Swift, Rust, BASIC, Python, or any other imperative or object-oriented programming language." It's a pretty broad range, but more importantly, notice that none of the aforementioned languages have much experience (or reason for it) with the lower-level details. (Back in my day, we learned C! And pointers! And we liked it!)

By the way, it's important to note that Hyde is no stranger to the subject. As he states in the Introduction, "This book is a sister volume to The Art of 64-Bit Assembly Language, which was, itself, a rewrite of The Art of Assembly Language Programming (AoA). AoA was a project I began way back in 1989 as a tool for teaching 80x86 (x86) assembly-language programming to students at California State Polytechnic University, Pomona, and the University of California, Riverside." 35 years, and dude's just been cranking one assembly language book out after another. (If I remember correctly, he also did a MASM Bible thirty or so years ago, which was one of the go-to books back in the day, too.) This is what I mean about "seasoned guide"--not only is Hyde an expert on the subject, he knows how he wants to get the material across to his audience/students, and he has a ton of experience doing so.

Lastly, this is officially Volume 1 of a planned two-volume set; on his website, Hyde notes, "The Art of ARM Assembly Language is actually going to be part of a two volume set. [Volume 2] will cover 32-bit ARM assembly language programming on 32-bit versions of Pi OS and on embedded ARM Cortex-M CPUs (such as Arduino/Teensy and Raspberry Pi Pico). Look forward to it sometime in late 2026." --Source

Hyde doesn't waste much time getting started right from the start, and the start here begins with the Introduction, where he recounts a brief history of ARM and then rationales for learning ARM and 64-bit ARM, respectively. In many other books, you can breeze past the Introduction/Chapter 0 without losing much, but honestly, I'd recommend you take the time to go through it--there's a lot of new products, terminology, and ideas in here that it really should be thought of as the first chapter of content. (Although, to be fair, arrays in assembly start at 0, just like in C, so maybe this is the first chapter!)

Hyde is also extremely straightforward and up-front about some of the decisions he makes in the book and why; for example, he doesn't cover Microsoft's toolchain, even though ARM is now running in Surface books. The reason? "In theory, it should be possible to apply the information in this book to ARM-based Windows machines (such as the Surface Laptop Copilot+). Unfortunately, Microsoft’s software development tools, particularly its assembler, is based on the original ARM assembly syntax defined by Arm (the company), not Gas [The GNU Assembler, which he explained a few paragraphs earlier --TKN]. While Microsoft’s armasm64 is a better tool in many respects (as it uses standard ARM assembly language syntax), everyone else uses Gas syntax. The machine instructions are more or less the same between the two sets of assemblers, but the other statements (known as assembler directives or pseudo-opcodes) are completely different. Therefore, example programs written in Gas will not assemble under armasm64, and vice versa. Since trying to present both syntax forms in example programs would be just as confusing as trying to teach 32- and 64-bit programming simultaneously, I stick to Gas syntax in my examples." I don't like that we're using the (Hyde's opinion) suboptimal tool, since he thinks armasm64 is the better tool, but I can respect the decision. More importantly, thanks, Hyde, for sharing the insight and the rationale--I can accept a decision like this much more easily when I know the reason for it.

Here's the TOC, and I am in no way going to go through each chapter; in fact, I'll just go over Chapter 1 (which is about as far as I've gotten, in any depth, because I want to go through and code along with the book) in any detail.

PART I: MACHINE ORGANIZATION

Chapter 1: Hello, World of Assembly Language

Remember when I said that Hyde starts running from the jump? He opens this chapter with some example assembly code that--literally--does nothing, as a way to explain some initial details. Linux and macOS, for example, treat names slightly differently, so he takes a moment to walk us through that. He shows the commands for assembling, linking, and running the example do-nothing program, then promptly demonstrates that the C/C++ toolchain will actually serve as a nice umbrella to do all that in a much simpler command-line. In a nice touch to show he's not in denial about what people want to learn assembly for, though, he immediately turns around and shows how to build a simple C++ program that calls that do-nothing assembly code. Fear not--he assumes readers have little to no experience with C/C++, and shows "just enough" to get the point across.

Once that's done, though, he covers some of the hardware basics about the ARM processor(s), and goes through some of the basics of how CPU and memory work at a low level. In some ways, this feels a little overkill--after all, he's clearly not assuming we are novice programmers, just new to assembly language--but at the same time, it's not so much overkill that the reader familiar with the idea can't just turn the page early.

He then goes over four or so ARM instructions (ldr, str, adr, and adrp), and their syntax and semantics. Almost immediately, we run into a problem--macOS and Linux don't like using absolute memory addresses, and require relative ones. (I can sort of understand this, since it offers the OS a bit more flexibility where to map a given module inside of a process space--but that's about the extent of what I think I know there.) So Hyde has some macros that effectively hide the difference; I kinda wish he didn't, so that I as the reader could be forced to understand the problem a bit more, but for a book that's looking to teach ARM, and not Linux/ARM or macOS/ARM, this seems reasonable.

More instructions are covered in some detail, though most behave as one would expect similar instructions from other assembly languages to behave.

Then, in Sections 1.9 and 1.10, Hyde discusses the ARM ABI (Application Binary Interface) and how to write ARM assembly to call C functions (most notably for I/O, a la printf). He ends Chapter 1 with a "Hello World" that mixes compilation with C and the ARM assembler, giving us a nice springboard from which to explore more ARM instructions in further chapters. Then, at the end of this (and I think every) chapter, he has a few "Test Yourself" questions to test your comprehension of the reading--these sorts of things are not my favorite, since I think they don't really serve the purpose they're intended to.

(With a book, so much of the experience is self-directed, I'd much rather see some suggestions on how to take the code we covered in the chapter and modify it in various ways--like, for example, "The printf function takes a variadic number of parameters to go along with the format string, so write a mixed-mode program that prints "Hello, %s, you are %d years old today!" where name and age are passed as parameters along with the format string." That forces(?) me to go and write that program, which builds on what I'm typing in, rather than testing my ability to go back and re-read a page in the book.)

Keep in mind, Chapter 1 spans 40-something pages (p3 to 44), and it's dense the entire way. Quite honestly, having read the chapter, I'm still going to go back and re-read/explore the chapter, this time typing (or, more likely, cutting-and-pasting from the PDF at least some of) the code he lays out in here, because I'm not going to learn ARM just by reading the book. Like most languages, it really needs to be experienced, not just observed.

Chapter 2: Data Representation and Operations
Chapter 3: Memory Access and Organization
Chapter 4: Constants, Variables, and Data Types

PART II: BASIC ASSEMBLY LANGUAGE

Chapter 5: Procedures
Chapter 6: Arithmetic
Chapter 7: Low-Level Control Structures

PART III: ADVANCED ASSEMBLY LANGUAGE

Chapter 8: Advanced Arithmetic
Chapter 9: Numeric Conversion
Chapter 10: Table Lookups
Chapter 11: Neon and SIMD Programming
Chapter 12: Bit Manipulation
Chapter 13: Macros and the Gas Compile-Time Language
Chapter 14: String Operations
Chapter 15: Managing Complex Projects
Chapter 16: Stand-Alone Assembly Language Programs

PART IV: REFERENCE MANUAL

Appendix A: The ASCII Character Set
Appendix B: Glossary
Appendix C: Installing and Using Gas
Appendix D: The Bash Shell Interpreter
Appendix E: Some Useful C Language Functions
Appendix F: Answers to Questions

I suppose if the book were going to lose some weight somewhere, the appendices could be trimmed down--does anyone really need (or want) a reference to the ASCII character set in a book, when Google is just six (or ten) keystrokes (or a single "New Tab" in an existing browser window) away? The Glossary is needed, for sure, and the "Answers to Questions" (did I mention he sprinkles little exam-like tests-for-understanding throughout the text?) are necessary, but "Installing and Using Gas" I would've expected in Chapter 1 or the Introduction, and I dunno if a primer on using Bash is all that relevant here, particularly since it covers the truly basic commands (pwd, ls, cd, mkdir, date, and so on). Frankly, a quick primer on Makefiles would probably have been a better use of the pages, if you ask me.

(Such a primer is discussed in Chapter 15, apparently, and Hyde has a note why shell scripts vs Makefiles: "If you have experience developing software by using the command line, you may wonder why I haven’t built the examples with a makefile. I discuss makefiles further in Chapter 15, but I’ve chosen not to use them here for a couple of reasons: (1) If you don’t already know the Make language, I’d prefer to put off teaching that until you’ve mastered a little more assembly language. (2) Using Make would mean writing a separate makefile for each example program. However, the build shell script this section describes works for nearly all the example programs in this book." I'm not sure I agree, but again, he has his reasons, and his explanation satisfies me that it's not a decision made lightly.)

Aside from the Glossary and Answers, though, some hidden gold lies in the "Some Useful C Language Functions"; these are quick definition and description of some of the most common functions in the world, the C standard library. And before you accuse me of being wishy-washy ("you don't want a Bash primer, but a C stdlib primer is OK?"), remember that Hyde is expecting the reader to be familiar with other languages, many of which have a string intrinsic or built-in type, and therefore those developers may not realize that C required us to work with strings "the hard way",as pointers to byte arrays that signaled termination with a NULL. (Back in my day, we just used pointers! And we liked it!)

Thoughts

As you probably could guess, I love this book. That said, this book is probably going to be on my list of "Books you should read when you have a bunch of free time (which for most people is 'never')". It's not that the writing is poor; Hyde has been here before, and knows what to include and how to present it. It's much more due to the fact that this is not top of my day job's requirements, and it's a lot to wade through.

In many respects, going down this path is the same as going down the path of building your own database, or your own RPC stack, or your own Javascript SPA framework. The knowledge gained here is foundational, and will help programmers with all sorts of different kinds of applications, but will also run the risk of feeling too abstract and removed from the problems of the day. (I say this because I've taught workshops at conferences on how to build your own bytecode virtual machine, and those were some of the evaluation comments I got as feedback.)

To wrap up: I love the book, and I think every programmer should read it. Candor compels me, however, to point out that what readers learn from this book is subtle, philosophical in nature, and hard to articulate to others until you've turned it over in your head a few times. I personally think it makes you a better developer, but you'll need to have at least a little interest in the subject of programming languages to start.

Meanwhile, I'm eagerly awaiting the physical copy of the book when it comes out, although I do feel kinda bad for the delivery driver who has to bring it to my front porch--it will not be easy getting down the path to my front door with a forklift....

Disclaimer: No Starch Press sent me a review copy of the book; no other compensation was provided.

Book Review: Effective C (2nd Ed)

Fri, 20 Sep 2024 00:00:00 +0000

No Starch Press sent me a copy of the book "Effective C (2nd Ed)", by Robert C Seacord, and overall, it's not bad, though I don't think it lives up to the "Effective" moniker established by Scott Meyers three decades ago.

Let me get one thing out of the way first: Despite the title, this is not an "Effective" book as originally coined by the master of the Effective-style books (Scott Meyers). Meyers' "Effective C++", published roughly three decades ago, set not only a high bar to hit in terms of quality and depth of technical insight, but equally so the format. When one picks up an "Effective" book, one expects to see "items" that each tackle a particular problem or concern with the language, typically written in an action-minded format. When you wrote a book in the "Effective" series, you were writing to that standard and that format.

This book does none of that.

In fact, it doesn't even seem to acknowledge the shoes it is either deliberately or accidentally stepping around in. While Meyers has been retired for a few years now, the author, the contributor and both the technical reviewers have been involved in C/C++ standardization efforts, and so most assuredly were not only aware of Scott (and his work), but were probably friends with him at some point--it's really quite surprising to me that they wouldn't have some kind of nod or acknowledgement of Scott's work(s) and some kind of explanation around the title. Maybe it's me, maybe it's the fact that Scott and I had some long conversations about defending the quality and approach of the "Effective" moniker back when I was writing "Effective Enterprise Java" in his series two decades ago, but it really strikes me as odd (and disquieting) to see this particular word choice here.

That doesn't make it a bad book, per se, but if you were looking for the next in the generation of mind-opening books in that series, you are not going to be happy with your purchase.

Physical

Any book that has a winged Cthulhu/mind flayer on the cover trying to look benevolent holding out a helping hand while a glass-helmeted robot sits on its shoulder reading a book is a sign that this is a book that isn't quite sure where it wants to rest in the world, if you ask me. What's more, the endorsement quotes also generate a slightly weird feeling, as two of the four are from security professionals--which is the first indication that maybe this book isn't quite aimed at the same category as one might think.

Opening into the front matter, it's clear that Seacord and his supporting staff (contributor and reviewers) are all very smart people--they work on popular C/C++ compiler toolchains, they sit on the standards committees, and so on. There is zero doubt that these folks know what they're talking about when it comes to C. But as you read through Seacord's bona fides you notice that he's done a lot of things around the intersection of C and security; for example, he wrote "Secure Coding in C and C++" back in 2013. His reviewer for the second edition comes from a similar background, and you begin to get a sense that maybe this book isn't quite for the application developer.

The other thing to note is that on the front cover, it very clearly and prominently displays a banner that reads "Updated to cover C23", the 2023 version of the C programming language standard. More on that later.

As you get to the title page, however, the subtitle threw me for a leap: "Effective C / 2nd Edition / An Introduction to Professional C Programming".

And this is where my main criticism of the book comes: The book's title leads me to believe that this is going to help an existing C developer become a better one, particularly with respect to the new C23 standard; instead, the book's apparent aim is to teach the basics of the C programming language to people who've never programmed with it before, and it's very apparent when you read the "Who This Book is For" section in the Introduction that Seacord is definitely starting from zero:

"This book is an introduction to the C language. It is written to be as accessible as possible to anyone who wants to learn C programming, without dumbing it down. In other words, we didn’t overly simplify C programming in the way many other introductory books and courses might. These overly simplified references will teach you how to compile and run code, but the code might still be wrong. Developers who learn how to program C from such sources will typically develop substandard, flawed, insecure code that will eventually need to be rewritten (often sooner than later). Hopefully, these developers will eventually benefit from senior developers in their organizations who will help them unlearn these harmful misconceptions about programming in C and help them start developing professional-quality C code. On the other hand, this book will quickly teach you how to develop correct, portable, professional-quality code; build a foundation for developing security-critical and safety-critical systems; and perhaps teach you some things that even the senior developers at your organization don’t know.

"Effective C: An Introduction to Professional C Programming, 2nd edition, is a concise introduction to essential C language programming that will soon have you writing programs, solving problems, and building working systems. The code examples are idiomatic and straightforward. You’ll also learn about good software engineering practices for developing correct, secure C code.

"In this book, you’ll learn about essential programming concepts in C and practice writing high-quality code with exercises for each topic."

I can't quite tell who Seacord thinks is his audience for this book. Programmers from other languages who want to learn C? System administrators who want to make the leap to programming? Hobbyists who want to learn how to program their Raspberry Pi devices to automate their doggie door? He ends the Introduction with this: "You’re about to embark on a journey from which you will emerge a newly minted but professional C developer." But honestly, I'm not sure that's a true statement, since (as we'll see in a second) the ToC is really about teaching the reader the C language, but not much of the supporting cast and ecosystem that most C developers would need to know in order to be able to operate in a professional setting.

Let's dig in a little deeper.

At 300+ pages (including front and back matter), Effective C ("EC" for short) is no pamphlet, but it's not going to be a paperweight on your shelf, either. That isn't bad--books should be judged by their content and not their count. And, in fact, the aforementioned "Effective" books by Meyers were always on the skinny side, yet packed to the brim with good stuff.

Chapter 1: Getting Started with C

Almost from the first paragraph, the inconsistency of the target audience comes to bear. Seacord opens by showing the traditional "Hello World" C program, which (for convenience) I repeat here:

#include <stdio.h>
#include <stdlib.h>

int main() {
    puts("Hello, world!");
    return EXIT_SUCCESS;
}

He then follows this with a quick description of the program, starting with, "The first two lines use the #include preprocessor directive, which behaves as if you replaced it with the contents of the specified file at the exact same location."

OK, what?

If this is a neophyte's introduction to the C language, they've almost never heard of the preprocessor before, and certainly have no idea why one would want to "replace (the text) with the contents of the specified file". He does go on to describe those two files as header files, but ends that paragraph with the lines "You need to include the declarations for any library functions that you use in your program. (You’ll learn more about the appropriate use of headers in Chapter 9.)"

Let's be clear about something: If the intended reader is brand new to C, they didn't need to know what those lines do, only that you need to "#include" them in order to make use of them. If the intended reader has some familiarity with C, then the deeper explanation perhaps is merited, but then do they really need to be told what "Hello World" is doing?

This example then continues into the next section, wherein Seacord shows using "cc" to compile the code into an executable and running it. Again, if this is for neophytes, that's appropriate, but he never discusses where "cc" comes from or what might need to be installed if the command fails on the neophyte's machine. (In fact, his only suggestion if it doesn't compile is to "compare the program text from Listing 1-1 to your program and make sure they are the same." Not particularly helpful.)

This whole chapter is all over the map like this; for example, he spends two-and-a-half pages on the differences between "Implementation-defined behavior", "Unspecified behavior", "Undefined behavior", "Locale-specific behavior", and "Common extensions", which considering just four pages earlier he was introducing Hello World and the main() function, just seems weird.

Chapter 2: Objects, Functions, and Types

Next, we discuss objects, which is an odd term for a language that is very not object-oriented. I suspect the term is used in the C standard because as a generic word to describe the things that variable names point to, but in a world where "objects" are commonly associated with object-oriented languages like Java and C# (and of course C++), it seems an odd choice to fixate on that term--again without much nod to any of the ways somebody might have heard it defined differently--in an introductory text.

He then introduces the notion of functions by creating a swap function, but at first does it incorrectly (passing the integers by value) before showing how to pass the integers by reference and thus, introduce pointers. This seems an obtuse way to introduce the subject, particularly in what is intended to be an introductory text, particularly when he's spent no time really talking about what an integer is or what it represents. In fact, the introduction of variables is as follows: "When you declare a variable, you assign it a type and provide it a name, or identifier, by which the variable is referenced. Optionally, you can also initialize the variable." That's it.

To be fair, he gets into the different primitive types of the language, (booleans, characters, integers, floats, and so on) later, but even here, he's clearly writing for an audience that isn't a beginner: "The signed and unsigned integer types are used to represent integers of various widths. Each platform determines the width for each of these types, given some constraints. Each type has a minimum representable range. The types are ordered by width, guaranteeing that wider types are at least as large as narrower types. ..." (He later states that he will go into "excrutiating detail" about integer types in the next chapter, which begs the question, why not cover that material here rather than later?)

I'll cease diving into examples at this point, because I think I'm making my point clear.

Chapter 3: Arithmetic Types

Chapter 4: Expressions and Operators

Chapter 5: Control Flow

Chapter 6: Dynamically Allocated Memory

Chapter 7: Characters and Strings

Chapter 8: Input/Output

Chapter 9: Preprocessor

Chapter 10: Program Structure

Chapter 11: Debugging, Testing, and Analysis

Thoughts

Seacord ostensibly wants this to be a beginner's book, but he keeps not writing for beginners. When writing a book for beginners, the author needs to take careful steps to introduce new concepts one at a time, ignoring details of implementation unless they are necessary to understand the concept. It's not easy writing for beginners, particularly when you (the author) are steeped in the myriad of implementation details on a daily basis.

Honestly, if I were an editor at No Starch Press and Seacord were asking my advice for a 3rd edition, it would be this: Write this same book, but assume the reader knows basic C. They know what a function is, they know the basic types, they know how to compile and link simple programs. Now, take them one level deeper in all of those things. Let another book (or YouTube tutorial, or hobbyist exploration and spelunking) teach them the core basics, and you focus on explaining the "why" and "how" of the language. For example, at one point when introducing enumeration types, you state, "For portability and other reasons (Meneide and Pygott 2022), it is always better to specify the enumeration type." Why? What are those reasons? The reference to the other work is always nice, but summarize those reasons and lay them out. (Ironically, much of that chapter is at the level I would expect of the work I'm suggesting he write, and could probably be kept as-is with a few minor edits. In fact, much of the remainder of the book could probably be pivoted to this new, differently-assumed, audience without requiring much work.)

I can't say that I can recommend this book to beginners--I'd have a hard time recommending this to some of my college students, in fact, and many of them have studied Java (and after taking a class with me, Kotlin and/or Swift). The prose is quite precise in places, and Seacord clearly knows the C language, the standard, the history, and the "whys and wherefores" of C. What I'm not sure he realizes is how to write for beginners.

If you're already familiar with the basics of C, this book can seriously help you "level up" in your C knowledge, and bring you closer into the world of how the C language is "put together" and operates. If you're familiar with another C-like language, a la Java or C#, this book could probably serve as a way to get into writing C, but there may be some terminology issues that I'm not picking up on. (The "curse of knowledge" hits me just as hard as it does Seacord, which is why I generally don't write beginner-level books!)

Disclaimer: No Starch Press sent me a review copy of the book; no other compensation was provided.

Book Review: Strange Code

Thu, 12 Sep 2024 00:00:00 +0000

No Starch Press sent me a copy of the book "Strange Code", by Ronald T Kneusel, and honestly, they had me at "esolangs". But more than that, the book opines on reasons why studying programming languages, including their esoteric kin, is a useful and productive endeavor.

First off, let's get something out of the way right now: Being the huge fan of programming languages that I am, I am almost always a fan of a book that talks about how to build a programming language. This one not only talks about some of the concepts involved in doing that, and shows how to build two custom languages (Filska and Firefly), AND talks about a few popular esoteric languages and how they warp your brain in a good way, it takes the unusual step (in Part II) of examining some languages that Kneusel describes as "atypical", which makes them unknown to most developers, which have some interesting and useful properties. His coverage of CLIPS, alone, is worth the price of admission here.

Secondly, I love this book because Kneusel's history so closely parallels my own:

My first exposure to programming came in the summer of 1980. It involved an Apple II+ computer. As principal of the local high school, my father was able to bring it home for us to play with over the summer.

In my case, my parents spent my father's Christmas bonus to buy an Apple II+ in 1979.

The Apple II+ included a version of unstructured BASIC in ROM called Applesoft. As far as BASIC programming languages go, it was somewhat limited. However, it opened a new world to me, and I ran with it, writing small program after small program as I explored what the Apple II+ was and could do. It could ask questions, interpret answers, calculate formulas, plot pictures, and make sounds. And all of that power was at the tips of my 14-year-old fingers.

My fingers were a little bit younger than his, but the excitement was the same. Dad showed us a game program, "Little Brick Out" (basically one of those early "pong but you're knocking bricks out" games) and then he showed us the Applesoft BASIC code that powered the game, and frankly, I was hooked.

Of course, I wanted one of these godlike machines for my own. My father, likely thinking it would put me off, told my younger brother, Bob, and me we could have one if we learned how to program it. Challenge accepted!

In my case, Dad was learning Applesoft BASIC, and worked out a barter deal with the computer shop whereby he taught public classes on the language, in exchange for hardware and software purchases. We went from the base unit by itself (hooked up to a tape recorder and a small color TV) to a green-screen monitor (soooo much easier on the eyes) and first one, then two floppy drives. I asked for--and got--a joystick for my birthday. But more importantly, Dad let me tag along and follow him to the store when he taught the classes, and although it was hard for me to pay attention during the class, the idea of programming was laid into my genetic structure early.

You can already see where this is going--Kneusel's history and my own are so similar I could've been his brother. I taught myself BASIC and (a little) 6502 assembly, as he did, but couldn't get to Pascal because we didn't have the necessary CP/M card to run it. (Which sucked because I really wanted to learn how to program something like Wizardry, which was my game addiction on that machine for many years.) I wouldn't get to Pascal until high school, when a friend "loaned" me his copy of Turbo Pascal 5.5 for my PC, and then later Turbo C/C++. But I digress--my point is, this book feels like "coming home" for me.

The book is a respectable 450+ pages (including front and back matter), so this is no "I can read it in the bathroom in a week" kind of book. This is rich, meaty, hearty fare that will require any programmer reader to spend some serious cycles to ingest and absorb.

More importantly, Kneusel lays out the intent of the book right in the first chapter: "The purpose of this book is:

"To give you a sense of where programming languages came from and provide some context for the languages we use today and the structures they contain.
"To explain some of the essential elements of programming languages, so you become familiar with terms like Turing machine and Turing complete.
"To expand your thinking about what a programming language can be and how it can express thought and process in creative and el egant ways. We’ll do this by exploring various programming languages ranging from the unusual to the downright bizarre.
"To have fun! We’ll create our own programming languages, and see how we can use them to implement some algorithms. We’ll even put one on a small computer for standalone projects (the BBC micro:bit)."

Clearly, this is not a beginning programmer's book, but fortunately, it also doesn't require the reader to have a postdoctoral degree in Computer Science, either. The book states, "If you are a student, you’ll hopefully see that programming can be more than what you may perceive it to be based on your introductory programming courses, as necessary as they are. If you’re a hobbyist, you’re likely already somewhat familiar with esolangs and are looking to feed your passion." but as I read it, it really felt more that this was for the professional developer who is looking to understand what's going on inside and/or below the languages they use, and to grok the "why" of certain languages. I don't know if the student has the necessary perspective to get that, and while the hobbyist might get something out of this, it's hard for me to say--I left "hobbyist" status behind when I took that first programming job fifteen years after getting that Apple II+.

The book uses Python as its implementation language (where it needs to provide implementation), which is a reasonable choice, given Python's popularity.

Part I: On Programming Languages

Chapter 1: A Cherry-Picked Review of Programming Languages

It's a lovely little bit of history and archaeology (Kneusel calls it paleontology, likening different language eras to different eras of dinosaurs), but there is one nit: he references the website http://www.info.univangers.fr/~gh/hilapr/langlist/langlist.htm, which unfortunately in my Chrome browser doesn't render. I have to assume the website did render at one point (it appears to be an https/http issue), but it does take a bit of the wind out of the reader's sails to get excited about seeing more languages and being presented with a blank screen.

His coverage of the foundational languages (FORTRAN, Lisp, COBOL) and the "Ur-Languages" (Algol, APL, BASIC, PL/I, Logo, Simula, Pascal, Prolog, Smalltalk, and Standard ML) is pretty decent, though in my opinion he clearly has the most fun showing off the last three (Prolog, Smalltalk, and SML). Those are very different ways of thinking about code, and all three are still wildly relevant today. (As a matter of fact, any of those could be used to build a modern web application, which is hard to say about Logo!)

Chapter 2: The Essentials of Programming Languages

It's a very basic treatise of what makes up a programming language. If you've taken a language survey course as part of a CS program, there's little to nothing new here; if you've never explored this space, it's a pretty reasonable introduction (and a lightweight one at that) to the topic. (I do love that he uses an Apple II+ BASIC program and its representation as the example, though, instead of something more modern.) He shows off UCSD Pascal p-code, the ancient predecessor to Java/JVM bytecode and .NET CIL, which for some will be the first time seeing something like that. (Matter of fact, now I want to find a reference doc on the UCSD p-code opcode set.) He covers most of the popular categorizations of programming languages (static and dynamically typed, the core data structures like arrays, lists, etc, and so on), but it's mostly a concept-heavy chapter that readers could probably skip and come back to if they aren't as interested in the broad categories of programming. (That said, you risk not having some of the definitions in your head as you move to later chapters--caveat emptor.)

Chapter 3: Turing Machines and Turing Completeness

This chapter feels a little bit like a digression from the main point; yes, it's helpful to discuss what a "Turing-complete language" means, but I'm not sure that we really needed this deep a dive or to explore a Python implementation of one. It's definitely the heaviest "academic" chapter in the book, and readers who are new to the material may want to come back to this one.

Then again, I've always been a little less interested in the theory and math around languages, so this may just not be a chapter for me.

Part II: Atypical Programming Languages

Chapter 4: Forth

Forth is an entirely stack-centric language, which feels like something of a waste... until you learn that most virtual machines (the JVM, the .NET runtime, and others) are all stack-based virtual machines. Learning Forth gets you a leg up on understanding how those machines can do all their wonderful things entirely in terms of an execution stack.

(True story, I have a professional colleague/friend from 20 years back who is a huge Forth nut, and is constantly threatening to write a book on Forth--Lionel, if you're reading this, you should totally write it, and No Starch Press might even print it!)

Chapter 5: SNOBOL

"SNOBOL is a text pattern matching lan guage developed in the 1960s. Modern programmers might find its syntax quaint, and maybe even a tad frustrating, but I suspect the power of the language will shine through in the end as we explore its features, some of which are still with us in modern languages like Python." With these words, Kneusel opens the chapter on SNOBOL (“StriNg Oriented and symBOlic Language”), and with a philosophy of "All the world's a string", SNOBOL is off and running. Which, if you think about it, is quite true--almost all the data we hand back and forth across network lines are strings (usually shoved into JSON or XML).

Lest you think this is purely an exercise in trying to shoehorn old languages into modern conversations, take a hard look at the part of this chapter entitled, "Machine Learning with SNOBOL": "the goal of this section is to build a complete SNOBOL application to classify datasets using a simple machine learning technique--a nearest neighbor classifier."

Chapter 6: CLIPS

This is one of my favorites; I discovered CLIPS when I was looking at logic languages, and in particular when I discovered the Java flavor of CLIPS, called JESS (Java Expert System Shell). (Sadly, JESS development seems to have stalled since 2006.) If you've ever spent any time exploring the world of "expert systems" or "rules engines", you owe it to yourself to read this chapter. (By the way, CLIPS has bindings for both Java and .NET, so the material in this chapter is actually directly applicable to your day job, if you're in either of those platforms.) In many respects CLIPS occupies the same space as Prolog, but the syntax is in some ways a little easier to parse, and there's less emphasis around "the cut" (a Prolog-ism). That said, the syntax is essentially Lisp, so if you're not a fan of parentheses....

Part III: Esoteric Programming Languages

Chapter 7: The ABCs of ABCs

Chapter 8: FRACTRAN

Chapter 9: Piet

Chapter 10: Brainfuck

Chapter 11: Befunge

These are all esoteric programming languages, selected because (a) esolangs are fun, (b) esolangs take some of programming's most cherished traditions and turn them upside-down or do away with them completely, leaving you to think whether you wanted to or not, and (c) no seriously esolangs are fun! Kneusel takes each esolang, explains them, discusses why it holds interest, then shows how to implement them in Python (and sometimes another language as well).

(By the way, if your religious upbringing makes it hard to pronounce the language discussed in Chapter 10, it's also commonly called "BF", which should be easier for the "fuck"-challenged.)

Oh, and just because it's fun to toss this off as the last comment in this section, the following is a legal Befunge program:

<v"How now brown cow?"0
 >:v
^,_91+,@

PART IV: Homegrown Esolangs

Chapter 12: Filska

Chapter 13: Using Filska

Chapter 14: Firefly

Chapter 15: Using Firefly

Filska and Firefly are Kneusel's new contributions to the field, and they're both... interesting. Filska asks, "what is it like to program in a language where each subprogram can manipulate only a single memory location?", while Firefly holds that "A Firefly program is a series of singlecharacter instructions that either move the firefly around the grid, set the way its luminous trail acts, or make the firefly sing a note." While they sound somewhat frivolous on the surface, Chapters 13 and 15 show how to build some pretty interesting non-trivial things using each language, and each one definitely qualifies as "makes you think" fodder.

Chapter 16: Going Further

This is more of a "Miscellaneous" chapter, in which Kneusel lists a few more esolangs, and then provides some resources around programming languages, including a number of solid books (many of which I've also found useful over the years).

Thoughts

As you probably could guess, I love this book. That said, this book is probably going to be on my list of "Books you should read when you have a bunch of free time (which for most people is 'never')". It's not that the writing is poor; Kneusel is clearly no neophyte author, as his bio states: "He is the author of Practical Deep Learning: A PythonBased Introduction (No Starch Press, 2021), Math for Deep Learning: What You Need to Know to Understand Neural Networks (No Starch Press, 2021), Numbers and Computers (Springer, 2017), and Random Numbers and Computers (Springer, 2018)." It's much more due to the fact that this is just a space where too few programmers really choose to congregate.

Frankly, I love the book, and I think every programmer should read it. Candor compels me, however, to point out that what readers learn from this book is subtle, philosophical in nature, and hard to articulate to others until you've turned it over in your head a few times. I personally think it makes you a better developer, but you'll need to have at least a little interest in the subject of programming languages to start.

Disclaimer: No Starch Press sent me a review copy of the book; no other compensation was provided.

Book Review: Dead Simple Python

Wed, 11 Sep 2024 00:00:00 +0000

I bought a copy of the book "Dead Simple Python", by Jason C McDonald, and I have to say, this is my new go-to book for book recommendations on learning Python.

For starters, this is not a pocket-sized book; weighing in at just under 700 pages (including both front and back matter), this sucker is quite possibly a deadly weapon in the right hands. That, to me, is a reflection of the Python ecosystem: rich, deep, and easily the equivalent of any of the other mainstream ecosystems out there (Java, .NET, Javascript, and so on). To try and cover Python by only covering the syntax would be doing the reader a disservice every bit as much as it would be do teach Java or C# without going over the standard libraries (java.util, System.IO, etc). Considering that much of the Python experience is heavily influenced by Guido's "Batteries Included" approach to the language, covering more than just the language seems an excellent choice.

It also reflects the goal of the book, which McDonald highlights very early on in the Introduction in the section "Who Is This Book For?": "If you are already familiar with another programming language and now wish to learn Python for the first time without slogging through beginner-oriented courses, this book is for you." McDonald has done the thing I frequently find authors fail to do with their books: Establish the target persona, and ruthlessly avoid anything that doesn't meet that target persona's needs. The professional programmer, who already knows about inheritance, classes, and flow control, doesn't need examples showing the concept of an if statement, they just need to see the syntax and any semantics that might be different from what they're used to. (And Python does have a few of those--I've never seen a language that has allows an "else" clause attached to a "while" loop before. Or "for" loops. Wild.)

McDonald's goal is also not just to teach the cold hard facts of syntax, but also the "Python way", which the community has captured in the Zen of Python, an official document that captures the essence of how good citizens of the Python community should think about things. McDonald frequently references the Zen, and liberally quotes from other books (which is a practice that I think is highly admirable--too many books seem to pretend that there is no other book out there but their own), which I find helps the new-to-Python reader know where to go in the larger community to see and feel more about how Python approaches things.

But if you're from No Starch Press reading this review, one quibble: The back of the book proudly proclaims "I Lie Flat!" and that the book won't "snap shut"; frankly, that's not true. As I write this, I have the book open to the "Brief Contents" page, requiring me to carefully balance the book's back half against the laptop screen so that I can use two hands to type while being able to read the contents to replicate the list in the post below. It's probably not the fault of the binding--after all, 700 pages is pretty heavy to hold open by itself anywhere before page 250 or after page 450--but it does like to smile innocently back at me while I glare at the back of the book because it just whoomped closed while I was trying to read the code while typing it in. Maybe just drop the logo.

As I said before, DSP is a hefty tome, and covers a wide variety of topics:

Part I: The Python Environment

Chapter 1: The Python Philosophy

Nice little read of how Python thinks about things.

Chapter 2: Your Workbench

This is your classic "Getting Started" section, describing all the different ways Python and its associated tools can be installed and used.

Chapter 3: Syntax Crash Course

It doesn't cover all the permutations--each of the various subjects get deeper coverage later--but it does get you the basics of Python's scripting syntax, such as variables, looping and conditionals, how to use objects, that sort of thing. In theory, if you had to suddenly support a Python script and had never looked at the language, you could probably use this as your "I only have time to read one chapter" candidate.

Chapter 4: Project Structure and Imports

Chapter 4's location struck me as odd at first; I wasn't really expecting a lecture on how to arrange the files in my project this early in the book. After reading it, though, I can see why it's here--so much of Python is reliant on conventions, and Python in particular really stresses things "be done a certain way" to work best. Getting that into a reader's head as early as possible is probably a good idea here.

Part II: Essential Structures

Chapter 5: Variables and Types

Chapter 6: Functions and Lambdas

Chapter 7: Objects and Classes

Chapter 8: Errors and Exceptions

These four chapters are your "deep dive" into the semantics of each subject.

Part III: Data and Flow

Chapter 9: Collections and Iteration

Chapter 10: Generators and Comprehensions

Chapter 11: Text IO and Context Managers

Chapter 12: Binary and Serialization

Chapter 10 I kinda wanted to be inside Part II, since I think of these in the same vein as lambdas, but it's not unreasonable that they appear here, after collections and iterators have been covered. Still, it feels a little out of place compared to the rest in here. (I also suspect that there's a balancing act at work here--each Part has four chapters, with exception of Part IV, which has five.)

Part IV: Advanced Concepts

Chapter 13: Inheritance and Mixins

Chapter 14: Metaclasses and ARCs

Chapter 15: Introspection and Generics

Chapter 16: Asynchrony and Concurrency

Chapter 17: Threading and Parallelism

If Chapter 10 felt like a sore thumb, then Chapter 13's place here in "Advanced Concepts" is the clown suit worn to a funeral--I have never thought of inheritance as an "advanced" concept, and I don't know that any "formally-trained" (Java, C#, C++, Smalltalk, ...) object-oriented developer would either. Personally I'd have preferred this to be in Part II, but the rest... they're definitely "advanced".

Matter of fact, unless you're seriously prepared to drop some serious mental acid and open your brain, I'd recommend passing on Part IV until you've had some time playing with the language under your belt. Don't get me wrong, this is the good stuff (and highlights just how "well put together" the Python environment is), but understanding some of this is definitely only for those who can explain the difference between contravariance and covariance in generic parameters in Java (or C# or C++ or....).

Part V: Beyond the Code

Chapter 18: Packaging and Distribution

Chapter 19: Debugging and Logging

Chapter 20: Testing and Profiling

Chapter 21: The Parting of the Ways

These four bring us back to practical terra firma again, with clear and direct discussion of each of these programming-adjacent topics that are really quite crucial to success in programming. It's also important to me that I point out that this is the first book that describes how to use pdb, the Python debugger.

Let me be more transparent: this is the first book that I've ever seen even talk about the Python debugger--in fact, I didn't know Python had a debugger (at least not a console-mode debugger) until this book. Made me take a second and look at my Python installation (Homebrew, on macOS) to see if there was a launcher script, and, sure enough... nope! Nothing. So then I flip to the back of the book to that chapter, and discover that pdb is, like so much else in Python, a Python module that ships with the base installation. Ah! That would explain why I've not seen it before. Cost of the book justified, right there.

Thoughts

As is probably obvious, I'm a fan of the book, and I plan on keeping it on the shelf nearby when I reach for Python as a solution to whatever problem I'm facing. There's enough reference material in here that I can see myself going to the book to look something up to refresh my memory rather than Googling it, particularly for things that are more conceptual than syntactic. The coverage is extensive but skips the trivia, and its emphasis on quoting the community and the Zen of Python helps the would-be Pythonista come up to speed on the "why of how" quickly. If you're a programmer wanting to learn Python (and you like books over videos!), this is where I'm going to send you.

If you're a data scientist or AI/ML researcher, I think there's still a lot here for you, but you may find about 50% of the book's content unnecessary--it's definitely written for those of us who would use Python to build applications. Unfortunately, it's not a matter of telling you which chapters to ignore, as they're all useful (except maybe you don't need most or all of Part IV). It's more the depth and assumptions made about how you're going to use Python that comes through in the text--as an application developer, it's in tune with what I care about, but if you're a data scientist, you may want to pivot to a different book.

If there was one criticism, I kinda wish there was a chapter or two that had a bit more of how to do the various things we want to do in Python: access a database (relational or otherwise), stand up an HTTP endpoint, that sort of thing. Wouldn't have to be much, just even a list of popular packages with one or two code snippets for a few of them, highlighting and showcasing just how mammoth and diverse the Python ecosystem is. That said, it's a nit, and definitely not something that should keep this book out of any programmer's hands.

Well done.

While No Starch Press has sent me review copies of books before, this was not one of them; this one I bought in a bookstore after flipping through the pages.

Manager Antipatterns

Thu, 29 Aug 2024 00:00:00 +0000

Ever had a manager who was clearly smart, but had no idea how to manage? Or one of those managers who was "laterally promoted" over from a team that was radically different from what your team did? There's a whole host of mistakes that companies often fall prey to with respect to those they have leading teams, and I thought it a good idea to collect them into one place, under the umbrella heading of "manager antipatterns". (Because "antipatterns" sounds better than "often-repeated mistakes that we really should have learned from so that we don't make them again, and yet...".)

Within each of these, I try to provide a barebones pattern form, but most importantly, advice on how to rectify the situation, split into three different forks: when you work for that manager, when you have that manager reporting to you, and when you are that manager.

A sample:

"The Absentee Manager": "What manager?"
"The Bomb Manager": "We need to toss it all and start over."
"The C-Buddy Manager": "Why yes, the CxO is my friend, and they asked me to take over this team."
"The Funnel Manager": "Wait until you hear what my boss told me today!"
"The Hammer Manager": "I only know how to use a hammer, so I'm going to hammer away with it"
"The Headless Chicken Manager": "OMG OMG OMG I don't know what to do OMG OMG OMG"
"The Laissez-Faire Manager": "I'm going to be the opposite of a micromanager, and leave them totally alone"
"The Lateral-Move Manager": "Let's take a manager from one group and move them to run a team in a totally different area of the company"
"The Perfectionist Manager": "Hey, it's MY team, so the work isn't done until I think it's perfect."
"The Smartest-Engineer Manager": "Let's promote the smartest engineer on the team to manager"
"The Sphinx Manager": "Riddle me this...."
"The Tech-Lead Manager": "Let's just have a developer be a manager and a developer at the same time"

The full list is here.

A Java Language Cumulative Feature Rollup

Wed, 28 Aug 2024 00:00:00 +0000

I found myself asking myself the question, "What's every new feature Java has introduced since the last time I really cared about new Java language features?", and didn't find an easy answer via Google. So, I decided to create that list.

Note that most JDK releases are a combination of new language features, new library features (and by libraries I mean the modules that ship with the JDK), and sometimes new virtual machine features to support either of those two. In the list below, I do my best to categorize which features apply where. I'm also going to elide out anything that seems relatively trivial or is a Preview feature in the release.

213: Milling Project Coin (9)
- Allow effectively-final vaiables to be used as resources in try-with-resources
- Allow diamond with anonymous classes if inferrable
222: jshell (9) (VM): "Provide an interactive tool to evaluate declarations, statements, and expressions of the Java programming language, together with an API so that other applications can leverage this functionality."
238: Multi-Release JAR files (9) (VM): "Extend the JAR file format to allow multiple, Java-release-specific versions of class files to coexist in a single archive." In other words, an MRJAR can sport different versions of a class for different releases of Java.
259: Stack-Walking API (9) (Library/VM): "Define an efficient standard API for stack walking that allows easy filtering of, and lazy access to, the information in stack traces."
261: Module system (9): Hello darkness my old friend....
266: More Concurrency Updates (9) (Library): Some enhancements around publish-subscribe (Flow and friends), the CompletableFuture type and some other enhancements. Looks to be entirely library-centric.
286: Local-Variable Type Inference (10): Use var as a type placeholder in local variable declarations. (What they should've done back in Java7 when they went the other direction.)
181: Nest-Based Access Control (11): "Introduce nests, an access-control context that aligns with the existing notion of nested types in the Java programming language. Nests allow classes that are logically part of the same code entity, but which are compiled to distinct class files, to access each other's private members without the need for compilers to insert accessibility-broadening bridge methods." I'm guessing this now invalidates/supercedes the need for the synthesized methods I always used to use as an example for why it was useful to look at Java bytecode. sigh.... I mean, it's best to close the hole, but now I have to find a new demo.
361: Switch Expressions (14): "Extend switch so it can be used as either a statement or an expression, and so that both forms can use either traditional case ... : labels (with fall through) or new case ... -> labels (with no fall through), with a further new statement for yielding a value from a switch expression." Functional programming infiltrates the language once again.
371: Hidden Classes (15): "Introduce hidden classes, which are classes that cannot be used directly by the bytecode of other classes. Hidden classes are intended for use by frameworks that generate classes at run time and use them indirectly, via reflection. A hidden class may be defined as a member of an access control nest, and may be unloaded independently of other classes." Now that's interesting, never heard of that before....
378: Text Blocks (15): "A text block is a multi-line string literal that avoids the need for most escape sequences, automatically formats the string in a predictable way, and gives the developer control over the format when desired." Looks like it uses the triple-quote "heredoc" syntax popular in other languages.
394: Pattern Matching for instanceof (16):
395: Records (16):
306: Restore Always-Strict Floating-Point Semantics (17):
409: Sealed Classes (17):
431: Sequenced Collections (21) (Library):
440: Record Patterns (21):
441: Pattern Matching for switch (21):
444: Virtual Threads (21) (Library): Fibers come to Java. Finally.

The full list, as given by the OpenJDK pages, are below.

The LTS-marked releases are the Oracle Long-Term Support releases.

JDK 9

102: Process API Updates
110: HTTP 2 Client
143: Improve Contended Locking
158: Unified JVM Logging
165: Compiler Control
193: Variable Handles
197: Segmented Code Cache
199: Smart Java Compilation, Phase Two
200: The Modular JDK
201: Modular Source Code
211: Elide Deprecation Warnings on Import Statements
212: Resolve Lint and Doclint Warnings
213: Milling Project Coin
214: Remove GC Combinations Deprecated in JDK 8
215: Tiered Attribution for javac
216: Process Import Statements Correctly
217: Annotations Pipeline 2.0
219: Datagram Transport Layer Security (DTLS)
220: Modular Run-Time Images
221: Simplified Doclet API
222: jshell: The Java Shell (Read-Eval-Print Loop)
223: New Version-String Scheme
224: HTML5 Javadoc
225: Javadoc Search
226: UTF-8 Property Files
227: Unicode 7.0
228: Add More Diagnostic Commands
229: Create PKCS12 Keystores by Default
231: Remove Launch-Time JRE Version Selection
232: Improve Secure Application Performance
233: Generate Run-Time Compiler Tests Automatically
235: Test Class-File Attributes Generated by javac
236: Parser API for Nashorn
237: Linux/AArch64 Port
238: Multi-Release JAR Files
240: Remove the JVM TI hprof Agent
241: Remove the jhat Tool
243: Java-Level JVM Compiler Interface
244: TLS Application-Layer Protocol Negotiation Extension
245: Validate JVM Command-Line Flag Arguments
246: Leverage CPU Instructions for GHASH and RSA
247: Compile for Older Platform Versions
248: Make G1 the Default Garbage Collector
249: OCSP Stapling for TLS
250: Store Interned Strings in CDS Archives
251: Multi-Resolution Images
252: Use CLDR Locale Data by Default
253: Prepare JavaFX UI Controls & CSS APIs for Modularization
254: Compact Strings
255: Merge Selected Xerces 2.11.0 Updates into JAXP
256: BeanInfo Annotations
257: Update JavaFX/Media to Newer Version of GStreamer
258: HarfBuzz Font-Layout Engine
259: Stack-Walking API
260: Encapsulate Most Internal APIs
261: Module System
262: TIFF Image I/O
263: HiDPI Graphics on Windows and Linux
264: Platform Logging API and Service
265: Marlin Graphics Renderer
266: More Concurrency Updates
267: Unicode 8.0
268: XML Catalogs
269: Convenience Factory Methods for Collections
270: Reserved Stack Areas for Critical Sections
271: Unified GC Logging
272: Platform-Specific Desktop Features
273: DRBG-Based SecureRandom Implementations
274: Enhanced Method Handles
275: Modular Java Application Packaging
276: Dynamic Linking of Language-Defined Object Models
277: Enhanced Deprecation
278: Additional Tests for Humongous Objects in G1
279: Improve Test-Failure Troubleshooting
280: Indify String Concatenation
281: HotSpot C++ Unit-Test Framework
282: jlink: The Java Linker
283: Enable GTK 3 on Linux
284: New HotSpot Build System
285: Spin-Wait Hints
287: SHA-3 Hash Algorithms
288: Disable SHA-1 Certificates
289: Deprecate the Applet API
290: Filter Incoming Serialization Data
291: Deprecate the Concurrent Mark Sweep (CMS) Garbage Collector
292: Implement Selected ECMAScript 6 Features in Nashorn
294: Linux/s390x Port
295: Ahead-of-Time Compilation
297: Unified arm32/arm64 Port
298: Remove Demos and Samples
299: Reorganize Documentation

JDK 10

286: Local-Variable Type Inference
296: Consolidate the JDK Forest into a Single Repository
304: Garbage-Collector Interface
307: Parallel Full GC for G1
310: Application Class-Data Sharing
312: Thread-Local Handshakes
313: Remove the Native-Header Generation Tool (javah)
314: Additional Unicode Language-Tag Extensions
316: Heap Allocation on Alternative Memory Devices
317: Experimental Java-Based JIT Compiler
319: Root Certificates
322: Time-Based Release Versioning

JDK 11 LTS

181: Nest-Based Access Control
309: Dynamic Class-File Constants
315: Improve Aarch64 Intrinsics
318: Epsilon: A No-Op Garbage Collector
320: Remove the Java EE and CORBA Modules
321: HTTP Client (Standard)
323: Local-Variable Syntax for Lambda Parameters
324: Key Agreement with Curve25519 and Curve448
327: Unicode 10
328: Flight Recorder
329: ChaCha20 and Poly1305 Cryptographic Algorithms
330: Launch Single-File Source-Code Programs
331: Low-Overhead Heap Profiling
332: Transport Layer Security (TLS) 1.3
333: ZGC: A Scalable Low-Latency Garbage Collector (Experimental)
335: Deprecate the Nashorn JavaScript Engine
336: Deprecate the Pack200 Tools and API

JDK 12

189: Shenandoah: A Low-Pause-Time Garbage Collector (Experimental)
230: Microbenchmark Suite
325: Switch Expressions (Preview)
334: JVM Constants API
340: One AArch64 Port, Not Two
341: Default CDS Archives
344: Abortable Mixed Collections for G1
346: Promptly Return Unused Committed Memory from G1

JDK 13

350: Dynamic CDS Archives
351: ZGC: Uncommit Unused Memory
353: Reimplement the Legacy Socket API
354: Switch Expressions (Preview)
355: Text Blocks (Preview)

JDK 14

305: Pattern Matching for instanceof (Preview)
343: Packaging Tool (Incubator)
345: NUMA-Aware Memory Allocation for G1
349: JFR Event Streaming
352: Non-Volatile Mapped Byte Buffers
358: Helpful NullPointerExceptions
359: Records (Preview)
361: Switch Expressions (Standard)
362: Deprecate the Solaris and SPARC Ports
363: Remove the Concurrent Mark Sweep (CMS) Garbage Collector
364: ZGC on macOS
365: ZGC on Windows
366: Deprecate the ParallelScavenge + SerialOld GC Combination
367: Remove the Pack200 Tools and API
368: Text Blocks (Second Preview)
370: Foreign-Memory Access API (Incubator)

JDK 15

339: Edwards-Curve Digital Signature Algorithm (EdDSA)
360: Sealed Classes (Preview)
371: Hidden Classes
372: Remove the Nashorn JavaScript Engine
373: Reimplement the Legacy DatagramSocket API
374: Disable and Deprecate Biased Locking
375: Pattern Matching for instanceof (Second Preview)
377: ZGC: A Scalable Low-Latency Garbage Collector
378: Text Blocks
379: Shenandoah: A Low-Pause-Time Garbage Collector
381: Remove the Solaris and SPARC Ports
383: Foreign-Memory Access API (Second Incubator)
384: Records (Second Preview)
385: Deprecate RMI Activation for Removal

JDK 16

338: Vector API (Incubator)
347: Enable C++14 Language Features
357: Migrate from Mercurial to Git
369: Migrate to GitHub
376: ZGC: Concurrent Thread-Stack Processing
380: Unix-Domain Socket Channels
386: Alpine Linux Port
387: Elastic Metaspace
388: Windows/AArch64 Port
389: Foreign Linker API (Incubator)
390: Warnings for Value-Based Classes
392: Packaging Tool
393: Foreign-Memory Access API (Third Incubator)
394: Pattern Matching for instanceof
395: Records
396: Strongly Encapsulate JDK Internals by Default
397: Sealed Classes (Second Preview)

JDK 17 LTS

306: Restore Always-Strict Floating-Point Semantics
356: Enhanced Pseudo-Random Number Generators
382: New macOS Rendering Pipeline
391: macOS/AArch64 Port
398: Deprecate the Applet API for Removal
403: Strongly Encapsulate JDK Internals
406: Pattern Matching for switch (Preview)
407: Remove RMI Activation
409: Sealed Classes
410: Remove the Experimental AOT and JIT Compiler
411: Deprecate the Security Manager for Removal
412: Foreign Function & Memory API (Incubator)
414: Vector API (Second Incubator)
415: Context-Specific Deserialization Filters

JDK 18

400: UTF-8 by Default
408: Simple Web Server
413: Code Snippets in Java API Documentation
416: Reimplement Core Reflection with Method Handles
417: Vector API (Third Incubator)
418: Internet-Address Resolution SPI
419: Foreign Function & Memory API (Second Incubator)
420: Pattern Matching for switch (Second Preview)
421: Deprecate Finalization for Removal

JDK 19

405: Record Patterns (Preview)
422: Linux/RISC-V Port
424: Foreign Function & Memory API (Preview)
425: Virtual Threads (Preview)
426: Vector API (Fourth Incubator)
427: Pattern Matching for switch (Third Preview)
428: Structured Concurrency (Incubator)

JDK 20

429: Scoped Values (Incubator)
432: Record Patterns (Second Preview)
433: Pattern Matching for switch (Fourth Preview)
434: Foreign Function & Memory API (Second Preview)
436: Virtual Threads (Second Preview)
437: Structured Concurrency (Second Incubator)
438: Vector API (Fifth Incubator)

JDK 21

430: String Templates (Preview)
431: Sequenced Collections
439: Generational ZGC
440: Record Patterns
441: Pattern Matching for switch
442: Foreign Function & Memory API (Third Preview)
443: Unnamed Patterns and Variables (Preview)
444: Virtual Threads
445: Unnamed Classes and Instance Main Methods (Preview)
446: Scoped Values (Preview)
448: Vector API (Sixth Incubator)
449: Deprecate the Windows 32-bit x86 Port for Removal
451: Prepare to Disallow the Dynamic Loading of Agents
452: Key Encapsulation Mechanism API
453: Structured Concurrency (Preview)

JDK 22

423: Region Pinning for G1
447: Statements before super(...) (Preview)
454: Foreign Function & Memory API
456: Unnamed Variables & Patterns
457: Class-File API (Preview)
458: Launch Multi-File Source-Code Programs
459: String Templates (Second Preview)
460: Vector API (Seventh Incubator)
461: Stream Gatherers (Preview)
462: Structured Concurrency (Second Preview)
463: Implicitly Declared Classes and Instance Main Methods (Second Preview)
464: Scoped Values (Second Preview)

JDK 23

455: Primitive Types in Patterns, instanceof, and switch (Preview)
466: Class-File API (Second Preview)
467: Markdown Documentation Comments
469: Vector API (Eighth Incubator)
473: Stream Gatherers (Second Preview)
471: Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal
474: ZGC: Generational Mode by Default
476: Module Import Declarations (Preview)
477: Implicitly Declared Classes and Instance Main Methods (Third Preview)
480: Structured Concurrency (Third Preview)
481: Scoped Values (Third Preview)
482: Flexible Constructor Bodies (Second Preview)

JDK 24

404: Generational Shenandoah (Experimental)
450: Compact Object Headers (Experimental)
472: Prepare to Restrict the Use of JNI
475: Late Barrier Expansion for G1
478: Key Derivation Function API (Preview)
479: Remove the Windows 32-bit x86 Port
483: Ahead-of-Time Class Loading & Linking
484: Class-File API
485: Stream Gatherers
486: Permanently Disable the Security Manager
487: Scoped Values (Fourth Preview)
488: Primitive Types in Patterns, instanceof, and switch (Second Preview)
489: Vector API (Ninth Incubator)
490: ZGC: Remove the Non-Generational Mode
491: Synchronize Virtual Threads without Pinning
492: Flexible Constructor Bodies (Third Preview)
493: Linking Run-Time Images without JMODs
494: Module Import Declarations (Second Preview)
495: Simple Source Files and Instance Main Methods (Fourth Preview)
496: Quantum-Resistant Module-Lattice-Based Key Encapsulation Mechanism
497: Quantum-Resistant Module-Lattice-Based Digital Signature Algorithm
498: Warn upon Use of Memory-Access Methods in sun.misc.Unsafe
499: Structured Concurrency (Fourth Preview)
501: Deprecate the 32-bit x86 Port for Removal

JDK 25

470: PEM Encodings of Cryptographic Objects (Preview)
502: Stable Values (Preview)
503: Remove the 32-bit x86 Port
505: Structured Concurrency (Fifth Preview)
506: Scoped Values
507: Primitive Types in Patterns, instanceof, and switch (Third Preview)
508: Vector API (Tenth Incubator)
509: JFR CPU-Time Profiling (Experimental)
510: Key Derivation Function API
511: Module Import Declarations
512: Compact Source Files and Instance Main Methods
513: Flexible Constructor Bodies
514: Ahead-of-Time Command-Line Ergonomics
515: Ahead-of-Time Method Profiling
518: JFR Cooperative Sampling
519: Compact Object Headers
520: JFR Method Timing & Tracing
521: Generational Shenandoah

These are the "diffs" list given by the OpenJDK pages on each of the LTS release pages. In essence, these are rollups from one LTS release to the next.

JDK 11 -> 17

Additions

403: Strongly Encapsulate JDK Internals (17)
390: Warnings for Value-Based Classes (16)

HotSpot JVM

386: Alpine Linux Port (16)
391: macOS/AArch64 Port (17)
340: One AArch64 Port, Not Two (12)
388: Windows/AArch64 Port (16)

Flight Recorder

349: JFR Event Streaming (14)

Garbage Collectors

344: Abortable Mixed Collections for G1 (12)
345: NUMA-Aware Memory Allocation for G1 (14)
346: Promptly Return Unused Committed Memory from G1 (12)
379: Shenandoah: A Low-Pause-Time Garbage Collector (15)
377: ZGC: A Scalable Low-Latency Garbage Collector (15)
376: ZGC: Concurrent Thread-Stack Processing (16)

Run-Time System

341: Default CDS Archives (12)
350: Dynamic CDS Archives (13)
387: Elastic Metaspace (16)
358: Helpful NullPointerExceptions (14)

Language

394: Pattern Matching for instanceof (16)
395: Records (16)
306: Restore Always-Strict Floating-Point Semantics (17)
409: Sealed Classes (17)
361: Switch Expressions (14)
378: Text Blocks (15)

Libraries

2D Graphics

382: New macOS Rendering Pipeline (17)

Cryptography

339: Edwards-Curve Digital Signature Algorithm (EdDSA) (15)

I/O

352: Non-Volatile Mapped Byte Buffers (14)
380: Unix-Domain Socket Channels (16)

Networking

373: Reimplement the Legacy DatagramSocket API (15)
353: Reimplement the Legacy Socket API (13)

Reflection & Method Handles

371: Hidden Classes (15)
334: JVM Constants API (12)

Serialization

415: Context-Specific Deserialization Filters (17)

Utilities

356: Enhanced Pseudo-Random Number Generators (17)

Tools

392: Packaging Tool (16)

Preview & incubating

Language

406: Pattern Matching for switch (Preview) (17)

Libraries

412: Foreign Function & Memory API (Incubator) (17)
414: Vector API (Second Incubator) (17)

Deprecations

HotSpot JVM

374: Deprecate and Disable Biased Locking (15)
366: Deprecate the ParallelScavenge + SerialOld GC Combination (14)

Libraries

398: Deprecate the Applet API for Removal (17)
411: Deprecate the Security Manager for Removal (17)

Removals

HotSpot JVM

363: Remove the Concurrent Mark Sweep (CMS) Garbage Collector (14)
410: Remove the Experimental AOT and JIT Compiler (17)
381: Remove the Solaris and SPARC Ports (15)

Libraries

407: Remove RMI Activation (17)
372: Remove the Nashorn JavaScript Engine (15)

Tools

367: Remove the Pack200 Tools and API (14)

JDK 17 -> 21

Additions

HotSpot JVM

422: Linux/RISC-V Port (19)

Garbage Collectors

439: Generational ZGC (21)

Serviceability

451: Prepare to Disallow the Dynamic Loading of Agents (21)

Language

441: Pattern Matching for switch (21)
440: Record Patterns (21)

Libraries

444: Virtual Threads (21)

Collections

431: Sequenced Collections (21)

Cryptography

452: Key Encapsulation Mechanism API (21)

I/O

400: UTF-8 by Default (18)

Networking

418: Internet-Address Resolution SPI (18)
408: Simple Web Server (18)

Reflection & Method Handles

416: Reimplement Core Reflection with Method Handles (18)

Tools

JavaDoc

413: Code Snippets in Java API Documentation (18)

Preview & Incubating

Language

430: String Templates (Preview) (21)
445: Unnamed Classes and Instance Main Methods (Preview) (21)
443: Unnamed Patterns and Variables (Preview) (21)

Libraries

442: Foreign Function & Memory API (Third Preview) (21)
446: Scoped Values (Preview) (21)
453: Structured Concurrency (Preview) (21)
448: Vector API (Sixth Incubator) (21)

Deprecations

HotSpot JVM

449: Deprecate the Windows 32-bit x86 Port for Removal (21)

Libraries

421: Deprecate Finalization for Removal (18)

JDK 21 -> 25

Book Review: The Nature of Code

Tue, 20 Aug 2024 00:00:00 +0000

No Starch Press sent me a copy of the book "The Nature of Code", by Daniel Shiffman, and I have to say, it's a lovely piece of work, both figuratively and literally.

Physical

Let's start with the physical (which is rare for a book review, I think): This book is gorgeous. It's clear that No Starch is trying to make this book more than just something that appears on your shelf, but instead takes center stage on your coffee table. A number of "design" books went after this motif, and The Nature of Code (TNoC) definitely feels at home with them. Paper is heavy, covers are solid, and the binding is strong. Definitely a book that can take some abuse as you cart it from desk to restaurant to bed to... well, wherever else you might read stuff while your body is preoccupied with other things.

Secondly, Daniel Shiffman (whom I've never met) clearly knows how to teach. His bio reflects that (creator of The Coding Train channel on YouTube, and associate arts professor at NYU Tech School of the Arts being two of the big credentials there), but there's a fair number of people I know who have a teaching credit in their bio that still don't know how to teach. My mom put it best once: "Some people know how to teach, and some only know how to lecture." Shiffman clearly falls in the latter category: he starts with some core functionality, then builds on that to the next section, then builds on top of that, and so on. The aim is to carefully bring the reader along on a journey, never too far away from where you started (at the start of the chapter) but far enough to realize that yes, you did learn something here.

Next, let's address the elephant in the room: Shiffman makes the material available over a variety of media, including the Web. (In the front matter, he even addresses how his goal has always been to have one set of source materials get transformed--dare I say transpiled?--into a variety of different consumable media, like PDFs, HTML, or printed book. Sentiments quite close to my own heart!) Should you buy the book even though you could read it for free? My answer here is biased and twofold: One, it's always good to support authors in situations like this, so buy the book so Shiffman gets his tiny, tiny royalty. (Seriously, book royalties are not why you write a book.) Two, while I like having books on my Kindle app on my iPad(s), because they travel soooo much easier than carting around physical books, there is still something visceral about turning the page that no e-reader has been able to duplicate yet. I suspect mine may be the last generation (Gen-Xer over ehre) that feels this way, but if you like books over online resources to learn something, then you already knew the answer.

In terms of the actual contents, TNoC is a hefty tome, weighing in at around 600 pages, and that's not an inflated number due to multi-page code listings (like you might find in certain self-published books). Twelve chapters (starting at chapter 0, as God intended!), covering:

Part I: Inanimate Objects
1. Randomness: Using different flavors to drive an object's motion (like a soccer ball through the grass)
2. Vectors: A foundational tool for pretty much everything else
3. Forces: Vectors are forces that can act on objects
4. Oscillation: Angles, trigonometry, and angular acceleration
5. Particle Systems: Now make thousands and thousands of those objects (particles) and turn 'em loose
Part II: It's Alive!
1. Autonomous Agents
2. Physics Libraries
3. Cellular Automata
4. Fractals
Part III: Intelligence
1. Evolutionary Computing
2. Neural Networks
3. Neuroevolution

It's pretty clear this isn't a tutorial on how to learn programming, but makes for an excellent follow-up to your traditional programming class, in that Shiffman's goal is to show how to model natural/physical systems in code.

Speaking of programming, the examples make use of p5.js a spirital cousin (if not successor) to Processing, the Java-based environment that allows "creatives" to express their ideas and designs without having to learn Java3D, JavaFX, or any other 3D graphics library + programming language.

Thoughts

Overall, as you've probably guessed, I like the book. This area--graphics and physical motion simulation and so on--is not one of those areas that I consider myself "skilled" in, so in many ways I think I'm in a target audience for the book, and one thing that strikes me as a deep positive is that it never tries to "talk over" me. In too many 3D graphics books, there's a tendency to start spewing Greek symbols all over the page and teach the math behind the code, but doing so before ever showing code. Shiffman doesn't do this--he shows code, then if he wants to go into the math, he does so in simple prose. Granted, I'm not much smarter about the math after reading the chapter, but if I wanted to learn the math, I'd go pick up a math text. Half the time, I don't care how a thing works, I just want to know (and see!) that it does.

If graphics and simulating natural systems appeals to you (and frankly I think anybody who's ever thought about diving into building a video game has thought about this at least once!), then I encourage you to "grab a copy of a bound collection of processed cellulose fibers, imprinted with symbolic glyphs via pigment-based transfer particles direct from me!".

Disclaimer: No Starch Press sent me a review copy of the book; no other compensation was provided.

The R-and-D Library Team

Wed, 12 Jun 2024 00:00:00 +0000

There are four different kinds of R&D teams, each with very different actions and goals, and each with very different outcomes. The success of the team often depends on aligning the activities of the team with the intended goals, and it's actually quite reasonable for a company to have two or more teams, each operating independently and towards different ends. In this post, I explore the Library Team.

As a reminder, there are four kinds of R&D Teams:

the Scout team, exploring what tech others in the industry have built (e.g., Flutter, SwiftUI, etc)
the Spy team, exploring what competitors have built
the Library team, building lowest-common-denominator libraries used (sometimes) within the company
the Practical Research team, building novel solutions to the company's (and others') problems

The Library Team

The Library team is the team given responsibility for managing reusable artifacts within the company--libraries, tools, frameworks, services (cloud or on-prem), whatever--because achieving actual reusability, it turns out, requires a degree of commitment of time and resources to maintaining and homogenizing those artifacts. Doing this work, at the same time a team is trying to manage a product release, ends up becoming a distraction to the product release cycle, particularly as the shared "thing" becomes more widely shared.

The Problem: In any given company consisting of more than one development team operating in parallel, there comes a point when each team creates their own independent solution to a particular sliver of the domain or infrastructure. Once the duplication becomes known, concerns about the duplication became a topic of discussion, and the teams spend some (sometimes tiny, sometimes significant) time talking over the issue, in what sometimes turns into outright negotiations.

But as the company grows, more and more teams come into play, and before long, it's something of a free-for-all as teams try to figure out what's shared and what's sharable. In some organizations, being the team that's sharing the library becomes a point of pride--or worse, metrics--and thus creates incentive to convince everybody else to use our stuff, as well as disincentive to use anybody else's stuff.

THe problem multiplies, however, when the teams that use another team's library start to discover bugs or a lack of particular features necessary to the adopting team's requirements. Pressure begins on the owning team to incorporate these new features--which the owning team really has no need for--and the inevitable conflict escalates to higher management to hash out. Quickly one or all parties realize that there is no real winner here: If the owning team is required to add the features, their own schedules and delivery are affected; if the adopting team(s) are told to add the feature themselves, the codebase now reaches a point where no one team has a real sense of how the code operates anymore; if the adopting team(s) are told that the library is "as-is" and no support from the owning team is available, the adopting team often uses that as a pretext to "go their own way" and "build it the right way". In essence, everybody loses.

Successful Execution: For successful reuse to occur, a library team must be formed with the explicit goal of being an "internal vendor" of software to other teams within the company. They own, operate, maintain, and innovate around the software library or libraries that are intended for use across the organization, and the relationship is formalized much as a formal relationship is built between a vendor and their customer: explicit expectations are written and signed, a contract (of sorts) around support requests and/or updates is created, and the library team begins to interact with the adopting teams around prioritization of fixes, tweaks, and updates.

The Library Team is a flavor of R&D, although it may not seem like it at first, since they are more focused on providing goods and services (in the form of tools, frameworks/libraries, and support) than most people associate with an R&D team. However, if R&D is defined as an effort intended to create competitive advantages for the company's developers through technology, then this team certainly fits.

In contrast to the perception that an R&D team is always about creating something bright and shiny and new, the emphasis for the Library team has to be on the ongoing support of the things created. Just as no language, framework, or tool would ever be adopted by a company without understanding (and leaning on) some kind of support structure and/or community around it, efforts to create internal reusable products have to have a surrounding suport structure. Later, as time passes and the usage of the internal product grows, the library team will take on some of the internal community support as well, engaging with internal developers the same way an external vendor's Developer Relations team would engage with developers.

Thus, the Library team's execution falls into a variety of phases:

Discovery. The library team is searching for new opportunities for reuse, by meeting and interacting with all of the development teams around them. Each time a team goes through a security review, architectural review, or any other sort of review, the library team (or a representative of) should be present to understand the architecture, discuss what options for security the library team can provide, and/or what features or infrastructure the product or service team is using or adopting. Each time a team releases something, the library team (or a representative of) should be present to understand what went into the release. In general, the library team should be on a constant hunt for things that could or should be reused across the organization.
Research. Part of this discovery process can come from looking at what sorts of things external vendors are offering, as well. If a mobile or UI control component pack offers some visual components that currently are missing (and requested), the library team can do the "buy vs build" feasibility analysis. If a new cloud service is available, the library team can examine it with the same eye.
Innovation. The library team is not entirely dependent on "outsiders" to come up with things to do, however, as there's likely plenty of ideas or feature requests for the existing shared components, so work can continue on developing those components.
Maintenance. Naturally, as bug reports come in, the library team will be responsible for triage and prioritization. The library team will also need to determine its release schedule, and coordinate with its downstream dependents to make sure they're not breaking anyone with their changes. (For this reason, any module under the ownership of the library team should follow a strict semantic versioning scheme, and use existing infrastructure like NuGet, npm, Maven repos, and/or the Swift Packaging system to make their modules available internally).
Reporting. Lastly, the library team should be gathering up statistics on the usage of each of the modules/components they manage, and actively look for "dead modules" that can be removed from their watch and deleted.

The R-and-D Scout Team

Wed, 12 Jun 2024 00:00:00 +0000

As a reminder, there are four kinds of R&D Teams:

the Scout team, exploring what tech others in the industry have built (e.g., Flutter, SwiftUI, etc)
the Spy team, exploring what competitors have built
the Library team, building lowest-common-denominator libraries used (sometimes) within the company
the Practical Research team, building novel solutions to the company's (and others') problems

The Scout Team

The Scout Team is what most corporate R&D teams go after: The hunt for new technology that will somehow provide the company with a competitive advantage over its competitors, despite the fact that the competitors are all themselves out there scouting the industry just like this team is. New programming languages, new databases, new architecture styles, new whatever-fill-in-the-blank, just so long as it is (a) new and (b) not yet mainstream, it's a candidate for the Scout Team's infinite desire to create prototypes that bear a remarkable resemblance to the demos that the new thing has on its website or in its documentation.

The Problem: The Scout Team suffers from several intrinsic flaws:

The Scout Team isn't always a great evaluator of technology. Honestly, it often comes down to the team lead's personal preferences and/or opinions. Do we scout Dart/Flutter, or React Native? Do we scout MongoDB, or CouchDB? Do we scout Pulumi, or Terraform? Some teams will try to do an apples-to-apples comparison of two relatively-similar technologies, but usually this effort just leads to the realization that "the differences are marginal" and, unfortunately, not particularly compelling. Which then begs the never-asked question, What would happen if we just threw a dart against the dartboard and picked one at random and tried building our prototype in that?
Any gained advantage is temporary. Great, you've identified that you can build your mobile apps 5% faster using React Native. You must now engage in a deep coverup of epic counterespionage, because if your competitor gets wind of the fact that you're using this technology (and they will, if they watch your website's Careers page, which some competitors will do), they have every opportunity to follow in your footsteps and begin using it as well, which effectively negates your advantage and renders all the work the Scout Team effectively moot. Given the cost of the Scout Team against the time where you actually had the advantage, the ROI on the Scout Team's efforts generally isn't there, particularly when you could play "second mover" and watch your competitors' Careers page, instead.

Successful Execution: If the Scout Team is to be successful, it must first begin with either a business problem to solve or a hypothesis it seeks to prove and disprove (yes, both). "We believe that we can produce customer-acceptable mobile applications in half the time if we use a cross-platform toolkit to build hte mobile app" is a reasonable hypothesis, because it leads to the natural question, "What does 'customer-acceptable' mean?" as well as "Well, how long does it take us to build a mobile app in both iOS and Android today?". Technology-centric folks (myself included) will feel more comfortable with formulating a hypothesis than a business problem, so either the team has to run its hypotheses past product owners or business analysts or even customers themselves before adopting one for scouting.

Then, once the problem or the hypothesis has been adopted, the team should move into a series of phases:

Discovery (1 to 5 days): What's out there? This shouldn't take longer than a week, maximum. For some topics, it might even be just a day. Google to your hearts' content, and capture the results in some kind of table to be able to do the best (highly-ignorant, because you really are just getting started here) comparison. Keep in mind, you're not at a decision point yet, you're literally just comparing the options. No code is written yet.
Discussion (2 hours): Once the discovery is finished, do a quick conversation (in person or online) about the options. All the attendees are given the table, encouraged to bring their own opinions, and as soon as the meeting starts, take a vote on the top 3 to explore. (I prefer "$100-voting" schemes, where you get to "spend" a fictitious amount of money/votes on up to three options, and the results are tallied. It's flexible voting, and allows those with really strong opinions to make the strength of that opinion known.) What stand out as the top three choices, given the problem or hypothesis? Limit yourself to three.
One will probably emerge as a majority favorite, another will be a lukewarm choice, and two others will be argued about extensively. Put a hard limit on this meeting, and if the team can't decide which one to explore as the third option, make the call to only use two--that'll usually coalesce agreement around a third (surprisingly quickly). If it doesn't, then clearly neither one has much momentum behind it within your team, so just do two.
Exploration (1 week): Now, depending on the size of the team, comes exploration, during which the thing is downloaded, installed, a few demos are built/explored, and initial conclusions are drawn. In order to avoid strong opinions coloring the analysis, however, make sure that whichever team member voted the strongest in favor of the technology is not given the technology to explore. That is to say, if Ted spent $50 on Flutter, Woody spent $25, and Scott spent $10, then Scott should be the one to explore Flutter. The reasoning here is simple--if Ted voted so strongly for Flutter, he runs a strong risk of being biased in favor of it, and will be less likely to spot the negative traits. Scott, however, feels less strong, and so will likely be more open-minded. (Yes, this system can be gamed if somebody really wants to try and game it, but honestly, if you can't trust your R&D team to be able to approximate some degree of unbiased investigation, then you probably have bigger problems.)
Keep in mind here, you're not building your prototype yet. You're literally just trying to get a sense for the "unboxing experience"--can we run demos right away? Are other dependencies involved? Do we have the right tools or infrastructure already, or would we need to pull down more in order to get even the simplest example up and running? This goes a long way to hint at what the overall developer experience will be.
Prototype Decision (2 hours): Once the team has had a little experience with each of the three options, it's time to decide which one to build the prototype in. The previous exploration step is essentially a preview for what comes next, but the team can't realistically build anything of substance if they're trying to do it across a large number of options, so reducing it down to one helps keep things focused. Meet with the team, with each of the three options presented, examples discussed, and make a decision out of the three.
Prototype (1-3 months): Now the time is right to build out the prototype, always with an eye towards the hypothesis or the problem. Don't just build a CRUD app--any tool can do that. Build something that has the difficulty characteristics that your business deals with. Got a mainframe you need to talk to? Figure out how that would work as part of the prototype. Complex workflow? Really convoluted price-calculation logic? All that has to be a part of the prototype, or it's not a successful prototype.
Presentation (1 week): Having spent the time building out the prototype, now the team needs to build out the presentation: Specifically, the questions answered, the issues addressed, the results, and so on. But keep in mind that at least half of this presentation would need to be forward-facing--that is to say, it needs to talk about what adoption would look like and how it would play out within the company. Which teams would adopt first? How would they go about adopting? What's the timeframe? What do the steps look like, going from zero to "adopted"? Without practical execution steps, your prototype is just a target, not a plan.
Execution (variable): At this point, having established a plan, the time has come to start integrating the new thing into the overall fabric of the team/organization/company. The time and effort required to do this will be variable, depending on the work required and how deeply it will cut through the rest of the organization, but this is the point to which the entire effort has been building. Integration, training, all of these things form the core of the execution of the adoption of the new technology, and now having established the start of the momentum to adopt it, it's time to carry it through. The full effect of this effort will likely require much more than the efforts of the Scout team, though--this is now a "digital transformation" effort, which means it's likely something that will need to be coordinated at an executive level.

The R-and-D Spy Team

Wed, 12 Jun 2024 00:00:00 +0000

As a reminder, there are four kinds of R&D Teams:

the Scout team, exploring what tech others in the industry have built (e.g., Flutter, SwiftUI, etc)
the Spy team, exploring what competitors have built
the Library team, building lowest-common-denominator libraries used (sometimes) within the company
the Practical Research team, building novel solutions to the company's (and others') problems

The Spy Team

The spy team's purpose is pretty much as its name implies: Take a strong look at what competitors and adjacents are doing in your industry, and how they do (or might) affect your company's activities. The spy team doesn't get industry press articles written about the things they are doing, but they do often adopt a technology quickly enough to make a difference without paying the cost of finding all the mistakes and wrong assumptions about it.

The Problem: Technology is often seen as a competitive advantage by a number of companies, but an equally larger number of companies don't want to be the "first mover" in a technology space because of the costs involved in discovering how to apply something to their day-to-day operations. When the Web first became a "thing", for example, many companies ran to the Web with an idea of bringing a corporate "home page" (which was the principal use case for the Web browser at first) to the rest of the world. MySpace was the first in that regard, and built quite the comprehensive set of individuals' home pages as a result. But companies pretty quickly discovered that the real power of the Internet-connected web browser wasn't in showing off one's questionable design skills, but in the brand-new world of being able to sell things to people. MySpace was the first-mover, but very quickly discovered they had no real business model to lean on, while Amazon went on to become one of the biggest tech companies on the planet. Lesson learned: Don't be the first-mover. Wait for the first-movers to figure out everything wrong with a technology, then leverage that knowledge and experience (which is free to you now) and be the second-mover, but the vastly more efficient adopter, of a technology.

Successful Execution: The spy team's execution model is often cyclical, in which the team chooses a "target" for their activities, and engages in some "creative reconnaissance" to determine what that company is doing. This doesn't mean decking out tuxedo-clad actors in all the latest spy gadgets or engaging in questionably-legal corporate espionage. In many cases, it's as simple as looking at the company's careers page (to determine what they're hiring by looking at the "skills desired" section of each job listing), or "view source" on the company's various web properties, or even looking on LinkedIN at current employees and discovering what their posts tell you about what they're doing or using at work.

Then, once something new has been discovered, the spy team kicks into high gear, looking at the technology and the company's public statements and conversations around that technology. Competitor just adopted React? Cool--see if you can identify which team inside the company is using React, what they're using it for, and how the adoption seems to be going. Are any of the company's engineers or technical executives speaking at conferences? See if you can catch the video recording of the talk, and use that as a guide for what they might be using it for, and what "minefields" they ran into when they were using it. Simultaneously, of course, begin your own investigation into React itself, but always with an eye back to what the competitor is doing and saying, so that you can avoid the mistakes they've made.

Once you have a good sense of the technology and how it might (or might not) apply to your own company's problems, you shift into Scout Team mode: prepare a presentation, define a prototype, and work out an execution plan for transitioning to using the new technology within your own company's fabric.

The R-and-D Research Team

Wed, 12 Jun 2024 00:00:00 +0000

As a reminder, there are four kinds of R&D Teams:

the Scout team, exploring what tech others in the industry have built (e.g., Flutter, SwiftUI, etc)
the Spy team, exploring what competitors have built
the Library team, building lowest-common-denominator libraries used (sometimes) within the company
the Practical Research team, building novel solutions to the company's (and others') problems

The (Practical) Research Team

The Research team (also more recently called "the innovation team" or the "disruption team" in more "hip" companies) is the one most people think about with respect to an R&D team. It's important to note, however, that the team cannot be just focused on research--like the Scout team, the Research team has to have a constant eye on what they build having a practical impact on the company that sponsors them. For that reason, I often make the conscious decision to put the word "practical" in front of the name, so that the emphasis is on practical research that will have a direct effect on the company's ability to execute.

The Problem: It has been proven, over and over again, that companies that don't spend time thinking about how to leverage technology to make their business run more effectively or more interconnectedly or more... well, pick your favorite adverb here, it probably applies... will find that they are being surpassed by competitors who have. This statement, however, is exceedingly vague, and is often vague on purpose--nobody exactly knows what the "next big thing" will be, so they don't know ahead of time what the right adverb is to use!

For example, there was a time when simply shipping something on the company's website was sufficient. Then it needed to be up there more quickly than the competitors. Then it needed to be faster and more interactive. Then the push was to get more users on the platform. And so on and so on. With each "internet generation" a new adverb became the watchword of the industry, and massive amounts of time and energy were spent chasing that adverb. (Case in point: For a while, the chief goal was "scale"/"scalability", and many companies went bankrupt trying to build a platform that could handle the "inevitable" challenges of having a million simultaneous users on the platform--even before they had their first user.)

If a Research team is trying to build things that will help the company go to "where the puck will be, instead of where it is now", as the famous hockey player is famous for saying, then the Research team is one that is firmly in the position of having to make fantastic guesses on a regular basis, and--worse--be right all the time or risk being accused of wasting time and money. (Picture this exchange, if you will: "What do you mean you went out and built tools for integrating with a blockchain?" "But boss, back in 2020, you said..." "Blockchain is dead! (I read it in Forbes!) You should be chasing after AI!" Now imagine what the exchange was like in 2016, 2012, 2008, ....)

The humbling truth for most businesses is that whatever the current industry hype is, it's probably not the thing that will most "move the needle" for the company. And, worse, you can almost be certain that your competitor's CEO is currently yelling at their staff about the exact same thing--in 2024, everybody is chasing AI, such that the memes ("Hey, we have a pitch meeting in five minutes; do me a favor, take our pitch deck and replace all the uses of 'blockchain' with 'AI', would you?") are omnipresent on the social media feeds. And yet, even as we know that everybody is chasing it, the biological imperative inside of our lizard brains keeps whispering, "Yeah, but what if they're RIGHT this time? We could be hopelessly behind!" (all the while trying desperately to make you forget that it whispered the same thing about blockchain five years ago).

In many respects, the problem with a Practical Research R&D team isn't the "research" part of it--it's the "practical" part of it.

Successful Execution: Executing a Practical Research team takes a lot more work than the other three kinds, because the practical research team has to do more in the way of research. (There's a certain logic to that statement, no?)

For starters, part of the research has to be about what will "move the needle" for the business as a whole. This is where some degree of familiarity with the domain (or having a subject matter expert on the team, even part-time) will help immensely with identifying what those areas might be. But keep in mind, part of the discussion can be "shifted left" in the pipeline--if there's no immediately obvious way to move the needle for one group in the company, what about doing so for a group that appears earlier in the pipeline?

Consider Facebook, a social media company. Most of the company's work is in building the web platform on which Facebook provides its "feed" to each of its users. But "shifting left" takes us to some of the infrastructure behind that platform: building the web components, integration with incoming data sources, and (for a while there, anyway) making it easy for people to build apps (like games) on top of the platform or connected deeply with the platform. What does "shifting left" here look like? Well, if someone makes it easier to build a webapp, with particular semantics for how data is managed by the UI code, you get... React.

Did React completely solve all of Facebook's problems? Probably not. (We can debate the issue for days or weeks if we want, but that would actually be missing the point.) More importantly, a team inside of Facebook asked themselves, "What is it something that we and/or our partners or customers do that we can improve?" And then they iterated. Over and over again, and eventually it made its way outside the company to be something that others found useful, and next thing you know, it's pervasive in the front-end industry.

Which brings up an important point about all of this: Progress isn't made in big-bang "Eureka" moments. (Neither is scientific progress, for that matter, but that's a different discussion.) Progress is made in incremental steps with lots of iteration. The shorter those steps, the more iterations you get, the faster you learn about the thing you're iterating on. (This has actually been discussed in certain circles as the "OODA Loop", as coined by John Boyd fifty years ago.)

Consider this story from the book Art & Fear:

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality.

His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the "quantity" group: fifty pound of pots rated an "A", forty pounds a "B", and so on. Those being graded on "quality", however, needed to produce only one pot – albeit a perfect one – to get an "A".

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the "quantity" group was busily churning out piles of work – and learning from their mistakes – the "quality" group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

As a practial research team, you need to put out lots of work, not just one "eureka" work.

Are you looking for examples of practical R&D? Consider some of the ideas being put forth by Microsoft Research, or Oracle Labs. (For the record, MSR produced F#, and Oracle Labs is the group behind GraalVM.) IBM used to have a similar section, alphaWorks, which unfortunately got shut down a number of years ago due to IBM's inability to actually be successful. Meta/Facebook has one, too. Does your company need to fund a multi-million (possibly -billion) dollar research group? Of course not. That's what companies with billions of dollars in their budget can afford to do. But a company of several hundreds in size can put together a single team of 5 +/- 2 developers to iterate on ideas to make the company's technology a better return on investment.

Which brings up the last point: A practical research team needs to be extremely visible to the rest of the company in terms of the work they are doing and the research they are carrying out. Because a team of this nature is usually a "net negative" in terms of the company's financials (they draw a lot of money in and rarely generate any money back), socialiing the work and demonstrating the current iteration is crucial to demonstrating the value of the team. In addition, it's important for people to get their hands on what the research team is doing--putting alphas and betas out into the world (or at least into the world inside the company, if there's no interest in open-sourcing the efforts) is important both to the visibility of the team as well as a means of obtaining feedback around the ideas.

The Four Kinds of Research-and-Development Teams

Sat, 1 Jun 2024 00:00:00 +0000

Many developers get really excited when they join an R&D team, because it signals in their minds that they're among an elite group looking to move the needle for a company. We imagine the business suddenly soaring in revenue and profit because of the whatever-thing we just built. We can hear--in the mind's ear--the kudos and praise raining down from the C-Suite as the whatever-thing we just built is rolled out company-wide and maybe, just maybe, industry-wide. We're excited!

Wait, what was it we were trying to do again?

In my last post, I talked about some of the problems with R&D teams and how they're run. What I didn't really get into in that post, however, was the idea that (I believe) there are four different kinds of R&D teams, each with very different actions and goals, and each with very different outcomes. The success of the team often depends on aligning the activities of the team with the intended goals, and it's actually quite reasonable for a company to have two or more teams, each operating independently and towards different ends.

In brief, there are four kinds of R&D Teams:

the Scout team, exploring what tech others in the industry have built (e.g., Flutter, SwiftUI, etc)
the Spy team, exploring what competitors have built
the Library team, building lowest-common-denominator libraries used (sometimes) within the company
the Practical Research team, building novel solutions to the company's (and others') problems

In each of those subsequent posts, I address each of these in some level of detail, but here's summaries for each:

The Scout Team

The Library Team

The Spy Team

The Practical Research Team

The Problem With Most R-and-D Teams

Tue, 28 May 2024 00:00:00 +0000

For many software developers, career success comes when they are placed on the "R&D" (research and development) team. It seems like a golden ticket kind of role: Spend time exploring new technology (which lots of developers enjoy doing), weighing in, offering insights, and just... it's like, all play, and no deadlines! And yet, strangely (or perhaps not), those teams never seem to last long--they go through one, maybe two iterations on something, and then the team is broken up and nothing replaces it for a few years until a new VP comes along and says, "Wait, who's tracking our future direction?" and forms one again.... Only to have it dismember again in a year or two. What gives?

Typically, the "failures" of these teams have nothing to do with the team's lead or its members; in fact, it's often counterintuitive that they would fail, since they're often made up of some of the most senior people in the company. In fact, in many cases, participation on this team is itself a reward for success elsewhere in the firm!

Why Build One of These?

We begin our analysis with the classic philosopher's question: "Why?" Or in this case, "Why does this team exist? What's the purpose of the team, and what are they asked to do?"

And, as with many of these philosophical questions, almost right away we run into an interesting problem: Nobody knows.

Most companies know, intuitively, that technology can be a competitive advantage, and almost every company that is over twenty years old has found themselves in a situation where they slept on a technology and suddenly found themselves on the wrong side of history when a competitor suddenly roars past them. Never mind the fact that the technology is only the surface symptom of why the competitor surpassed them, clearly the fault is that "We aren't studying the future enough! We aren't protecting our future! We need a team to do research and development!"

Armed with this "realization", a VP or CTO acts as the sponsor/champion for the founding of an R&D team, a team leader is selected for the role, and the team is told... pretty much nothing solid. "Go evaluate the industry! Find us that competitive advantage that we need! Go 'future-proof' our tech investment!" and other such empty statements are offered up as the vision or goal of the team, and they are turned loose.

NOTE: Keep in mind that "R&D" here could easily be replaced with "innovation"--as in, the "R&D Team" could easily be called the "Innovation Team" and staffed with not just engineers but also product and UX folks, but the goals are still equally vague. It simply highlights that most companies don't know how to Innovate any more than they know how to Research & Development.

The CTO might check in once or twice a quarter, particularly around review time, but for the most part the team is left alone, to their own devices, to char their own way. "These are the best and the brightest in the company," any doubters are told, "And they need time and space in which to work." If the doubters persist, they're told stories about teams at big companies that turned out the most amazing things by being left alone--Smalltalk and the DynaBook at Xerox PARC, or the iPhone at Apple, or any of a dozen other stories (only some of which actually have any bearing in fact, but that never stops the stories from being told). The R&D team itself will often get into the mix, too. "We need to behave like a startup," the R&D team tells anyone who comes knocking. "That means we have to go out there and Innovate. R&D. Discover. Disrupt, even!"

After a while, though, enough questions are asked that the CTO or VP is finally ready/willing/able/persuaded to take a deeper look at what the team is doing. "Can you present some of your findings to the Board/Company/Department sometime next month? We're all eager to hear what you've come up with." And the R&D team, just as eager to show off the fruit of their labor, agree to put together a presentation.

The Presentation

THe appointed hour comes around, and the R&D team proudly stands up to present what they've spent the last six, nine, twelve, sometimes even eighteen months working on. And it's....

"We're really excited to show you how Flutter can be used to completely change the way we build software. We've combined it with Firebase to produce this amazing prototype demonstrating...."

It's one presentation after another about existing technologies out in the world, just outside the mainstream. Elixir. Flutter. Kotlin. Two years ago it would've been "Cloud Native". Five years ago it would've been "Serverless". Today it's probably something in the AI space, such as demonstrating how to build a chatbot or use a vector database.

The CTO is excited. Wants to be excited. Tells everyone else they need to be excited by this. But the rest of the Board or the rest of the company are demonstrably less so. This doesn't really showcase how the company will catch up on their competitor, particularly because the competitor doesn't seem to be using that....? If the CTO or VP are in a strong position, they might double down on the R&D team, but in time, before long, somebody points out that this team is "the most expensive team in the company" and pressure builds to break the team up and put all that brainpower and skill to use on other teams that are sorely lacking in such. Before long, the R&D team is gone, the former team's members are either bitter or sad, and sometimes they start sending out resumes shortly thereafter. The CTO or VP move on at some point, which--if the team isn't broken up by then--almost guarantees the replacement is going to take a look at this huge "investment" and its return, which is pretty close to nil, and break them up at that point.

And all of the findings are essentially lost to the company, but helpfully captured on each members' resume.

Analysis

Many readers are already jumping out of their seats, screaming, "Of course the team failed, they weren't researching the right thing!" But as other readers will point out, that's not the real problem. The real problems are that:

The team was never given any goals. What problem are they trying to solve? "Innovate" isn't a goal, it's a vague word that could mean anything to anybody at any time. Worse, the term's shared definition and context can easily change over time. Sure, it's entirely possible the team is versed enough in the company's business domain to know intrinsically what the problems are--just as it's entirely possible that the CEO, who doesn't actually program for a living, knows programming enough to know what language and stack the company should use. (Without the regular feedback that comes from doing the job, any "knowledge" about a job should be treated as suspect and untrusted until proven otherwise.)
The team had no business problem guiding it. Technology, as any halfway-trained product owner will be quick to point out, has to solve an actual business problem in order to have any value to the business. What's the business problem that needs to be solved? What does solving that problem look like? The best R&D teams are, in fact, run like a startup, complete with pitch: "This team is going to (blank). Our customer (people inside the company) faces pain in the form of (blank), and we aim to solve for this pain by creating a solution that (blank)." The R&D team has to have a driving goal, a purpose, or it risks wandering around in the technology jungle for a while before finally being declared "missing in action" and rescued (disbanded).
The team has no checkpoints or accountability points. Even in those rare cases where the team has goal, it frequently lacks any sort of checkpoint or accountability. "We're innovating! You can't put a schedule on that!" True, but you can put a periodic checkpoint to see how the team is faring against the goal, and/or force the team to present their findings on a much more regular basis. It's a simple agile problem--without feedback the team cannot know how well they are actually attacking the problem. In a startup, you get those checkpoints via "seed rounds" and "Board meetings" and "revenue targets"; in a larger firm, you'll have to create them. The R&D team needs to show progress on a regular basis towards the goal that was set, or it will be assumed to be lost, regardless of its actual progress or status.
The team had no plans beyond the research. For most of these presentations, the R&D team creates a prototype, a wonderful example of using the thing(s) they've researched, and they show it off during the presentation. "It's amazing, it's wonderful, it cuts our development time by (some wild-eyed number)!" Which is all great, but did anybody on the team stop to think about how this would/could get rolled out within the company? Like, if the research is on Elixir, what's the plan if the CTO says, "Great! We're in!"? As the natural experts within the company, the R&D team is now going to be tasked with figuring out the rollout, starting with which projects will get updated, which teams get training, and which projects will see the perceived benefits of the new thing right away. What are reasonable timeframes? What are the likely obstacles groups will hit? Is the intent to freeze all development across the company and do a rewrite of everything in Elixir? How reasonable is that? How possible is that? It's not enough to just say, "Look there!"; the R&D team also needs to have some kind of plan around how this would actually work.
The team's prototype doesn't actually address the hard problems. The prototype usually bears a slight resemblance to what the company does, but usually it's an idealized version of the problem, with none of the ugly parts. "It's just a prototype," they'll say when challenged on that point. Unfortunately, these prototypes, which often are built to be basic CRUD apps over a data store of one form or another, show off how well the new thing does the easy parts. Folks, any technology can make the easy parts look easy--any prototype that's actually gong to be useufl needs to show how well it handles the Hard Parts. This was one of the major pain points among the Ruby-on-Rails and NodeJS crowds back in the early stages of each of those technologies--yes, they made it absurdly easy to build a CRUD app, but then again, so did everything else. Show me the RoR app that streamlined how I talk to a mainframe, or how the NodeJS app can talk to our legacy COM/DCOM or CORBA objects. And they didn't--or couldn't--and as a result, it took a very long time for either of those two to break into "the enterprise", but were overwhelmingly adopted by startups, who lacked those brownfield obstacles. (Then, of course, when the startups were bought, they had to learn how to do those integrations, but by that point the die was already cast.) If you build a prototype, it has to demonstrate how to handle the hard problems, not the easy ones.

Even then, however, the R&D team can still get lost if they aren't run properly or in the right direction. But starting by addressing some of these points will help at least get the team to a 50/50 shot of succeeding at something.

I have much deeper thoughts about the different kinds of R&D teams and how to think about creating a group that focuses on innovation (to the scale of something like what Facebook created with React or what Netflix did with Chaos Monkey), but that's going to have to wait for a new post.

An Engineering Manager Challenge

Mon, 20 May 2024 00:00:00 +0000

My LinkedIn feed recently brought me a question posted about an interview question the original poster faced:

"You're the tech lead and your team is getting stretched thin. You decide to add resources (sic; not my choice of words here) but you can afford 1 senior full-stack developer or 2 junior full-stack devs. Which do you choose and why?"

It's a decent engineering manager question. So....

tl;dr I think the issue is an ROI calculation, based on short-term relief vs long-term relief. It's an immediate-cost vs long-term ROI analysis, in other words, but measured in the currency of time/effort, rather than money. (The capital expenditure here is way smaller than the operational cost, when you start adding up your teams' time.) The longer the window, the more the juniors make more sense.

Now, let's dive deeper into why I think this.

Context

Let's make a few assumptions here, that (1) the juniors we'd hire are going to come with basic skills in our tech stack (i.e., they know how to write React, Java, and SQL, but have no clue about our codebase), and (2) the senior would not only have the basic skills in the stack, but be of sufficient skill in our stack that she could debug into the stack itself if need be. That may be an unrealistic assumption, both in terms of what we (collectively) mean by "junior" and "senior", but clarifying what "senior" and "junior" means here is going to make a huge difference in the outcome. In particular, we'll want to spend a moment thinking very consciously about the senior--if our tech stack is React/Java/MongoDB, and we find a senior who is most comfortable in Angular/C#/MSSQLServer, would that individual still qualify as a "senior" in this discussion? For my money, the answer to that is going to be very much dependent on what our stressors are, and where we could see new hires jumping in.

Along similar lines, let's also assume that our tech stack is fairly mainstream. If we're using something that's too bleeding edge, the junior/senior split isn't going to matter as much, because each will still likely need to spend time coming up to speed with our stack. If we're using Elm, Jolie, and ArangoDB, both juniors and seniors are going to spend half their time looking up how to do common things in the unfamiliar parts of the stack. (And God help us all if there's Perl in here somewhere.)

Lastly, I think it's important to assume that the rest of the team is going to welcome the newcomer(s). If the team is stressed, that's not always a given, particularly since the newcomer(s) will (a) disrupt the existing team dynamics, and (b) likely have lots of questions and/or suggestions that were asked or discussed a while ago. If the team has no patience, bringing anybody in is going to be a bit rough, on both the existing team and the newcomer(s). If the team sees the newcomer(s) as a welcome relief, they'll be much more likely to go whatever extra mile necessary to make them feel welcome.

EDIT (22 May 2024): A few comments from my LinkedIn post to this blog entry pointed out there were a few things more that should've been here:

"There is no such thing as a full-stack developer." While I agree with the statement, it's often important to take the context as-is and work within it, particularly when (as was the case for the original poster) the question comes in an interview. Moreover, as much as I disagree with the use of the term, it is a standard, and it does lead to the next important assumption....

"... I would hire a frontend and a seperate backend developer." That's actually a really good point--it often makes more sense to have some degrees of specialization within the team, and once we examine the full-stack assumption on its own merits, we open up the idea that maybe it's actually better-and-easier to hire people into particular roles. This is a really good point, which still falls outside the framework of the question, but definitely something to consider in a real-world scenario.

"As an Engineering Manager, you can negotiate the scope and schedule...." Another great point, assuming you actually have that flexibility. I would amend the statement to read "As an EM, you should have the ability to negotiate the scope and schdule...", but unfortunately there are some projects where scope and/or schedule are constrained in ways that aren't easy to amend. (If an app needs to be ready before the Super Bowl, it's not like we can change the date of the Super Bowl!) All that said, if you have the flexibility, that's another lever to slide, because if you can slide the scope or the schedule, you can potentially relieve some of the pressure that leads the team to feeling stretched. That also said, headcount is also a pretty precious commodity, so if you have it, there's a strong drive to exercise it and hire, because if you solve the problem without hiring, there's a near-1.0 chance that it'll be taken from you and given to another team. It's not necessarily fair, but it is a truth of even small companies.

"I'd also consider the makeup of the team. If you have 3 juniors.... If you have 3 seniors...." I actually can't believe I missed this in my analysis, and yet, I can; I typically look to hire "diamond" or "pyramid" teams, that are either balanced senior-midlevel-junior (sort of a 1:2:1 ratio) or junior-heavy (1:2:3 ratio), but that's no excuse in forgetting to question the assumption.

"How big is the team? What is its present composition, output, turnover, etc?" This is more of the previous, and all are good points to consider.

"If the team is feeling stretched thin, it's because they have too much work to do, and the planning isn't getting done correctly already." That's a pretty good point, and also highlights another assumption that may not be legitimate to assume, in that there's actual deadlines here. It's entirely possible/reasonable that the team is simply stretched thin over a constant workload (think a support-ticket bug-fixing maintenance team, for example), and there aren't immediate deadlines that are in jeopardy.

Most of these assumptions don't, in large part, change the analysis of my answer, although the front-end/back-end distinction could very well weigh in significantly. Lesson learned, and thanks for the comments!

Assumptions set, let's analyze.

Analysis

Juniors are initially a drag on the project velocity as the others on the team help bring them up to speed. Over time (months), however, they can end up significantly increasing team delivery velocity as they begin to jell with the rest of the team and start pulling their own weight. This can be eased some by making sure their initial assignments are appropriate to developers with less experience (both in general and with the codebase). Best approach I've seen is to have them take on bugs rather than building features while they're coming up to speed.

However, keep in mind that the senior is going to need onboarding time, too! No developer, not even the mythical "10X Developer", is going to be able to jump into a brand-new codebase and immediately add value without requiring other team members' help/assistance. Usually the senior can come up to speed faster (weeks instead of months), but the onboarding cost is never zero.

We also need to consider the auspices under which we're bringing them in--specifically, is this a contract basis, or are we making them full-time offers? Contract vs full-time is an important distinction here, too. If this is just a short-term contract, don't bother with the juniors for the same ROI-based reason. However, I think a contract approach here is wildly inappropriate, because all you're doing is kicking the "stretched thin" feeling down the road for a time, and eventually you'll be right back to square one, and all the time invested in the onboarding will have been a waste. So either hire them full-time, or don't bother.

Also, factor in the interviewing and hiring time. Interviewing and hiring seniors takes much longer than hiring a pair of juniors--this holds true regardless of the market. Seniors are also more likely to have multiple interviews running simultaneously, and thus a higher chance of rejecting an offer or ghosting partway through. Plus, juniors are generally more intrinsically motivated to demonstrate they were a good bet, so they'll be more likely to take on the challenge more enthusiastically. (In fact, the hard part might be getting them to actually ask questions, because they want to prove they don't need the handholding, when it would be far more efficient if they just ask.) As for determining their skillsets, juniors will take a little more time to interview, since they'll more likely be unwilling to admit what they don't know, whereas most seniors are (generally) comfortable with elucidating their less-comfortable spots. Juniors also will potentially have blind spots outside the tech, and may take longer to adjust to the culture; but while seniors may be more accustomed to adjusting to new cultures, may also be more intransigent and stubborn if they decide they don't like the culture. That can play out in the team interpersonal dynamics in a big way.

Remote will have a play in here, too, and generally it's not a great story: You're probably going to want people who will be willing to "come into the office" frequently, in order to get direct exposure to others on the team in the most high-signal way possible. If the whole team is remote, then you're going to need to figure out how the newcomer(s) are going to have easy access to other team members in order to ask the questions they're inevitably going to have. This means you're probably going to need them to "buddy up" with existing team members, either by virtually pairing together on code, or else by having them share a Teams/Zoom call that's muted for much of the day but still live. (Slack or Discord just may not be quite at the high-interactivity and low-barrier-to-question level that an open channel or an in-presence "swiveling chair" situation would provide.)

In short

If the timeframe is less than a month, don't hire anyone--you won't get the ROI. If the timeframe is less than 6 months, hire the senior--you won't get the ROI from the juniors in that short a timeframe. Otherwise, hire the juniors.

Once you've hired, pair the newcomer(s) with a rotation of the existing team members for the first several weeks (seniors) or months (juniors) or until the newcomer(s) are able to complete work without requiring more than one question per unit work (story, bug, etc). I think the superior approach is to have them fix bugs in the code to start (the first two or three weeks or months), then have them add functionality to existing features, and only after that do you have them create new features.

But as with all answers of this type, context will influence and matter greatly on the outcome. Caveat emptor.

Software Architecture, In Practice

Mon, 26 Feb 2024 00:00:00 +0000

One of the more curious things I've found, both in my time as a consultant and as a software management executive, is the striking differences between what the title "architect" means at different companies. Strangely (or perhaps, not so strangely), the nature of the job changes significantly depending on the size of the company and how "interwoven" the various software development teams are against one another. The larger, more interwoven the company, the more your job as an architect is "referee or facilitator between teams" than it is "decider of technical strategy and direction".

Tell me if this story has ever happened to you: You are set up as an architect on a project. Cool, you think. I'm going to make some solid decisions, make sure that they're well-reasoned, approachable, nothing too overkill, all the things that make an architecture a good one. You spend some time analyzing the problem domain, you build out a few proofs of concept and a prototype or two, and your team is off and running. Knowing that your project doesn't exist in isolation, of course, you make room for some interoperability capabilities--let's say through an API. You make the firm decision that any access to your system has to come through the API.

One day, you get a call. Another team is working on a project that runs in parallel to yours, and they want to talk about how you've built some of your data-access code. Sure, you think, That makes sense. They don't want to reinvent the wheel if they don't have to, and I'm here for it--that's good architecting! You meet, and that other team comes away with some ideas of how to best connect to a database like yours--and the internal Git repo URL for your project, so they can see how to use it.

Another team, meanwhile, has also heard of your project's work, and wants to discuss whatever-it-is you're using for your API layer. More importantly, they want to understand how you're doing some of the validation logic. Sure, you think, They're trying to avoid reinventing the wheel, too. You meet, and once again they walk away with the Git repo URL for your project, because they wanted to see how to use that whatever-it-is.

But you're starting to get a weird feeling in your stomach that all these teams are too alike. You mention it to your boss, who nods thoughtfully and asks, "It's almost as if all these teams are building the same thing over and over again, no?" Somewhat uncertain, you nod, because, well, yeah, it kinda does. "Thanks for bringing this to my attention, I'll check with a few others and see if they get the same feeling. Good work." Feeling good that you may have done the company some good, you go back to your team and hunker down.

For a sprint or two.

Then, one day, you get called into an Engineering All-Hands. The Senior Engineering VP opens by saying, "We have great news. Thanks to some proactive investigation," and she gestures at you, "We have identified a major source of waste in our organization, that of duplicating efforts across many teams, and we are forming a group to combat it. We will be pulling all the architects together into an architect group, where we can share and spread architectural knowledge across the teams and create opportunities to aggressively share code." On the one hand, cool, but on the other hand, you're a little worried because your project--ahem your former project--is somewhat like those other projects, but it's kinda not. But, hey, you get a cool new title--"enterprise architect"--and it'll give you a chance to rub shoulders with other architects, so... yay?

After a few weeks' of reorg madness, things settle down, and you start having meetings with your former project team and the other project teams that you'd met with before, around the database and the API. Under the SVP's new mandate, the teams are breaking out these into separate microservices, and you're now in the uncomfortable position of trying to help your former team modularize code that was never intended to be modularized, for the use by teams that have problems that are similar--but definitely not identical--to theirs, with all the weird corporate battles around ownership and time allocation that come along for the ride: Who is responsible for bug fixes against the shared module/microservice? Who owns the decision-making around what can and cannot go into the microservice API? Each of the various teams look to you to champion their cause over the others, because after all, you were the one that helped them back when all this got started.....

When a company gets to be of a particular size, the different project teams often find themselves needing to collaborate in various ways, but (it seems) often don't feel comfortable doing so at just the "programmer" level--it's almost as if we don't trust programmers, even senior ones, to make decisions for their team. That's the Team Lead's job!

But the Team Lead is too busy being the Lead--meetings with the boss, with the team as a whole and also as individuals, and responding to fires that pop up every so often--to really sit down with the other teams to sort out technical issues, so...

Then one day, an executive realizes, "Hey, we have an Architect, and they're (supposed to be) the smartest technical person in the room, so let's have them sort these integration issues out! Even better if they're actually not attached to any one team but are instead part of an "architect team" or part of the "Enterprise Architecture" org. Then, when we disagree, we're not disagreeing with our teammates, we're disagreeing with somebody outside of the team who clearly doesn't understand the team's issues as well as we do." Granted, they may not think of it in precisely thsoe terms--particularly that last bit--but the thought still pops up. (Even better, some organizations make the Team Lead be the Architect, or vice versa, which means it's really a no-brainer for that executive.)

At this point, however, the role has transitioned away from the "solver of technical problems" and moved squarely into the "referee of team dynamics" and "arbiter of integration". Once in this position, most enterprise architects find themselves in a position that is more organizational design than it is software design.

And then, once that realization hits, the unhappy enterprise architect has two ways out: get promoted to become full-time management (embracing the organizational design side of the story) or get out of enterprise architecture entirely, either by transitioning to an IC role writing code again, or to a smaller organization wherein they can be "just" an architect again.

For a little while, anyway.

The root of the problem? In the previous section it's actually easy to spot: "trying to help your former team modularize code that was never intended to be modularized". Remember, microservices were an organizational design solution to a particular problem at Amazon: that of removing all dependencies. According to Working Backwards, Jeff Bezos spent a lot of time thinking about this.

Speed, or more accurately velocity, which measures both speed and direction, matters in business. With all other things being equal, the organization that moves faster will innovate more, simply because it will be able to conduct a higher number of experiments per unit of time. Yet many companies find themselves struggling against their own bureaucratic drag, which appears in the form of layer upon layer of permission, ownership, and accountability, all working against fast, decisive forward progress.
The answer lies in an Amazon innovation called "single-threaded leadership," in which a single person, unencumbered by competing responsibilities, owns a single major initiative and heads up a separable, largely autonomous team to deliver its goals.

In essence, the whole "microservices" approach popularized by Amazon was, in fact, not an exercise in software design, but in organizational design--by creating teams that were focused on a single objective, with an explicit agenda to eliminate dependencies that they might have. The elimination of dependencies, it turns out, is precisely the thing that holds up most teams, which (as any architect knows) is also the same principles around how to get the most throughput from a scalable system. "The enemy of scalability is contention." The more often a process has to content for a shared resource, the worse the performance curve gets.

For a team to truly be the master of its own destiny, it needs to be focused on one thing ("single-threaded"), with zero dependencies on external resources (databases, message queues, etc). It admits, up front, that there will be some violation of the Don't Repeat Yourself principle, but like all of the practices listed in the Pragmatic Programmer, they were intended as heuristics, not ironclad algorithms.

All of this, though, is really tangential to my main point: When you take an architect position, you really need to understand, up front, what role your position is really taking. Are you tied to a single-threaded organization? Then you are probably a software/system architect, focused on achieving a specific and concrete result. If you're matrixed across several teams, though, you're much more likely to become an organizational architect, focused more on helping teams navigate their cross-dependencies with one another.

Of course, there's always the third option, in which you are matrixed but have the authority to dictate to those teams the technical decisions you feel best suits their combined problem set. But, in all honesty, that position, I think, is a corporate legend: Everybody's heard of it, everybody yearns for it, but nobody's ever seen one in the wild except at an extreme distance, much less been in one. I used to think we called those roles "CTO", but having spent some time in and among CTO roles, I'm less convinced than ever that it actually exists.

Why Candidates Don't Trust Recruiting

Mon, 12 Feb 2024 00:00:00 +0000

If you've spent any time on LinkedIn at all the past few years, you've seen the incessant posts about people "looking for their next challenge". Many of these are followed, sometimes weeks or months later, by additional posts (sometimes by the same people) about being really really really desperate to find something. Sandwiched right in between them are posts by LinkedIn "influencers" trying to tell us the right sequence of events, the "magic sauce", to get that next gig. Then, every so often, there's a recruiter in the feed that bemoans the behavior of other recruiters, says, "Do better", and disappears. Yet, nothing ever gets better.

To all the recruiters in the world: The truth is, while some of you are certainly predatory and untrustworthy, I firmly believe the majority of you aren't trying to take advantage of anybody. That's the good news. The bad news is, we still don't trust you. And the main reason we don't trust you is because we don't trust your process.

Don't get me wrong, there's plenty of bad--or at least thoughtless--actors out there in the world. I recently interviewed with a CEO for a CTO position in which we would be working quite closely together. Post-interview, a week later, no response. She sent out an email right after the interview with the full job description, but has since offered no insight as to whether the search is still going, whether there will be another interview... nothing.

I don't think she's being deliberately cruel (far from it, she seems like a good person), but it's certainly sent something of a message. Even if she comes back next week with a "Great, we'd love to schedule another round..." I don't know that I'd be all that interested to continue. Sure, she's busy--but so is everybody.

Relationships, professional or personal, take work. This is a position wherein two people are going to be working closely together, and if you don't have the time and energy to lay down a good foundation for a good working relationship, then how can I be sure you're going to put the time and energy into a good working relationship once we're on the job together?

What's worse, I don't even know if we're done or if we've even started--I have zero insight into what the interview process is. I don't know if we had a good conversation, a bad one, a "meh" one that didn't really take me out of the running but left me out of the top consideration.... nothing.

I have ZERO insight into what's happening there. And in the absence of information, people tend to assume the worst, and then make judgments and decisions based on their assumptions. Then recruiters wonder why nobody's really warming up to them, even though half (or three-quarters) of the time, the recruiter is actually on the candidate's side!

FAANG

Big companies are much, MUCH worse. While they have a better chance of describing the overall process to you (in some few cases, they'll actually have it posted either as part of the job posting or as part of the application process), the process itself tends to lend itself to opacity--multiple rounds, each with multiple interviewers, often each asking a spate of questions that they themselves aren't 100% familiar or comfortable with.

Depending on the position, the nature of the interview will be different. Some of these interview rounds will be "system design" interviews, in which the candidate is presented with an open-ended question, such as one I recently went through, wherein you are challenged to "Please design an Identity and Access Management (IAM) system." You get maybe a paragraph's worth of specification. The interviewer (or interviewers) are often there to see... what, exactly?

Or perhaps you are doing a "culture fit" round, where the interviewers are asking you about the company's cultural values/positions/statements. They ask questions about your thoughts on... what, exactly? Whether you agree with the company's values? How you would align to them? The minefields are so deep here.

Or, if the position is coding-related in some way, you might be asked to write some code. Of course, these are often entirely "context-free", so as to not have to worry about the business domain. This means--typically--a question specifically around algorithms. How do you insert a node into the middle of a linked list? How do you do a breadth-first search of a binary tree? How do you optimize a search of a HashMap-of-String-to-String? All of these being problems that pretty much nobody actually writes code for in 2024.

For each of these, the "right answer" can take one of several forms: the interviewers are looking to "see how I think" by watching how the design or code or cultural thought evolves in response to insights gathered and the questions I ask along the way; the interviewers are looking to "see how good an architect/coder/person I am" by keeping their thoughts to themselves until we have a finished product; or the interviewers are looking to "see how collaborative I am" by watching not what's produced, but the assumptions I make along the way (instead of asking the interviewers--the customer/client proxy stand-ins--questions).

Best part is, the interviewers often have their own opinions about what the "right answer" is for these (something something something Kubernetes, something something something comPLETE), and even if you present a viable alternative, if it's not aligned at least a little with their preconceived notion of what the "right answer" looks like, you fail.

And even more better, nobody will ever say ahead of time what the round is looking for. You're left, as a candidate, to try and figure it out on the fly. Guess wrong, and you don't pass. Don't pass the round, you don't pass the interview.

And you will never, ever, EVER know why.

It's not the recruiter, it's the process

Look, recruiters, you can be as charming, heartwarming, and convivial as any human can possibly be, and all of it isn't going to matter in the long run.

The problem is that the nicest recruiter in the world doesn't address the primary frustration that candidates experience: A complete lack of visibility into the recruiting process, both at a high level (what the process is like) or at a more tactical level (what's expected and/or being examined in each round).

Look, put it like this: Imagine somebody sits you down in a room, and tells you that the test you're about to take is going to determine your (and your family's) financial future for the next half-dozen years or so. They then walk out of the room, without telling you what the test is on, what a passing grade looks like, or the subject you're about to be tested on. Or whether the test will be written, oral, or athletic. Or even how many tests there will be. And when it's over (whenever it is, in fact, over, because remember, they can always call you in a few weeks and tell you "it's time for round two", or round three, or round twelve, or....) you won't actually be told your results unless you're hired/passed.

Repeat this a few times, and it really won't matter what the personality of the person bringing you into the room--the process itself will break even the most cheerful, optimistic, skilled, and confident candidate.

Cheating With Chat-GPT

Thu, 1 Feb 2024 00:00:00 +0000

A colleague pointed out a semi-scientific study about interviewees cheating with ChatGPT hosted by interview.io; the results are not particularly great if you're relying on LeetCode-style problems: "Predictably, the [interviewee group using common LeetCode problems without modification] performed the best, passing 73% of their interviews. Interviewees reported that they got the perfect solution from ChatGPT. ... In our experiment, interviewers were not aware that the interviewees were being asked to cheat. As you recall, after each interview, we had interviewers complete a survey in which they had to describe how confident they were in their assessments of candidates Interviewer confidence in the correctness of their assessments was high, with 72% saying they were confident in their hiring decision."

Whoops! Maybe it's time to unit-test your interview process?

These are not reassuring numbers. When almost three-quarters of the interviews in which the interviewees were told explicitly (by the experimenters) to not rely on their own skills, but exclusively on ChatGPT are considered "confident", you really have to wonder: How good is your interview process? How many false positives are you allowing through? How many false negatives?

This sort of scientific testing (although, to be fair, this really needs to be done many more times, with different interviewees and interviewers, for the results to be considered scientifically conclusive) of interview processes really needs to happen a lot more. And while I'll step aside and let more-qualified psychological-testing masters tell us how to create tests of this nature industry-wide, I think there's value in considering the more localized test--that is, testing the recruiting process within your own company.

But let's forget the "cheating with ChatGPT" for just a moment--because we hadn't really solved interviewing even before ChatGPT got into the mix. Most companies, their interview processes are broken--hilariously so.

Lots of technology companies not only don't believe their interviewing/hiring process is broken, they think it's magnificient. As a matter of fact, I've been told point-blank (as a candidate, no less!) that "Our interview process is one of the top 10% across the industry." (Soooo many questions went through my mind when they said that, like, how did you measure that? Who else participated? What was the metric used here? And did you measure this yourself, did you use a neutral third-party, and did you ask the candidates their opinions? On that last point, by the way, the answer was, not surprisingly, "no".)

The basis of any test--unit- or otherwise--is the relatively simple concept that we want to validate the algorithm by putting known inputs into one end, and comparing/seeing the known/expected outputs at the other end. With code, we can do this pretty easily, since (most) code is deterministic: put a character string into the parameter of a "toLower()" function, and we expect that the result will be a character string with all the upper-cased ASCII characters to be converted to lower-case. Put the same input in, you'll always get the same result.

With humans, though, this is trickier. Humans are most certainly non-deterministic, so the same inputs can yield vastly different outputs. (Insert Dr Ian Malcolm's explanation of Chaos Theory from Jurassic Park here: "The conditions... never repeat and vastly affect the outcome".) Imagine I come to work every morning and greet the security guard at the door with the same message. "Morning, Joe." Some mornings he may nod back. Some mornings he may smile. Some mornings he may take that as an invitation to a longer conversation. Why is Joe so wildly all over the map in his responses? Because some mornings, he had a great morning: His alarm went off on time, he woke up refreshed, his spouse made him a lovely breakfast, whatever. Some mornings, Joe has a lousy morning: his alarm failed to go off, he didn't sleep a wink, his spouse is sick, whatever. Some of it even has to do with the number of times we've done this particular exchange--in the beginning, he may have felt I was just being polite, but then he may have noticed that I said something similar to someone else, which started a longer conversation about kids, weather, or sportsball, and Joe may be deciding that I use this as an open invitation to a deeper conversation. So many different conditions deeply affect Joe's response, and I neither control, nor have any visibility, into any of them.

Now take this to your interview process: If you're like most companies, you have a team of people you draw from to conduct interviews. There's variation #1, because two different people will test and interact with candidates differently. Many companies mix-and-match the teams of people doing the interview (often just grabbing people at random when the interview time comes up), which creates yet another significant variation, as the team dynamics can change how the team comes up with an answer. The team often comes up with questions on the fly, or worse, grabs standard LeetCode problems for the candidate to solve, interacting with the candidate the entire time (all the while pushing, prodding, rewarding, or punishing with their gestures, facial expressions, words, or outright statements)

We can normalize some of this--having the same interview questions, the same process, even creating a rubric for the answers--but no matter the effort, you'll never know if you're successful unless you test it.

Which brings me to the point: Unless you test your interview process, you'll have no idea of how good, or bad, it really is.

Testing Interviews

Before I go any further in this essay, I want to make two points clear: One, I have never had the opportunity to do any of this--the orgs I've been a part of, where I wanted to do it, explicitly shot down the idea for one reason or another; and Two, I absolutely am going to keep looking for opportunities to do this, because I firmly believe that it is a necessary prerequisite to being able to feel confident about an interview process.

Hypothesis: That a deterministic result is obtained when a candidate is put into our hiring/interview process. (To be fair, a perfectly deterministic result is never going to be possible with humans--there's really just too many variables in play. So let's settle for a "reasonably" deterministic result, at least until we get some experience under our belts, hmm?)

Experiment: We will take a candidate, and ask them to perform the same steps of our interview process as we would any other candidate.

Caveats:

We do not wish to change the process; changing the process would change the algorithm, and yield an untrustworthy result.
We do not with to change the interviewers; these are also part of the process. However, if the interviewers know that this is a test, they will very likely behave differently than they would in a "live" setting. Therefore, the interviewers must be kept unaware that a test is under way.

In order to be able to "know the outcome" of the experiment, we need to work with "known" inputs ahead of time--that is to say, if we put a candidate into the system who is a highly-venerated, highly-respected engineer, our interview process should identify them as a highly-senior engineer (even if we don't pick up on all the accolades or successes); similarly, if we put a random individual "off the street" with no relevant skills whatsoever through our process, we should be able to identify them as such and yield a "no-hire" response.

Readers familiar with my writing will probably already be thinking what I think is the key to this process: Work with known-quantity candidates, a la, friends and colleagues of the hiring team that are known to be solid performers, for the "yes-hire" known inputs. For the known "no-hire" known inputs? Hell, use paid actors. But to be certain, give the "known-yes-hire" candidates full transparency into the process and what questions might be asked, and give the "known-no-hire" candidates explicit instructions on how to answer any questions in the negative.

Doing this, either the candidates have to be unknown to the interviewers (which unfortunately means you can't really use other engineers from your company, unless your company is extremely large), or you have to hide their personal and physical characteristics from the interview team (which you probably should do anyway, but that's another discussion for another day). The ideal scenario would be use people who have already been hired by your company, so as to minimize the ambiguity of the "outsider-as-a-yes-or-no" (some people would insist that even somebody like Vint Cerf is not qualified to work at this company in the network team), so tweaking your process to allow for that kind of deception is preferred.

By the way, if you do this several times a month (say perhaps one in every five "live" canddiates, maybe one in ten), you can slip a "known-no-hire-using-ChatGPT" into the cycle and see how well your process responds.

Critics will be adamant that this is an absolute waste of time. I recall when people said the same thing about unit tests, and before that, QA. If it's important to you, it's important to measure it, examine it, and critique it.

But hey, if you want to run your interviews with a close-to-0% confidence that you're getting the right results.... you do you.

We Need to Talk

Mon, 15 Jan 2024 00:00:00 +0000

For many years, people have ranted and railed against the dread managerial "Do you have a second to chat?" opening that has no context around it. Just Google "Can we talk" and you get something in excess of 4 billion responses, many/most of which talk about how to respond to those openings, coupled with a stern shaking finger for those who use them. But what should we use instead?

Imagine, for a moment, you're a manager. Your job, more than anything, is to keep a steady and close pulse on the team as a whole. You need to talk to them, listen to them, respond to them, and connect with them at both an intellectual and emotional level. In many respects, you are the connection they have to the rest of the company (particularly in remote-work situations), and much of what they know about the rest of the company will flow through you. Communication is essential.

But, truthfully, there's another part of the job that we often ignore in conversations like these, and that's that managers are often privy to information that they are not permitted to share with anyone, or at least not until some particular deadline has been passed or some kind of criteria has been met. Confidentiality, thy name is "manager"; you must never discuss personal information about one of your team members to anyone else, even if you know (or, more often, think you know) that the individual doesn't consider it personal or is comfortable with you discussing it with others. That's theirs to tell, not yours. There is so much you cannot discuss until it's "done": upcoming promotions, demotions, hiring, firing, reorganization, performance, it's a whole list of things that must--by both law and company policy--be kept close to your chest until "the right moment".

(We will also leave alone for the moment that "the right moment" is often decided long ahead of time, and by people who aren't you or even in the room with you. Complain as much as you wish, this is not likely to change. Ever.)

All of this puts a natural tension on the free and open communication that any reasonable and/or compassionate manager wants to have.

Let's assume for the moment that you have news to share with one of your team. For the sake of argument, the actual news itself doesn't matter--while it would be tempting to say, "Oh, it's terrible news about layoffs" or "It's that dreaded separation/termination conversation", the opposite is often true, too, for purposes of this blog post. Either way, as your manager, I have a piece of news that I need to share with you.

Now, keep in mind a few key considerations:

I really need to have the conversation in person. Because of the sensitive nature of the news (again, regardless of whether it's a promotion or a termination), I need to be looking at you in the face when I deliver it, for a variety of reasons. Sometimes, we have to make do with a Zoom call because we're not in the same timezone, but then it's all that much more imperative to do this "face-to-face". Doing this over an email or a chat window is just classless and incompassionate at best, legally-liable at worst.
This isn't just normal day-to-day news. Often it has a time component to it, and I need to get the news to you as quickly as possible. I need to do this because...
The rumor mill will be flying fast and furious. Again, the kind of news doesn't matter--somehow people know they're being promoted even when their hiring manager hasn't heard the news yet--and so it's imperative that we get together to discuss this as quickly as possible, before rumor catches up to you and gives you a false report that I would then have to spend as much time deconstructing and denying as I do giving you the real news and explaining the consequences of that.

Assuming I've been diligent about my communications with you, I have a couple of options available with which to deliver this news:

Our weekly 1:1. This is often a great time/place to do it, because it's just the two of us, and anything we say here can often be done in confidence. That said, however, if the news is team-wide, I am now trusting that every single person on the team will be trust-worthy and refuse to share the news with anyone else before I've had the chance to discuss it, and that everybody on the team will be disciplined about their communication and not accidentally leak the information to somebody else on the team, who will silently resent being "the last to know" (which could very well be the case, if they're the last 1:1 in the week).
A weekly team huddle. The weekly team huddle is often the best place to deliver team-wide news all at once, but only if it affects the entire team--individual layoffs, individual performance actions (PIPs, etc) are entirely inappropriate to bring up at the team huddle, as would be any sort of salary adjustment (whether up or down), as it's only going to invite comparisons. You're going to end up dealing with some of that in any event (people on the team will, probably, talk to one another about raises and such, regardless of what you'd prefer), so you certainly want to be prepared to handle those discussions, but having those conversations out in a public form (the team huddle) is pretty much the worst place possible to have them.

Beyond those, when do you have regularly-scheduled meetings with each of your directs? Most manager's don't, and so now we get in to the world of the dreaded unscheduled chat.

Look, let's be face it: Any time you schedule something with your direct that isn't part of a regular cadence, it's going to raise some eyebrows. And as soon as their eyebrows go up, so will their anxiety. It's a natural human response: The brain is basically hard-wired to assume the worst of a situation. (Want to really watch this in action? Send your spouse or significant other a text message with just "Can we talk?" and then don't send anything after that. Or worse, start to type a follow-up, then erase it without sending, then start to type a follow-up, then erase it without sending, and by about the fifth or sixth iteration of this, you'll have effectively pushed them into a psychotic break.)

If you really want to try and ease your direct's anxiety, you can try to provide some context. "Hey, can we chat? I need to talk to you about your promotion." That certainly provides some context to the conversation, but it also runs the risk of misinterpretation on their end. "I need to talk to you about your promotion" can be interpreted by the confident as "I need to tell you that you got your promotion!" which, when it turns out that they didn't get it, feels like an utter rejection and low blow. "I need to talk to you about your promotion" to the anxious can easily be "I need to tell you that you blew it and didn't get the promotion", but they're also just as likely to interpret it as "I need to tell you that you blew it and not only did you fail to get the promotion, the CEO is furious at your impertience and wants you fired", ridiculous as that may sound. (Anxiety and unwarranted assumption really knows no boundaries.)

The other danger of providing some context about a conversation is that unless you can do so without giving away part of the conversation, you run the risk of the worst of all situations: When the direct replies back with a response that demands more information or explanation. Now you're in exactly the situation you didn't want to be in, trying to explain or discuss a sensitive topic over a context-less medium like text, email, or instant messenger chat. If you try and punt on the conversation, your direct stews and simmers and gets angry or upset; if you don't, you're unable to read them effectively (and vice versa) and it becomes easy to really blow the whole thing into a Really Big Deal. Definitely not high on the Ted Lasso Scale of Senstivity and Compassion.

I'll be transparent here: I personally have never found a way to ask a direct for an unscheduled meeting without triggering those negative feelings, and I really don't think there is one. Thus, my only real advice or suggestions here is as follows:

Understand that your direct is going to be on edge. If you give them the context-free request, they're going to be assuming the worst. It's human nature, and you can't really avoid it.
Minimize the window of discomfort. For everybody's sake, do not ask to chat with somebody on Friday, and then schedule the meeting for Monday afternoon. Do the request first thing Monday, instead.
Be up-front as much as you can. Don't schedule-bomb somebody and silently drop a meeting on your direct's calendar without at least telling them in an email, "I need to chat with you about something, and it's a little urgent. Saw you had some free time this afternoon, grabbed it for a conversation. Talk more then." Yeah, your direct is going to start doing weird gyrations in their head as they try to mentally go over everything they've done in the last few days that might trigger this, but remember, you can't help that. And, if you just silently schedule-bomb them, it'll just be that much more surprising and panic-inducing.
For the love of all humanity, don't just cancel the meeting without reason or ghost it. Seriously. That's also a recipe for a psychotic break, and honestly, I think it's grounds for justifiable homicide in twelve states. If you put the meeting on the calendar, be there, or at the very least postpone or reschedule it with a reason. And keep in mind, you're leaving your direct twisting in the wind during the interim--if that doesn't bother you, then maybe you lack the compassion necessary to be a good manager.

And, hopefully it goes without saying, if you can provide context to the conversation, by all means do so! Then you avoid all of this, and the world is a bright and happy place again. Most of the time, anyway....

The Subtle Power of Teams

Wed, 3 Jan 2024 00:00:00 +0000

A post on LinkedIn got a little close to my heart, showing a picture of Joe Flacco (an almost-retired NFL player, for those who aren't in to sportsball) and asking, "Are you overlooking experienced candidates because of their age? Take a lesson from Joe Flacco, who at 38 was written off by experts as too old and past his prime. Despite sitting at home hoping for a chance, the Cleveland Browns saw potential and gave him a chance to prove himself." And as one who's cough not exactly fresh-out-of-school myself, I thought it was worth a comment.

It always surprises me how often companies and hiring managers think, "I know! Let's save a few bucks on somebody who's never seen anything, will overreact to the tiniest of crises and completely miss the subtle cues that something serious is about to go down. No WAY that's going to come back and haunt us!"

Look, I get it: With the rapid iteration cycles in tech, it's hard to believe that somebody who's spend 45 years writing COBOL will have much to offer to your "full-stack" Javascript team. But you're missing the forest for the trees: Most 30-year veterans haven't spent all that time in just one stack or one company. The diversity of their experience vastly dwarfs all five of your recent college grads, and even if they can't pull all-nighters like the young'uns, it's because they usually don't have to.

Don't get me wrong, I'm not advocating for a complete reversal of position, or that you need to toss your current crew. Don't make a whole team out of 45-year COBOL veterans, but don't make one out of Javascript-slinging GenZ'ers, either. Your team should be a mix of skills and experience, using diversity of team composition to cover as many possibilities as you can.

To borrow another analogy, which Diagnostic Medicine doctor would you rather have in charge of your company's medical case? Cameron? Chase? Foreman? 13, Kutner, or Taub? "Maybe," you think, "he's suggesting we want Dr House, sure I get it." Nope. I want him AND his whole team--there's a reason House needs a team, no matter how smart he is, and he's definitely smart, because he knows he needs a team that thinks differently from him and challenges his perspective and assumptions. That's (literally) the point of the whole show.

As my dad used to say, "Youth and boundless energy lose to old age and treachery every time." ;-)

2024 Tech Predictions

Mon, 1 Jan 2024 00:00:00 +0000

It's that time of the year again, when I make predictions for the upcoming year. As has become my tradition now for nigh-on a decade, I will first go back over last years' predictions, to see how well I called it (and keep me honest), then wax prophetic on what I think the new year has to offer us.

As per previous years, I'm giving myself either a +1 or a -1 based on a purely subjective and highly-biased evaluational criteria as to whether it actually happened (or in some cases at least started to happen before 31 Dec 2023 ended). If you want to skip to the new predictions, scroll down to the next major heading.

2023 was a tumultuous year, for sure--lots of tech layoffs in the early part of the year as big-player after big-player announced whole trunks of the org tree no longer had a job. It definitely created some chaos and consternation among our industry, for sure, and that was all before the collapse of Silicon Valley Bank, which sent a huge ripple through both startup and established player alike.

By the way, as of this writing, despite multiple false starts, I'm still looking for my next great adventure (ideally as an Developer Relations or Engineering leader-of-leaders), so if you find my analysis here to be interesting or intriguing--even if you disagree with it--perhaps there's a role in which I can do this kinds of strategic and executive thinking on your company's behalf? Would love to hear from you.

In 2023...

To start, I wrote:

First off, the pandemic hasn't left us, and CDC is starting to talk about masking up again. Despite desperate efforts to pretend we're clear of COVID (and related viruses), we're not, and it's going to continue to rear its head in ugly ways going forward. I don't think we're going back to mandatory stay-at-home policies, mind you, but it's going to be a couple of years of really bad winters, medically speaking. The "taper down" I talked about last year definitely seems to be in full swing (or slope, as the case may be).

Masking is still a thing in much of the Pacific Northwest, and they're still required in most medical offices, but beyond that, COVID or no COVID, people have made their choice. Aside from the performative video piece, nobody is going around ripping masks off of other peoples' faces. Periodically, somebody catches COVID and is down for a week (minimum, usually two), but that seems to be a risk we're willing to accept as a nation.

Secondly, even though the Ukraine conflict continues, and inflation reared its ugly head during the end of 2022, the economy doesn't look as bad as we thought it would six months ago, and I think things are going to "perk up" once we get through the holidays. 2022 sucked, but there's a lot of reasons to think 2023 won't be nearly as bad. (Of course, we thought that about 2022 compared to 2021, and about 2021 compared to 2020, so....)

Statistically, the economy has recovered. Inflation is coming back down, unemployment is actually back to reasonable numbers, and the stock market is doing... better. Nothing is back to its pre-2019 levels, but frankly, we'll probably never be there.

Meanwhile, the war in Ukraine continues, and a new one has opened in Gaza. Regardless of which side of the political spectrum you're on, everything feels terrible, and the last time that happened, Jimmy Carter was President. Are we due for another "Morning in America" moment? Probably not--we're all too cynical to the media for Reagan's watershed speech to work again, but something is going to give soon, and we will all rush to embrace it, because it just doesn't feel as good as it should right now.

Meanwhile, on the tech front....

"Blockchain-related losses are going to be huge tax write-offs. It's already starting, and other companies will follow suit as soon as they see others doing it." Going to give myself a +1 on this; most big-tech firms have divested their blockchain development efforts (and advertising), and the only ones really left pursuing it are startups (most of which seem desperate to exit from what I can see). Blockchain is pretty much dead as a tech, it seems, unless you're so deeply wrapped up in it that you need it in order to survive. Tally: +1.
"***Blockchain-related businesses are likely to be the target of lawsuits.***" Well.... Sam Bankman-Fried comes to mind immediately, not to mention most of his inner circle. That said, there were fewer lawsuits than I expected, particularly against the media personalities that shilled for crypto (Ashton Kutcher, Matt Damon, etc), but I probably shouldn't have expected them in the first place--a friend pointed out that the famous almost never get sued for hawking a bad product. Still, lawsuits and jail time, I'll take the (debatable) +1 here. Tally: +2
"The fallout from the cryptocurrency nightmare is going to stain 'Web3' for a good long time. .... Web3 is not going to be a fun place for a while." That definitely qualifies as a +1. Every one I know, be they developer, architect, tech manager, you name it, they're all staying well clear of anything "Web3" or "blockchain", even when the job market was at its worst. Reddit is filled with threads of curious developers asking about the "negative feelings about Web3" and most future-looking Web3 discussions seem to tie it closely to social media, which is also taking major hits right now in the courts of public opinion. Tally: +3
***Distributed systems start to embrace contract-driven development cycles.***" This one is harder to judge--OpenAPI hasn't reached quite the level of ubiquity that I would expect to call this a solid +1, but at the same time, there's a lot of groups/teams that are starting to look for better ways to interact with an HTTP-based API than the traditional "just build out the URL and HTTP header by hand...." approach. Lots of the contract seems to be getting buried inside official SDKs released by the API provider, which more or less provides the same kind of "contract" approach, albeit with a lot more work on the part of the API provider. Still feels like we're moving in the same direction on the ferris wheel, though, so... +1. Tally: +4
"***Microservices start to give way back to deliberate monoliths.***" Well, talk about interesting--not more than a week after I wrote last year's predictions, I wrote another piece, "You Want Modules, Not Microservices", and boy did that resonate--it got picked up by a bunch of news and journal services, and much of the feedback/commentary was... actually agreement (modulo the odd conspiracy theorist who is still convinced that monoliths are somehow the gateway drug to 5G nanobots in your blood controlling your mind at the every whim of Bill Gates and the Greater Dolphin Cabal of the North Atlantic). Several companies turned around and de-micro'ized their architecture back into a monolith, in fact, including Amazon. Turns out, all that traffic across the network bouncing between a bunch of services is bad! Who could've guessed? +1, bringing my Tally: +5
"***Security companies are going to need to OSS their offerings.***" Checking the Auth0 GitHub repositories.... Nope. It was a long shot, and it definitely didn't happen. -1 means I'm now at Tally: +4
"Work-From-Home (WFH) continues to normalize and become "just another location". (Probability 0.7)" Well... this one's harder. For each "It's Time to End the War on Remote Work", there's a "Remote Work is Still 'Frustrating and Disorienting' for Bosses". Studies continue, and likely will for the remainder of this decade, and while many companies are quietly just sorting it out internally, other companies are making great noise about return to the office (or just "work more hours", which seems to have had exacty the effect you'd expect), but when they do, it generates a ton of bad PR and the executives are usually obligated to walk it back. It looks like 2024 is going to (hint, hint) see a drawdown in the attempt to get people back into the office--but we'll leave that for the next section. For now, I'll give myself a +1 simply because it does feel like it's becoming more and more normal to expect a remote gig, and that's really the definition of the word "normalize". Tally: +5
"***Somebody is going to sue over state income tax.***" You can see where I was coming from on this one, but the SCOTUS' continued unwillingness to take a stand on it surprised me and continues to surprise me. Technically, I get a point for this one, since... well, tax law professor, New York, you get the picture, but this has been an ongoing fight since well before the pandemic, and it doesn't seem to have "ballooned" into the bigger fight that I thought it would be, so at best I feel it's a 0. Tally: +5
"***Low-code/no-code is going to accelerate further.***" A lot of the low-code/no-code efforts got shoved off to the side of the mainstream tech media in 2023 because of the incredible tech hype and media storm around generative AI and natural language models, so any efforts at advancement here kinda got derailed. However, the generative AI hype wave caught up a few low-code/no-code entrants along the way, and now you can build workable prototypes from napkin sketches, apparently. (No word on what happens if these AI-powered prototypes have access to the Internet, though.)
By the way, those Boston Dynamics robots are getting closer and closer to mounting guns, regardless of what the manufacturer says, enough so that states are considering outlawing them entirely. Because nothing stops a T-800 like a cease-and-desist order. (I kid, I kid. Actually, no, it's more "I cry, I cry", because it's going to happen whether BD wants it to or not.)
+1 means Tally: +6
"***Internal tech audits will become relatively popular.***" If it happened, everybody kept reallllly quiet about it, which you would kinda expect. But if it didn't show up in the tech press, did it really happen? -1 means Tally: +5
"***Muskian-style management will be studied, cargo-culted, and lead to the failure of more than one company.***" groan. sigh. Lord save me. +1 I hate that I was right about this one. Tally: +6
"***2023 will be the year we try to figure out what to do with all this AI stuff.***" Nope. Nope, nope, nope, nope, nope. -1 We haven't even figured out the difference between natural language models and generative AI, or the copyright implications of training a model on copywrit sources, or... pretty much anything beyond "Slap some AI on it, and it will sell!". We haven't figured anything out here, and it was horrendously naive of me to think we could in just a years' time. Tally: +5
"BONUS PREDICTION: 2024 will be the year that we start putting AI into some of our apps--like using ChatGPT for chatbots--and 2025 will be the year that we start ripping it all back out because we completely got it wrong. (We always do this. And I see no signs that we'll stop in my lifetime.)" OK, that one, I got right. But it was like candy from a baby--no points.
"***Companies will accelerate their drive to become 'tech companies'.***" A fair number of firms continue their "technology transformation" efforts, but nobody's really any faster or slower--or better--at it than they were before. -1 This was probably complicated by....
"***Hiring will accelerate at the end of 1Q/2023.***" I kinda think I should reset all my points back to 0, because holy crap, was I wrong about this. So deeply, incredibly, soul-wrenchingly wrong. A year ago, I actually wrote this: "Even as companies are engaging in layoffs, these aren't the savage cuts that we saw back in the Dot-Bomb era (circa 2001) or the Great Recession (circa 2008) era." Maybe back in 2022 that was a reasonable take, but then 1Q/2023 came around, and all that went out the window as big firm after big firm just pruned at their org tree, and did it savagely. This layoff tracker says it perfectly: "So far in 2023, there have been 1,992 layoffs at tech companies with 428,335 people impacted (1,183 people per day)." The graph shows 108,000+ in January 2023 alone. Tally: 0 So, so wrong.
"***Entry-level positions are going to be easier to come by.***" Ironically, this one actually seems to be taking shape. Even as a lot of companies laid off in 2023, many of them started hiring again almost immediately (2Q, 3Q), but almost exclusively for individual contributors (ICs) and many of those roles were entry-level positions. It still isn't a great job market by comparisons against the average from the 2010s, and definitely not by comparison against the pandemic era, but it's not hopeless if you're a tech IC right now. (Management, on the other hand.... that's where a lot of the struggle still lies.) I'll take the +1 since I'm desperate for points now, bringing us to Tally: +1
"***Platform-oriented development is going to begin making some waves.***" This one is a little tricky to judge. On the one hand, we can see companies developing platforms all over the place, particularly as exemplified by... well, pretty much every startup that's gotten a round of funding since 2019. But my prediction is a little vague, and speaking honestly, I was expecting to see a bit more of a push from multiple places in the developer ecosystem, which didn't happen. So... -1 takes us to Tally: 0
"***Cloud will begin to shift.***" Now, this one was interesting. When a major talking-head like DHH talks about moving off the cloud, it sends waves, and sure enough, companies began to do the math and realize that cloud companies are charging them at rates that generate (surprise!) a profit... to the cloud company. +1 See, that was always going to be the endgame here, because while the economy-of-scale graph does mean that you can generate more of a good/service per unit if you do it at larger amounts, that graph does eventually flatten out, and more importantly, multiple larger businesses began to realize that they were probably large enough to be able to take advantage of that same graph, themselves--once they'd put the initial investment in, they could, in essence, run their own private cloud, and pay even less for it than going through Azure, AWS, GCP, or any of the others. It will always be cheaper to make it yourself, once you've put the investment in to be able to make it yourself at scale. It's the same reason McDonald's makes its own buns, beef, and vegetables--it's the technology equivalent of controlling your supply chain. Tally: +1
"Elon will sell Twitter (perhaps involuntarily) at the end of 2023, to another tech firm (like Microsoft or Oracle). ... And that's how the grand MuskTwitter experiment will end. Not with a bang, but with a pathetic whimper."*** Well, it's the end of 2023, and... checking... nope, Musk appears to have no intention of selling off Twitter-I-mean-X-sorry. But the quality of the Twitter timeline keeps dropping, my friends and follows (the people whose opinions and insights I respect enough to follow) are slowly dropping off as well, and I'm about one more unsolicited DM from Nikki Haley away from doing so myself. So, first, I take a -1 on the prediction, but in the meantime, Musk continues to drain away everything that was actually good about Twitter. Tally: 0
"Meanwhile, alternatives to Twitter will continue to sprout up, but none will actually be able to be everything Twitter was, partly because Twitter was (at the time) unique in its approach to social media, and as a result it created a "perfect storm" that led the platform to its current (well, pre-Muskian) level of success. That will be nearly impossible to recreate." Yeah... +1. Sorry, Mastodon, Threads, BlueSky, and every other Twitter clone that's been released in the last year, but none of you are showing me the critical mass of "interesting mess" that Twitter became. It used to be that people I followed were also commenting and RT'ing on people they followed, and made it easier for me to find new and interesting voices to follow, and the clones all just don't seem to "get" that. I suspect, going forward, that before long my only social media presence will be LinkedIn and this blog, which will make me feel a little sad and alone, but frankly I'll probably feel a lot less depressed about the world around me once I do. Tally: -1
"RSS will make a comeback." Nope. -1 It hasn't, and it probably never will by this point. Sorry, folks, but RSS is essentially dead as a protocol of widespread adoption beyond where it's already in use now. Tally: 0

In summary:

twenty-one predictions,
a bunch of hits,
a few fails but mostly a massive fail,

... leaving me with what I consider to be a completely missed round. Yikes. (I am really annoyed at how wrongly I read the room this time last year on the job market--for personal reasons, yes, but also because I really do pride myself on my ability to do these sorts of professional forecasts, and that was a huge misread.)

2024 Predictions

With that settled up, let's take a look at what I think will happen across calendar year 2024. As I've done with my predictions, I'm including a probability score with each one, in much the same way intelligence officials do in their assessment reports, to give you a sense of how confident I am in the prediction.

Hiring will open up. (Probability p 0.8)

The interesting thing about the layoffs is that while they continue to trickle out from various companies all the way to December of 2023, it's a trickle compared to the flood that happened in 1Q, particularly in January. More interestingly, the kinds of people that are going to be hired will shift slightly--right now (end of 2023), companies are focusing on hiring ICs to fit into existing holes in their current org tree. But with the new year, and a bit of optimism in the market, they'll start growing new branches to that tree, and that will require more in the way of engineering management and "ancillary units" like developer relations, R&D, and so on. It'll take a while, and it'll never be like 2019, but before too long things will get stronger.

WebAssembly will gain more traction. (p 0.5)

I haven't really paid much attention to WebAssembly for the past few years, because I wasn't sure where things were going. The 1.0 specification is done, but it's pretty minimal compared to other richer, more full-featured virtual instruction sets (like the JVM or the CLR). Truthfully, it's a ridiculously bare-bones spec. But, even as I say this, there's some interesting infrastructure that's targeting LLVM more and more (most notably LLVM and GraalVM), making it easier and easier for WASM to gently "slide in" to a deeper role within a dev toolchain and do some interesting things. That said, though, WASM still has yet to really demonstrate where it's real value lies--why write something to WASM binary, which generally means targeting the browser, when there's so many Javascript-centric options already? Where's the compelling value-add from the developer perspective? This is the hinge on which WASM's broad-scale applicability matters: If WASM can show how using language-X-compiled-to-WASM enables a new kind of feature or a faster/easier workflow to get to "done", then WASM could very well begin to emerge as the infrastructure backplane for browser-based applications all over the Internet. I don't know what that compelling reason is yet, though, which is why I leave this at a 0.5 p.

Generative AI will lose its luster. (p 0.7)

It already has started, in many ways--memes now are common across the Internet showing ChatGPT conversation windows in which it responds to the question, "How do I use GitHub?" with "In order to push code to the remote server, issue the command git branch..." in a completely erroneous and mistake-riddled answer. The stories of the lawyers who used ChatGPT to write legal briefs that turned out to be filled with inaccuracies and errors--for which they were disbarred, by the way--have cast some serious doubt on the ability of these generative AIs to "fire all the peons" whose work ChatGPT and its ilk were supposedly going to replace.

We will begin to disambiguate between generative AI and large language models (p 0.5)

It happens with AI every decade or so--something new comes along, it takes our imagination by storm, and we start chanting "AI! AI! AI!" like an amped-up-on-booze-and-caffeine-pills crowd at a wrestling match. Then, as we get disenchanted with the results, the various AI firms--who have yoked their success to the success of their AI work--will start to point out that they were never "AI" companies, they were "large language model" companies and therefore never a part of the hype machine that's now augering in.

Custom AI models will begin to gain traction. (p 0.6)

Making use of a large company's model for your natural language analysis needs is great, but over time, companies are going to figure out that the cost of that model will probably be smaller if it can be hosted locally, rather than constantly going to the cloud and back again. (See the economies-of-scale discussion earlier, as well as the performance and reliability implications, a la "The Eight Fallacies of Distributed Computing".) Slowly, quietly, companies with some room in their budgets are going to start looking to develop their own models, and in many cases may find that the model's they're building don't need to be quite so large (and heavy), that in fact a "medium" language model, or even a "small" language model would work, allowing for a local presence on the same node as the rest of the processing. OpenAI and other firms are going to combat this by constantly releasing new models with new excitement and fanfare, but like the cloud itself, the basic path here will be "start with the hosted LLM, then as your success and budget grows, look to build--and tune--your own". It'll be "Buy vs Build" decision-making, all over again.

New and interesting languages will begin to make waves. (p 0.6)

I've been saying this for years, but I keep holding out hope. Frankly, I think some of the development will be to raise the abstraction levels of what we do, because ChatGPT's success at writing code is due to the fact that 95 times out of 100, we're writing the same basic CRUD crap over and over and over again. The Pareto Principle holds here: 80% of all mobile applications (as an example) are doing the same stuff, it's just that it's a different 80% from all the rest. A language/stack that can embrace an onion-style architectural approach (allowing for "trap doors" to drop down a level of abstraction when/if necessary, such as how C/C++ allowed for inline assembly way back in the day) will be the answer here.

The other element I imagine will/could be interesting to explore will be the intersection of what natural language models and compiler front-ends will look like--if we can get a language to start looking more like a natural language, it might enable some interesting DSL-type scenarios for end-users, and reduce the demands/pressures on developers to respond to every little change.

Cracks in the "full-stack developer" facade will grow. (p 0.7)

The demand for the "full-stack developer" is mostly a cop-out on the part of managers, who don't want to artificially restrict their employee search because "hiring is hard". So, instead of carefully considering what skills the current team lacks that needs shoring up, they instead scribble "full stack developer" on the job description, and proceed to rattle off every single technology that the team has ever used.

AR/VR will start showing glimmers of life (p 0.5)

If AR/VR is going to achieve anything of any substantial value, it really needs to do so soon, before it is relegated to the "Interesting Things That Were Out Of Their Time" bin, like the Apple Newton and the Xerox PARC. This year might--might--be the year it presents us with something, particularly given the rise in interest in wearables. Certainly, with Meta going all-in on the whole "Metaverse" thing (whoever came up with that should be fired), there's going to be no shortage of money thrown at the problem, the big question is, will there be any actual innovation around UI? Because so long as the AR/VR thing is solely game-centric, it's fairly safe to assume that the AR/VR will go right into that bin without too much hesitation. I know, games are huge industry (measured in the trillions now?), but it's not enough to support hardware; witness the difficulties even conventional gaming hardware (for example, joysticks--today's flight sims are vastly more complicated than the ones twenty years ago, yet nary a joystick to be found) has had over the decades. If AR/VR (which requires hardware specific to it) is going to reach a point that justifies buying specific hardware for it, it has to be for something more than just gaming (and please don't say "education", because if there's one place in the world that has almost no budget, it's education).

"Car apps" will begin to become "a thing" (p 0.6)

More and more vehicles out there are coming with computers in the console, offering easy connection to your mobile device. What's been lacking has been any sort of apps for it, beyond the basics (maps, audio entertainment, etc). Now that there's more of a target market, perhaps we are reaching a critical mass of potential customers to justify the investment in building apps specificall for vehicles.

If this does start to take hold, it will be interesting to see what sorts of apps get built (I could imagine some CRM apps for certain kinds of salespeople, for example) and what form user interaction takes hold (voice control and interaction would be very important for drivers, for example). Frankly, though, the hard part will be the core innovation itself--what sort of apps do we want when we're driving?

Kotlin will remain an "Android-only" language (p 0.5)

This one is a hard one to call, but I'm flipping a coin and landing on the pessimistic side of the "Will Kotlin break out of Android-only?" question. For those who weren't following the Kotlin story at home, Kotlin has recently done a couple of things to make it more than just "that language you use for Android" by developing Kotlin Multiplatform and Kotlin Native. These open up the Kotlin language for use in more situations, but their success will really hinge on how many developers actually want to use Kotlin in more places than just their Android codebase.

Historically, multiplatform languages have not done well, with the sole (arguable) success being that of Java--whose "Write once, run anywhere" campaign didn't really accomplish much. (Remember, most of Java's success was in the server room, not the desktop.) Native might have more possibility for success, but either one gaining any traction would be an interesting development and potentially grow Kotlin beyond just "an Android thing".

C# and Java will continue to "pile on" with features (p 0.8)

Both languages have reached a point where the weight of the language is really beyond the ability of any one person to keep track of what's there, yet each time a new release comes out, it comes with a whole slew of new proposed features designed to bring new syntax into an already-existing part of the language or an attempt to cram yet-another-interesting-idea into an already "overloaded-collection-of-features"-oriented language. (Can you really look at either C# or Java and tell me that they are "object"-oriented anymore? You can go quite a ways with either and never see or hear an object anywhere along the way by this point.)

Look, maybe this is my "Gitoffamahlawn" moment to these predictions, but both of these languages are old enough to drink, they each spun up powerful and resilient platforms that have a number of newer, simpler, more-concise languages, and at a certain point in time, it's reaasonable to say, "This one is done. There's nothing new we need to add." To keep cramming new stuff in is a terrible addiction--it's a way to show that "We're still hip! We're still doing cool stuff (...and therefore justify why companies should still be paying us license fees!)" We keep telling startups not to do it, but our two major providers of software development tools can't seem to get the lesson themselves.

Spend some time with F#. With Clojure. With Kotlin. Play with some of the new ideas (Mint, Wasp, Io, Alloy). Or even go back and experiment with some of the classics (Smalltalk, Self, LISP, Scheme). But, for your own sake, stop enabling the relentless pounding of the feature surf on the beach of your brain by breathlessly hanging on "What comes next", because in time, you'll be battered into sand.

Crypto falls even further. Blockchain struggles to reinvent. (p 0.7)

Look, faith in cryptocurrency is really at an all-time low, except for the people who are caught holding the bottom of the Ponzi pyramid, and they desperately want to hype the hell out of it so they can pass the bag on to the next level down. That won't change. Blockchain, the world's worst-designed distributed database, will continue to wrestle with the goal of finding a reason to exist, and hey, could maybe even find one bullseye, given enough darts. (Many companies that invested in Blockchain have either decided to walk away from the dartboard or else that they're going to load up on a lot of darts--and maybe a shotgun or two--until they hit that bullseye.)

I really would love for these two to just go away entirely, but they won't, because nothing ever dies completely. But this is the last year they're going to get much--if any--serious time in the tech press, despite the fevered efforts of a precious few on LinkedIn. (Yes, Andreeson-Horowitz is going big into it; yes, they've been big into it since it's start; yes, there will always be people who see what they want to see, instead of following actual evidence; and yes, folks, there's still time to dive in and turn your profit, just don't be the last one holding the pyramid up!)

And, for me, 2024 will be the last year I talk about this in any form.

Phishing attacks go ballistic (p 0.7)

Thanks, AI! With all these generative natural language tools (like ChatGPT), it will be easier than ever to generate emails intended to trick users into surrendering their credentials to fake sites. That will lead to more breaches, more private info leaks, and more of all the wonderful things that come with those.

What will make it even worse will be all the security companies that roll out "AI-based" security tools, claiming that the AI will somehow be able to better protect against those very same attacks--but while generative AI can create really good-looking human-facsimile output, it's not always great at recognizing artificially-created human-facsimile input. Which means the attackers just got a boost to their toolset, and the defenders will be looking to try and keep up this year.

Databases will incrementally improve (p 0.7)

... as opposed to releasing something drastic. There is some interesting ideas percolating over in the world of DBOS, spearheaded by one of the principals in the Postgres world, but I have a feeling that it'll take a little bit for those ideas to brew before there's really something there to evaluate. (Hope I'm wrong, though.)

Meanwhile, though, players in the RDBMS world will slip a new feature into the mix by polluting the relational model just that wee bit more, the NoSQL players will pick just a tiny bit more from the relational world and add that, and developers will continue to choose which player they want to play with based on criteria that touches maybe 1% of the feature set of the player they choose.

I'm publishing a book (p 0.8)

I and a few other folks are working on a book based on the Developer Relations Activity Patterns catalog that I drafted a ways back, expanding on the material there in a big way. That'll come out in 2024, almost guaranteed.

Fifteen predictions for the next twelve months. Let's see how I do this time.

Talk to you all next year... if not sooner.

Tabletop RPG IT

Mon, 4 Sep 2023 00:00:00 +0000

With the enforced hiatus I've been on for the past 18 momths, I've had some opportunities to engage in a few projects that otherwise would've gone undone or unexeplored. One of these projects is actually indulging in a favorite pastime of mine: TableTop Role-Playing Games (or TTRPGs, for short), which I've enjoyed since 1978. As somebody who runs a TTRPG campaign for some of my friends, I find that wading through piles of books, some of which deprecate material found in another, to be really, really ancient and antiquated, and the IT professional in me says, "There's got to be a better way."

Let me take a few moments to spin up the context: I've been an avid TTRPG (tabletop role-playing game, as in D&D or Pathfinder, sitting around a table with friends, as opposed to video gaming versions thereof) player since 1978, when I got the "blue box" edition of D&D. When the pandemic started, some college buddies thought it would be a good opportunity to do some online gaming, and I offered to GM.

For me, that meant an opportunity to exercise my creativity, as I've always preferred to run campaigns out of my own worlds, rather than using the prepackaged ones that come from the vendors. This would be no different, largely because I wanted to provide a world that was tailored (somewhat) to what my players would be interested in, but also allow me some room to deviate from the traditional and explore some new ideas. For example, I've come recently to realize that "players" and "monsters" aren't all that different from one another, aside from history, so I've come to embrace the idea of "monstrous player characters"; you want to run a minotaur barbarian? Go for it! Kobold sorcerer? Why not? Orcish cleric of healing? Makes total sense.

This also means I like to make use of other homebrewed content, such as what one might find on DM's Guild or RPGDriveThru, but that raises an interesting problem: with all the different sources, it can be awkward to reconcile them all, meaning either I the GM have to have every sourcebook and homebrew document ever used anywhere, or I have to have some way of creating a "single source of truth" for the different content in my game. Spells, creatures, races, classes, all of it has to be in one place so I only have one place I need to consult to keep track of it all.

Enter GitHub.

Source Control

Although it's most often used for source code management across versions, tools like Git and the like can be used to keep versions of anything (including binaries, though less effectively), and I've long used Git for holding copies of all of my professional writing. This blog, for example, is completely inside a Git repo--and, in fact, was at one point entirely CI/CD-driven. (It's not currently because I ran into some weird configuration issues with my Azure Website, and I haven't taken the time to fix it yet, instead choosing to use the uber-simplistic website-hosting provider Site44, which turns a Dropbox folder into a website root, perfect for static-site-generated sites like this one.)

So, the Github repository for my world was born. I found a pretty nifty world map editor to create the world in, generated one with a nice mix of land and sea (I like having worlds with a good mix of both, because that opens up more varied adventuring possibilities), and lo! The world of Azgaarnoth was born. (The editor is called "Azgaar's World Editor", so I just borrowed the first part of the editor name--seemed like a reasonable homage to me.) All of the files are written in Markdown, and the built-in VSCode editor in every GitHub repo means that if I can access the page (and log in), I can edit it right there in the browser, or I can work with it offline on a laptop for more heavy-duty editing.

First steps were to lay out some basic history (8,000 years of it!) and geography, which meant taking some of what the map editor had generated for me, creating some points of conflict (this part of the world started as a single empire, then had a massive invasion by foreign entities, which stalemated) extracting some consequences across history (with no quick conquest or defeat of the invader, people got frustrated which led to civil breakaways from the empire), and laying out some rough guidelines.

But.... there's more I wanted to do. "Single source of truth" really means "single" in my mind, and that holds for some game-system information (such as races and classes) as well as the world-specific stuff. So I created some Markdown versions for races and classes, which wasn't all that hard. But I also wanted to have a complete reference for every spell and creature and magic item in the world.

The GM's Spellbook

Spells are one particular point of fun, because while the core D&D 5th Edition rules has a pretty decent collection, there's always room for more, particularly if you want to bring some fun flavor that the players haven't seen before. (And my group, we've all been playing since a young age, so it's often quite the stretch to bring something they've never seen before.) This meant bringing all of the core spells into Markdown format, but also all of the homebrew and 3rd-party purchased content as well, most of which is distributed as a PDF. But I also want to be able to sort through it all without too much difficulty, too.

And here's where things get interesting. A given spell has a lot of interesting metadata on it beyond its name: its level, the "kind" of spell it is (illusion, divination, etc), the classes that can cast it, and so on. Any one of these could be used to organize the master list of spells, but the truth is they're all going to be useful at some point.

Sounds like a database to me!

But truthfully, I didn't want to build this on top of an existing database like Postgress or MySQL; for starters, the RDBMS doesn't really "fit" with storing what is essentially a text document with a few fields of metadata around it, and if I moved all the content into the RDBMS from GitHub, I lose the "quick editing" ability that GitHub provides, plus now anybody who wants to edit or add new spells to the list has to be SQL-trained to do it (or I have to build an editor). I kinda still want the source of truth to be GitHub, rather than a more formal RDBMS.

Actually, why not leave GitHub as the database? Each entry is a Markdown file with predetermined blocks in it for various data elements ("Duration", "Casting Time", etc), with each of those elements identified by being bolded in Markdown format. Since each spell needs to be uniquely named, that becomes the "primary key" for the spell, stored in a flat list (directory). Thus, a spell looks like the following:

#### Magic Missile
*1st-level evocation* (Sorcerer,Wizard)
___
- **Casting Time:** 1 action
- **Range:** 120 feet
- **Components:** V, S
- **Duration:** Instantaneous
---
You create three glowing darts of magical force. Each dart hits a creature of your choice that you can see within range. A dart deals 1d4 + 1 force damage to its target. The darts all strike simultaneously, and you can direct them to hit one creature or several.

***At Higher Levels.*** When you cast this spell using a spell slot of 2nd level or higher, the spell creates one more dart for each slot level above 1st.

... and is stored in magic-missile.md. So long as the formatting doesn't deviate from the above, I have some easy parsing markers to work from.

Now, I need some kind of tool that knows how to parse these into a form that can be manipulated and queried. Let's be honest--any language would've worked for this (the parsing here is not complicated, and each spell is essentially a flat record), so I decided to go with Python because why not?

Thus was born "SpellTool": a Python script that rips through a directory filled with these Markdown files, parses each into its own "Spell" object, and allows for some predetermined sorting (such as finding all the spells that can be cast by Sorcerers) and output. It's not a full-fledged database (I can't do ad-hoc queries on it--yet), but it's not that far off from it. Obviously it's not nearly as fast or storage-efficient as a B-tree-based system would be, but it's also directly editable by myself or other "domain experts", which is every bit as valuable, if not more so.

But the Python fun doesn't quite stop there.

Enter Mkdocs

One thing I found from my players is that as much as I love the GitHub interface, they were a little less fond of it. Honestly, that's not unreasonable--GitHub was not designed for the casual individual. Plus, the URLs that GitHub was putting up for each of the spells was... clunky. And truthfully, the Markdown format is not the easiest thing in the world to read if you're not familiar with it. (It's not hard to read, mind you, just... awkward.) All of this goes away if I convert the Markdown to HTML and put it all on a simple website (such as the aforementioned Site44).

Fortunately, Python has a great tool, MkDocs, which was obviously intended as a documentation management tool for Python libraries, but it serves my purposes perfectly well, since it takes a collection of Markdown files, transforms each into HTML, but also recognizes relative links and honors them. Plus, any links that don't seem to line up with an actual file are flagged during the process (since they'll likely be a broken link after the translation is done), giving me some "compile-time warnings" along the way.

The results, by they way, are what's on the website now. If you've ever spent any time looking at Python docs, you'll recognize the format instantly, and the "Next/Previous" links are pretty broken, but I can either live with it as-is, or figure out how to customize or turn those off later.

Enter npctool

The most recent thing I've wanted to do (for years, actually) is create a tool that generates NPCs (non-player characters) for use during the campaign. After all, why should players be the only ones with all the goodies (in the form of special abilities, spells, and so on)? Some of my most successful campaigns have been because the PCs are standing against a villain who is sporting some of the same kinds of abilities as they are.

But as the players get higher in level, quickly putting together an NPC that's their peer (or superior) is tricky, because the game system is really expecting that you painstakingly craft each level of experience. Most GMs just "fake it" or hand-roll an NPC and use that singular NPC for a while. Me, I like having a fair number of them, partly because it's reasonable to expect that PCs will have more than just one nemesis across their careers, but also because it keeps things varied and interesting to have some different "feels" in the campaign. For example, in the current campaign, the first adventure was the party had to go up against a would-be lich who swiped the lair of a real lich. (Lots of undead.) The second adventure saw the PCs going out to a recent shipwreck to rescue an item before it was lost. (Lots of seaborn creatures.) Each had some different NPCs involved, but you can bet that as we move into the third adventure, the NPCs who weren't outright killed in each adventure are figuring out how to "get back at" the PCs.

In essence, what I needed was a tool that could quickly rip through each level of an NPC's career, answering whatever customization questions that arose, and when the NPC was "finished", calculate all the necessary "to hit" and damage and other level-variable calculations ahead of time. Thus was born the "NPCBuilder", another Python script that will generate NPCs either interactively or from a simple text script:

Decide which?
1: Ability Scores
2: Gender
3: Race
>>> 1
You chose Ability Scores
Method:
1: Standard (<function generatenpc.<locals>.selectabilities.<locals>.standard at 0x1074d8900>)
2: NPC (<function generatenpc.<locals>.selectabilities.<locals>.npcstandard at 0x1074d8860>)
3: Hand (<function generatenpc.<locals>.selectabilities.<locals>.handentry at 0x1074d8ae0>)
4: Randomgen (<function generatenpc.<locals>.selectabilities.<locals>.randomgen at 0x1074d8a40>)
5: Average (<function generatenpc.<locals>.selectabilities.<locals>.average at 0x1074d89a0>)
>>> 5
You chose ('Average', <function generatenpc.<locals>.selectabilities.<locals>.average at 0x1074d89a0>)
Decide which?
1: Gender
2: Race
>>> 1
You chose Gender
Choose gender: 
1: Female
2: Male
>>> 2
You chose Male
Decide which?
1: Race
>>> 1
You chose Race
Choose a race: 
1: Siren (<module 'Sirens'>)
2: Kobold (<module 'Kobolds'>)
3: Dwarf (<module 'Dwarves'>)
4: Reborn (<module 'Reborn'>)
5: Half-Ogre (<module 'Half-Ogres'>)
6: Elf (<module 'Elves'>)
7: Tabaxi (<module 'Tabaxi'>)
. . .
33: Sahuagin (<module 'Sahuagin'>)
34: Triton (<module 'Tritons'>)
35: Genasi (<module 'Genasi'>)
>>> 6
You chose ('Elf', <module 'Elves'>)
Subrace: 
1: Dark (<module 'Elf-Dark'>)
2: Bright (<module 'Elf-Bright'>)
3: Shadow (<module 'Elf-Shadow'>)
4: Wood (<module 'Elf-Wood'>)
5: Shadowmark (<module 'Elf-Shadowmark'>)
6: High (<module 'Elf-High'>)
7: Winged (<module 'Elf-Winged'>)
8: Fey (<module 'Elf-Fey'>)
9: Sea (<module 'Elf-Sea'>)
10: Wild (<module 'Elf-Wild'>)
>>>

The scripted version is essentially a simple "pipe" of answers to the prompts that come up, so that generating an NPC along particular lines (say, an 8th-level Sahuagin Cleric of the Tempest Domain) can be generated by using the following script:

Ability Scores,Hand,13,11,12,12,13,9,Race,Sahuagin,Cleric,Medicine,Religion,Tempest,Yes
Cleric,Yes
Cleric,Yes
Cleric,WIS,CON,Yes
Cleric,Yes
Cleric,Yes
Cleric,Yes
Cleric,WIS,CON,No

... where each "Yes" at the end of the line is the answer to the prompt "Do you want to add another level?". The above script, then, produces the following output:

>### Name
*Medium humanoid (Sahuagin) Cleric (Tempest) 8, any alignment*
>___
>- **Armor Class** 10 (DEX (+0))
>- **Hit Points** 55 (8d8 + 8)
>- **Speed** 20 ft, swimming 40 ft
>___
>|**STR**|**DEX**|**CON**|**INT**|**WIS**|**CHA**|
>|:-:|:-:|:-:|:-:|:-:|:-:|
>|15 (+2)|11 (+0)|14 (+2)|12 (+1)|16 (+3)|9 (-1)|
>___
>- **Proficiency Bonus** +4
>- **Saving Throws** Wis +7,Cha +3
>- **Damage Vulnerabilities** 
>- **Damage Resistances** 
>- **Damage Immunities** 
>- **Condition Immunities** 
>- **Skills** Medicine +7, Religion +5
>- **Proficiencies** Light armor,Medium armor,Shields,Simple weapons,Martial weapons,Heavy armor
>- **Senses** darkvision 60 ft, blood 20 ft, passive Perception 13
>- **Languages** Common,Sahuagin,Aquan
>___
>***Blood Frenzy.*** You have advantage on melee attack rolls against any creature that doesn't have all its hit points.
>
>***Limited Amphibiousness.*** You can breathe air and water, but need to be submerged at least once every 8 hours to avoid dehydration. For each 8 hours period you do not submerge, you suffer one additional level of exhaustion.
>
>***Shark Telepathy.*** You can communicate simple ideas with sharks. They may understand you, but you have no way of understanding them.
>
>***Blood in the Water.*** Your specialized sense of smell can detect blood. You are aware of creatures within 20 feet that have hit points below their maximum, and can bleed. When in water, this range extends to 60 feet.
>
>***Channel Divinity: Destructive Wrath.*** When you roll lightning or thunder damage, you can use your Channel Divinity to deal maximum damage, instead of rolling.
>
>***Thunderous Strike.*** When you deal lightning damage to a Large or smaller creature, you can also push it up to 10 feet away from you.
>
>***Channel Divinity (2/Recharges on short or long rest).*** See below for the details of each use.
>
>***Divine Strike.*** Once on each of your turns when you hit a creature with a weapon attack, you can cause the attack to deal an extra 1d8 thunder damage to the target.
>
>#### Actions
>***Bite.*** Melee Weapon Attack: +6 to hit, reach 5 ft., one target. Hit: 1d4 + 2 piercing damage.
>
>***Channel Divinity: Turn Undead.***  You can use one of your uses of Channel Divinity to turn undead. Each undead that can see or hear you within 30 feet of you must make a Wisdom saving throw. If the creature fails its saving throw, if it is a CR of 1 or lower it is destroyed; otherwise it is turned for 1 minute or until it takes any damage.A turned creature must spend its turns trying to move as far away from you as it can, and it can't willingly move to a space within 30 feet of you. It also can't take reactions. For its action, it can use only the Dash action or try to escape from an effect that prevents it from moving. If there's nowhere to move, the creature can use the Dodge action.
>
>***Cleric Spellcasting (Wis, at level 8. Recharges on long rest).*** 4 cantrips known. 11 spells prepared. Spell save DC: 15, Spell attack bonus: +7
>
>Spells always prepared: [fog cloud](http://azgaarnoth.tedneward.com/magic/spells/fog-cloud/),[thunderwave](http://azgaarnoth.tedneward.com/magic/spells/thunderwave/),[gust of wind](http://azgaarnoth.tedneward.com/magic/spells/gust-of-wind/),[shatter](http://azgaarnoth.tedneward.com/magic/spells/shatter/),[call lightning](http://azgaarnoth.tedneward.com/magic/spells/call-lightning/),[sleet storm](http://azgaarnoth.tedneward.com/magic/spells/sleet-storm/),[control water](http://azgaarnoth.tedneward.com/magic/spells/control-water/),[ice storm](http://azgaarnoth.tedneward.com/magic/spells/ice-storm/)
>
>* *Cantrips:* 
>* *1st (4 slots):* 
>* *2nd (3 slots):* 
>* *3rd (3 slots):* 
>* *4th (2 slots):* 
>
>#### Reactions
>***Wrath of the Storm (3/Recharges on long rest).*** When a creature within 5 feet of you that you can see hits you with an attack, you can cause the creature to make a Dexterity saving throw (DC 15). The creature takes 2d8 lightning or thunder damage (your choice) on a failed saving throw, and half as much damage on a successful one.
>
>
>#### Bonus Actions
>***Sharks Frenzy (Recharges on short or long rest).*** You can make a special attack with your Bite. If the attack hits, it deals its normal damage, and you gain 2 temporary hit points. These temporary hit points disappear when you finish a long rest.
>
>***Channel Divinity: Harness Divine Power.*** You can expend a use of your Channel Divinity to regain one expended spell slot, the level of which can be no higher than 3.
>

#### Description
***Race: Sahuagin.*** A fish-like monstrous humanoid species that lives in oceans, seas, underground lakes, and underwater caves.

***Class: Cleric.*** Clerics are intermediaries between the mortal world and the distant planes of the gods. As varied as the gods they serve, clerics strive to embody the handiwork of their deities. No ordinary priest, a cleric is imbued with divine magic.

***Divine Domain: Tempest.*** 

***Magic Item: Uncommon Permanent.***

***Magic Item: Uncommon Permanent.***

The magic items at the bottom are entirely level-based, as described by the 5th-ed DMG. As the GM, of course, I reserve the right to amp up (or down) the NPC's inventory as feels right, always keeping in mind, of course, that players love to loot the bodies of their fallen opponents. Spells are something I always figure I'm going to want to decide post-creation, so I don't bother trying to assign (or prompt for) those ahead of time. The rest, though, that's all based on the various benefits of race (and subrace, if applicable) and class/subclass.

Gleaned from where, you ask? Why, from the Markdown describing each race and class itself, of course.

Enter Literate Markdown

Before you start getting excited that I've somehow built a Large Language Model for parsing Markdown and understanding 5th-ed D&D rules, let me cool you off ahead of time: Nope. Instead, I decided that the quickest/easiest thing would be to write those "rules" in Python. However, after a first effort where I painstakingly flipped between Python script and Markdown files, I realized I really wanted the Python script rules to be much "closer" to the code and vice versa, ideally, right next to each other.

A number of years ago, Donald Knuth promoted a concept called "literate programming", in which documentation and executable code were intermingled ("woven") together into a single file. In most modern programming languages, we can have extensive documentation blocks that are essentially well-formatted comments (Javadocs, for example), but I wanted something of the opposite--documentation with well-formatted code blocks.

Well, as it turns out, Markdown supports that, so in theory, I could do my own "Literate Python" if I could somehow get a Python interpreter to skip past everything that wasn't in a Markdown triple-backtick code block, like what we see here:

# [Orc](../Creatures/Orcs.md)

```
name = 'Orc'
description = "***Race: Orc.*** "
type = 'humanoid'
```

* **Ability Score Increase.** Your Strength score increases by 2, your Constitution score increases by 1, and your Intelligence score is reduced by 2.

```
def level0(npc):
    npc.description.append("***Race: Orc.*** ")

    npc.STR += 2
    npc.CON += 1
    npc.INT -= 2
```

* **Age.** Orcs reach adulthood at age 12 and live up to 50 years.

* **Alignment.** Orcs are vicious raiders, who believe that the world should be theirs. They also respect strength above all else and believe the strong must bully the weak to ensure that weakness does not spread like a disease. They are usually chaotic evil.

* **Size.** Orcs are usually over 6 feet tall and weigh between 230 and 280 pounds. Your size is Medium.

```
    npc.size = 'Medium'
```

* **Speed.** Your base walking speed is 30 feet.

```
    npc.speed['walking'] = 30
```

* **Darkvision.** You can see in dim light within 60 feet of you as if it were bright light, and in darkness as if it were dim light. You can't discern color in darkness, only shades of gray.

```
    npc.senses['darkvision'] = 60
```

* **Aggressive.** As a bonus action, you can move up to your speed toward an enemy of your choice that you can see or hear. You must end this move closer to the enemy than you started.

```
    npc.bonusactions.append("***Aggressive.*** You can move up to your speed toward an enemy of your choice that you can see or hear. You must end this move closer to the enemy than you started.")
```

* **Menacing.** You are proficient in the Intimidation skill.

```
    npc.skills.append("Intimidation")
```

* **Powerful Build.** You count as one size larger when determining your carrying capacity and the weight you can push, drag, or lift.

```
    npc.traits.append("***Powerful Build.*** You count as one size larger when determining your carrying capacity and the weight you can push, drag, or lift.")
```

* **Languages.** You can speak, read, and write Common and Orc.

```
    npc.languages.append("Common")
    npc.languages.append("Orcish")
```

My answer to this lies in the fact that Python, as a source-executable language, knows how to take source and turn it into an executable module; if I can figure out how to tap into that, I can write my own skimmed-down "loader" that knows how to parse a Markdown file, skip everything that's not in a code block, and take the result and feed it to the standard Python module-loading mechanism:

def loadmodule(filename, modulename=None):
    def parsemd(mdfilename):
        pythoncode = ""
        with open(mdfilename) as mdfile:
            lines = mdfile.readlines()
            codeblock = False
            for line in lines:
                if line[0:3] == "```":
                    codeblock = not codeblock
                    continue

                if codeblock == True:
                    pythoncode += line

        if SAVEPY != None and SAVEPY in mdfilename:
            with open('./Python/' + os.path.basename(mdfilename) + '.py', 'w') as pyfile:
                pyfile.write(pythoncode)
        return pythoncode

    def builddict(module):
        global classes
        global races
        builtins = {
            "allclasses": classes,
            "allraces": races,
            "traits": traits,
            "spelllinkify": spelllinkify,
            "choose": choose,
            "replace": replace,
            "random": randomlist,
            "dieroll": dieroll,
            "min": min,
            "len": len,
            "print": print,
            "types": types
        }
        for (key, value) in builtins.items():
            module.__dict__[key] = value

    literatecode = parsemd(filename)
    if len(literatecode) > 0:
        if verbose: print(literatecode)
        if modulename == None:
            modulename = os.path.splitext(os.path.basename(filename))[0]
        module = types.ModuleType(modulename)
        builddict(module)
        exec(literatecode, module.__dict__)
        return module
    else:
        warn(f"{filename} has no literate code")
        return None

The heart of this loadmodule function is in the exec call five lines from the end--that's the Python3-preferred mechanism for transforming source into an executable module. Thus, this whole thing is broken up into three parts: the parsemd nested function rips through a Markdown file and keeps only those lines inside the triple-backtick blocks, the builddict function creates a dict (key-value pairs data structure) containing the "globals" that will be in scope when building this module, and the body of loadmodule itself calls parsemd to get the code, builddict to build the globals (with my own set of top-level functions for each module to be able to rely on), creates a ModuleType object into which to pour the executable code, and execs it.

Voila. Literate Markdown.

From here, I can have each race, subrace, class, subclass, and even feats and/or backgrounds be its own Python module. If I then couple some conventions (every module has a top-level "name" and "description" field, for example, along with "levelN" functions that define what that class or subclass does at level N) with this, I have an interesting set of modular... er, modules... that independently describe how they affect an NPC at each discrete level of experience.

And, if I add more classes (unlikely), races (distinctly possible), subclasses (very likely), or feats (also very likely), all I need to do is drop the literate Markdown into the appropriate directory, and my npcbuilder will pick them up automatically. (Note that if a given Markdown file has no code in it, I don't try to load it, and if there's a "SAVE_PY" environment variable set, I save the extracted Python in it for easier debugging.)

Current state and TODOs

The one thing I dislike about this so far (aside from the fact that I really wish that LLM did exist, so I wouldn't have to write out the literate code by hand!) is that now the descriptions are clogged with code blocks that appear in all the HTML. I think this is pretty easily fixable--MkDocs has hooks for "plugins", which can customize the output, so I think I can write a plugin that discards all code blocks (since I don't need them, since this isn't a code project it's a D&D project).

There's a few other things that I find myself wishing I had, such as:

a city-generator tool (in progress) that will use some seed info and randomization to create the outlined sketch of each of the several-hundred-or-so cities I have on the map, not to mention the unmentioned thousands of towns and villages that aren't mentioned anywhere on the map.
Each city, of course, has NPCs at various places within it, so of course I want to call out to my npctool to generate each of those. In fact, I kinda want to figure out how to tie all of these tools together more coherently into a well-oiled package, but I don't quite know (in my head) what that looks like or how they should work together.

But hey, I'm not hired anywhere yet, so there's more to keep me busy.

Own, Collaborate, Inform

Mon, 4 Sep 2023 00:00:00 +0000

tl;dr While having some conversations with a client, we got to talking about teams, processes, and how to partition work. I realized, as I was talking, that while RACI is a reasonable way of thinking about such things, it's a little complicated, and I prefer a slightly simpler model: OCI.

For those who aren't familiar with RACI, it's an acronym that stands for "Responsible, Accountable, Consulted, Informed", also known as a "Responsibility Area Matrix" in some circles. The idea is that for a given process or area of work (dare I say "value stream"), you line the individuals or the teams involved in a column on the left-hand-side, then across the top you create columns labeled "Responsible", "Accountable", "Consulted" and "Informed".

As you go through the different teams, or the different steps in a process, you examine the "players" in the left-most column, and determine whether they are responsible, accountable, consulted, or informed about the work being done. In essence, we are figuring out the degree of involvement around a given process or work, as part of coming to an understanding around who is doing what. (But not necessarily when--that adds a whole new dimension to the problem.)

More thorough explanations can be found on Usemotion and Asana.

My issue comes with some of the separation of the columns; specifically, I think it allows for too much overlap and conflates some concepts. Consider the eplanation of each of the four terms from the Usemotion blog (linked above):

Responsible These are the individuals, pairs, or groups who do the work to complete the task or deliverable. Each task should have at least one responsible person.

Accountable These are the individuals, pairs, or groups who delegate and review a team’s work. They are typically in a leadership or management role and ensure that the responsible person or team knows their expectations and completes the work on time. Each task should have only one “Accountable” person.

Consulted These individuals provide input and feedback on the work being done. They have a stake in the outcome because it could affect their current or future work. Not every task needs a consulted person, but the project manager should consider all possible stakeholders when creating the RACI chart.

Informed These are the individuals or groups that need to be kept up-to-date on the progress of a project but don’t need to be consulted for decision-making. They might include the heads or directors of teams or the senior leadership in a company.

But... who "owns" the thing?

Goals, Microservices, RACI, oh my

One of the things I've discovered in my time managing teams and consulting to executives is that teams (and people within teams) function best when they have clear, distinct goals. Ideally, that should be a singlular noun: goal. When the team or the individual knows what they're doing such that it can describe it in a single sentence (without being a runon!), you have a much clearer sense of who's doing what.

In short, this is called "ownership", and it's a large part of what the whole microservices architectural model was trying to drive towards, what Amazon now refers to as a team being "single-threaded": they own their inputs, they own their outputs, they own the maintenance, the whole nine yards. "No shared anything" is another way of referring to it. It has nothing to do with the size of the service, or the team, but whether they have a clear sense of what they work on.

(By the way, that doesn't line up well to the use of single-function microservices at all well, does it? That's because the first use of the term "microservice" is more about organizational design, whereas the single-function AWS Lambda or Azure Function is more about distributed system design, and we need to stop pretending that the two are synonymous.)

The Case

Suppose you have two teams, each supporting their own service, and both make use of a shared database. Let's call the first team the "Sales Microservice" team and the second team the "Marketing Microservice" team. Clearly both will have need of much of the same data--Sales will need customer information (to know where to ship the product, for example), and Marketing will need customer information (to know if the customer has bought into the latest iteration of our bright and shiny product or not).

The data people in the back of the room are already starting to squirm uncomfortably, because they can see all sorts of opportunities for violations of data integrity: if we don't put this customer information into a shared database, there's all kinds of opportunities for trouble. "If the customer moves, we might send the product to the wrong place!", they'll whimper, "Or we'll send them a coupon for 25% off the thing they just bought. So many ways to embarrass ourselves, we just have to have the database shared between these two teams."

Fair enough. But who owns it?

"Nobody owns it," they respond, "It's a shared database."

OK, but who makes sure it's still running? Who's paying for the rack or the cloud or whatever hardware and software we're using to make use of this database? Because somebody is going to do those pesky maintenance things, right?

The heads are starting to turn, and they're beginning to realize where this is going. (Actually, if they're data professionals, they already know the right answer: The dedicated Database Group owns it!) For any given hardware or software "thing" inside the company, somebody has to own it, or it's going to fall apart eventually.

And let's not get started on what happens when the Sales team wants to break a column or a field or a table or a document up into smaller parts, but Marketing doesn't see the need. One way to solve this, of course, is to put the shared database into the Database Group, and now each team has to bring their requests for modification to the Database Group, who will schedule the modifications to happen, please be sure to allow up to 4 to 6 weeks for us to process your request, and make sure your business case has been approved by the VP of Database Engineering ahead of time, who will need to schedule time with both the VPs of Marketing and Sales as part of the vetting process....

Yeah, now we see why shared databases aren't quite as popular as they once were. We may have traded one problem off for another, but that's a different discussion for another day. Let's try this again: Sales and Marketing each need customer information.

Enter RACI

Traditionally, this is where the business consulting excitedly jumps into the room and starts waving Excel spreadsheets around: "We can do a RACI exercise and figure out who is in which column!" The theory being, of course, that once we've established who is Responsible and who is Accountable, we'll have some clear indications about who is going to do what or decide what.

Except....

Have you ever been accountable for something that you weren't responsible for? As in, your work/outcome/bonus depended on the work that somebody else was doing? This is the classic definition of stress, and in general it works against the notion of "autonomy", a critical aspect to creating a dynamic and viable team. Consider Asana's description of these two roles:

Responsible. This person is directly in charge of the work. There should only ever be one Responsible role per task so you know who to go to with questions or updates. If a task has more than one Responsible person, you can lose clarity and cause confusion. Instead, aim to add additional collaborators as some of the other RACI roles, which can have more than one person.

Accountable. The Accountable person is responsible for overseeing overall task completion, though they may not be the person actually doing the work. There are two ways to assign an Accountable role. Sometimes, the Accountable is the project manager (or even the Responsible, though in that case the person is taking on two different roles during the task workflow). In these cases, the Accountable is responsible for making sure all of the work gets done. In other cases, the Accountable is a senior leader or executive who is responsible for approving the work before it’s considered complete. Like the Responsible role, there should only ever be one Accountable.

I mean, would you want to be the project manager in this scenario? You're Accountable for the work done, but not being Responsible means you're not doing the work, you're just checking up on everybody who's doing the work. And if that Accountable individual is a manager, you have a recipe for micromanagement in a coffee cup.

This does not strike me as a great state of affairs.

Enter OCI

I think the original terms of Responsible and Accountable felt intuitive to the folks who first started using the RACI diagram, but as we've gotten more familiar with autonomous and self-organizing teams, particularly in the view of how we like to organize development teams around microservice architectures, I think we need to revamp the acronym a little. Thus do I present the OCI diagram: Owner, Collaborator, Informed.

Owner. If you own the thing, you own the thing: You are responsible for it, you are accountable to it, and any buck having to do with the thing stops with you. This is your baby, and you get all the credit and all the blame for it. No questions, no exceptions.

This is, remember, what the microservice approach was after in the first place: single-threaded. If I own it, I can make changes to it, because if it blows up, it's on me.

But we rarely exist in a vacuum, which is why we have....

Collaborators. In the case of a microservice or a database, this will oftne be the consumers of the thing. I own its implementation, but I have responsibilities to others, and we negotiate some kind of deal (formally or informally) around what their needs are and the degree to which I satisfy them. They're not being "consulted", which isn't always clear as to the degree of the relationship--do they get to veto an idea? do they get to weigh in on the idea? As a collaborator, it's more clar that they're involved, in some capacity, and therefore they have some say in it.

Informed. Lastly, there's those folks who aren't involved, but need to know because they do some other things around it. Perhaps they consume the API provided by the service, or they kick of a new process every time this process reaches a certain step. And so on.

You can start to see how the OCI approach begins to work for both processes and software: in a customer complain scenario, when a complain comes in, a Customer Service rep often takes the complaint and passes it off to the relevant part of the company. After all, they're Responsible for that which the customer is complaining about. But if we turn this on its head a little, and ask, "Who owns this complaint?" we get to what I think is the better approach: The Customer Service Rep owns this complaint, and is both responsible and accountable for its resolution.

You might well argue that I've changed the focus of what the RACI discussion should've been, and that may be true. Half the time, though, as a philosopher, the goal is to ask the right question--and often enough, I think the right question is, "Who owns this?"

Good fences

One of my favorite "isms" (pithy sayings that denote or describe something I think is an important idea or philosophy) is that "Good Fences Make Good Neighbors".

When I was just starting fifth grade, we moved to one of those half-acre lots that are the definition of the sprawl in LA. One of the very first things we did was to put up a wall between ourselves and our neighbor, Reg. I had no real idea why--Reg was a nice enough guy, very pleasant and chatty (as LA neighbors go, anyway--you generally don't talk much to your neighbors in LA), and whenever a ball sailed over the wall, he was happy to either grab it (if we was outside) or leave his gate unlocked so we could go grab it.

So why the wall, I asked my Dad. "Because good fences make good neighbors", he replied. "This way it's clear which part of the property belongs to whom, and if there's a problem with the sprinkler system, for example, we can know whose system its with and who needs to fix it." But there's a small strip of lawn in the front right between the two driveways extending out to the street, and we didn't run the all all the way down that, why not? "Well...." and then he changed the subject.

Later, when I was given ("gifted", my father said proudly, as he handed off a chore to me that he hated doing) the responsibility of mowing the lawn, I was told that Reg and us traded off mowing that little strip. No formality to it, just if you were out mowing and it hadn't been mowed yet, you just took the ten minutes necessary to mow it, and figured roughly that it would all come out even between us in the end.

(Persoally, I tried to make sure I always started mowing the lawn after Reg did, so that I had that much less chore to do. Pretty petty, but hey, i was 13, and mowing the lawn in 110-degrees-F heat is not my idea of a fun time on a summer Saturday afternoon.)

When I go visit the Midwest or back East (in the US), I see all these homes with no fences between them, and I'm always curious as to who mows what between the two homeowners. I ask folks, sometimes, if I happen to know the people living there, how they know who mows what. Every time, I find that they've answered the "OCI" question in their own informal terms; they usually say something along the lines of, "Oh, I just mow to where I think the property lines are" or "Well, we kinda agreed where the dividing line was." (One friend told me everybody on the block contributes to an enterprising kid's college fund by paying him to mow everybody's lawn--which neatly answers the question of "Who owns mowing this?")

In a business setting, you ask "Who owns this" not because you're looking to start (or end) turf battles between teams; you ask because that way, if I know the ball landed in your part of the yard, I can either tell you (if I think you didn't know already) or I can trust you to toss it back over the fence (if I think you know already) when you get around to it, and vice versa. It's about making it clear where the boundaries are between our teams, so that we're each clear on who's doing what.

And sometimes, the act of going through one of these "ownership"/OCI discussions reveals that there's gaps--and that's the best kind of result, because now we have spotted a hole that we didn't know was there before, and can (imediately!) move to address it. The act of drawing the fence means checking the property lines, and every time you do that, ou discover something new about your property, your neighbors', or both.

Good fences make good neighbors.

You don't want passion

Sun, 25 Jun 2023 00:00:00 +0000

tl;dr Recently, while sitting in the speaker ready room at VSLive Nashville, I had a lovely conversation with Angela Dugan and Amber Vanderburg about something I've been saying for a few years now: I don't want to hire passionate people, I want to hire professional people. Turns out, Harvard Business Review agrees with me.

Passion is, by definition, illogical, and often leads people to act without thinking--we call them "crimes of passion" for a reason. Mostly, though, the distinction here is that the professional will do the parts of the job they dislike just as they will the parts they like--the professional knows it's a job, not a life choice or an identity.

Buy vs Build... Over Time

Wed, 22 Mar 2023 00:00:00 +0000

It's the age-old question of our industry--do we buy something to take care of a need, or do we build the thing ourselves? No matter which way you go, it seems like somebody comes around later and makes it clear you chose wrong. The deep secret, however, is that no matter which way you choose, you're likely to be wrong later, not because you chose wrong, but because the context of the choice changed.

This particular post, as is common, was inspired by a LinkedIn post:

At Tonic, we once tried to build our own integration engine. It looked simple enough. And it worked. The only problem was what we built lacked a UI. And our goal was to separate integration work from engineering work (can’t do that without a UI). To achieve that would have needed to rebuild the equivalent of MuleSoft or Coreppoint. After a year, I made the team do a formal evaluation of external options and we picked an integration engine and sunset our internal product. post

I know a lot of engineers will look at this and blame the author for having chosen poorly. I argue that they chose correctly.

Buy vs build

It seems like such a (relatively) easy choice: If we buy something, we don't have to build it. Vendors provide support, including bug fixes. This saves our engineering hours for the real problem we're trying to address. Plus, the purchased tool will almost-always be more polished and feature-rich than what we build. But buying something costs money! And if we build something, we can tailor it specifically to our needs, with no need for workarounds and weird adapters that turn an otherwise straightforward process into a Rube Goldberg machine.

So, to decide which is the "right" choice, we need only figure out:

if the vendor product has the features we're looking for, for at least the near- and short-term future,
if the vendor product is at the right price point,
if the vendor product will be supported (and how) for the length of time we need (in other words, is the vendor likely to go bankrupt),
and if the vendor product can integrate into the rest of our software development fabric (processes, tools, languages, databases, etc).

And yet, stories of having made the wrong decision are so, so common. Why?

Decisions are always made within a context; most of the bullet items above don't really address the context of the decision directly, but rely on that context indirectly. Price, for example, can be a very complicated thing, including up-front and recurring costs, but also support and maintenance investment of time and energy. (Ask any IT leader about the costs of installing an Oracle database on-premise, and you'll get an earful.)

In the beginning of a project, we may not have enough experience or we may not have the bandwidth to really understand the nuances of the choice (whichever it is!). If the project doesn't go very far, then a "buy" decision was probably unwise, because now we're left with a product we don't need, or it provides capacity and features that we've never taken advantage of yet still paid for. If the project suddenly sees huge growth and potential further growth with additional needs, then the "build" decision was a bad one, because now we need those features and polish.

Rather than lament the "wrong" decision made a year ago for a buy-vs-build decision, accept that you can only make the best decision you can given your current context, and accept that it's entirely possible--in fact it might even be likely--that the context will change and lead you to making a different decision down the road.

And that's OK. Because we're always making decisions without perfect information (in this case, we can never know the future context or how it will differ from the current one), we have to accept that some of our bets won't pay off the way we thought. Waiting for the context to change, however, is never a great answer, because the context is always changing.

Opportunity Cost

When I was a younger man, back in college when it was much more common to put your own desktop computer together yourself out of parts, friends and I would debate whether we wanted to buy a new computer rig. New motherboards were coming out, new graphics cards, faster hard drives, you name it. One friend always held off, saying, "There's a faster version coming out in a month, and if I buy now I'm just going to regret not having waited." I said the same for a while, until I realized that I was never going to buy anything, because there was always going to be something faster coming down the pipeline. Yes, Apple has released the M2 MacBookPro--but if we wait until next year, they'll have an M3 MacBookPro, and it'll be so much better because (something something something).

In economist circles, we call the cost of making a decision and losing the benefit of choosing otherwise the opportunity cost. Consider a simple decision: My wife wants to go to a Korean BBQ joint that just opened up, but I want to go to the steakhouse I like. If I choose to insist on my choice, the opportunity cost of not choosing to go to the Korean place will be (a) I won't discover that it's actually a pretty good place, (b) I won't get a chance to watch K-Pop band music videos, and (c) I may end up sleeping on the couch tonight. (Of course, my wife will be paying on opportunity cost of not choosing to go to the steakhouse, but that's a different conversation, and one she has to have with herself.)

In the case of buying computer hardware, the opportunity cost of waiting is to spend a whole year continuing to work/play on the even-older rig that I was looking to upgrade or replace. But opportunity cost cuts both ways! Choosing to "buy now" means that I may have to watch with a certain amount of envy as my roommates get to play a game that's even faster on their upgraded hardware, at least until I can do my upgrade, and then they get to salivate over my new purchase.

Within the "buy v build" decision, the real focus should be on opportunity costs: If we do build something now, what are we paying in opportunity cost? If we do buy something now, what can we not buy later, because we spend the money? For a lot of startups, for example, money is extremely tight, so that "what can we not spend money on" becomes a very important factor. For a lot of VC-backed startups, on the other hand, money isn't the critical-path resource, and now the real conversation becomes "what engineering hours would we not have, preventing us from building out some feature or another" if we build.

Summary

The buy-vs-build conversation, as you can see, is never just about the vendor product feature set and the time/energy it would take engineers to build the same. It's about the context of the situation, and the opportunity cost we pay when we make the decision. Throw in the fact that the project's or company's context can change (sometimes drastically and sometimes instantaneously), and you have a recipe for a basic realization: You're never going to be always right. Or, to be more accurate, the decision you make today has absolutely no guarantee that it's going to be the right decision the further away we get from the decision point. The best we can do is make the decision for today, accept that we pay an opportunity cost in doing so, and keep an eye for when the context has changed enough such that we need to consider revisiting the decision.

AI-generated Applications

Fri, 17 Mar 2023 00:00:00 +0000

tl;dr With the recent spate of AI-driven engines (like ChatGPT) that have been shown to be able to build applications from a complete spec, a lot of folks are having a bit of an existential crisis (or else gleefully pointing out somebody else's existential crisis). Nowhere is this more pronounced than in tech--the very industry that gave us ChatGPT may now be eating itself for breakfast! (Click here to find out more!)

As is common, a post on LinkedIn triggered this thought; this time, it's from my friend Aaron Erickson, who wrote:

"I give it 3-5 years before AI makes low end "give me a spec and I give you code" style contracting that job shops do utterly obsolete. If all an agency does is write code to a spec, your business model is about to become irrelevant." --post

This was in turn in response to an earlier post which said, "Last night I used GPT-4 to write code for 5 microservices for a new product. A (very good) dev quoted GBP 5k and 2 weeks. GPT-4 delivered the same in 3 hours, for USD 0.11. Genuinely mind boggling."

No, it's really not that all that mind-boggling. It's an indicator of something I've been believing (and speaking about) for many years: Our tools are not elevating our game.

Levels of Abstraction

Part of the goal of any programming language is to take abstract concepts (like functions, procedures, objects, and so on) and transform them (compile them) into low-level constructs that a CPU knows how to execute (machine code). Different languages choose different levels of abstraction as first-class concepts; Smalltalk chose a very high level of abstraction (objects and messaging), whereas C chose a very low level of abstraction. In fact, C was originally intended as a "portable assembler" for most UNIX machines--hence it's heavy use of pointers and memory locations as part of its "surface area" when programming in it. It's why C was known as a "systems" language--you could build operating systems with it.

C++ chose to bring object-orientation to C, but for the most part, Bjarne Stroustrup wanted to keep the flavor and abstraction level of C intact. "If you don't use it, don't pay for it" was a common theme in the days of C++, and so for example, compiled classes wouldn't contain a v-table unless the class had an actual virtual method in it somewhere. References were pointers but without the ability to pointer-math them. (It's OK, you could always convert it back to an address with a cast.) And so on.

Other C++-inspired languages, such as Java or C#, followed the C++ model pretty closely, but even languages like Python adhere pretty closely to a lot of the things C++ and C held up. Slowly they've been taking steps away from the low-level nature of early C/C++ code, but still there's a ton of features in each of these languages that reflect an ancestry of a "portable assembler".

And we're using those languages to write microservices and full-stack applications?

Degrees of Differentiation

Meanwhile, in the world where microservices are being built, we have dozens of languages and frameworks that all "make it easier" to write microservices, and almsot without exception, most of these microservices are all doing roughly 80% the same thing: take an incoming API request, do some logic to it, store it into a database behind the service. We debate over which languages to use, which frameworks to use, and which databases to use, but fundamentally it's all the same thing: we're rebuilding the same stuff with some (maybe) different "business logic" scattered in there somewhere.

The desktop, mobile, and web applications we build, meanwhile, all very much do the same things and display the same things and validate the same things. And in each case, we may argue over which web framework to use, which unit testing tools to use, and so on, and so on, ad infinitum.

A long time ago, early in my career, I went to work for a company that had a team building an application to help nurses in a call center take phone calls for a variety of things: referrals to doctors, general guidelines around whether to go to the ER, questions about their insurance plan, and so on. This was the first generation of a now-ubiquitous feature for insurance companies, but the Director of Development summed up the C++/Windows 2-tier client-server application and Oracle database very simply: "It's just dialogs over data."

Folks, it's almost all just dialogs over data. The dialogs have gotten more sophisticated and have different kinds of input to handle, but fundamentally, every health care, every financial, every e-commerce app... they're all just "dialogs over data".

And when you see that, you begin to realize that maybe instead of trying to write the next amazing object-oriented language that incorporates this feature from someplace else or adds that idea from some other language, we need to start thinking about how we elevate the abstraction level entirely.

What about....

Think about this for a second. Imagine we want to build a system that allows attendees to sign up for talks by speakers at a conference. The object world spots three clear objects there: Attendee, Speaker, Talk. Three classes, with properties and methods. A few restrictions, obviously, such as limits on the number of attendees in a given talk, or that Speakers can "share" a Talk (give it together), and so on. We could easily come up with a half-dozen or more "business rules" without thinking about it too hard, and I'm sure we could get that number up to two dozen or more if we wanted to add a few more concepts (like Room, or prerequisite Talks before you can attend a particular Talk).

How long would it take you to build this application, from scratch, as a "front-end" (I'll let you pick web, mobile, and/or desktop) talking to a "back end" storing data in a database?

And yet, we've built something like this dozens of times before.

And we're wondering why an AI engine, which does really well keying off of thousands of examples of near-exactly-the-same-thing, knows how to build one of these?

Surprise: This is a large part of the reason low-code/no-code tools are creating a huge surge in interest right now. The low-code/no-code tool takes an opinionated "line" through all the different layers in an application stack, and suddenly you can turn out an application with some slightly-adjusted business logic that is ready to go in moments. How hard would it be to see a "low-code/no-code microservice" tool before long? Honestly, I'm kinda surprised that it hasn't already made it out the door yet.

Steps forward

Aaron's right--if we don't elevate our game, we're going to find that AI can churn out these kinds of applications with ease; in fact, it'll probably get so good it'll have one meta-pattern stamped somewhere in its silicon axons and dendrons and it'll just "fill in" the salient spots. All the way up until I can't anymore, I mean.

One of the more interesting exercises to play is to take a look at Naked Objects. It's not a fun place for programmers to hang out sometimes, because it suggests that somebody could build an app with this in a fraction of the time, feels like low-code/no-code, what happens when we need to customize something, and so on.

But let's stop for a moment and consider: Naked Objects are a re-imagining of what Smalltalk (and Self) tried to build, which was an entire environment that contained front-end all the way through to storage. Gemstone, for example, had automatic data storage baked in to its Smalltalk development environment, so building an application there meant you had literally the entire "full stack" in one place, ready to go.

Ironically, so did Microsoft Access. And Visual Fox Pro.

In a previous post, I talked about what I want to build, and this kind of an environment is high on that list of desirables. I don't think it needs to be an industry-spanning "take over the world" kind of language/tool/environment, though; I think it can actually be built out of existing off-the-shelf frameworks and tools, tuned to a particular corporate domain, allowing for developer and non-developer alike to create applications and extensions to the core platform.

It's simply a question of when we will stop chasing after assembly language, and genuinely embrace a higher level of abstraction.

Embracing "Old" Tech

Wed, 1 Mar 2023 00:00:00 +0000

tl;dr It's a common problem, and one we've seen in the industry several times--a technology (language, framework, platform, whatever) is making a significant, backwards-incompatible change, meaning any code written in "v.N" (the current, stable version) will require some coding changes if the project wants to move to the next "v.N+1" version. During v.N+1's incubation, everybody gets excited about the new features, and nobody wants to start a project on v.N. And that's a flawed take.

This particular post was inspired by Cory House's Tweet below:

I feel like @nextjs is in a weird spot at the moment.

I can’t recommend Next 12 because it’s “old” tech. You’ll want to convert to 13 as soon as it’s stable.

And, I can’t recommend Next 13 because it’s in Beta.

Thus, I can’t recommend Next at all until 13 is stable.
— Cory House (@housecor) March 2, 2023

... but to be fair to Cory, this is hardly new. I saw the same thing happen when AngularJS became Angular, when C++ broke from C, and quite frankly it's the discussion every time Microsoft announces a new presentation technology (WinForms vs WPF vs MAUI vs oh hell I've lost track where we are now).

The conundrum is clear: Do we actually want to create a new project using "old" tech? And my response is equally clear: Of course you do.

The Fallacy of the New Tech

A long time ago, I blogged about the Fallacies of Enterprise Computing. The very first fallacy is this:

New technology is always better than old technology

And I tried to be pretty subtle about it, too:

"This cannot be emphasized enough: This is fallacious, idiotic, stupid, and brain-dead."

Well, subtle for me, anyway.

I'll leave it to you to read the full text of the fallacy, but to be fair, there's a slightly different scenario here.

The Fallacy of the New Version

First off, let's be really clear--anything that causes the reaction Cory describes ("Thus, I can't recommend Next at all until 13 is stable") is something that every framework/language/platform author desperately wants to avoid, because it hurts their adoption. This is partly why these authors avoid backwards-compatible changes entirely, and ironically, it hurts their long-term viability.

Look at Python2 vs Python3, for example. Python3 has cleaned up several significant contradictions in the language that Python2 was stuck with, and frankly, despite all the hullablloo around it, it was a necessary shift. (And yes, there's been a fair amount of words written on that transition.)

But in the long run, Python3's language model is much easier to understand for those changes, and overall it hasn't hurt Python3's adoption or use, because nobody stopped using Python2. (In fact, you can argue Python has the opposite problem.) It's clear that people can, and have, continued to work with the old version of Python and been quite successful--in fact, it's not unreasonable to suggest that this has been true for any v.N-vs-v.N+1 debate, at least the ones that I'm aware of. (Yes, Virginia, there are new WinForms apps being built even today!)

Why should Cory keep recommending and using Next 12?

The New Version Isn't Guaranteed to Work

Any time a tech is making a backwards-incompatible change, there's a strong chance (guarantee, really) that the new version is going to have bugs. Subtle bugs. Nasty bugs. Bugs that will only be found as hundreds/thousands/millions of users will find. There's a reason why my dad, the pilot, said "You never want to fly the model A of any airplane" and in our industry we said "Never use any Microsoft product until version 3".

Waiting for Next 13 to be stable could be an unacceptable wait time.

Time Stands Still for No Woman (or Man)

How long will you need to wait for the stable version to be ready? Will your customers/clients be willing to wait that long? Chances are good that putting a "hold" on the project's development until that threashold is crossed is an unacceptable delay. Hey, even if it's six months, you risk the budget allocated will go away, and let's not even imagine for a moment that you can have a dev team sitting on their hands all that time. They will get allocated to a different project, and that one will very likely not be willing to send them back when you're finally ready.

And what's the cost of starting the project now with Next 12? "We know we will be taking on technical debt--we'll have to rewrite parts of the code when the new version comes out."

You're always going to need to rewrite parts of the code. Get over it.

The Fallacy of Writing It Once

Here's a fact: things change. Technologies will grow and evolve. Platforms will emerge, platforms will go away. Languages will be new, mainstream, then legacy. Frameworks... well, we all know the lifecyle of the Javascript framework closely resembles that of the fruit fly. The point is, the things we use to build applications are in a constant state of evolution and change.

And even, if by some miracle, the development world (particularly the open-source part of it) were to suddenly stand up, declare "We're finished! It's all done! There's nothing new to add!", the business world doesn't. New competitors, new businesses, new ideas, new partners, new everything means that as soon as you ship, your legacy begins.

Michael Feathers is quoted as saying that legacy code is any code which is untestable; personally I think that legacy code is any code that's been deployed. It's all legacy code; the key question is how much "legacy" are we talking?

Or, in other words, what's the technical debt for this particular release?

Technical Debt is not Always Bad

We developers have come to regard technical debt as universally bad--there are some who will be quick to point out that ANY amount of technical debt is bad and therefore to be avoided like the plague. This is, in many respects, the implicit heart of what Cory's tweeting about above: "I know that migrating from Next 12 to Next 13 will require change, which means you will begin your project with technical debt even from the first line of code. Therefore, I cannot in good conscience recommend a path that incurs technical debt."

This is silly--every project starts with technical debt, and it's my assertion that you cannot build software without incurring technical debt. Some debt you won't discover until later in the project, but it's a fallacy to assume that any deployed software--even "Hello world!"--has no technical debt. There's always something.

And in the words of eXtreme Programming and the agile movement that followed it, "embrace change".

A Use-Now, Upgrade-Later Strategy

To Cory, I would say this: Go ahead and use Next 12. Even if you end up having to rewrite a significant chunk of the code, it's a known tech debt that you capture, document, and even perhaps start some parallel development against betas of Next 13 to make sure you're not coding/architecting yourself into a corner. But the show must go on, and so must the development, so use Next 12. Maybe consider some abstractions to hide the pivot points that the 12-to-13 migration will require. Maybe you find that 13 is more stable than you thought and you can embrace it even before it's shipped in a stable form. Any way you slice it, you are moving forward, a lot of code will be built that won't be affected by the migration, and you will even be able to ship something before Next 13 comes out.

And if you find, during that parallel 13-based development track, that 13 isn't stable at all, and you want to wait until 14 or 15, you haven't crippled yourself by waiting.

But What About Alternatives?

"Hey, you're forgetting that we don't have to use Next! We could use...." Yes, you can, absolutely. But that takes the conversation into a different dimension: Should we use Next 12 or should we use (some competitor)? That's an entirely different question to what Cory was tweeting originally (v.N vs v.N+1). It's a fair one, but changes the basis of the conversation.

To be fair, it's not an unreasonable one--perhaps there's a close competitor that can do everything Next 12 can do. In which case it's a near-guarantee that you won't even need to think about upgrading to Next 13, which could very well be a win! Yay!

But to avoid using a thing at all because a new version of it is coming out soon... that's a mistake.

What has tech brought us?

Tue, 21 Feb 2023 00:00:00 +0000

tl;dr Tech has brought us many wonderful things, but if we take a step back from the hustle and bustle of recent releases, media excitement, and vendor advertising, we begin to realize that the tech-fueled vessel sailing to utopia that we were ready to embark on just a decade and a half ago... has kinda sunk.

Recently, a LinkedIn post captured an accurate (if depressing) realization about the technology industry as a whole; I've excerpted parts of it in order to weigh in:

As far as I can tell, the only improvement in my quality of life resulting from the massive improvements in tech is that I can unlock my door and turn on my lights from anywhere in the world using my phone. Which is cool!

Let's stop and consider the past decade: 2012 - 2022. There's been no real sea-change in hardware, other than "smartphones got faster CPUs, more storage, and more cameras." Laptops are basically laptops, servers are basically servers, and while both got faster and more (RAM, storage, bandwidth, whatever), they're entirely unrecognized from what they were ten years ago.

Consider the app stores: What is in here that dramatically changed anything? Most of the wildly successful apps are directly tied to commercial consumption: Everybody has a rideshare app (Uber, Lyft, etc), everybody has a food-ordering app (Doordash, Uber Eats, etc), and pretty much every restaurant or grocery store has some kind of app, either in-house or a service, that coordinates ordering and "take away" service.

But what about medical apps? Nutrition apps? Apps that will help you navigate financial problems? A few companies are promoting their apps as doing such, but they inevitably get sucked into the commercial vacuum and end up being a mouthpiece for a retail entity--Truebill, for example, got sucked into the vortex that is the Rocket Mortgage family of companies, and was rebranded as Rocket Money (or something).

Consider the FAANG: Google, for example, has released a number of new things in the last decade, which, by the way, have almost universally been unsuccessful, meaning Google's only real success stories are Google search, GMail, and Android, all of which were either before 2012 or right close to it. Nothing Google has done in the last decade has left a mark.

Microsoft has Azure, but each successive wave of Windows? Meh. Office? Meh. Even their principal developer stack, .NET, hasn't made any significant definitive mark except with the people already using it.

Netflix? Canceling their custom-created content as fast as they release it--but the basic service is still the same. Facebook? "The Metaverse!" and 'nuff said about that one.

Apple? MacBooks are now running ARM CPUs which is pretty much just "more" like we said earlier. Phones are still handheld touch computers; Pads are still larger handheld touch computers. Heck the largest iPad is essentially a touch-screen laptop just without the keyboard. They've even discontinued the iPod entirely. Arguably, their most revolutionary product in the last ten years has been... earbuds. Wireless earbuds.

Social media? I think by now we've come to realize that social media has as many nuclear-bomb-yield minefields as it does benefits.

The proposition that tech improvement = humanity improvement was always dubious and extremely risky. In fact it’s quite likely that the opposite is true, tech improvement = humanity regression.

In particular, tech utopia was always fake news. Crypto was never going to bank the unbanked. FAANG was never going to solve the energy crisis or make our economy more inclusive. Weird little app X from San Francisco transformed nobody’s life outside of the equity holders for more than a few minutes.

It is true social indicators are at lifetime highs but they’ve barely moved in these last 15 years of tech’s golden era, and many indicators have steadily trended down since November 2016. Tech utopians claiming credit for any of that progress sounds like theft, and shirking any responsibility for its decline abdication.

Consider this: Most of us--particularly the Millennials and the GenZers--are actually less happy because of our phones. We spend more time looking for "likes" online than we are in talking to friends at a restaurant about what we like. We've substituted social media for actual socializing. Tinder taught a whole generation how to "swipe right", Netflix brought us "Netflix and chill", but neither actually helped a generation find actual love or happiness.

And it wasn't for lack of investment....

Amazon’s operating expenses for the last 3 years totals USD 1.3 TRILLION. What do we have to show for that spend? Weird crappy speakers cluttering our houses and deliveries of shit we could’ve just as easily bought at a store. ... For comparison, Amazon spent almost as much in the last 3 years as America spent on national defense during the lowest 3 defense spending years of the Obama administration (average 500b).

And that's only Amazon's investment. That doesn't begin to consider what Google has thrown into its products--how much has Google thrown at self-driving cars, trying to one-up Tesla? Microsoft has at least avoided THAT mess, but has its own AR/VR/XR (extended reality) mess that it's starting to draw down from. Which is nothing compared to Facebook's mess. Or the ongoing dumpster fire that is Twitter--but we already talked about social media above. (44 BILLION USD, all so that Elon Musk could make sure to show up in everybody's timeline--so long as you didn't block him, which a lot of us did a loooooong time ago.)

Tech will improve humanity if we build tech for the purpose of improving humanity AND validate its delivery when it’s done. Tech will not improve humanity if we build tech for any other reason, or build it for the right reasons without associated social indicator metrics to measure against.

Here's a question for us all going into 2023 as the layoffs put a LOT of tech talent onto the street with nothing to do for the next few months: What could tech do? What kind of things could a technologist, armed with some free time and some money, build to make the world a better place? Or even just make a small sliver of the world a better place? How could we leverage what we know about building software platforms, and turn that into societal good instead of just more dollars/Euro/yen?

If I’m wrong about tech utopians, I’d expect the mass layoffs to lead to a cascade of new products built for social good. If I’m right about them, expect more tech that is very useless but very attractive for VCs to one up each other.

I hate to be a betting man betting on either of these outcomes, but here's my hope: That the technologists who found some degree of power and flexibility in the pandemic, and refused to let it go when their companies ordered them back to the office, will now find the means to explore some ideas that aren't just "X, but for Y" kinds of tired VC pitches. Don't build "Uber, but for pets". Build a platform that allows those who are in need of social services to find the nearest one that offers the aid they need. Don't build "Doordash, but for rock-climbing equipment". Build a platform that makes it easier for teachers to wade through YouTube to find actual educational supplemental content (and deliver it without the YouTube algorithm leading kids to white supremacy videos).

Let's build some tech for social good.

Not just greed.

A DevRel Activity Pattern Language

Mon, 20 Feb 2023 00:00:00 +0000

tl;dr Developer Relations consists of a number of different activities that accomplish different things, but nobody's ever really sat down and talked about when to use which activity over another. I thought it was time to bring all of them together into a single, cohesive "whole". Being a fan of patterns, I thought the best way to do that was to weave them together into a pattern language.

If you're already familiar with pattern languages and how to use them, or you're just impatient, go ahead and jump to the language. If you're a more patient sort, or if you're not sure how to use pattern languages, read on. (And if you come back here after taking a quick look at the catalog to see a little bit more about how to use it, I promise I won't tell anyone.)

What is a pattern language?

Wikipedia defines it thusly: "A pattern language is an organized and coherent set of patterns, each of which describes a problem and the core of a solution that can be used in many ways within a specific field of expertise. ... A pattern language can also be an attempt to express the deeper wisdom of what brings aliveness within a particular field of human endeavor, through a set of interconnected patterns. ... (O)rdinary people can use this design approach to successfully solve very large, complex design problems."

In this scenario, the "very large, complex design problem" is that faced by a Developer Relations team, particularly its leader (team lead, Director, or VP): Of the large number of different activities we can do, which ones best accomplish our goals? Where can we put our precious resources (time and money) to get the most return? How best do we interact with the developer communities we serve (for the particular definition of "best" we care to define)?

In essence, the pattern language offers at least a modicum of structure: to conduct a Developer Relations campaign, first identify the persona of developer you seek to reach, the results (metrics, perhaps?) that will tell you you have reached them, and the activities that will help you best achieve those results. (This is a pretty high-level oversimplification of running a DevRel organization... but not too high-level, and not too oversimplified, I think.)

OK, what's a persona?

In a nutshell, your target. Your audience. Your chosen developer demographic. It goes by many names, but in essence, it's a description of the kind of developer you're trying to reach. It may be easier to explain by example. Imagine you are working for a NoSQL database company, one that historically has been used for one purpose (perhaps as a caching layer) but has recently made some technical changes that enable it for use in a variety of other scenarios. As the VP of DevRel, your job is to bring the idea of using it in those other scenarios to light within the developer communities.

Consider these personae:

Tina is 25 and a front-end developer at ACME Manufacturing. She knows Javascript (React, mostly), CSS, and HTML, and works on software that provides UI to the various services that manage ACME's supply chain system with their partners. She has been coding for two years, having graduated from a coding bootcamp after college, and she has recently started dabbling with the use of document-oriented NoSQL databases over HTTP from her front-end applications.

Abdi is 32 and a back-end developer at Netflix, where he focuses primarily on building ultra-reliable systems that never crash--or when they do, recover nearly instantly. He is a CS graduate from the University of Washington, been building back-end systems since he graduated, and spends nearly as much time reading about enhancements to the Gossip protocols as he does tips on how to better program in Rust (which is what he uses on the job).

Shawon is 45 and runs an engineering team at Publix Markets (a grocery store chain). The team builds software to help the store manage its inventory and supply chain, and its most recent "big project" was the pandemic-inspired "grocery delivery service", in which customers can shop online, have store employees pick out the relevant items, and either have them ready for fast pick-up at the store location, or use a delivery driver to bring them to the customer directly. Shawon's background is as a former developer (C++, back in the day), but she hasn't touched code directly in ten years.

Which of these is more likely to be interested in your messaging, enough to take a harder look at your product/service? Now, where does that individual go for information? Which developer portals, which conferences, which magazines (print or online), and so on? Establishing your target persona (or personae, there may be several) helps streamline the decision-making process around "which developers" you want to reach.

Note: While it might be tempting to "re-use" personas like the ones above, don't. Take the time to build your own--it serves as a useful reason for the DevRel team to get into a room with all the other teams (Sales, Marketing, Product, Engineering, whatever) and talk about exactly who this product/service is for. Some of your peer teams will have already done this exercise, some won't, but everyone--including Engineering--will benefit from the time spent building them.

Wait, so I'm supposed to use all of them?

Oh, Lord, no! The pattern language is an attempt to categorize and summarize all of the different kinds of activities, but it's in no way intended as a "check every item off on this list" kind of collection. In fact, I'll argue that even trying to use them all is itself a huge red flag! Look at your goals, look at your available resources (bandwidth and budget), and make deliberate choices selectively that help get you closer to your goals.

Keep in mind that most DevRel team strategies will want to make use of several activities simultaneously, so choose ones that have a good mix of reach and interactivity; Conference Sessions have good interactivity, but limited reach, so consider pairing them with Live Streaming to boost reach, and perhaps advertise both using Social Media. And so on.

So... show me!

The DevRel Activity Pattern Language is not "finished", in that no pattern language is ever really finished--there's always realizations and/or innovations that can be captured, analyzed, and categorized. Pattern languages are intended to be somewhat informal, yet hopefully comprehensive.

Also, if you've done DevRel for a few years now, you may find yourself a little underwhelmed. It is not uncommon for people who have "lived this" for years to look at a pattern language and growl, "But I already know all these things! I've done this before!" The same was true of the original Design Patterns book thirty years ago.

How do we use this?

What a pattern language does is provide us with a unified lexicon--a glossary--that we can all agree on without having to define our definitions every time a new person joins the meeting, as well as a framework by which to think about the choices we make and how they might easily work together with other choices and/or directions. Additionally, it can help give newcomers to the field a "leg up" on understanding the breadth and depth of the space, and provide a reference tool as they wade through this new space we call Developer Relations. And, honestly, I'm looking forward to using this as a reference when hashing out my next DevRel team strategy--and I'm hoping you will, too.

What if I'm not the DevRel leader?

There's a couple of ways it can still be useful: as a list of all the possible things you could be doing, so that you can suggest a few during team meetings; as a list to discuss with the rest of your team so that you can have more focused conversations about goals and objectives; and maybe even a way to think about your own activities when writing up your accomplishments for the year/quarter/resume.

Wrapping up

If you find it useful, I'm glad; if you don't, I'm sad, but thank you for your time anyway. And if you're really excited by this, perhaps you want to consider contributing some time to it by dropping me a note with your feedback? (And yes, I'd love to consider your company hiring me to run your DevRel org!)

Thanks

Much thanks to Chris "Woody" Woodruff, Scott McAllister, and Matthew Groves, for their feedback, ideas, challenges, and suggestions.

When to Command (and When to Collaborate)

Mon, 6 Feb 2023 00:00:00 +0000

tl;dr Managers and leaders are often told contradictory things when discussing how to interact with their teams, and those new to the role often mistake collaboration for abdication of authority.

Before we dive too deep, let me make one note about glossary here: Leaders are those who are in a position of informal authority, whereas managers are those in a position of formal authority. In other words, a leader might recommend somebody be promoted, or fired, but only a manager can actually provide the promotion or start the termination. The reason for the difference is clear--where leaders may have some kind of popular or moral position, managers have accountability to their teams. A leader might not get promoted when the people who follow them aren't successful, but a manager can (and does) get fired.

(If you think of the term "manager" as a perjorative and/or insult, implying that managers are just shufflers of paper--figuratively or literally--then we have strongly different definitions.)

Team Interaction

As mentioned earlier, new managers/leaders are often given conflicting information regarding what the role of management/leadership is like. Perhaps most famous (and most widely misunderstood) of those aphorisms is Harry Truman's "The Buck Stops Here". It's often taken to imply that Truman was the one responsible for making all the decisions, referencing a line from one of his speeches: "The President--whoever he is--has to decide. He can't pass the buck to anybody. No one else can do the deciding for him. That's his job." In truth, it actually meant that he held himself responsible for those things which happened on his watch, as evidenced by another speech: "You know, it's easy for the Monday morning quarterback to say what the coach should have done, after the game is over. But when the decision is up before you -- and on my desk I have a motto which says The Buck Stops Here' -- the decision has to be made."

What Truman understood, as most successful Presidents have also figured out, is that while the President will receive lots of insight and advice from a whole army of people much smarter than he (or she, in the future) will ever be in particular subjects, the President is ultimately the one that must "own" the decision. That doesn't mean he is responsible for coming up with all of the details of the plan; far from it.

Usually, during the "normal" course of events, analysts within the various Departments of the United States government will write "position papers", describing the benefits and drawbacks of a particular proposal. Analysts are constantly at work writing these papers for a variety of positions, some real and some hypothetical. (Military planners do the same, often preparing for situations that may seem nonsensical in the moment--how might the US Army invade a friendly nation?--but then become useful when the context of the situation changes--such as when the US invaded Panama during Manuel Noriega's rule.) When the President needs to make a policy decision around a particular issue, position papers are requested or retrieved, and the authors are sometimes asked to present their paper to the President's staff or to him directly. The staff can debate, ask questions, float ideas, and do all the things that we've come to expect of any sort of conference meeting, but in the end, there's only one voice that truly matters: the President's.

The President of the United States makes the decision on the policy, and relies on his staff to carry out the execution of that policy. If that policy accomplishes its aims, the President reaps the reward of having made the "right" call. (Except from the opposing party, who will take great pains to point out all the ways that call led to undesirable consequences.) If that policy has negative side effects, or the policy fails to accomplish what it intended to do, the President has to defend that decision in the press to his constituents (and to the opposing party again).

He "owns" the decision: He makes it, and then owns the ramifications of that choice.

Ownership

This notion of "ownership" is frequently the hot potato of corporate communication. It is, I believe, the main reason that so many people want nothing to do with leadership or management roles--it's one thing to be responsible for your own work; to then be asked to make decisions for and about others, that scares people. When you couple that fear with the accountability that comes with making a decision, that drives many away entirely.

You can be many things in Western corporate life, but one thing you cannot be is wrong.

As a manager on a team, you own the decisions made by that team. The buck stops with you: whatever happens with this decision, you are the one that must answer for the consequences. If the decision leads to positive outcomes, you get to claim the credit. If the decision leads to negative outcomes, you must accept the blame.

Ownership is not unilateralism

Because of this, many new managers often feel that they are the unilateral decision-maker. Any decision must be their decision. Because, after all, if the team's performance reflects directly on the manager, and the team's performance is rooted in decisions the team makes, then the manager has to be involved in every decision.

Thus does the new manager descend directly into micromanagement, and finds simultaneously that (a) they have zero time to do anything but make decisions, and (b) their team feels entirely powerless and meaningless. Employee surveys and feedback tell the new (micro)manager that their team lacks innovation or enthusiasm--which only reinforces to the micromanager that the team needs to make less decisions, because if they're not doing well, how can they be trusted to make good decisions? The spiral continues until the team breaks somehow--either team members are fired, or quit, or the team is broken up, or the manager is fired or (worse) promoted.

Not falling into this trap is the first bit of advice given to new managers: "Collaborate. Listen to your team. Hold meetings and be the last person to speak your opinion." And so on.

Death by Consensus

This then leads to the second pitfall of new managers: no decision can be reached until the team collectively agrees to it. Meetings among the team take on the nature of an academic board of discussion, in which each of the professors who are experts in their own field weigh in with their opinions and analysis, with the intent that logic and deductive reasoning and debate will reveal the truth in time. "If everybody is convinced," the theory goes, "then collectively the right decision has been reached".

This "management-by-consensus" says, in essence, that it's not the manager that makes the call, it's the team that makes the call. After all, if the decision goes well, that means that the team collectively gets to share in the credit, so what's not to love, right? (And, less-often mentioned, if there's blame to get apportioned, well, the team as a whole shares the blame, so unless the company is going to fire the entire team over it....)

Management-by-consensus is essentially an abdication of authority.

Consider an average scenario: An issue has come up, and a decision needs to be made. The consensus manager, seeing this, calls the team together and presents the issue. The team debates for a while, and after some period of time, one of several outcomes emerge:

Only one person in the room has an idea of what to do, and everybody in the team likes it. Consensus achieved!
One person has an idea of what to do, and others in the room contribute modifications to the plan, which everyone agrees are improvements. Consensus achieved!
No one in the room has any ideas. A few half-hearted ideas are tossed out, but nothing gains much traction. Somebody makes a suggestion that seems like the least-worst option, so everybody agrees to that as the decision. Consensus achieved!
Multiple people have ideas of what to do, and hearty debate ensues. At the conclusion of the meeting, one plan is agreed by everyone to be the clear winner. Consensus achieved!
Multiple people have ideas of what to do, and hearty debate ensues. As the debate wears on, though, no one plan emerges as the best chocie. A decision needs to be made, however, so individual parts of different plans are each combined into a single plan that everyone in the room can grudgingly agree to, since it seems to be the best compromise. Consensus achieved!
Multiple people have ideas of what to do, and hearty debate ensues. As the debate wears on, though, no one plan emerges as the best chocie. A decision needs to be made, however, but nobody can agree on how to achieve that compromise. The decision is postponed until the team can meet again. By agreeing to defer the decision to a later meeting, the team has reached agreement. Consensus achieved!
Multiple people have ideas of what to do, and hearty debate ensues. As the debate wears on, though, no one plan emerges as the best chocie. A decision needs to be made, however, but one individual (or more) believes that their plan is the superior choice, and will not agree to anything other than the selection of their plan. In essence, they hold the meeting hostage until everybody agrees to their plan. Consensus achieved!

If you're starting to notice a pattern here, that the "consensus achieved" isn't always a positive thing, you're picking up on the underlying problem of management-by-consensus: consensus often undermines the very confidence in the decision that it was intended to create. The team may have agreed, but individuals are often frustrated, hurt, or even angry at the decision (or the lack thereof).

What's worse is that the manager, by having abandoned their ownership of the decision, has lost the respect and confidence of the team--that consensus manager is now, at best, an arbitrator, and at worst, a spectator.

The Buck Stops Here

As a manager of a team, you are (literally) paid to own decisions. There will come times when the team will need a decision to be made, but cannot agree on what that decision needs to be. Some may have an agenda they wish to see fulfilled, some may genuinely believe they have a better perspective; their intentions do not matter. What matters is that they disagree on what the decision will be. In those moments, the decision is--and must be--yours to make. The buck stops with you, and that means that at times, you will need to make decisions that may be unpopular with some of your team. You don't do it capriciously or maliciously, and you certainly don't do so in such a way that somebody takes it personally, but you have to do it.

You are the manager. It's your decision to own.

Two interesting stories are told about this phenomenon:

"When Dwight Eisenhower served as the Supreme Commander of Allied Forces during World War II, he faced the dilemma of sending many to their deaths with the invasion of Normandy. Balancing many variables, the window to go or wait came down to a poor set of choices. On the day planned for the invasion, the weather was uncooperative and the fog and cloud cover would keep Allied bombers from supporting the invasion force. Eisenhower’s team was split over the decision to invade or postpone. As was observed and reported, Eisenhower stared out the window into the fog for a long moment, turned around and said, 'We’ll go.'"

Eisenhower had, in the room, some of the best, sharpest military minds, and the combined might of millions of men and women, the political will of two empires (one rising, one fading), and all the best minds that could be brought to the problem. They poured millions of man-hours into the analysis of the problem. In the end, though, it came down to weather: Would the weather hold up well enough to permit the landings? Some said yes, some said no, and either could've been right. In the end, Eisenhower didn't call for a vote, nor did he continue to wrangle his staff to reach a consensus. The decision was his, and he made it.

What history often doesn't tell, when telling that story, is that Eisenhower then retreated to his office to write two letters/speeches: one that praised the troops for the successful landings, and one taking full responsibility for the failure of the Allied forces to land on the Normandy coast. Eisenhower knew, full well, that credit for the success lay with the troops who stormed the shores and manned the ships and flew the aircraft. Failure, however, was his, because his was the decision that sent them.

You could argue that Eisenhower was a brilliant strategist who had a feel for the vibe of the campaign. You could argue that he had a greater sense of his troops' capabilities. You could even argue that it was a coin flip result, and he got lucky at the right time. It doesn't really matter for our purposes why the D-Day landings were successful--all that mattered is that Eisenhower knew the decision--and the accountability for that decision--lay with him, and him alone.

The buck stopped with him.

Another, similar, story carries within it a slightly different lesson:

"The Illinois football team was on the brink of beating number one ranked Ohio State. It was 4th and 1 in Illinois territory with over 8 minutes to go and the Illinois coach decided to punt. Ohio State called a timeout and during this interval, the Illinois QB convinced the coach that they had not worked hard, practiced early and long for months (and for much of their lives) to give the ball back to Ohio State and lose with a potential drive. The coach reversed his decision, Illinois went on to make the first down, and ultimately eat up the clock en-route to the major upset of the year."

Notice here the coach (the manager) made a decision, but was convinced by others on his staff to make a different decision. As the manager, if you have hired your team well, you are not the smartest person in the room. Other people will have different perspective. They will have vision or skills or connections that you lack. They may well (and most likely will) be the ones who are executing on the decision, and if they believe they can pull it off, they might well be able to.

Had Illinois failed to convert, it's reasonable to assume that many would have held the coach responsible for that decision--even though his quarterback, the one executing the play (along with 10 other players) were the ones actually "doing the work". The manager (coach) reversed his decision, and the team responded with a win. (Literally.)

As a manager, being open to your team's contrarian positions and thoughts help you shape your own decision-making. Creating a psychologically safe environment in which your team can bring their dissenting opinions to you--and be heard--is a major part of the manager's job. You want, and need, the experts on your staff to bring their expertise and provide you with as much relevant information and evidence as they can. The team may even agree on what the decision should be. But, ultimately, the decision(s) rest with you. The accountability of those decisions are held up to you.

If you are the manager, you own the decision.

Crisis scenarios

It is important to note that there are times when extenuating circumstances require that the traditional pace and process of decision-making be cast aside for something more streamlined. These are often (if not always) crisis situations, and during those times your management and leadership skills are put to the test.

During those situations, many of the normal rules of management have to be set aside. During a crisis, your team will be looking to you not just for decisions, but much, much more. They will be unsure of themselves, particularly if this is their first crisis, and many will be afraid to make a decision because they believe it's better to do nothing than do something wrong (which often is the exact opposite of the truth). During those times, you'll need to find your "command voice"--that tone of voice that makes it clear that you are in control here. No yelling (unless it's necessary to be heard), no blaming, and no distraction.

The practical details of how to handle a crisis vary from situation to situation, but in general, think about the following:

Dive into details. During normal day-to-day operations, you'll often let your team handle any small details that come into their areas of expertise, but during a crisis it's usually important to understand any and all of the details that surround the crisis. And equally important to discard those details that aren't involved in the crisis. (Knowing the difference between the two is a hard but necessary process of filtration that you'll develop both in the moment and over time.)
Be present. Your presence, as the visible (figurative or literal) icon of where the buck stops will help the team focus on their individual portions of responsibility. You may also want (or need) to take a hands-on role during the crisis, but remember that you are the authority figure here, and you need to be available for more questions and/or decisions. This is not the time to be delegating to somebody on the team--own it.
Ask for questions, ask for input, but don't ask somebody else to make the decision(s).

Crisis situations will often showcase your leadership skills, but in the end the success or failure of the crisis situation's resolution will hinge on your management skills, the formal authority vested in you, and your willingness to own all the decisions made.

Summary

When taking on a management role you are accepting the accountability of what happens on your team's watch. The team will be filled with experts and expertise, but ultimately the decisions that the team needs made will be ones that you will have to make. Sometimes the team will come together collectively to figure out the best decision to make, but ultimately, it's not theirs to make or own--it's yours.

Keep in mind, by the way, that there are a few situations where you may deliberately play "the Management Card", and veto or overrule your team's consensus because you believe you have a different answer than what they can see. This is certainly your prerogative as the manager, but doing so spends a great deal of your management credibility, and had better pay off in ways that build (and enhance) that credibility--in other words, you better be damn right if you do this. (And if you end up being wrong, you'd better be the first person to admit it!)

Leadership might mean helping the team come to the best decision, but management means owning them.

The Player/Coach Fallacy

Fri, 20 Jan 2023 00:00:00 +0000

tl;dr Many companies look to hire individuals who are both leader and individual contributor (IC) on the same team. These are often referred to as "Player/Coach" kinds of roles, and people in these roles often find a distinct lack of success over time.

The problem is that this role sets people up for failure:

The role will inevitably gravitate (and spike) to one of either the leader role or the IC role. An Engineering Management Player/Coach, particularly if they are a new leader, will often index hard on their engineering skills, and spend too much time writing code (and "fixing" code written by others on the team, which undermines confidence and trust). A DevRel Player/Coach, particularly if they are a new leader, will index hard on giving presentations and doing webinars and writing blog posts and such. In both cases, the Player/Coach, being more recently a Player (and likely a successful one at that, hence the promotion), will neglect their Coaching duties in favor of "getting things done".
The Coaching expectations then don't get met. The team finds that their leader has no time for 1:1s with them, and/or team members miss opportunities to engage in their own career growth because the Player/Coach is spending too much time Playing. In some cases, the hiring of the rest of team never happens, all because the Player/Coach doesn't have the time (because they're "too busy doing the IC work").
Separations follow. Either the team members slowly drift away to other teams where they can get better support, or the Player/Coach gets frustrated that their performance reviews (and/or anonymous survey results from the team) are sub-par, or the Player/Coach's manager takes steps to "correct" the situation. In the worst of these cases, the whole plan for the team is abandoned.

You cannot have two Priority-One goals at the same time. If you are there to be an individual contributor, management tasks distract you from your purpose; likewise, if you are there to be a manager, IC tasks distract you. No role is ever 100% one and not the other, but there should never be ambiguity as to which is your primary focus.

Now, having said all this, it's still a common scenario for hiring managers to want to hire a "Player/Coach", particularly in the early stages of a team's formation. The idea, simply put, is that until the team is built, the Player/Coach will need to be "hands on" until the team's construction is complete and the Player/Coach can move to full-time coaching.

If that's the case (and this is speaking from having done this three or four times now), then here's the way to do it, assuming that no team actually exists at the time of the Player/Coach's hire:

The first six months are spent being 100% IC. This gives the Player/Coach the chance to identify the goals for the team, spent a little "time in the trenches" to see what their team will need to be doing and how to do it, and get to know some of the other teams around them.
The next six months are spent hiring. This means the principal focus is on staffing up the team, but since that likely won't take 100% of the Player/Coach's time, the remaining time is to continue the activities that were started in the prior six months. As each team member is hired, the Player/Coach will need to put more and more of their time into onboarding, mentoring, and managing, leading to the point where....
The team is staffed, and now the entire focus is on management. Once the team is hired and in place, the Player/Coach needs to hand off all of their IC activities to members of the team (regardless of the Player/Coach's personal feelings about some of the activites--love them or hate them, they all need to be turned over), and from this point forward, focus exclusively on growing and improving the team.

I've done this several times now, and if I were to do it again, I would consider any of these to be "red flags":

The timeframe for beginning the team hiring is beyond six months. Organizational inertia is a subtle and ugly thing. The Player/Coach is often very good as a Player, and the organization will often be very, very incentivized to keep them in a Player role. In which case, simply admit that you don't need the manager, hire a senior or Principal-level IC, and move on.
The team size is not going to be larger than 2. Setting up a situation in which a manager is managing a single person is probably the most ridiculous thing any organization can do. It's often the result of poor planning and/or failure to secure a real commitment to the team's existence--"We want to see how this goes and grow incrementally" is often the explanation, and it's weak sauce. Either there is a need, and the team needs to be built to fill it, or there is no need. Companies that build a team of manager+1 are often feeling inconclusive about whether the need exists, don't have the insight into their own company to be able to validate whether the need exists, or don't have the budget to afford the team. All of these are red flags that the team likely won't be around a year from now.
The company won't commit to headcount for the team until they've seen the Player/Coach in the role for a while. This is because the individual has zero credibility within the company, and it's a terrific way to set them up for failure. Without any sort of "executive support" to help them build their own cred, the Player/Coach will have to fall back to IC activities to build that cred, and doing so will only emphasize that "we never needed a team after all". It becomes a self-fulfilling prophecy.

It's tempting to look at the idea of the Player/Coach and believe that it's manageable, particularly in the early days of a team's formation, and conclude "We can make it work." I agree, but only when there's a clear transition plan in place (ideally enforced by paperwork, so the plan can't go awry when the situation changes even slightly). Anything else is likely to lead to a boulevard of broken dreams.

The Full-Stack Developer Fallacy

Thu, 19 Jan 2023 00:00:00 +0000

tl;dr Lots of companies are spending exorbitant amounts of time trying to track down and hire "full stack" developers, and finding them difficult to find. This is probably because there is no such thing, and reveals a deep weakness in the hiring manager's thinking.

Definition

As we begin, let's start with a working definition of the "full-stack developer" (or FSD, for short). Tossing this into Google yields (for me) the top result of:

"Full stack technology refers to the entire depth of a computer system application, and full stack developers straddle two separate web development domains: the front end and the back end.
"The front end includes everything that a client, or site viewer, can see and interact with. By contrast, the back end refers to all the servers, databases, and other internal architecture that drives the application; usually, the end-user never interacts with this realm directly.
"The easiest way to put the full stack into perspective is to imagine a restaurant. The front end encompasses the well-decorated, comfortable seating areas where visitors enjoy their food. The kitchen and pantry make up the “back end” and are typically hidden away from the customer’s view. Chefs (developers) gather permanently stored materials from the pantry (the database) and perform operations on it in the kitchen (the server), and then serve up fully-prepared meals (information) to the user.
"Front end developers work to optimize the visible parts of an application for web browsers and mobile devices. Front end platforms are usually built with HTML, CSS, and JavaScript; however, they can also be made via pre-packaged code libraries or content management systems like WordPress. Back end developers, in contrast, refine the software code that communicates with servers, databases, or other proprietary software that conveys information to front end interfaces.
"Those knowledgeable in both front end and back end are called full stack developers, meaning they are well versed in both disciplines."
(from https://bootcamp.learn.utoronto.ca/blog/what-is-a-full-stack-developer/# )

... which seems a reasonable working definition. However, I would argue that for most companies, particularly those trying to find FSDs for microservices-based teams, there's a few more disciplines left out of the above definition:

Testing: Writing unit- and integration tests.
Database: Defining and maintaining databases, including schema and data migrations.
Deployment: Familiarity with tools like Docker, Kubernetes, Istio, and the other services and products that are used to deploy the service/application.
InfoSec: JWTs, OAuth, OpenID, and other tools of the trade to keep malicious-intent individuals at bay.

... which is in addition to the original two disciplines (front-end/presentation, and back-end/distribution), each of which is arguably made up of other disciplines (front-end usually requires some degree of UI/UX design, and back-end often requires familiarity with distributed systems principles).

(By the way, we're also not really taking into account that a web-based FSD is likely to need a different set of skills than a mobile FSD. Some areas, like the back-end, might share many of the same skills, but the closer you get to the front-end, the more likely divergence is going to occur.)

Which, if you're thinking about it, is a relatively astounding amount of stuff for any single person to have stored away in their head, much less accessible at any given point in the day.

Jacks-of-All-Trades

The intent here is to hire people who are "jacks of all trades", meaning they know a little about a lot of things.

Historically, recruiters have talked about different "shaped" people:

"I"-shaped people, who have a great deal of knowledge in a very narrow range of topics (think about the developer who knows everything there is to know about a particular database, but nothing about HTML, CSS, Javascript, or even other databases),
"_"-shaped people, who are the original jacks-of-all-trades, with much breadth but little depth, and
"T"-shaped people, who have some depth in a small number of topics, but then a little breadth in more topics.

It may sound like the FSD is a "_"-shaped person, with wide breadth, except that often recruiters are looking for significant depth to go along with all that breadth. "Candidates must have 10 years HTML, 12 years CSS, 27 years with Istio, ..." and so on. In essence, the goal for the FSD-candidate search is to find "box"-shaped people, who have width breadth and wide depth.

That... just isn't going to happen. Not naturally, anyway.

Any developer who's been in the industry for more than a few years generally will have their experience skewed in some particular direction down one (possibly two) of those above disciplines. Some will really like database and back-end systems (that's me, honestly), while others will find front-end and testing interesting.

(Note that I'm picking these combinations somewhat at random--I've met people whose happy place was any combination of the two, including front-end and InfoSec.)

If we take the Malcolm Gladwell measurement of "expertise" as being 10,000 hours in any particular area, and we assume a working year of 2,000 hours each (50 weeks at 40 hours each week), that's five years to achieve expert status in that field. Now, if we assume that you work full-time in one discipline, having depth in all five disciplines means that you aren't a qualified full-stack developer for 25 years.

And, of course, nothing will change during that time to invalidate what you learned two expertises ago.

Hiring Manager Fail

How did we get here? There's a few theories, but my personal one is that it's a failure on the part of the hiring manager. A huge failure.

Look at it this way: Imagine you're running a team building a mobile application. Your team currently consists of the following six people, each of whom has distinctly different skills (all of which your team is using):

Alfred: HTML/CSS, Selenium testing
Betty: HTML/CSS/Javascript, OAuth/OpenID
Cassandra: Postgres, OpenAPI, Spring
David: Postgres, Azure/AWS, Docker
Esther: Docker, Javascript, JWTs
Francisco: Kubernetes, Istio, AWS

Now imagine that your boss comes to you and (given the current climate as I write this) demands that you lay one of these people off. Which do you choose? Anyone you let go is going to leave a huge hole that might be partially plugged by one of the others on the team, but not completely. You might be able to send the rest to some training classes or online training videos to learn what is now no longer covered on your team, but that's going to take time, and while they're learning you're still vulnerable there, and that assumes that others on the team want to learn whatever it is you're missing.

Now, let's look at it if you have a team of FSDs:

Amanda: Full-stack
Bob: Full-stack
Charlie: Full-stack
Denise: Full-stack
Evan: Full-stack
Freida: Full-stack

Well! Now when the request comes down from above, you don't have to play jigsaw puzzles with the team--you pick the one that's either the most expensive or the most junior (or is on a PIP or is the biggest pain in your--), and voila, problem solved. Or, if one of them chooses to leave, again, you just put out a request to Recruiting to hire another one of those FSDs, and you're back to full strength!

What's more, when the team is doing on-call rotation, each team member has the same skill set coverage, so that there's no "fallback" or "chain" that needs to be followed depending on what went wrong; in the first team, if Betty got paged because of an escalation having to do with the Postgres database, she lacks that skill, and somehow we need to get either Cassandra or David on the line, even though they're not on call this sprint. Bleah.

So, from a hiring manager's perspective, the FSD approach makes their life much, much easier.

And frankly, that's weak management--the manager's job isn't to make their own life easier, their job is to make the team's life easier. Yes, it's going to be a pain to juggle responsibilities and/or get team members cross-trained in other areas so we always have a "principal" and "secondary" expert on any tool or library or technology the team uses. Yes, when one of them leaves, you're going to have to lean on the other(s) to help fill the gap, and do more training to get yourself back to a "principal"/"secondary" expert within the team.

But when those people leave, now finding somebody who has one or maybe two disciplines under their belt means the candidate hiring process goes from "years" to "months". Plus, it simply acknowledges the reality that nobody is a "box-shaped" individual, no matter what their resume states.

Don't try to hire full-stack people; build full-stack teams.

A DevRel Activity Ontology

Sat, 14 Jan 2023 00:00:00 +0000

tl;dr: Newcomers to Developer Relations are often curious as to what, exactly, makes up the range of activities that a Developer Advocate (and related individuals) undertake. In this post, I look to provide an ontology and means by which to examine what artifacts satisfy what needs, and how it isn't just about what artifacts a DevAd produces.

First of all, if you haven't seen James Ward's "Seven Artifacts of Developer Advocacy Projects", I highly recommend it before continuing onwards here.

Let me begin with my working definition of Developer Relations:

That part of the company that engages with developers, both internal and external to the company, for the purpose of using that relationship for net gain to both sides. This can take the form of many things, but ultimately, (a) it's aimed at developers, (b) it's intended to facilitate greater positivity to one or both sides, and (c) it centers around developer activities of one form or another (that is, it could be technical, "human skill", or process, but it kinda-sorta has to be around the world of software development).

NOTE: Some folks will take issue with the part of my definition that starts it off: "This is that part of the company....", arguing that it's not limited to companies, i.e., open-source communities engage in Developer Relations. That may be true, but what we're seeing is that by the time the OSS community gets to the point of formalizing their relationship to the developers using their products/tools, they've usually formed some kind of foundation and it becomes somebody's job to take that relationship on.

Keep in mind that in James' post, he appears to be thinking that "for any given DevRel project, these are seven things that can come out of each." So, for example, if the DevRel team works on a code sample (maybe it's a React example of how to use the company's SDK), this is the list of things that can all be obtained from that single investment of time and energy:

Code Samples
A Blog Post
A Presentation and/or Video
A Hands-On Workshop
Broad Social Media Reach
Product Feedback
Enriched Community, Partner, or Customer Relationships

... and he's right, to be sure. Matter of fact, I'm a huge proponent of that mindset--if you're going to spend the time to learn a thing, amortize the cost by spreading the benefit of that learning around and into as many things as you can. I've written magazine articles (or series of articles), blog posts, training classes, conference workshops, breakout session conference slides, and more, out of learning just one new thing (like a new framework or a new language or a new library).

An Activity Ontology

But there's a deeper element here, around how we choose each activity and what we get out of each. As I see it,each of these provides value along three different "axes" of value to a DevRel team (and to the company that sponsors them):

Reach: How "far" does this artifact go? How many people can see it and/or consume it? Those things done over the Internet tend to have a large reach (particularly if the artifact is someplace where Google can find it and pop it up during search results), whereas those things done in person (such as the hands-on workshop) will have very short reach, since participation requires physical presence. For example, a blog post can echo across the entire world within minutes, and even across time itself--certain blog posts just keep getting rediscovered by new readers. That kind of reach is just not possible with an in-person workshop done at a conference event, even if the workshop has a thousand people in it (which, by the way, is categorically impossible to do--a group that size isn't doing a hands-on workshop, they're watching you lecture).
Interactivity: This is the fidelity of communication--how "conversational" is the artifact? This in many cases is in inverse proportion to reach, but not always. The blog post doesn't really allow for great conversation (yes, you can open up comments on the blog, but we all know what happens when you do, and it's not pretty), whereas a workshop really requires a high degree of interactivity with the attendees. The blog post author doesn't learn much from their audience when posting the blog--the workshop facilitator, however, can learn all kinds of things from the attendees via the questions they ask, the problems they run into, the questions they don't ask, and so on.
There's an important note to the points of Reach and Interactivity, though: Although the blog post and the in-person workshop may seem like they are mutually exclusive--and for some tools they are--they don't always have to be. I can do an "in-person workshop" over something like Zoom that can reach across the world, and if we record it, we get more Reach. I can make sure the blog post has comments enabled and people assigned to react and respond to the comments--or we can take the blog post to Hacker News or Reddit or StackOverflow, and make sure we keep a sharp eye on any discussion that emerges there, and increase our Interactivity. Reddit isn't as interactive as an in-person workshop, but that doesn't mean it is entirely a one-way communication, either.
Direction: One thing James hints at, but doesn't really call out strongly, is the idea that DevRel is a circular exercise: DevRel should be talking to both those developers inside the company as well as those outside of it. This means the usual activities of bringing development discussions to developers that might be customers of the company's product, but also bringing feedback from those developers on the outside back into the company for further discussion and/or examination.

Let's also put something out here: DevRel can be just as much about mentoring and catering to the developers inside the company as it does outside of it. Particularly with the efforts around "inner-sourcing", wherein companies (particularly large ones) practice the same processes and practices for open-source projects on internal projects, it can be important for a DevRel team to be just as focused on bringing awareness and engagement to developers internal to the company as external. Matter of fact, I first engaged in this with my DevRel team at Smartsheet, where we took over the internal-facing "Lunch Talks" (also known as "Brown Bags" at other companies, where an internal employee talks about a subject over lunch) and internal training; did the same thing at Rocket Mortgage, where we created the "Tech Speaker Series" (where we had industry names do virtual all-day workshops for our engineers during the pandemic). So not all activities of DevRel are externally-facing, and realistically, any activity that is typically done externally-facing can also be done internally-facing.

A DevRel Activity Catalog

So with these three axes in mind, we can start to categorize some of the activities that a DevRel team does (whether around a particular project or not), with the idea that if your company is looking to enhance its reach, it's interaction, or change/adjust its direction of engagement, we can choose activities accordingly.

The Foundation

First, we have the primary colors of the DevRel spectrum:

Code

Also known as: Samples/Examples; SDKs; Tests
Reach: High - Code can go anywhere a browser can be fired up, and since most languages are freely available, anyone who can see the code can usually grab tools to build and run it.
Interactivity: None-to-Low - Yes, I can file bug reports and raise issues on GitHub, but those are properties of GitHub, not the code. The code itself is pretty much entirely on its own, and the recipient is left to figure out what the code is trying to tell them.
Direction: Code can be passed to anyone, internal or external, so long as there aren't legal entanglements around it.

Presentation

Also known as: YouTube; Webinars; Conferences (sessions); Meetings (customers or others)
Reach: Depends If the presentation is "live"/in-person, the reach is constrained to the physical space the audience can fit within. If the presentation is "broadcast" over video-capture platforms (like Zoom), then the reach grows significantly. If the presentation is recorded, that reach now extends beyond the moment in which the presentation is being given, adding longevity to the reach.
Interactivity: Depends The traditional presentation is one-way (from the speaker to the audience), and is often referred to as "eyes-forward" or "lecture-style". While a presenter can certainly gain some insight and feedback from questions asked during the presentation, most presentations are 75%+ presenter-talking, with some being 90%, 95% or even 100% presenter's voice, leaving only a small amount of room for that feedback. If the presentation is recorded or broadcast, the interactivity drops significantly.
Direction: Bidirectional It all depends on the audience composition.

Writing

Also known as: Documentation; Articles; Blog posts
Reach: High Thanks to the power of Internet protocols (such as RSS) and search engines, writing can be made available through a variety of mechanisms and consumed by literally any of millions of people online at any given moment.
Interactivity: None-to-Low While it's certainly possible for written pieces to have a "comments section" wherein readers can offer up thoughts and feedback, experience has taught us that unmoderated comments are usually a fast track to noise and spam, decreasing the quality of the feedback to zero quickly.
Direction: Bidirectional Writing is commonly used to reach both an internal and an external audience.

Social

Also known as: Twitter/LinkedIN/Facebook; GitHub; StackOverflow; Forums; Conferences (floor); Conferences (sponsorship)
Reach: High - The various "communication platforms" of the community are neither writing nor presentation but fast becoming its own medium. The longevity is debatable, and the risk can sometimes be high, but the reach can be astounding. Keep in mind, though, this isn't limited to just the traditional social media platforms--GitHub long ago figured out its role in life is as a developer social platform. And those are just the online platforms--analog platforms (like conferences) have been around for a long, long time.
Interactivity: High - These platforms don't accomplish anything unless people are using them, and if they're using them, by definition there's interactivity there. Interactivity is off-the-charts high here.
Direction: Mostly external While certain social media players (cough Facebook) experimented with specific instances of their platform running internally to the company (that is, on the company's intranet), it does not seem to have "taken off" in a widespread manner to replace the more traditional tools like "internal websites-plus-email-plus-Slack/Discord/chat".

It's important to realize that anything can be amplified by announcing their existence over social channels--the ubiquitous "Hey come to my talk in room 5 at the Ritzy Glass Hotel on the 30th of February" blast over every social media platform, that's table stakes. Being on the floor at a conference talking with developers is intrinsically a high-interactivity activity, while sponsoring at a conference (as in, name on the banner above the floor and on the website) is lower Interactivity, but higher Reach.

The Hybrids

Then, we have the secondary colors of the spectrum, made up by blending two of the above together:

Code + Writing

Also known as: Tutorials; Reference API Documentation; Books; Articles; blog posts
Reach: High
Interactivity: Low to none

Code + Pesentation

Also known as: Workshops; Hands-on-Labs; Code Review; Training
Reach: High
Interactivity: Low to none

Code + Social

Also known as: Gists; Posts
Reach: High
Interactivity: Low to none

Wrapping up

Keep in mind, this isn't an exhaustive list of everything a DevRel team might do. For example, part of what isn't on this list is "Participation in unowned OSS projects"--that is to say, OSS projects that aren't "owned" or founded by the company. (Microsoft currently participating in the development and enhancement of the OpenJDK is one such example.) In these cases, the goal here may be to help the company's DevRel activity by being a good player in a partner's ecosystem, or it may be to support some other corporate goals.

Now, based on the DevRel team's goals, different activities (and their variations) can be selected deliberately to match. If the team needs to broaden developers' awareness of the company and/or its tools (sometimes referred to as "brand marketing"), high-reach activities are called for, without as much concern for interactivity. If the company wants to engage more deeply with the community--perhaps as a stepping stone to creating a formal "close friends group" (such as Microsoft MVP, Google GDE, and other programs), then activities high in social factors should be evaluated. Need to help Sales boost some numbers? Social will help, ideally in combination with high interactivity, so that potential sales targets can be more easily identified and pursued. (Saving salespeople from making cold calls/emails--and your community members from having to receive them--makes everybody happier.)

There's probably deeper ontological analysis lurking in here, and maybe even a full patterns catalog, but releasing earlier and oftener is a bonus, so... have at! :-)

Where does DevRel fit on an org chart?

Sun, 8 Jan 2023 00:00:00 +0000

tl;dr: The Developer Relations org isn't exactly Engineering, but it's not entirely Marketing, and it often isn't really Sales. So if you're a company looking to find a home for your nascent (or currently-existing) DevRel team, where do you put it?

Let's consider some of the traditional places a DevRel team appears first.

Engineering

This often seems like the most natural place for a Developer Relations team--after all, developer advocates often come from engineering backgrounds, most DevRel activities involve code in various ways, and, after all, their target audience usually work in Engineering departments themselves, so.... This is the right place to put them, right?

Unfortunately, while this might be a reaonsable second choice, it's not ideal. Engineering teams' sense of success is to ship things, often "at scale", and that mentality often carries over into the DevRel team, leading the goals of the team to be centered around shipping things (blog posts, code samples, etc). Worse, the emphasis of "scale" leads engineering management to concentrate on actions that reach the largest number of recipients, regardless of the quality of the connection. DevRel needs to be a loop, connecting with developers outside the company, and bringing that information back around internally and the team can't do that if the conversations are always one-way with little to no loopback.

"But, what about things like documentation, samples, tutorials, and so on?" These are all good things that need to be done, particularly if the company promotes a product or an API (whether that's the principal item for sale or not), but these can be done by Technical Content Creators, such as Technical Writers do for documentation.

DevRel inside Marketing

It's a common place to put the DevRel team, particularly in developer tools/products/services companies. The reasoning here is pretty straightforward: if the company wants to connect with developers in order to convince them to buy/use the "thing" (the tool/product/service), put them in the Marketing team and let them do the Marketing things to attract them. Go to conferences, write the blogs, but more importantly, go out there and show off the "thing" and get developers excited about it.

It's not optimal to put the team here, though, for a variety of reasons.

Marketing is often a "one-way" exercise, the goal being to create awareness of the product and bring them to the "top of the funnel" (meaning the "sales funnel", the euphemism for the collection of people who move through the process around purchasing a product). Developer "relations" means having a relationship with the developer community, not just talking at them.

Practically speaking, the goals of the Marketing team will often differ in substance from what a DevRel team needs to do. For example, Marketing's metrics will often revolve around "page hits" or "eyeballs" or "site stickiness", all of which revolve around the number of people and amount of time that arrive at the website or booth to find out more information. There's little to zero measurement around the depth of the connection, and even less emphasis on taking the feedback from the developer community.

What's more, the lines of communication between the Marketing team and the Engineering team are often quite "distant". If the DevRel team cannot get feedback from the developer community outside the company to the developer community inside the company, then they're not really "relating" to developers anymore, and they're really just Technology Marketing or Developer Marketing.

SIDEBAR: The bigger the distance between any two nodes in the organization chart tree, the harder it is to communicate on a regular basis. By this, I mean, if Alice and Bob both report to Mary, their distance is 0 from each other; if they both don't report to somebody until reaching the CEO on the org chart, then they're essentially at opposite ends of the Earth from one another. That distance makes it nearly impossible to set up a regular conversation, because the day-to-day of those two orgs is enough to create organizational resistance in a variety of ways. Try it if you don't believe me sometime--set up a regular recurring 1:1 with somebody from Legal or somebody from the shop floor or some other department that's miles and miles away from you on the org chart, and see how long your 1:1 lasts.

While it makes sense for the DevRel team to be in constant communication and alignment with the Marketing team, DevRel needs to be independent enough to not be held to 'top-of-funnel metrics' and instead held accountable to things that promote total engagement with developers.

DevRel inside Sales

In some organizations, particlarly the ones that sell products/services to developers, there's a natural tendency to put the DevRel team in Sales, since the DevRel team relates to developers, and the target demographic for buying the product or service is developers, and DevRel team members are developers, so.... Natural fit, right?

Oh my goodness, no.

First off, developers grow up to learn a pathological distrust--if not downright dislike--of anything that smells like Sales. Let's be honest, a lot of that is learned behavior from their predecessors (aka me and my generation), which may not be fair. (Probably isn't fair, to be honest.) Part of that distrust stems from sleazy Sales campaigns that my generation was exposed to; I still remember the day the Symantec Java IDE hit the streets and its salespeople boasted--without a hint of caution or hesitation--that their IDE would enable a "400% productivity boost" just by using their IDE instead of any of their competitors. It was complete and utter fabrication with zero basis in reality, but it's not the first time--nor will it be the last--that a sales pitch aimed at words like "productivity" or "scalability" or "cost savings" as tangible reasons to purchase a product. Salespeople have an unwholesome reputation, that they will say or do anything to "close the deal", regardless of the factual accuracy of what's being said, and developers want zero part of that.

When the DevRel team goes out into the community to connect with the community, they need to build relationships based on a basis of mutual trust: That I as a DevRel, I'm a developer too, and I know where you're coming from and I'm here to help you be better at what you do. Theoretically, salespeople should be coming from a similar perspective in a variety of other domains; it's why we have commercials in which we see people facing the camera talking about the product--you're supposed to empathize with who they are, either because they're who you are, or they're who you want to be. When that trust breaks down, then the messaging is entirely lost. In fact, if that trust breaks down far enough, it itself becomes an obstacle to the sale; for years, I told people to steer well clear of the Symantec IDE just because I was so disgusted with that sales tactic.

In many scenarios, a developer-type individual can go in to a sales call as a supporting character to the salesperson's meeting. This is the world of Technical Sales, and in many cases is used to help flesh out the details of the product and/or provide reassurance of what the product can and cannot do, particularly if it's a deeply-technical product. So long as the salesperson/TechSales tandem is in sync with one another, having the TechSales there, keeping the conversation "honest" and being forthright about the details, creates a deeper sense of trust between the buyer and the salesperson, and enables deals where it otherwise would fall apart. And Dev Advocates can certainly take the role TechSales for that meeting--any Dev Advocate who's been in the industry for more than a half-decade has done that at least once--but it's not at the forefront of their job description, and it should never be anything more than a supporting role in the sales process.

Again, if we consider the metrics, salespeople often fall under the ultimate metric: commission. And if your DevRel team falls under that metric umbrella, they're not going to be concerned with building trust in the community (and making it easier to land sales), they're going to be held accountable to going out and making sales of their own. It completely undermines their ability to build relationships, and reduces them to the transactional mindset that permeates sales organizations so deeply. And most relationships--whether professional or personal--fall apart under even the tiniest stress when reduced entirely to a transactional basis.

DevRel inside HR

Believe it or not, a (admittedly small minority) group of folks think the DevRel team belongs inside the same organization that handles recruiting. "If the DevRel team is about building relationships with technology communities," the thinking goes, "Then it makes sense to think of them as an adjunct to our other department that provides resources to the humans of the company."

On the one hand, there is something to this argument. Frequently, the most powerful recruiting team in the company is the DevRel team--they are out and about in the community, they speak the same language as the developers the company is courting, and they have the trust of fellow developers that recruiters (sadly) lack within that crowd. It's a very strong advertisement: An attendee has just seen a Dev Advocate give an amazing talk at a conference, thinking, "This person is one of the Cool Kids!", and when they get to the booth, they find out that Dev Advocate's company is hiring, and thinks, "I want to go work where the Cool Kids work!" Many are the stories of a prominent open-source developer being recruited by one of the FAANGs because the conversation started at a conference, with one of the Dev Advocates, who could "walk" them to the right hiring manager or recruiter.

Additionally, if the DevRel team is being faithful to its charter, it is looking to connect with not only the developers outside the company, but also the developers within the company. In some companies, the DevRel team takes on some of the internal developer relationship activities, such as scheduling and arranging internal conferences or meetups, or acts as the point of contact for acquiring training for an internal team. (After all, the Dev Advocates are out and about in the community, rubbing shoulders with the people writing the books and recording the videos, so they'll be best-positioned to know who the right people are if we want "only the best" for our internal training needs. And we do want only the best, right?)

Unfortunately, while these are both important aspects to the DevRel team, we still fall afoul of many of the same previous arguments: the metrics in HR are often entirely "out of whack" compared to those we'd like to see for a DevRel team, and the HR team is a little too far removed from Engineering, Marketing, or Sales, for the DevRel team to be effective at accomplishing its other goals.

Even worse, in a lot of companies, the HR team is a cost center, meaning the company is looking to keep the costs involved to a minimum. DevRel activities require budget--conference sponsorships, conference participation, hosting for blogs--and time--for samples, tutorials, posts, videos, and so on--that are critical to the DevRel team's success, yet don't fit in to traditional HR targets. Over time, the DevRel team becomes less and less "technical" and more and more "organizational", and the participants end up being Technology Trainers or Technical Recruiters.

DevRel inside the Office of the CTO

Companies which have had sub-optimal success with the DevRel team in any of the three above locations often cast about in some consternation--where do we put a team whose amibitions and activities don't closely align with neither Engineering, nor Marketing, nor Sales? How many other departments are left?

In many cases, particularly younger companies (aka "startups", though many companies characterized as such have long left their startup days behind them), the CTO takes a personal interest in seeing a DevRel team chartered and founded. The CTO has oversight over a number things, all of them related to "Technology", so it seems to reason out that the team should be part of the CTO's "pet project division", often known as the "Office of the CTO" (or OCTO).

And, truthfully, in the early days, the team performs well here. They have a direct line to Engineering through the CTO, and the CTO can sponsor their inclusion in activities that bring them close to Marketing and Sales; the Marketing teams often need to consult the CTO on messaging for the technology things the company is doing, and Sales will often reach out to the CTO for the Technical Sales support it needs in tough sales cycles. They can also work closely with Recruiting to help hit recruiting targets.

Seems like a perfect fit!

Ironically, the fact that the OCTO gives the DevRel team to be flexible and free also often undermines the team's long-term success. So long as the CTO is personally engaged in seeing the team succeed, the DevRel team can do some good things, but without the CTO's constant support, they often start to lose momentum and ground, and eventually wither away, because the rest of the company doesn't know how to interact with them on an organizational footing.

Each department within an organization needs to know how to interact with its peer organizations, and often this is done (somewhat indirectly) through the establishment of processes and metrics. Departments hold each other indirectly (and sometimes directly) accountable through these: Marketing needed a feature, so it used an internal process to put it on Engineering's backlog, who then connected with Sales to see how the feature was received, all of this described and measured via a variety of metrics to help with decision-making. When the DevRel team is a "special project" of the CTO's, those processes and metrics have a tendency to be short-circuited, in the interests of making the CTO happy, and the rest of the company never quite gets into a rhythm of where and how DevRel fits in.

Then--as inevitably happens--when the CTO no longer is putting the DevRel team on their "short list" of personal projects, either because the team has been around long enough that it feels like they should be able to survive "on their own", or because the CTO has moved on and a new CTO is now prioritizing other things, the DevRel team has lost their sponsor and suddenly they have a short window in which to figure out where and how they fit into the organizational mix.

DevRel goes... where, then?

The DevRel organization's goals are not the same as Engineering's goals, nor are they aligned perfectly with Marketing's or Sales'. It isn't unusual for organizations to "break out" of one or another this way--a long time ago, Marketing and Sales used to be unified into a single organization, until the realization became widespread that their goals were different. The same often happens with the Public Relations team (now often referred to as "Communications" or "Comms"), which has deep ties to Marketing, Sales, but also Legal and HR. Legal used to be buried inside of Operations. And so on. Each top-level organization should be (needs to be) able to pursue its goals independent of the others, and the leader at the top of that organization needs to be able to meet with peers to accomplish this.

If the goal of the DevRel team isn't completely aligned with any of Engineering, Marketing, or Sales, then where in the organization does it go? The conclusion drawn here is actually not surprising, once we stop to think about it: Developer Relations deserves to be its own top-level organization within the company.

Just as companies are realizing the benefits of having a "Chief Marketing Officer" and a "Chief Communications Officer" and a "Chief Information Security Officer", a "Chief Developer Relations Officer" reporting directly to the CEO and providing the guidance and leadership to the CxO suite around the art and science of Developer Relations makes sense. Alternatively, if the "CDRO" title seems a touch pretentious, and the company lacks a CMO or CISO, then it's probably more appropriate to call it a "VP, Developer Relations", again reporting directly to the CEO or CTO as a peer to the VP of Marketing, VP of Sales, and VP of Engineering. (Who reports to whom depends on the organization, but in general the VPs all end up sitting in the same room together anyway, so at that point it's a little moot.)

This is not to say that DevRel is ever going to be at the same size as any of the others--it's hard to imagine a DevRel team that will span hundreds, much less thousands, of individuals. But the same is actually true of a lot of departments within an organization. Legal, for example, is generally not going to be hundreds of people in size except in the largest of companies. PR, similarly, can be made up of a dozen folks supporting thousands or tens of thousands.

The DevRel team being at a level reporting directly into the C-Suite means that now the CDRO or VP of DevRel establishes high-level goals and metric targets suitable for the team, but in conjunction with the overall goals of the company as a whole. If the company needs to make a major Marketing push because Engineering is releasing a new version, DevRel can adjust goals to provide tutorials and samples, as well as work directly with Marketing to get the Dev Advocates out into conferences and podcasts to talk about the new release and features. It can allocate some time to go with the Sales folks to help sell the new version. And so on. By allowing the DevRel team to draw its own goals and objectives--not to mention argue for its own budget and headcount alongside those teams--in direct consultation with its peers in Engineering, Sales, Marketing, HR, and the CEO and CTO, the DevRel team establishes itself as a team player with a clear mission and well-understood set of activities and metrics.

Every company needs a DevRel team, and it's high time we saw the importance of the developer community as large enough to merit its own organization dedicated to interact and grow with it.

You Want Modules, Not Microservices

Mon, 2 Jan 2023 00:00:00 +0000

tl;dr Architecture is hard sometimes--people keep offering up some new idea that quickly becomes the mainstream "way to do it" without any context or nuance, and the industry, desperate to find ways to improve their architecture, snaps it up without hesitation. Microservices was the latest in the trend, and it's time we dissected the idea and got to the real root of what's going on.

At the heart of microservices, we're told we'll find...

... Lots of Good Things (TM)!

"Scalability": "Code can be broken into smaller parts that can be developed, tested, deployed, and updated independently."
"Focus": "... developer focuses on solving business problems and business logic."
"Availability": "back-end data must always be available for a wide range of devices... ."
"Simplicity": "provides simplified development of large scale enterprise level application."
"Responsiveness": "... enables distributed applications to scale is response to changing transaction loads... ."
"Reliability": "Ensures no single point of failure by providing replicated server groups that can continue when something breaks. Restores the running application to good condition after failures occur."

These all sound relatively familiar, I'd imagine, but the fun part about those six quotes is that two were taken from microservices literature (blog posts, papers, etc), two from twenty-years-ago EJB literature, and two from Oracle Tuxedo, which is forty-plus-years-ago technology. Can you spot which went to which?

We have a tendency in this industry to re-use our hype points over and over again.

"Those who cannot remember the past are condemned to repeat it." --George Santanyana, The Life of Reason (1905)

With respect to the microservices hype, one company's blog post offers 10 reasons to charge into microservices:

They promote big data best practices. Microservices naturally fit within a data pipeline-oriented architecture, which aligns with the way big data should be collected, ingested, processed and delivered. Each step in a data pipeline handles one small task in the form of a microservice.
They are relatively easy to build and maintain. Their single-purpose design means they can be built and maintained by smaller teams. Each team can be cross-functional while also specialise in a subset of the microservices in a solution.
They enable higher-quality code. Modularising an overall solution into discrete components helps application development teams focus on one small part at a time. This simplifies the overall coding and testing process.
They simplify cross-team co-ordination. Unlike traditional service-oriented architectures (SOAs), which typically involve heavyweight inter-process communications protocols, microservices use event-streaming technologies to enable easier integration.
They enable real-time processing. At the core of a microservices architecture is a publish-subscribe framework, enabling data processing in real time to deliver immediate output and insights.
They facilitate rapid growth. Microservices enable code and data reuse the modular architecture, making it easier to deploy more data-driven use cases and solutions for added business value.
They enable more outputs. Data sets often are presented in different ways to different audiences; microservices simplify the way data can be extracted for various end users.
Easier to assess updates in the application life cycle. Advanced analytics environments, including those for machine learning, need ways to assess existing computational models against newly created models. A-B and multivariate testing in a microservices architecture enable users to validate their updated models.
They enable scale. Scalability is about more than the ability to handle more volume. It’s also about the effort involved. Microservices make it easier to identify scaling bottlenecks and then resolve those bottlenecks at a per-microservice level.
Many popular tools are available. A variety of technologies in the big data world, including the open-source community, work well in a microservices architecture. Apache Hadoop, Apache Spark, NoSQL databases and many streaming analytics tools can be used for microservices. We are also proud to partner with Pivotal in this area.

Let's take a second and examine each of those, but this time in light of prior art:

They promote big data best practices. Pipes-and-filters architectures have been a part of the software scene since the 70s, when Unixes promoted several ideas:
1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
They are relatively easy to build and maintain. See the Unix philosophy, above.
They enable higher-quality code. If focusing on one small part at a time helps improve quality, then see the Unix philosophy, above.
They simplify cross-team co-ordination. This one is interesting; it suggests that "service-oriented architectures (SOAs) ... typically involve heavyweight inter-process communications protocols"--like JSON over HTTP? Or is that taken to mean that all SOA requires SOAP, WSDL, XML Schema and the full collection of WS-* specifications? Ironically, nothing about a microservice in any way prevents it from using any of those "heavyweight" protocols, and some microservices are even suggsting the use of gRPC, a binary protocol that bears closer resemblance to IIOP, from CORBA, which was the "heavyweight protocol" predecessor to... SOAP, WSDL, XML Schema, and the full collection of WS-* specifications.
They enable real-time processing. Real-time processing has actually been a "thing" for quite a while, and while many such systems use a pub-sub or "event bus" model to do it, it hardly requires microservices to do it.
They facilitate rapid growth. "Reuse the modular architecture"--do we even have a count of how many different things have all promoted "reuse" as a selling point? Languages certainly have done it (OOP, functional languages, procedural languages), libraries, frameworks.... One day I want to see something hyped that explicitly says "Screw reuse. We don't care about that."
They enable more outputs. "Data sets often are presented in different ways to different audiences"--that sounds a great deal like the Crystal Reports home page.
Easier to assess updates in the application life cycle. The need to "assess existing computational models against newly created models" for machine learning and advanced analytics environments... kinda sounds like a large pile of action words thrown together with little substance behind them.
They enable scale. How funny--the same was said of EJB, transactional middleware processing (a la Tuxedo), and mainframes.
Many popular tools are available. I don't think I have to really work hard to point out that tools have always been available for every major hype that's come through our industry--particularly after the hype has taken root for a while. Most readers won't even be old enough to remember CASE tools but maybe they'll remember UML.

But the discerning reader will notice that there is a pretty common theme to about half of the points above--the idea of creating and maintaining small, independent "chunks" of code and data, versioned apart from one another, using common inputs and outputs to enable a larger integration of the system. It's almost as if...

At the heart of microservices, we find...

... modules.

Yup, the lowly "module", that core concept that has been at the heart of most programming languages since the 1970s. (Even earlier, though it was harder to do with older languages that didn't incorporate the module as a first-class core concept.) Call them "assemblies" on the CLR (C#, F#, Visual Basic, ...), "JARs" or "packages" on the JVM (Java, Kotlin, Clojure, Scala, Groovy, ...), or dynamically-link libraries from your favorite operating system (DLLs on Windows, sos or as on *nixes, and of course macOS has the "Frameworks" tucked away inside the /Library directories), but at a conceptual level, they're all modules. Each has a different internal format, but each serves the same basic purpose: an independently-built, -managed, -versioned, and -deployed unit of code that can be reused.

Consider this working definition of a module, quoted from one of Computer Science's foundational papers:

"A well-defined segmentation of the project effort ensures system modularity. Each task forms a separate, distinct program module. At implementation time each module and its inputs and outputs are well-defined, there is no confusion in the intended interface with other system modules. At checkout time the integrity of the module is tested independently; there are few scheduling problems in synchronizing the completion of several tasks before checkout can begin. Finally, the system is maintained in modular fashion; system errors and deficiencies can be traced to specific system modules, thus limiting the scope of detailed error searching."

This comes from David Parnas' seminal paper, "On the Criteria To Be Used in Decomposing Systems into Modules", written in 1971--over 50 years ago at the time of this writing. The well-defined "separate, distinct program modules" covers about half of the suggested benefits of microservices, and we've been able to do that for fifty years.

So why the hullabaloo over microservices?

Because microservices were really never about microservices, or services, or even distributed systems.

At the heart of microservices, we should find...

... organizational clarity.

Amazon, one of the first companies to openly discuss the microservice concept, really wasn't trying to push the architectural principle as much as they were trying to push the idea of an independent development team whose blockers were few and far between. Waiting on the DBA team for schema changes? QA needs a build to test so they can find bugs? Or are we waiting on the infrastructure team to procure a server? Or the UX team to create a prototype for the presentation?

SCHHHLLLUURRRRRRRPPPPpppp...

That sound you hear is the development team aggregating ownership of any and all of those dependencies that could (and frequently would) block them from moving forward. It meant that the teams were a small microcosm of the average IT team's various parts (analysis, development, design, testing, data management, deployment, administration, and more). It did mean that now teams either had to be assembled from a variety of disparate skillsets, or else we had to require the complete set of skills in each team member (the so-called "Full Stack Developer"), which meant that hiring these folks became infinitely trickier. It also meant that now the team was responsible for its own production outages, meaning the team itself now has to be given on-call responsibilities (and the commensurate payroll and legal implications that go along with that). But, when all that was navigated, it meant that each team could build their artifact independently of one another, constrained by nothing other than time and the physics of how fast fingers can fly over a keyboard.

In theory, anyway.

At the heart of microservices, we often find...

... the Fallacies of Distributed Computing.

For those not familiar with them, the Fallacies were first coined by Peter Deutsch in a presentation to his peers at Sun--back in the 80s. They reappeared in the 1994 seminal paper "A Note on Distributed Computing" by Ann Wolrath and Jim Waldo, and they both essentially say the same thing:

"Getting distributed systems right--performance, reliability, scalability, whatever "right" means--is hard." (loosely paraphrased)

When we decomposed the system into in-memory modules running on a single operating system node, the costs of passing data across process or library boundaries was pretty negligble, even fifty years ago. When passing that data across network lines, though--as most microservices do--adds five to seven orders of magnitude greater latency to the communication. That is not simply something we can "scale away" by adding more nodes to the network; that actually makes the problem worse.

Yes, some of that can be made less relevant by hosting the microservices on the same machine, usually by loading them into a cluster of virtual machines running containerized images of the independent microservice. (As in, using Docker Compose or Kubernetes to host a collection of Docker containers.) Doing so, however, adds latency between the virtual machine process boundaries (because we have to move data up and down the virtual networking stack, in accordance with the rules of the seven-layer model, even if some of those layers are being entirely emulated), and still creates the reliability issue of running on a single node.

What's worse, even as we start to wrestle with the Fallacies of Distributed Computing, we begin to run into a related, but separate, set of problems: The Fallacies of Enterprise Computing.

At the heart of microservices, we need...

... to start rethinking what we really need.

Do you need to decompose the problem into independent entities? You can do that by embracing standalone processes hosted in Docker containers, or you can do that by embracing standalone modules in an application server that obey a standardized API convention, or a variety of other options. This isn't a technical problem that requires abandoning anything that's already been built--it can be done using technologies from anywhere in the last twenty years, including servlets, ASP.NET, Ruby, Python, C++, maybe even shudder Perl. The key is to establish that common architectural backplane with well-understood integration and communication conventions, whatever you want or need it to be.

Do you need to reduce the dependencies your development team is facing? Then begin by looking at those dependencies and working with partners to determine which of them you can bring into the team's wheelhouse. If the organization doesn't want to officially break up the "skill-centric" ontology of its org chart (meaning you have a "database" group, a "infrastructure" group, and a "QA" group as peers to your "development" group), then work with the senior executives to at least allow for a "dotted-line" reporting structure, so there's individuals from each group that are now "matrixed" in on a single team. But, most importantly, make sure that team has a crystal-clear vision of what it is they're trying to build, and they can confidently describe the heart of their service/microservice/module to any random stranger walking by on the street. The key is to give the team the direction and goal, the autonomy to accomplish it, and the clarion call to get it done.

It really boils down to these two things, which really, really, really have nothing to do with each other except tangentially.

What I Want to Build

Mon, 2 Jan 2023 00:00:00 +0000

tl;dr What would you do if you could do anything? Or, if you're a developer-manager-type, what would you build? Given that we're starting 2023, I thought it a reasonable time to take a stab at putting it out there into the universe.

Part of the motivation here comes from the interview cycles I've been through; I've been asked the "What do you really want to do?" question many times, and it's always tricky to answer in the moment. Being the developer-manager type, I thought it might be useful to answer it more easily by casting the question in the light of what would I really want to build?

For the record, if you're reading this because you're a recruiter (and either you found this on your own or I sent you here), this isn't necessarily what I expect to build in my next job, but if any of these sound interesting, I'd love to put the team together to make it happen. But if your needs are different, let's still talk--these'll keep for a while, or I'll tackle them as a side project. shrug

Big Projects

Let's start with the epic-level ideas that are probably way too big for any one person to tackle.

One common theme you'll find to a lot of these epics is the idea of elevating the abstraction level: Martin Odersky (creator of Scala) once commented that one of the key goals of Scala was to get programmers to worry less about the physical details of programming (what's a field, what's a method, what's a reference, etc) and focus more on the concepts they were trying to express. That idea has stuck with me ever since.

A platform-oriented language

We've had object-oriented languages for decades now, and the functional/functional-object hybrids for at least a decade, and neither really "moved the needle" on programming productivity or elevating the levels of abstraction. So what I'd like to spend some time doing is building a language that knows how to build code for a platform; that is to say, a language that has some interesting higher-level abstractions than your typical object language:

Service-oriented primitives. Ballerina and Dark each have some interesting exemplars here, allowing us to build first-class service endpoints. Theoretically any wire protocol could or should be useful here, but one of the key things I'd like this language to support is the Restful Object or the Hypertext Application Language specifications, making it self-descriptive to navigate through the entities online.
Persistence. Honestly, there's so many different tools to persist an object to a data store, and most of the time any thoughts of optimizing the storage or retrieval is just wasted mental energy. Let's just let the compiler generate the details for loading and storing entities.
Design-by-Contract primitives. Some languages have explored the idea of language-level primitives for invariant, pre- and post-condition enforcement (C++20 is incorporating it, even), and I see no reason why it can't elevate into a first-class contract.
State-machine implementation. So much of our implementation work is managing state transitions within an object or family of objects, so let's elevate that to a language level and let the compiler sort out the details.
Docker container artifact output. Forget the traditional binaries we've been churning out--the common output format is now Docker. Just have the compiler spit that out and be done with it.
Observability and manageability built-in. Just like virtual machine runtimes put hooks in to allow for easier debugging, profiling, and management, put the necessary hooks in to the generated service code to allow syslogd or Grafana or whatever the current standard-of-choice is to observe these things.

The goal here would be to build a language (on top of one of the existing enterprise ecosystems--likely the JVM or the CLR/.NET ecosystem, though there's really no reason it couldn't be NodeJS or Python or Ruby either)--that allows us to go from "single source file" to "working full-stack service" with a single compile/assemble step. Right now, building services is too hard, requiring too many different pieces assembled using too many different configuration files and languages, and while the flexibility is great across the entire industry, for any given company, there's likely a number of decisions ("corporate standards") that could reduce that flexibility significantly enough to allow for a language (which can always allow for exceptions if the language is designed to do so) to be able to generate most of that out of source.

And man, imagine how quickly you could innovate if you had this.... Build a fully ready-to-deploy service from a single source file?

An application-oriented language

Remember the "good old days" of Visual Basic and Visual FoxPro and Paradox and a bunch of those other 4GLs, when a single developer could build a working application, database through UI, all by themselves? We've lost that somewhere along the way, and as a result, what we call the "citizen developer" is now forced to learn Python and NoSQLs and HTML and CSS and JS and and and....

Bleah. Building a user-facing app today takes way too much work, because the developer building an app today has about a thousand different physical concerns to deal with before ever dealing with the logic and concept of the app itself.

There's ways to build an app on modern infrastructure and standards that could simplify all of this. The "low-code" movement is trying to capture this, but they're doing it with drag-and-drop application generators that don't show you code (or very small parts of it) or that generate vast amounts of code. Give us a nice high-level language, perhaps drawing signficant inspiration from Smalltalk and Io, that work off of a simple mental model, provide significant high-level out-of-the-box "primitives" (like, forget "string"--how about "Name", "SSN", "Date", "Age", and other really common data types that we always somehow want to reduce down to "string"s or "int"s?) complete with built-in validation and rules, and let them build applications that know how to present and store themselves? Make persistence an implementation detail (but cloud-friendly so it settles naturally on top of the existing clouds), make user interface a core concern that intermingles with the low-code logic (which gets replicated across both client and server tiers), and you have a very interesting language/platform for people to build some "personal applications" that would never, ever, EVER need to scale to Google levels.

Think Naked Objects meets Self meets a very reduced set of Python, over the Web and mobile devices, persisting as an implementation detail tucked away from the citizen developer (but knows how to export and/or provide an HTTP-based API for high-code access). The foundations are there, we just need to put the parts together.

An open RPG fantasy engine/app

I was a huge fan of the AD&D computer video games of the 90s: Pool of Radiance, Secret of the Silver Blades, and so on. It probably was helped by the fact that I owned a copy of the first Wizardry and Bards Tale that were the AD&D video games' spiritual predecessors (which was only fair, since they both drew from the tabletop RPG as inspiration), but I hated the fact that once you'd played the game, you were essentially "done" until a new game came out.

Meanwhile, over the years since, games like Battle of Wesnoth changed the story--a game engine could provide the mechanics and display and so on, while scenarios could be built against an open specification, allowing for literally an infinite number of possible plays. Plus, if we're going to go "open", let's open it up the entire way, to allow for expansion spells, skills, classes, anything we can imagine.

Mechanically, the classic fantasy RPG is not all that complicated once you pull the rules out of the equation, and the rules themselves operate by some core mechanics that (I think) can be extracted into an inference-like engine that can recognize how to deal with various combination scenarios. I'm thinking that the D&D 3.5e ruleset is the best way to go here--it's not the latest (so hopefully avoiding some of the copyright issues that might arise), and the principal critique was all the special-case scenarios in the rules that would be perfect to be captured in a game setting.

Essentially, I'm thinking a JavaScript engine hosted inside the game, with the user interface interacting with the engine, JavaScript because it's a popular language understood by many and therefore representing a lower barrier-to-entry for those gamer-types who want to populate the ecosystem. "Modules" (in the spirit of what we used to call the pre-canned adventures TSR used to sell) would be bundles of JavaScript files written by third parties, along with images or audio files for new monsters or scenes.

The various interactive-fiction engines also hold some level of intrigue here, as this goal is in large part the same as theirs, but they focus on the interactive storytelling, not the mechanics of hack, slash, loot, level-up, repeat. Still, if I do go down the path of thinking about a custom language for this thing, the TADS, Inform7, and Dialog languages all have their appeal as models for how to build out the non-data parts of the module.

The UI would be very simplistic: probably console for the v0.1, then v1 moving to the forward-facing view popularized by Wizardry and Bard's Tale, then maybe v2 shifting combats to the isomophic grid view, but probably not much further than that. The goal here would be to focus on the extensibility of the game, not trying to compete with Baldur's Gate or later.

Long-term, this could even morph into some kind of multi-player thing, but frankly, if I wanted to play D&D with friends, I would grab the books and head down the hallway in the college dorm--today, I'd just send them a Roll20 link. The video games were for when I just wanted to hack-and-slash my way through the dungeon as a loner late at night instead of doing my homework.

A database built from GC principles

With the rise of NoSQLs--by which I mean non-relational databases--as a storage concept, and the prevalence of needs to track data's usage and ownership, there's a small part of me that wonders at the power of bringing some of the mark-sweep semantics of automatic memory management into the database tier. In particular, I think it would be powerful if data could know its origins--where it came from, who owned it, who touched it last, and so on--in a way that most databases currently lack. It wouldn't likely be the most high-performance system in the world, but by this point we've seen that performance is a relative term; we can afford to spend some additional cycles and some additional bytes on overhead if it means having necessary business functionality baked in (like being able to make audit tracking so very much easier).

Some of this is derived from the "worlds" concept defined in Alessandro Warth's thesis on OMeta: "We can think of a program as an agency set up to accomplish or reach one or more goals. For some goals--such as adding up numbers--there is a simple straight road in most programming languages. Other goals--such as finding the square root of a number--will require some search, but can still be homed in on very effectively. For more complex goals--such as certain kinds of parsing, problem solving, and editing in a user interface--it may not be possible for the program to simply home in on the desired result; instead, it may need to experiment with multiple alternatives, sometimes making mistakes that require retraction in order to make fresh starts.

"For some goals of this latter kind, it might be a good idea to use backtracking, as in Prolog (it would also be a good idea to automatically undo asserts that are backtracked over, which does not happen in Prolog). For problem solving on a larger scale, it might be a better ploy to use "possible worlds", perhaps even parallel possible worlds that can report progress to a coordination agent that is trying to make choices about best paths. If the problem terrain is really rough and tricky--a veritable "elephant to blind men"--then we might want to use the metaphor of dispatching "scouts" (or "scientists") with walkie-talkies who can simultaneously explore different parts and intercommunicate to gradually build up a better model of the obstacles and the possible routes around them.

"Unfortunately, the nature of state in traditional programming languages makes this "experimental" style of programming impractical. For example, because the state of a program is scattered around the computer’s memory in several kinds of data structures (e.g., arrays, objects, and activation records), it is both messy and difficult to undo the side effects of an action that has been performed by the program. Also, because there is only one "program state", a program cannot explore multiple alternatives concurrently.

"My approach to solving this problem was to introduce a new language construct that reifies the notion of program state as a first-class object. I call this construct a world. All computation takes place inside a world, which captures all of the side effects--changes to global, local, and instance variables, arrays, etc.--that happen inside it. Worlds provide multiple views on the state of a program, and mechanisms for interacting among these views. They subsume the mechanisms of backtracking, tentative evaluation, possible worlds, undo, and many similar control and state regimes.

"We shall see that while it is often convenient for programmers to use worlds directly, there are also cases where worlds are better suited as a kind of "semantic building block" for higher-level language constructs."

It got me to thinking--if worlds need to track changes and be able to backtrack, it means that state is using state to track changes to its state. Generally we just think of databases as a repository of standalone facts with little to no additional supporting metadata. But what if we start thinking about data in a database as a derivative of this "world" concept, where state knows its part of a larger state, including information about when/where/who/how it is used or from?

If, for example, a data element is deleted, we could track all of the places this data touches and/or is incorporated, and that in turn could/should trigger discussions about what to happen to that state--perhaps even automatically, without requiring programmer interaction. One could argue that this is simply making use of a graph database, but I think there's a notable distinction between the "shape" of the raw data (as presented to the user of the database), and the internal implementation and tracking information metadata that could be used to drive these sorts of things. Much as an object in memory in Java or C# has additional information (metadata) about it that isn't visible to the Java/C# developer, yet is highly necessary to enable things like synchronization, automatic memory management, and more (type information, for example, is necessary when calculating how much space to allocate for a new object).

A persistent programming language

MUMPS has fascinated me since I first discovered it (recently), in that global variable declarations in MUMPS are, in fact, allocated space on disk.

To quote Keanu: Whoa.

That's some interesting thought right there: I allocate a variable, and it's immediately (and silently) stored to disk. Updates write to disk. Accesses read from the disk (but could also be cached, since the compiler is the one entirely in charge and has knowledge of when the variable is written).

And if the state of the program is entirely managed in terms of these persistent variables, the entire state of the program is automatically preserved in the event of a crash.

What if we took a traditional OO language and added this concept of "persistent types" to go along with the "transient types" we already use? (It would probably be in the form of a lexical modifier--string name being a traditional transient variable, while string^ name is a persistent one.) You probably wouldn't use it for all of your variables, but for your "domain" and/or "transactional" data? I think that would solve a lot of interesting problems right out the door.

There'd be a few things that would need to be sorted out (identity semantics being one!), but it's definitely worth playing around with further, particularly for those programs which really need to be able to pick up again after a crash and don't have heavy persistence needs on that tier (because they store their data long-term in a database on a server, for example).

A service choreography engine

Lots of people built microservices, and then found that they needed to build services that choreographed process flow through those services. The danger here, of course, is that "choreography" is yet another word for "chatty distributed system", and a major cause for performance nightmares in your microservice fabric.

Part of the problem here is that applications built out of microservices want to retain control over the steps of the choreography, yet the thought of baking them into a traditional programming language that then in turn must be tested (with mocks and fakes and such) just makes the developers groan.

What if we could abstract away the details, and hoist a "script" into the service fabric to do the choreography? Testing the script would be a helluvalot easier when the service mechanics are already all tested and deployed, and ideally the script would run out in the same tier as the services--that is, in the cloud.

This speaks to the idea of an "orchestration" or "choreographer" service, accepting scripts as input, making the service calls, capturing the responses, and evaluating those results to decide where to go next with calls (if any). It feels somewhat like a "workflow service", a la Camunda, but not something that's end-user-facing; more a developer-facing service that has dynamic overtones allowing it to be more flexible than a traditional compiled service.

(Frankly, this one feels a little smaller than the others, I think.)

Smaller Projects

There's some smaller projects, some of which are really no bigger than an application demo, but interesting to me nevertheless, that I have on my TODO list.

Hearts Scorer

My family plays the card game Hearts a lot--my folks taught it to me, my wife knew how to play already when we met, and I taught it to my kids--and I'm always the one who handles the scoring. But math is hard--so let's build an app! Sell it on the AppStore for $0.99, just because.

Spades Scorer

Same idea, different card game. Likewise, sell it.

Tic-Tac-Toe

I've been working on a libgdx version of TicTacToe (for mobile devices, but easily for desktop or Web too), and a Ballerina implementation of a TicTacToe REST server. Time to marry the two up, and include some AI so people who want to play single-player can do so.

"GSpaces"

A while back, academia posited this concept of "distributed spaces" or "tuple spaces" (as part of the Linda programming language), and with the rise of non-HTTP-based distribution tools (gRPC, Orleans, etc), I'm curious to resurrect the idea and see how far it flies. You wouldn't replace your database with this, but for transitory state and/or state that wants/needs to live in memory across clusters, this could be an interesting tool.

And there's more....

... but most of the other ideas are just one- or two-sentence ideas floating around in my head.

Somebody got a million bucks they want to throw my way so I can explore any of these? Could be fun...

Every Company Needs a Developer Relations Team

Sat, 31 Dec 2022 00:00:00 +0000

tl;dr: In my 2022 Tech Predictions, I asserted that more companies would be building DevRel teams, and I've repeated that in my 2023 Tech Predictions. I have reasons for that; I've been at many wildly-different companies, many of which with wildly different business models, and I've concluded that every company (that is in any way associated with software beyond a cursory level) in the post-2000 era has need of a Developer Relations department or organization. That said, DevRel is not the same at every company, and varies with the kind of things the company does to make money.

These are bold words, and often generate reactions ranging from quizzical to "are you trying to sell me something?". I stand by them, though, because every company in the post-2000 era that does something with software--whether they buy it, use it, or build it--engages with developers in some way, and therefore needs to relate to them.

If you're still willing to hear me out, let's revisit my definition of Developer Relations:

That part of the company that engages with developers, both internal and external to the company, for the purpose of using that relationship for net gain. This can take the form of many things, but ultimately, (a) it's aimed at developers, (b) it's intended to facilitate greater positivity to one or both sides, and (c) it centers around developer activities of one form or another (that is, it could be technical, "human skill", or process, but it kinda-sorta has to be around the world of software development).

Let's also consider a simple categorization of companies, according to my own (very biased) experience and perspective:

Developer-facing product companies. Your Microsofts, Googles, JetBrains, and so on. These companies make and sell products that are sold directly to developers, making them the target audience and the source of interest and income for the company. These are the ones that pioneered the Developer Relations concept, and for that reason, these are often the companies that come to mind when thinking about DevRel. Wikipedia calls these "Developer-First" companies, and says they have a "Business-to-Developer" (B2D) business model. This makes sense, although I think there's still value in thinking in terms of companies buying the developer-centric product, as opposed to developers buying it for themselves. (But maybe that's splitting hairs.)
Product companies where developers "value-add" indirectly. These are companies Consider, for a moment, one of my previous employers, Smartsheet. This is a product that does not sell directly to developers--in fact, it often finds itself in categories where professional developers are explicitly not allowed or desired (e.g., "low code" tools or "citizen developer" products). Wikipedia thinks of these as "Developer-Plus" companies, and thinks of them as business-to-business (B2B) or business-to-consumer (B2C) models, and says "While the primary focus of Developer-Plus companies is to create and sell products for businesses or consumers, they also make products or services available to developers which benefit or enhance their strategy including: opening new market channels, creating new use cases, contributing to innovation strategies, or optimizing/enhancing existing products."
Companies where integration is a key core of the business. Rocket Mortgage would fit this definition, since much of their ability to sell mortgages (which is clearly a non-developer-facing product!) depends on their ability to integrate with the other various financial players out in the world. Consumers won't see code, they won't touch code, but the value of the product will definitely grow (or shrink) depending on developers' ability to connect "our stuff to other peoples' stuff" (such as banks, escrow companies, title companies, credit organizations, and so on). It could be argued that this is more "Developer-Plus", but we're starting to stretch the definitions some here; for some companies, being able to provide integration capabilities is "table stakes" and means the difference between a partnership or sale and a waste of salespeoples' time.
Consulting companies. This is its own category primarily because there's so many of them, and they all have the same basic goal: Sell their developers to companies that either don't have developers, don't have enough developers, or don't have the right developers. The company's core business model is in putting their smart developer people to work for you, regardless of what sort of developer people you already have--either in conjunction with your smart developer people, or as a complement to your overworked developer people, or as leaders to your less-smart developer people, and so on.

Each of these companies needs a DevRel team, but for different reasons:

Developer-facing product companies. Pretty clearly the DevRel team's job here is a combination of marketing,sales (technical pre-sales, actually), and engineering:
- Marketing: Brand recognition (getting them to understand that the product exists), product-fit awareness (having enough knowledge what the product does to be able to judge if it's worth exploring for a given problem), and getting them to try the product.
- Sales: Working with developers to get past any obstacles they may have to buying the product, such as overcoming concerns about compatibility with other products, or the product's applicability for use in various scenarios, and so on. (If you're not sure this is really a DevRel role, consider how many blog posts and/or presentations you've seen describing how Vendor Product X works in Vendor Cloud A or with Vendor Database G.)
- Engineering: Providing developer support if/when the product doesn't execute as expected, or there's unexpected outcomes, or unanticipated obstacles, and so on, as well as providing examples for developers to follow so that their onboarding with the product is easier.
For the most part, these are pretty self-descriptive, and I doubt most developer-types reading this would disagree strongly with this characterization.
Consulting companies. The consulting company needs DevRel because consulting companies that cannot convince potential clients of the consulting company's technical skills and apititude don't stay in business for very long. We are smart consultants! Your company needs us! Thus, the consulting company's DevRel team needs to:
- Marketing: For a consulting company, marketing here really means brand recognition ("We are smart people! Doing smart things!"), so that clients will already know about the company--and how smart their people are--by the time a salesperson connects with the potential client. ThoughtWorks, for example, coasted for the better part of a decade on the various activities of its Principal Scientist, Martin Fowler, who brought incomparable amounts of brand recognition to the company's name and brand. After a while, others at the company began doing some of their own brand recognition efforts, but Martin clearly was the cornerstone of the company's brand efforts for quite some time.
- Sales: Most of the time, when a consulting company is looking to sell their services to a client, the client needs to be sure of a couple of things: one, that these people are not only smart about technology, but also about the kinds of problems the client is facing; two, that these people can work well with the client's people to get the project done; and three, that the consulting company has some success under its belt, such that this feels like a "good bet". In these situations, it helps immensely to have one of the developer-types be in the room during the conversation, to speak (as a reasonable proxy) to the skills of the consultancy and offer up some examples of when/where the consultancy was able to succeed.
- Engineering: Coming back around to the "bidirectional" nature of DevRel, in a consulting organization the DevRel team should be talking to your consultants (and their clients) to get a sense of what sorts of problems are being found routinely at a variety of different clients, so that your consulting company as a whole can be thinking about how to gain a competitive advantage over other companies by building reusable assets that can accelerate the teams' productivity across both current and future clients. (Not doing that? Maybe you should be!) At the very least, the DevRel team can be serving as an incubator for the use of new languages and platforms to that same purpose.
- Recruiting: In many consultancies, the best source of consultants is through the same venues and activities that DevRel engages--so consultants are encouraged to do those activities as a recruiting effort.
Note that in most consultancies, the current DevRel activities are generally done by the consultants themselves--Principals, Software Architects, senior consultants, and so on--but very, very few consultancies actually accept that this is a role in its own right, and most senior consultants who do this work do so during their non-billable time. In fact, even though most consultancies accept that Sales and Marketing are both activities that require being outside the umbrella of "billable hours", anything that requires technical skills means those Sales and Marketing folks have to beg time from consultants--who are in turn now caught on the horns of the "If I do this it's non-billable time, and my end-of-year bonus is tied directly to my billable time percentage, so either I sacrifice my bonus or I'm doing it off-the-clock as a volunteer effort."

NOTE: If you're a consultancy, stop doing this! Or, rather, consider how much more effective your company could be at finding and placing consultants if the people doing it were doing it full-time, just as your Sales and Marketing folks are. Not to mention the opportunities it gives you to rotate consultants through this team so that they get some experience doing something other than code--it helps grow the company in a variety of ways.
Product companies where developers "value-add" indirectly. These companies don't sell to developers, and in many cases, aren't even in the "tech business" at all; think about Netflix or Hulu for a moment. They sell video streaming, which is obviously technology, but they don't sell to developers--they have no offering that a developer or development team can purchase. Yet, Netflix is commonly considered to be one of the FAANG companies, where Hulu--whose business model is exactly the same--is not. The key difference is because Netflix chose to put some effort into the OSS community. Why do this? Netflix offers an API that allows people to build "value-add" using the Netflix platform; quick challenge, without Googling, does Hulu have an API? (Hint) And hey, while we're at it, what about Ford? You know, the cars? Do they have a public API? (Hint)
If you can think of a company that's larger than a thousand people in size, there's a good (better than 50%) chance that that company has an API. Even a soft drink manufacturer has an API. Chances are strong that your executive team has been looking for ways to turn your company into a "tech company" and provide a "platform" for people to build with. (By the way, most executives have no idea what that platform should be or look like, they only know they need one. Talk to me if you're an executive and want to figure out what one for your company should look like and provide.) Even if you're not looking to build a public API, chances are even stronger that you're building a private one--which means you need a DevRel team.
- Marketing: If you're doing a public API, you want developers to know that this API exists, even if they don't directly have access to it, because it creates credibility around your efforts when partners and/or paying customers want to use it. If you're doing a private API, your efforts might actually be more internally-focused, the "inner source" model, where your efforts are now more about making other people inside the company aware of the API and what it does. (One of the best reasons to inner-source? So that other teams don't waste time and effort duplicating the API!)
- Sales: Sales is about convincing people to use your product; in an external-facing API, some technical type will likely need to sit with potential partners and customers for your paying API, answering questions and exploring challenges and/or obstacles regarding the purchase of the API. For internal APIs, there's less pressure "to buy", but not by much--different teams will need to know that an inner-source API can do what they need, and there's likely to be just as much negotiation at work, for example over which team will add missing functionality or be responsible for added features when they need fixes and such. Having a DevReal team to take on some of that work can be invaluable.
- Engineering: Speaking of fixes, if your public API is now a part of a financial contract, then the partner or customer has to have somebody to call in the event that something doesn't work right (or doesn't work the way they think is right). In developer product companies, they're very quick to spin up a technical support team, but for non-developer product companies, where does that tech support team sit? Right there in the DevRel team, if you ask me--particularly if the engineering effort to build out the APIs is scattered across several organizations internally.
- Recruiting: Same story as the consulting DevRel team, really: If Ford wants to attract new developers, it helps to have some caché to do it, and that comes from being out in the world doing interesting things where developers can see them. Once again, that's the realm of a DevRel team.
Companies where integration is a key core of the business. You're not Ford, you're a bank or some form of B2B business, where the company's main line of product requires integration to work or optimize. A lot more companies are in this bucket than you might imagine--airlines, for example, are always looking at various other systems they want or need to integrate with, starting with other airlines (as they share loyalty programs, for example), rental car companies, hotel chains, and other players in the travel industry. If the business is a restaurant chain, developers can make use of APIs to automate the supply chain, tying the restaurant's inventory to orders, for example.

Here, the company's goals are to build developer tooling that makes it as easy as possible for other companies' IT to mesh with their own, to enable the integration scenarios that every CEO dreams about. That means building the platform that has copious entry and exit points, dozens of ways for integration to occur, and a plethora of documentation for the integration partners' developers to consume while working to make everything work. And the Developer Relations team will be squarely at the center of it all (working, ideally, with the integration partners' DevRel teams, because their CEOs all read this article, too).
- Sales and Marketing: This is probably something of a combined effort, in that making potential partners aware of the integration capabilities (Marketing) and convincing them to integrate (Sales) are likely to be wrapped up into the same organization within the company (called, for example, Strategic Partnerships). Here the DevRel team will be a technical voice in the conversation, offering up explanations of the company's platform and its technical capabilities, usually in meetings with technical folks from the other partners.
- Engineering and Support: This platform needs construction, and it needs people to make sure the platform remains up and running, and both of those mean working with partners to determine their requirements, their capabilities, and providing a single point-of-contact for the partners to call in the event things aren't working "correctly" (where that term can mean "not according to spec" all the way through "not according to what I think should be the spec"). The DevRel team--relating to the developers both within their own company as well as those at their partners--is ideal for managing these relationships.
- Recruiting: Like all companies that use software, they have a need to bring in more developers. (The demand for software is infinite, but supply of those who can provide it, finite.) DevRel reaches out into the broader world, discussing the company's platform, the company's goals, and just in general making it very clear that there's a bunch of Cool Kids Doing Cool Things (TM) at this company, and wouldn't you like to be one of them?

Now, keep in mind that when I talk about the role of developer relations, I'm not just speaking of the activities of developer advocates; a modern DevRel team will have a variety of different folks on a DevRel team, and in many cases those activities will be different for the different roles on the team. DevAds will often go and speak at conferences, for example, whereas Technical Community Managers will be taking more of the social media management on themselves, while Technical Content Engineers will be focusing on content creation (such as documentation, videos, and so on), and Developer Support Engineers will be the first line of support for external developers, and so on. These roles overlap, to be sure, but it's the rare case where one person will do everything. (Frankly, those cases should probably be reserved for those companies that are just getting started and need somebody to kick things off in the DevRel space--just like the startup's technical co-founder writes the code, builds the website, manages the bugs, and so on. Over time, you need to scale up and out, and that means more people.)

Every company that uses software, needs Developer Relations.

2023 Tech Predictions

Thu, 29 Dec 2022 00:00:00 +0000

As per previous years, I'm giving myself either a +1 or a -1 based on a purely subjective and highly-biased evaluational criteria as to whether it actually happened (or in some cases at least started to happen before 31 Dec 2020 ended). If you want to skip to the new predictions, scroll down to the next major heading.

In 2022...

... I wrote a lot of stuff. (That always seems to happen when I do these.) So let's get to it; as I do each year, I'll include the full text of what I wrote in each bullet point first, then put the Result after it with whatever insights or comments seem relevant. (Arguably none of them are, but hey, it's my set of predictions, so....) As always, evidence is entirely anecdotal and from my perspective, so caveat emptor.

The major thing that kicked off in 2022, of course, was the conflict in the Ukraine. Russia's decision to move armed troops across the Russo-Ukrainian border was really precedent-shattering on a number of fronts, and significantly shook a world order that had made the assumption that "major-power" armed conflicts were a thing of the 1900s. The "McDonald's Rule" of International Relations ("No two countries, each of which having a McDonald's within their borders, have ever gone to war against one another"), a bastion assumption of economic liberalism, has suddenly been thrown for a loop, and it leaves a lot of people wondering what to make of it. It combined with the overall economic gray doldrums that had been starting to beat its way through our consciousness to make 2022 feel pretty grim.

On the tech front, the big news in the second half of the year was the collapse of first one, then many, cryptocurrencies and exchanges, and literally billions--if not trillions--of dollars of value disappeared into the ether. Whether this will end up being the precipitating event of a Great Depression (like the 1929 Stock Market Crash) or a momentary blip on the radar of history has yet to be seen. Generally, that amount of wealth just going up in smoke has some pretty negative ramifications, but it's questionable whether that "wealth" ever actually existed. To be fair, people that paid USD (or EUR or YEN or ...) for tokens have now lost their original investment, but it seems that the original investment was much smaller than many media pundits are quoting; it's much more sensationalistic to talk about the "trillions of dollars lost" by speaking of the supposed wealth of the tokens, rather than the "millions of dollars of investment lost" from those who speculated. This one will probably take a few years to sort out before we really know what its impact was.

Meanwhile, among my 2022 predictions....

"5G rollout is going to be a lot more contentious than we expected."

Result: -1. 5G rollout generated a few more warnings, but for the most part, not much really kicked in. There may be more movement next year, but as a prediction, it definitely didn't generate much news, reaction, or practical issues.
"Demand for full-stack developers will continue to leave companies high-and-dry."

Result: +1. Yup, the demand got white-hot, all the way up until recently, when the whole "recession" thing kicked in. Even now, as layoffs are taking place, some of the very companies that laid people off are still hiring. It's the weirdest economic situation I've ever lived in, and I lived through the Stagflation and Oil Embargo of the late 70s.

As to my predicted second-order effects from this one....:
- "Work-from-home will become normalized as part of the landscape."
  
  Result: +1. Yup. In fact, the "remote work" thing is now normalized in a way it wasn't before, even despite some companies (most of whom I think are trying to justify the huge investment in land and leases and office space they're still obligated to pay for) strongly trying to get people back into the office.
- "Companies will start "growing" their own developer staff."
  
  Result: 0. Some of the bigger companies dove into this in the start of the year, but leveled off or backed away from it as the year wore on and the Gray Clouds of Recession loomed more menacingly overhead. They haven't gotten away from it entirely--and possibly never will, if they've got the infrastructure already set up and in place--but it's nowhere near as highly-established as I thought it would be.
- "Automation tools and platforms will climb in use."
  
  Result: +1. I'll give myself a point here, but let me be clear: There is no way I was thinking anything along the lines of ChatGPT or OpenAI or any of that stuff. I mean... wow.
- "No-code/low-code tools and platforms will surge."
  
  Result: +1. Do a Google search on either "low-code" or "no-code" and then take a step back as the results flood your machine. Layoffs have hit some of the big players in this space (Airtable, for example), but it's early enough for some of them that they're just charging forward damn-the-torpedoes-style.
- "Developer market will grow. Slowly."
  
  Result: +1. Lots of people are posting to LinkedIN and other places about joining the tech field, and the bootcamps and self-study programs continue to grow (Pluralsight's 20% layoff notwithstanding).
- "Job descriptions will emphasize more about skills (including human skills) and less on degrees/certifications."
  
  Result: 0. Well, on the one hand, (some) companies did revise their job descriptions to be a little more "growth mindset"-focused, but the interview processes didn't exactly shorten up--if anything, as the economy started to get more bleak, interview processes stretched out. (The cynic in me wants to point out that stretching out the interview process is a great way to see if the company is going to lay off; if we don't hire them, this is one person they don't have to lay off.) If companies are putting any sort of metrics on their interview pipelines, it's not many companies doing it, and their metrics aren't reaching the folks responsible for refactoring the process, it seems.
"Remote tooling vendors will look to improve." (0.6)

Result: +1. All of them rolled out various features, and just about every IDE now has some kind of "remote connect" feature allowing developers to pair- or "mob"-program together remotely. We can argue about the efficacy or efficiency of doing so, but the tools vendors definitely saw the writing on the wall and responded.
"'Older' tech will start to see a surge of interest and rise in adoption."

Result: 0. It's not widespread yet, but... check it out: a startup around hand-curating news sent to you over text message. If that's not a direct appeal to "let's simplify all this, shall we?"--and a direct rejection of AI-curated social media feeds masquerading as news--I don't know what it is.
"Theranos will teach us something... but we'll not be sure exactly what."

Result: +1. Elizabeth Holmes got 11 years (appeals pending). Her partner got 13 (again, appeals pending). Clearly "the system" wanted to send a message about fraudulent claims around startups, but overall, the reaction in the startup world has been... mixed. There's still the incessant hype around Miracle Founders (I mean, how much more smoke do we need to blow up Elon's ass already?), and much is being written, even as little is being done. Hate to give myself a point for being right about no progress, but here we are.
"NFTs, crypto; corporate promotion of NFTs followed by a huge backlash/abandonment."

Result: +1. I kinda want to give myself more than one point for this one, given how the whole thing collapsed in spectacular fashion. I mean... FTX, Bitcoin, Terra, Celsius... and not just because it was speculation in the industry, mind you, but outright malfeasance and/or stupid server errors, demonstrating the lie that "the blockchain is forever". On the one hand, yay me for getting this one right, but on the other hand... yikes. Banks and other financial firms have regulation to prevent precisely these sorts of things from happening, and "absolutely nobody saw it coming" that these similar kinds of problems would appear in cryptocurrencies? People lost their entire life's savings out of this mess. It's more than a mess, it's a scam, and people not neck-deep in it are starting to realize it--including the feds.
"More non-developer-sales companies will build Developer Relations teams."

Result: +1. DevRel is a hot commodity among companies right now, even despite the layoffs. Lots of companies are looking for "Head of DevRel" candidates, signaling a desire for a "DevRel IC who can later build out the rest of the DevRel team (in a few years, when we're sure this DevRel thing is going to stick....)".

By the way, if you're curious, here's why every company should have a DevRel team...
"Open-source backlash/flight/pressure grows."

Result: 0. There've been a few OSS packages that went the way of the Dodo bird, but for the most part, the OSS backlash I'd anticipated (both in terms of using OSS and on the part of OSS maintainers demanding more support or threatening to bail entirely) didn't really play out as strongly as I'd thought it would.
"New programming languages are going to start the R&D cycle on languages again."

Result: +1. Hello, Carbon. Hello, Cadl. Inform7 was open-sourced (April 2022). I can feel the steam building around languages, and I don't see it going away--in fact, I think it'll get even stronger. (But at the same time, I wish the industry were a little bit more open to new players here--it's hard getting any interest in some of these.)
"Some company is going to be really happy they took me up on that hypothetical in the second paragraph [about hiring me]."

Result: -1. :-( Yeah, we covered this already.

In summary:

sixteen predictions,
four 0s,
and two -1s

... leaving me with ten +1s, or something around 60%. Above average this go-round, it seems.

2023 Predictions

With that settled up, let's take a look at what I think will happen across calendar year 2023.

First off, the pandemic hasn't left us, and CDC is starting to talk about masking up again. Despite desperate efforts to pretend we're clear of COVID (and related viruses), we're not, and it's going to continue to rear its head in ugly ways going forward. I don't think we're going back to mandatory stay-at-home policies, mind you, but it's going to be a couple of years of really bad winters, medically speaking. The "taper down" I talked about last year definitely seems to be in full swing (or slope, as the case may be).

Secondly, even though the Ukraine conflict continues, and inflation reared its ugly head during the end of 2022, the economy doesn't look as bad as we thought it would six months ago, and I think things are going to "perk up" once we get through the holidays. 2022 sucked, but there's a lot of reasons to think 2023 won't be nearly as bad. (Of course, we thought that about 2022 compared to 2021, and about 2021 compared to 2020, so....)

Specifically, though, I think 2023 will show us the following:

Blockchain-related losses are going to be huge tax write-offs. It's already starting, and other companies will follow suit as soon as they see others doing it. In a related note...
Blockchain-related businesses are likely to be the target of lawsuits. (Probability 0.8) Individuals lost a ton of money in NFTs and other such schemes, and they'll likely look for lawyers to help them go after the companies that sold these worthless things--and the various personalities that helped sell them. (Note that one such lawsuit was dismissed, but the judge left it open to be re-filed with new evidence, which the plaintiffs have already declared they will do.)
The fallout from the cryptocurrency nightmare is going to stain 'Web3' for a good long time. (Probability 0.8) The criminal trials are just beginning, more cryptocurrencies and brokerages and stores are collapsing every day, and the various governments are just starting to stir on the subject; the crypto stories are nowhere close to being over, and a lot of investors are going to want to steer very clear of this whole space until things settle in. Same thing happened in the early "recreational cannabis" days--and still isn't entire clear, while the feds take their time trying to figure out where they want to sit on the issue. "Web3" is not going to be a fun place to be for a while.
Distributed systems start to embrace contract-driven development cycles. (Probability 0.7) HTTP-based APIs have become a staple of integration and partnership now for several years, but as companies find that they need to evolve their public-facing APIs, they're running into the core problems of doing so: Namely, that partners and clients get really upset when they have to rebuild API clients from scratch to match all the parameters correctly. JSON-over-HTTP was widely chosen because it wasn't as complicated or complex as something like IIOP (the old CORBA protocol) or XML-serialization-over-SOAP, but we're now starting to feel the pain of doing something "simple" but "by hand", as opposed to have a specification document (aka an interface definition file) that we can code-generate from. That's going to cause another half-spin on the Ferris Wheel of Technology History, and things like OpenAPI/Swagger and AsyncAPI are going to start to become much more popular and much more necessary in production-quality API consumption.
Microservices start to give way back to deliberate monoliths. (Probability 0.6) The microservices hype train has just about run out of steam, as companies that embraced the microservice ideal saw some negative ramifications from having a distributed system spread out across 1700 different nodes exhibit less-than-stellar characteristics. As influential articles emerged last year, we also saw anecdotal evidence that microservices were less an architecture and more a fad--nobody could ever define exactly how granular a "microservice" was intended to be, for example. Given that the shine has come off (starting as far back as 2019), we're going to see some organizations start to compose multiple existing microservices together back into a single node, and new project will be less aggressive about trying to "microservice all the things". This could in turn have some interesting implications for the single-function services (a la Azure Functions, AWS Lambdas, etc), since they were a natural tool to use when "microservicing", since you really can't get more micro than a single function. Right? (Don't answer that--I'm sure somebody out there has figured out a way to make four services out of a single function invocation.)
Security companies are going to need to OSS their offerings. (Probability 0.5) Those companies who sell security-related software or cloud offerings sell their wares based on the premise of trust--namely, that you can trust that they will keep their secrets safe, and therefore by extension your secrets. Sadly, it's becoming more and more clear that those secrets (such as passwords or code) are hard to keep safe. Therefore, these companies are going to need to OSS their stacks, slimming down the surface area of the data they need to keep secret, in order to rebuild some of that trust. (Sadly, I kinda expect that this won't happen, because too many industry executives still believe that secrets are in the code, when most security and cryptography experts say the opposite--spread the code far and wide, but keep the keys secret. Considering cryptography experts have about twenty centuries' worth of greater experience, I'm inclined to agree with the cryptography experts, myself.)
Work-From-Home (WFH) continues to normalize and become "just another location". (Probability 0.7) As much as companies would like to get people back into their (very expensive) offices, there's a lot of companies that did a lot of remote hiring during the white-hot labor market during the pandemic, and it would be a pretty big bait-and-switch (not to mention terrible PR!) to have promised "remote" employment back then and go back on it now. Not to say that a lot of companies won't, but doing so would be essentially a huge black mark against that company within the tech sector, and make it that much more difficult to hire technologists for the rest of the decade. As a result, while some companies are going to continue to be "remote-always" (because they most likely don't have an office lease somewhere that they need to justify, or they've never built a corporate HQ someplace), and some companies are going to be "office-always" (because they do!), many, if not most, companies are going to simply make "remote" another location in their "Locations" dropdown on the application web form.
Somebody is going to sue over state income tax. (Probability 0.7) One of the things that those companies who offer remote options is the fact that the legal/tax situation around remote work is murky at best. Some states have interesting laws on the books regarding employees who work within their borders, and other states have no state income tax at all. When I was at Rocket Mortgage, I discovered that there were two states in the US that Rocket simply did NOT want anybody working from, because of the byzantine tax implications it would cause us. (Don't ask me for details, I don't have them.) At some point, this is going to culminate in a legal action--against an employee, against a company, and maybe even against one or more of the state governments--because somebody didn't understand the rules, or found the rules to be prohibitive, or is facing a LOT of money lost for one reason or another. That's going to start in 2023, take many years to wind its way through the courts, and gather a lot of interesting "takes" from the business community, ending in who-knows-what kind of solution. But it'll start in 2023.
Low-code/no-code is going to accelerate further. (Probability 0.7) With an economic contraction of any form, getting your Amazing Innovation Project(tm) funded so you can hire software developers to build it will be that much harder. You'll start looking for ways around that obstacle, and as a result, interest in low-code/no-code tools will really begin to accelerate even further. Additional prediction: In 2025, interest in integrating those low-code/no-code solutions into the larger fabric of the company's IT will skyrocket. (Personally, I think this will bleed into new language development, too, but seeing the results of that will probably take longer.)
Internal tech audits will become relatively popular. (Probability 0.4) During times like this, when we're not quite so "move fast and break things", we have a tendency to slump in the chair and look around the room at the collateral damage that the previous "go-go-go" era caused. At a company level, that means starting to take some deep audits of what we have, and some efforts to "clean house" will result. External purchases will be paused while execs look to integrate existing things rather than building new things. Regardless of what comes out of the analysis, it starts with the IT audit, though, and as a result, it's likely to be a pretty high-priority project on everybody's road map for 2023.
Muskian-style management will be studied, cargo-culted, and lead to the failure of more than one company. (Probability 0.6) Love him or hate him, Elon Musk has propelled himself into the center of all things "management academia" with his handling of the Twitter purchase and takeover. Personally, I think he's flailing and thrashing and in general way out of his depth, but one thing is for sure: His rampant (and some suggest ill-advised) decisions about Return-To-Office and refusal to honor certain debts (like employee severances and expenses) are being watched very carefully by other CEOs around the industry, some of whom already are signaling that they're thinking about following suit.

At the very least, there's going to be a book or two that will be published on all this, probably due to hit shelves somewhere at the end of the calendar year, and it will garner all sorts of reviews from established periodicals. Some will be critical of Elon's approach, some will seek to find some sort of "underlying meaning" behind his actions that we can carry away insights from, but all will be looking, pointing, whispering, and taking notes.
2023 will be the year we try to figure out what to do with all this AI stuff. (Probability 0.8) ChatGPT and OpenAI have completely taken the media and social media world by storm, but it isn't clear yet how any of that actually helps. Like most applications of AI, it will likely be a mix of human creativity and AI computing power, but how? With all the image-recognition stuff that was the darling of social media, we ended up doing.... what... with it? It certainly hasn't changed anybody's life that I know. That's what we'll spend 2023 trying to figure out.

Oh, and the whole "GitHub Co-Pilot is going to take your job, you silly programmers" hype? Hot air. Whipped-up hot air. With all the nutrition and substance of cotton candy. Look, CoPilot might be able to take certain things on its own hands, but (1) it's proprietary to GitHub, (2) it's built on top of open-source projects whose contents are not acknowledged, violating license, and (3) it's not going to be able to bring the creative element that humans bring. It's a legal mess, and given the deep pockets GitHub is now blooming from (that is, Microsoft), it's ripe for some major legal action before it can safely take anyone's job. (Not to mention, somebody still has to drive CoPilot.) Worst case? CoPilot elevates the abstraction level, bringing a more natural-language element to programming languages--because somebody still has to figure out which algorithms and how to "codify" business rules. (FWIW, people said programmers were obsolete with the release of Visual Basic 3 and other high-level 4GLs; how'd that turn out?)

BONUS PREDICTION: 2024 will be the year that we start putting AI into some of our apps--like using ChatGPT for chatbots--and 2025 will be the year that we start ripping it all back out because we completely got it wrong. (We always do this. And I see no signs that we'll stop in my lifetime.)
Companies will accelerate their drive to become 'tech companies'. (Probability 0.6) For quite a few years now, there's been a drive among a number of firms to be seen as a "tech company", with all the perks and benefits of being one, but with many (if not most) of the companies seeking to make this transformation unsure of what that exactly means. Particularly with the projected "soft" IT market next year to start, it will be a good time to take a moment to take stock, reshape the thought process, and then jump into it, particularly because...
Hiring will accelerate at the end of 1Q/2023. (Probability 0.8) Even as companies are engaging in layoffs, these aren't the savage cuts that we saw back in the Dot-Bomb era (circa 2001) or the Great Recession (circa 2008) era. Both of those were about the industry figuring out that whole lines of business were suddenly seen for what they were (unprofitable--Pets.com, anyone?) and making some fundamental adjustments to our thinking. This time around feels more like a "correction", more of a "We know there's something in this line of business, but we didn't quite nail it and we need to re-examine and re-allocate". People I've talked to at both Microsoft and Amazon feel like as soon as this "organizational refactoring" is finished (which shouldn't take too long after the holidays), things will start to spin back up again. Which will also mean that...
Entry-level positions are going to be easier to come by. (Probability 0.5) When the market gets soft like this one appears to be, companies often start looking for ways to do take smaller risk in their hiring by hiring people newer to the workforce--a la junior and associate developers (and their kin), who they can hire at half of the cost of a senior software developer, thus risking less against their budget in the event the hire doesn't work out. (It doesn't mean companies will make it any easier for those early-stage developers to get hired, though, because they'll still be trying to put candidates through seventeen rounds of interviews over a six-month period before offering...)
Platform-oriented development is going to begin making some waves. (Probability 0.6) As companies start looking for ways to create new revenue opportunities for themselves, or at making the transition into becoming a "tech company", they're going to start looking very hard (once again) at how to build platforms out of their business, and the software they seek to build will follow suit--in other words, developers will be asked to build platforms rather than just hooking into one.

In fact, there is a reasonable chance (and maybe this is more of a 2024 thing) that libraries/frameworks/languages will emerge to make it easier to integrate two platforms together more seamlessly and easily, rather than the current crop of tools designed to consume a platform's offering.
Cloud will begin to shift. (Probability 0.5) TikTok is now an enemy of the US state, because it appears to be a tool of the Chinese state. Russian hackers and bots are shadowy figures looking to destabilize just about anything they can in the West, or so the pundits would have us believe (and which is probably at least partly true). And massive data breaches continue to plague comapnies left and right (lookin' at you, Twitter....). Cloud is not responsible for these things, and in many cases will be a powerful tool to fight against them, but the US government is signaling that it is going to start cracking down on some aspects of how the virtual world operates--and that could mean some guidance and/or restrictions around where data gets to live and where it gets to be processed. Even before the Feds tell CEOs where to put it, though, CEOs are going to start asking themselves the same questions, and this will lead to some "reallocation" of assets in and around various clouds. It's not likely cloud usage will shrink, but how we use cloud and where the cloud servers that we use are located will become more important.
BOLD TWITTER TAKE: Elon will sell Twitter (perhaps involuntarily) at the end of 2023, to another tech firm (like Microsoft or Oracle). (Probability 0.6) The current model is unsustainable. Twitter has no source of income: advertisers are bailing, and his "verified checkmarks" income is nowhere close to what he needs to keep things going even under the new staffing levels. (And let's just leave aside the existing debts and/or expenses from the pending lawsuits that are likely to slam into him in 2023--I'm sure he can get Tesla or SpaceX to cover those, no matter how illegal that may be....) Twitter's losing people in droves, there's no new signups to replace the losses, and it's now become clear that where Twitter might have had a "cultural monopoly" on the idea of "short-form mass-widespread posting" (as opposed to some of the other forms of social media), that monopoly has now been shattered as non-technical folks start to pick up on terms like "Mastodon" and "Post". Twitter needs users, and users are very quickly learning they don't need Twitter. Without stable leadership and a massive infusion of cash, the platform will plunge in value until one of the other tech giants just shrugs and says, "We can pick it up out of petty cash--how can we not buy it now?"

And that's how the grand MuskTwitter experiment will end. Not with a bang, but with a pathetic whimper.

Meanwhile, alternatives to Twitter will continue to sprout up, but none will actually be able to be everything Twitter was, partly because Twitter was (at the time) unique in its approach to social media, and as a result it created a "perfect storm" that led the platform to its current (well, pre-Muskian) level of success. That will be nearly impossible to recreate.
BOLDER SOCIAL MEDIA TAKE: RSS will make a comeback. (Probability 0.4) RSS, the protocol that make blog-aggregators work to consolidate/federate all the blogs you wanted to read into one place, will be adapted or adjusted to work with all the differnet social media sites you want to read, so that social media aggregators can now bring all the social media you want to read into a single place. And if it's not RSS, it'll be a new protocol that does basically the same thing (because our industry loves building a new protocol to do the same thing an old protocol does, only "better"). Maybe ActivityPub but I won't hold my breath.

Talk to you all next year... if not sooner.

Of Cost Centers, and Competitive Advantages

Wed, 28 Dec 2022 00:00:00 +0000

tl;dr: The recent collapse of Southwest Airlines' operations that forced them to cancel 2/3 of all of their flights across the country, which current analysis suggests was due to "outdated IT systems" (specifically their employee-scheduling system), reminded me of a conversation over dinner with some speaker-friends a few years back. Companies can be classified into two categories: Those that view IT as a "cost center", and those that view IT as a "competitive advantage". How do you see the difference, and if you're a part of management, how do you influence the outcome?

Cost centers

For those unfamiliar with the terms, the official definition of term "cost center" is "a department or other unit within an organization to which costs may be charged for accounting purposes." This is frequently the case for internal (enterprise) IT software development teams, who "bill" other departments for development of projects that those other departments are looking to have built. More importantly, these organizations often don't add directly to the company's income or profits, which then leads to some less-than-ideal implications.

For example, if the Accounting department needs some custom software built to help track costs and an off-the-shelf solution doesn't work, it's pretty common for Accounting to "hire" a team from the IT department to build that software and "pay" for it out of their budget. Since this new Accounting software won't really add any new lines of income to the company, or grow an existing line of income, the only "measurable metric" for the IT department is how much it draws away from the income--in other words, how much does it cost? And if more cost is bad, then the natural growth/improvement approach here is to look for ways to minimize costs.

That statement is dangerous.

On the surface of things, sure, it makes sense to minimize costs--if my team can get a project done faster, then we cost less and all is good. Of course, because there is no countervailing metric by which to measure the quality or effectiveness of the project, it becomes something of a runaway metric: We can cut costs by reducing the amount of work we do, or by paying more junior people less to do it, or even by removing people entirely! This approach works all the way up until Accounting complains about it, at which time we can take on a new project with them to address the issue. Our metrics look good: we cut our costs down, and we got a follow-up project out of it.

Internal accounting like this leads to some really screwed-up results sometimes, particularly for those companies that insist on being "metrics-facing" or "data-driven".

(Yes, I know Amazon right now is strong on using data to drive their decisions, and everybody loves them for it--and yet, "data-driven" was a thing twenty years ago, too, and it was a synonym for "metrics-facing", and it led to the kinds of results I just described.)

Competitive advantage

If your IT department is a "competitive advantage", it means your IT department is at the forefront of how the company is able to accomplish things that other companies can't (or can't as quickly). Rocket Mortgage, for example, was the first mortgage lender to put the mortgage application process on a website, thus opening up a whole new line of income for the company. Those of you readers who are about 30 or older probably remember a time when flying a commercial flight meant arriving at the airport, going to the airline counter, giving your name and ID, and receiving printed boarding passes before going through security. Then an airline realized that the process of verifying the ID and the name and printing the boarding pass could be done by a kiosk. Then another airline realized that it didn't really require being in the airport to do that, and boarding passes could be printed ahead of time on your own computer and printer. Then yet another airline realized that you didn't even need the paper, and the "airline app" was born, using QR codes and scanners to let you on the airplane.

By the way, it should be pretty obvious that the first company to do something like this has a competitive advantage; by the time the third or fourth competitor in the same space does it, it's table stakes--it would be a significant missing feature if they didn't have that feature. Imagine being the airline today that requires printed boarding passes available only at the airline counter.

The point here is that these approaches and these strategies are never static--today's "IT as a cost center" company is yesterday's "IT is a competitive advantage" company, and vice versa. Consider: after building the website to start mortgage applications, what has Rocket Mortgage done to leverage IT as an advantage? Delta was the first to roll out their "Fly Delta" app, but aside from piling more features into the app (airport maps, access to in-flight entertainment options, tracking frequent-flier status, and so on), what else can the airline's IT group do to hold an advantage over their competitors? (Answer: I was recently on a Delta flight where the in-flight monnitors had Bluetooth, meaning for the first time I didn't have to use my corded Bose speakers to watch a movie. Smooth.)

Life wasn't always smooth for Delta's IT. They've clearly made a strong effort to redress that, and use IT as a differentiating factor from their competitors.

Take careful note that being a "competitive advantage" company doesn't always mean not having legacy systems--Delta still uses some of the some infrastructure that all the airlines share, and always will. Lots of banks still use the COBOL-based mainframes to exchange electronic records that maintain the accounts between them. This hasn't stopped Chase from having apps that allow you to deposit checks using your mobile phone's camera, however.

Knowing the difference

The real key is, where are your IT projects coming from, and where are they going? If your project is customer-facing, there's a strong chance it's going to be in "competitive advantage" territory... but not always. If a project is to replace a legacy system whose functionality growth has stagnated, that can in turn unlock some new features and/or speed of execution that would be impossible to achieve with the legacy system. And not all internal-facing projects are "cost center" candidates; building a platform that allows the company to more quickly and easily integrate with their supply chain, for example, can help speed the delivery of product to customers, and/or open up new sales opportunities or options or products.

The bigger question is, as it so often is in business, what is the company's strategy around this? If the project aligns with the company's overall strategy, then it is likely to be a competitive advantage project. If it doesn't, it's probably falling into the "cost center" bucket.

Ironically, just about anything can be a competitive advantage within a company. Those companies that really go above and beyond to take care of their employees often see a significant competitive advantage from doing so. Those that invest into their infrastructure often see it pay off when the investment creates resiliency that their competitors lack, and allow them to keep going when something disastrous strikes. For many years, for example, the key competitive advantage McDonald's had (and arguably still does) was the heavy degree of investment into their supply chain, which helps them isolate against supply-chain shocks. (Which I think, in late 2022, we can all better appreciate the need to consider.)

Implications

Southwest pretty clearly considered its IT to be a "cost center", which doesn't surprise me in the slightest, given both the fact that they've not really used IT as a competitive advantage anywhere in their execution. (Ironically, Southwest used to be one of those companies that invested heavily into their employees, and still might, but that's a different discussion for a different day.)

Note that the people running Southwest are not stupid--if somebody had come to them a year ago and said, "You are going to have to cancel 2/3 of your entire flight catalog for several days if you don't fix this scheduling system", they would have immediately made that project a top priority. But that's still a cost-center mentality, because they would be comparing the cost of that project against the cost of those cancellations (which has a pretty clear winner). But in the absence of something to compare the cost of that project against, a cost center mentality says "let's not incur the cost" and postpones it or duct-tapes it or addresses only the most critical flaws. Which, then, leads us to a system that is only marginally able to resist a system-wide shock.

Like the "Blizzard of 2022". Or the "Pandemic of 2020". Or the "Server Fire of 2016". Or ....

Speculations

Note that one of the reasons companies often don't engage in exercises of IT competitive advantage is that it can be so damn expensive to do so. Given that any non-trivial software project can take a team of five or more upwards of nine to twelve months before something tangible is able to be rolled out for people (customers or otherwise) to start using, you're looking at a pretty large bet on something that--if truly innovative--has no guarantees of actually yielding anything more than a yawn (or worse, a new legacy system that has to be maintained).

If you're an IT manager or organization head, then, your goal is to find ways to enable IT teams to be able to get something up to a usable state as quickly as possible--the traditional "Lean MVP"--so that the business can find out if this is something that will "move the needle" for the company or not. Part of that means putting some infrastructure in place to enable that fast development--what we now refer to as a "platform". Build your platform to allow for people to innovate on it, and they will.

And the next thing you know, suddenly, your IT team is in the "competitive advantage" side of the ledger.

Hello Cadl

Thu, 15 Dec 2022 00:00:00 +0000

This one just crossed my feed today: Cadl, "... a language for describing cloud service APIs and generating other API description languages, client and service code, documentation, and other assets."

In other words, you write this:

import "@cadl-lang/rest";

using Cadl.Http;

@server("https://example.com", "Single server endpoint")
@route("/example")
namespace Example {
  @get
  @route("/message")
  op getMessage(): string;
}

... and when you "compile" it, you get an OpenAPI (or gRPC, or some other service-oriented protocol) specification out the other end. From there, it seems, you use other tools to go from OpenAPI to one of the service client or implementation languages (C#, Java, Python, Ruby, whatever suits your fancy), so long as there's an OpenAPI code generator for it.

First take: this seems so much less feature-rich than something like Ballerina, but I suppose it will appeal to those who prefer to get all their Azure-related tooling from Microsoft and/or prefer their stack to be entirely JS-based. (Cadl seems to be a product of TypeScript.)

Will plan on investigating this more, soon.

(Cross-posted to my Dev.to account, mostly as an exercise to see which one gathers traffic.)

Speaking Tips: Mistakes happen; get over it

Thu, 5 May 2022 00:00:00 +0000

I can't say how many times I've talked to new speakers who watch speakers and are amazed at how effortless their flawless presentations seem to go. Folks, part of the reason they go that smoothly is because they're practiced, but part of it is also that the speaker is often ready for the mistake to happen.

A long time ago, when I was but a wee speaker learning the Art of the Technical Talk from the DevelopMentor Grand Master of Speakers (Ron Sumida), one of the things he told me was that "Everything that happens in that room, happens because you intended it that way." If a demo fails, you intended it that way, because you wanted to give the room a chance to see the failure, and then you intended to show them how to recover from it. If a slide is in the wrong order, it's because you wanted to get that point across before moving on to a different point. If water starts pouring out of the ceiling....

Yeah, that really happened to me once. While teaching a DevelopMentor open-enrollment class at the corporate office down in LA, one day I'm in the middle of class, just turned around to start doing a demo when water starts pouring out of the ceiling--literally through one of the ceiling tiles--like somebody had just turned on a faucet right above us.

That was not intended.

I ran out of the room, grabbed one of the fisherman-buckets in the lobby in which we kept snacks, dumped out the snacks, ran back to the room, and put it under the pouring water. Not a perfect solution, but it allowed me to deal with the problem temporarily, then get the class back on track. This time, to the soothing sounds of water rushing by.

Another time, I was doing a talk at Devoxx (back when it was still called "Javapolis") in Antwerp, and it was not going well. It was supposed to be a talk on the various ways that Microsoft Office and JavaEE (then called J2EE) could work togther (and you'd be surprised at how many different ways they could--it was pretty seamless), and just about every demo was bombing for one reason or another. I'd even carefully put all the demo code into a VMWare VM, so that it would be isolated from any other changes or environment updates on my laptop, but just one demo after another wasn't working. And it was odd--certain files were truncated in weird positions. Finally, in a fit of exasperation, I said, "OK, folks, let's try one more demo, and then if that doesn't work, I'm just going to surrender my laptop to the Demo Gods and go the rest of the way without demos."

And that's when the VM blue-screened in front of everybody.

Did I mention that Javapolis/Devoxx is held in a movie theater? You have never seen a blue screen of death until you've seen it on the big screen.

Fully expecting to get hammered in evals, I was surprised when I heard that I was actually doing quite well. I ran across one of the attendees from that talk later in the conference, and I flat-out asked him, "What did you think of my talk?"

"Oh, I loved it, I thought it was great!"

I was flabbergasted. "If you don't mind me asking, why? Why did you think it was great? Every single demo bombed!"

"Oh, but you never let it stop you. I believe you when you say the demo is supposed to work, so when it didn't, and you just kept going, it was great!"

I don't know if that attendee could see the shaft of enlightenment settle from the clouds above (Javapolis was still a smoking event back then, and there was always a thick haze of cigarette smoke near the ceiling in the hall), but in that moment, I truly understood Ron's precept.

It's not that you intended the demo to bomb, or the water to start pouring out of the ceiling.

It's that this is your room, and you are in full control of this presentation. It's really about your confidence in your knowledge, your skills, and your talk coming through: "I am fully ready to deal with whatever may come, and I will guarantee you that nothing will derail us from me taking you on this journey with me."

Not even a tornado warning, drunk co-presenter, or gastrointestinal difficulties. (All of which have happened to me, by the way.)

Everybody who has ever attended a tech talk has seen a demo bomb. Your demo failing is not anything new to them. It's really not. (What, you think you're special, that your demo is somehow better or more amazing than anybody else's demo? It's just a demo, fercryinoutloud!) And your demo is not the centerpoint of your talk, and it will fail in ways you could never have predicted. And that's OK.

But what if your demo is the one thing they were hoping to see? Hardly likely, but let's even assume that it is: You think those attendees whose career hangs on seeing your demo working aren't able to come up to you later and ask, "Do you mind if we try that demo again, I'd love to see it working" and sit with you at a table working through it if they cared that much? (Ironically, the only time that's happened to me is after that very same Javapolis presentation, which was when we discovered that my VM's virtual disk had corrupted. Fortunately I had a copy of the VM on another external hard drive, so it was easy to spin that one up and show the demo off as it was meant to be seen. They went away happy, and I went away educated.)

The key thing is, if you do live code demos, you need to be comfortable with two things:

Be ready to debug on the fly. You will forget a semicolon, you will misspell the name of the method you need to apply, you will forget that the file you're trying to load is one directory up or down from where you're looking to load it from, you will make mistakes. Most of them will be simple, stupid, and fixable. And frankly, there's value in showing attendees how to debug the very same kinds of mistakes that they're going to make when they get home and try to repeat your demo. But you have to be comfortable debugging on the fly. Even then....
Be willing to abandon the demo. Sometimes, it's just not going to happen. It's more subtle than you thought, it's trickier than you thought, or worse yet the version of the library you're using silently upgraded without you realizing it, and now they're looking for a differently-named method or the file is in a different place. Sometimes, it's just going to take more time than you've got to debug.
And in that moment, you have a choice--stay focused on the room, abandon the demo (my usual choice of words is, "Do you believe me that this is supposed to work?" which usually draws a chuckle or two), and move on with your talk. Or, you can stay focused on YOU, and your need to get this demo to work, and abandon the rest of the talk.

Remember, the audience is here to hear a story. They don't need your demo to hear the story. The demo is there to help them visualize and see the story play out in front of their eyes--but it's not the story as a whole, and if they have to use their imagination a little, that's not a terrible thing.

Mistakes will happen. Get over it. Fast.

PostScript. If you really don't want to deal with failing demos, try the "Milli-Vanilli" pattern from Presentation Patterns: record a video of the demo on your machine, and embed the video in your presentation and play it during your talk. That way you can talk about what you're doing in the video while it's playing, and you don't have to try and split your brain coding and talking at the same time. Of course, never discount the possibility that the video could fail during your talk... at which point you look at the audience and say, "Do you believe me when I tell you that this video was supposed to work...?"

The Founding of Solidify/US

Fri, 29 Apr 2022 00:00:00 +0000

tl;dr: Solidify, a Sweden-headquarted consulting company, is creating a US division, and they've asked me to build it out. Hence, I am announcing the foundation of Solidify/US, a software development consulting company with deep expertise in DevOps (AzureDevOps and GitHub) and a focus on helping companies maximize their software development efforts.

EDIT: 15 December 2022 Sadly, it was a short one. With the dark clouds of the global economic slowdown on the horizon, as well as the uncertain clouds of war hovering over Ukraine, the folks at Solidify decided that it was a little too risky to take the chance on starting a business here in the US. Thus, I continue my search for my next new adventure.

It's been an interesting road since Rocket Mortgage and I parted ways back in October 2021. I've interviewed with a number of companies, some big and some small, some product-focused and some consulting, and kept finding something missing in each. In many cases, they chose not to move forward with the process, and a few of those genuinely stung--I could see some potential in those. It's hard, going through the interview cycle, particularly at the more senior executive levels of roles I was trying to reach for. Some companies were name-recognizable, others were companies nobody (including myself) had ever heard of. CTO roles, VP roles, a few Director roles, and every once in a while a recruiter tried to talk me into being an individual-contributor-Architect again or an IC Developer Advocate. A quarter of them, I turned away; the rest, either it wasn't a great fit (culturally or financially), or they chose to go a different way.

Some rejection was expected--hell, more than one of them was a long-shot "what the hell" kind of exercise--but it begins to wear down on you after time.

Then, around the Feb/Mar timeframe, I connected with these guys out of Sweden who were asking if I had any interest in joining up with them to do some consulting, particularly around DevOps and GitHub. In particular, they said, they were long-time Microsoft partners, with a great relationship with the Microsoft Sweden field office, and wanted to explore the idea of opening an office in the US to better serve customers (and hopefully break into the much bigger market!) here.

Huh.

See, I'd done the consulting company founder thing before, with iTrellis, and while I thoroughly enjoyed my time doing it, I had reached the conclusion that I didn't really want to join into another consulting company again unless it was at a pretty senior level, where I could make use of (and follow through on) a number of the things I had learned about organization design from my time at RM. Other consulting companies had talked about "maybe someday" being able to indulge me in taking ownership of things like onboarding, mentoring, and other talent development kinds of activities, but for now, could I put in some billable hours? Several small groups and I spoke, but couldn't find the senior-level fit (usually financially). Some larger consulting groups were interested, but didn't hire in anybody above a senior consultant level--I'd have to "prove my worth" and internally promote to get to the levels I was interested in.

Solidify was different--they were asking me to essentially own the US side of things, with their backing and support, and with the idea that if things go well, Solidify/US and Solidify/Europe would be mutually self-supporting partners. Obviously for the first engagement or two, I'd probably need to be at least a little billable, but as soon as was reasonable, I'd be looking to grow the business rather than "doing the work".

It quickly became apparent that if I did this, it'd be like spinning up a consulting company from scratch, but with the resources and connections of an established group to help with the bootstrapping and smooth over some of the rough patches.

Huh.

As it turns out, some of my friends and speaker circuit colleagues are neck-deep in GitHub activities, working specifically to help grow the GitHub community--the very same space in which I'd be looking to create connections and find clients. In other words, I'd get a chance to work closely with some folks that I deeply respect, admire, and enjoy. While partnering with a CEO and CTO in Sweden who are both quite delightful people to talk to, I'm discovering. It doesn't hurt either that Magnus and Mathias and I are of the same mind on a number of important points--like billable-hours targets not being 110%, like an emphasis on growing juniors into mid-level and seniors, like putting company time and resources behind ongoing training and development, like engaging with developer communities and speaking at developer conferences....

And, as I thought about it all, to top it all off, now is a really interesting time to be a software developer: AI/ML is often a key centerpiece of many IT strategies yet it's still cordoned off from the general software development pipeline, budget, and strategy; data engineering is often now expected to be a full partner in the development of software yet nowhere does it appear on a DevOps roadmap; DevOps itself is struggling to figure out how to incorporate other silos (like Product or Support, to the left and the right of the DevOps space on the timeline) into the mix even as InfoSec keeps popping its head up every so often to point out when another company got hacked; all this in a time when "low-code"/"no-code" tools keep capturing the interest of "the C suite" and upper-echelon executives even as companies look to hire more and more "full stack" developers who apparently need to know everything about everything....

Huh!

Is there really any way I say no to this?

Fast-forward to now, and here we are: Solidify/US is in the process of being built out--we're meeting with prospective clients, signing agreements (through Europe at the moment, until Solidify/US has its paperwork ducks in a row), and before too long, I'll be out looking for new clients and new consultants.

If you've an interest in consulting, with an eye towards DevOps (but frankly, I don't see that as the limit to what we do) and an emphasis on the GitHub ecosystem and the Azure platform, you might want to drop me a note over email or on LinkedIn sometime in Q3 or Q4 of this year.

If your company is struggling to try and make sense of all these new forces in software development, and you could use with a partner to help you sort out what matters vs what's hype, and how any of these forces affect your current plans and pipelines... well, I'd love to talk to you.

Should be an interesting journey.

Why the Bug Tracker

Sat, 26 Feb 2022 00:00:00 +0000

tl;dr: Allen Holub is a smart, experienced developer whose seen a lot of things in his time. But his take on the bug tracker--that it "is a symptom of a deeper problem--insufficient focus on quality" misses the mark badly on a number of fronts, even as he is slightly right about it.

First, the tweet reads:

I do wonder why anyone thinks that it’s a good idea to not just fix a bug the moment you become aware of it. Bug-tracking systems have always struck me as weird. Don’t track them; fix them.

Then, the post goes on in greater detail. Let's take some of this apart.

You do not need a bug tracking system. In fact, a bug tracking system is a symptom of a deeper problem--insufficient focus on quality.

It's a bold opening, and certainly one designed to catch peoples' attention. This is what comes from being a professional content creator, and I'm just as guilty of it at times.

Look, I get where he's coming from: I've worked on teams (particularly back in the early days when I was doing C++ on some large projects), where the bug tracking database seems to only ever get larger and more complicated and certain bugs just never get fixed and nobody ever seems to care. I remember looking through bug databases at employers and marveling at reports that were from over a year ago, and the features they referred to didn't even exist anymore--or were so deeply different that the bug just didn't matter anymore. It was a HUGE source of noise for me as a developer.

But...

In general, I fix bugs the moment they appear—not in a drop-everything sense, but as soon as I can get to it—usually within a few hours, but sometimes in a day or two when I’m done with the current story. Of course, if I’m working in the code when I notice a bug, I just fix it right then. The software I write has no known bugs in it as a consequence (and yes, I never don’t use TDD and I always add real tests as I work—these are two different things). I can trust the code as I’m working, and I can verify that I haven’t broken anything as I work by running the tests every few minutes. Takes all the pressure off and radically improves development speed.

I don’t use a bug tracker because I don’t need to. There are no bugs to track.

I know that people who haven’t tried it will be skeptical that writing a bunch of tests increases your speed, but programming has never been about the speed at which you type. Time spent typing a test is nothing when compared to overall development time, which is mostly thinking. Having the tests in place eliminates all the staring-at-the-code-wondering-if-it-will-work time and lets you focus on the problem you’re solving rather than whether the code works or not.

These three paragraphs are essentially a time-honored argument that "TDD and tests mean your code is better quality". It's an old argument. It's an argument that has some truth to it. However, the fact that Allen "doesn't need to" use a bug tracker because "there are no bugs to track" makes it pretty clear that he's never worked on a project that is sufficient size and scope that it's more than just him working on the project. (Full disclosure, I've never worked with Allen, and I'm not aware of his current work status or his work history--this is all analysis, not fact.) He's never stepped into a project that existed before he was able to bring his TDD discipline (and it is a discipline, folks, it requires a tremendous amount of willpower to do successfully) to bear.

Frankly, it's the argument of somebody who's essentially saying, "I'm better than all you idiots who can't see the clearly-superior way of working" and while it kinda fits in with his irascible "grumpy elder statesman" persona, it's not a great way to win friends and influence people.

Part of Allen's assumption here, too, is that nobody who's not a programmer will ever find a bug, and I don't know that he's that arrogant to assert that. That means that if the bug is to be fixed, there must be some place where people who are not programers can let the programmers know that "the bug exists, and these were the conditions in which we saw the bug (to the best of our memory anyway)". They aren't in a position to fix them.

Additionally, Allen seems to assert that he has complete discretion over how he is going to spend his time; he gets to decide whether he works on the bug or a feature next. As much as we'd like to assume that programmers will always prioritize what is in the business' best interest when prioritizing their work, we already know that's not always the case. And, in all fairness to programmers, they may not have the necessary context to choose between getting feature A out the door as a higher priority than fixing bug B, particularly if bug B only occurs sporadically and to a small audience, and feature A will win a million-dollar deal. And, to the programmer's chagrin, that is a perfectly reasonable decision to make. Programmers don't like it, particularly those who value their craftsmanship, but when you value your craftsmanship over business value, you're already on shaky ground.

Lastly, bug trackers serve as a means of information transfer--on a large project, I may not have the knowledge of the code in which the bug occurs, and for me to come up to speed on that area of the code well enough to be able to diagnose the problem and create a useful fix... that could take orders of magnitude longer than if I just note the problem and let somebody who has knowledge of that code attack it.

(Side note: How many of us have worked with people who've spent so much time in all parts of the system so that we can just describe the problem and they can go "Hmm.... yup, I know where it is, here's how you fix it"? One of my best friends was that guy on a team we were on twenty-five years ago. I've seen it at client engagements since then, too. And you know what happens? Everybody comes to that guy for every bug, and soon nobody bothers trying to learn the system. Let that person help drive the fix, but don't let them do the fix.)

The power of the bug tracker is also well beyond the moment the bug is discovered and its fix. Bug trackers can also provide a database and history, documenting what the problem was and how we fixed it, so that future iterations of teams can do searches through the database to find common scenarios and see if there's a larger problem that could be fixed, either cutting down the time required to diagnose a bug, or potentially finding a solution further upstream that prevents a whole class of bugs.

(I am well aware that not many teams use a bug database that way. That doesn't mean it's not a powerful tool for those teams who do.)

But the next paragraph starts opening up some interesting avenues.

If you have so many bugs that you need to track them, it seems to me that you have serious problems in the way you do development. Fix that first. You can start by throwing away every bug report older than a month. If it’s important, it will come back. Then distribute the bugs out to the teams. In other than a very small organization, the teams will need to work out some way to fairly route bug reports. Throwing dice works pretty well. Assign a number to each team. Throw the dice. You can round robin, too. The team can fix it whenever it’s convenient, but there should be an upper limit on that—a few days, maybe. If your stories are properly sized, that means you can finish the current story before you move to the bug fix. If you’re interrupting people, you’re doing it wrong. A bug is rarely a crisis.

It's a throwaway line, but "throwing away every bug report older than a month" hides a much more interesting take that could've been much more valuable: what is the time a bug spent in a bug tracker before it is closed? If a team has a bug tracker with bugs in it that stretch back by more than a month, then clearly those bugs have been deprioritized, and they should be moved into some kind of state indicating that; some teams call that a "Won't Fix" (and please by all that's holy include a reason for it so people can get the full history as to why that decisions was made). A team can and should set up some kind of review board or process (or discipline, it doesn't have to be all that formal, if the team is disciplined enough to follow through) for bugs in the tracker.

See, here's the thing that I think Allen has accidentally stumbled upon: The distance between the bug's discovery and it's resolution is a useful signal. If it takes your team less than a day to fix a bug, you're in a great place. If it takes your team less than a day to resolve that a bug will never be fixed, you're also in a great place, because now you won't waste time looking at it every single time you go into the bug database. If it takes you weeks, or months, to figure out what you're going to fix, however, you're in the same conceptual place as those teams that spent weeks or months gathering requirements.

Once again, "being agile" means identifying your source of feedback, and tightening up the loop of feedback-to-action.

And that has nothing to do with a bug database.

Book review: Supercharge Your Applications with GraalVM

Wed, 23 Feb 2022 00:00:00 +0000

A while back, Packt Publishing asked me to do a review of the book "Supercharge Your Applications with GraalVM", and while I wrote it up, I never pushed it to the blog. tl;dr, it's a decent book--a good intro to GraalVM, with some interesting tidbits in there, but not deep enough for the expert. (Full disclosure: Packt sent me a copy of the book in exchange for a review.)

For those Java developers who’ve never heard of or never explored GraalVM, the latest virtual machine technology project courtesy of Oracle Labs, it may be unclear why the world needs another virtual machine--after all, HotSpot did wonders for most Java applications when it came out in JDK 1.2 (circa 2000), and the caretakers of the JVM have been tinkering and making significant changes to the performance ever since. Garbage collectors, bytecode translation, you name it, they’ve tinkered with it. So the average Java developer could be forgiven if they were to see GraalVM, and think, "Yeah, whatever."

A B Vijay Kumar’s book, Supercharge Your Applications with GraalVM, will go to great lengths to dissuade Java developers of that naïve notion. While the book itself could use with a bit better "overarching story", explaining how the different features of GraalVM tie together and present a coherent whole, it’s not like the GraalVM documentation does a much better job of that; instead, the various features are laid out, and it’s left to the reader (or user of GraalVM) to figure out where and how each significant piece of the platform "fits".

The first criticism I might have of the book is that it doesn’t always seem to consistently know who its audience is. Chapters 1 and 2 go over some of the core basics of the JVM; Chapter 1 covers the C1 and C2 bytecode compilers, for example, which for any student of the JVM is pretty redundant. Chapter 2, though, introduces JITWatch and the +XX:PrintCompilation flag, which is generally something reserved for the serious VM explorer. While I suppose it’s possible one could go from "lightweight intro" to "serious player" in a fairly fast manner, doing so in two book chapters seems like a leap.

Chapter 3 begins the serious exploration of GraalVM itself, getting into the architecture of the system. It’s also the first hint the reader gets that GraalVM is more than "just" an upgraded JVM; in fact, it’s fair to suggest that GraalVM may very well be the last virtual machine anyone would ever need to write, thanks to its "Polyglot" support: GraalVM can run not only Java bytecode, but Python, R, Ruby, even C/C++ and Web Assembly (WASM) code. This support, known as "Truffle", is what gives GraalVM the ability to do ahead-of-time (AOT) native compilation of Java applications, which is what most of the Java industry conference speakers focus on, particularly since that native AOT compilation is what will make microservices spin up faster. However, that’s merely the surface of the subject.

Chapter 4 then dives into getting GraalVM installed and hooking up the various performance monitoring tools (like VisualVM); this chapter felt a little out-of-place to me, since I would’ve expected to see it earlier--or later--in the conversation based on the previous chapter. He also introduces another tool, the Idea Graph Visual izer, which analyzes how Graal JIT-compiles code. This doesn’t exactly feel in line with the "getting started" vibe from the earlier parts of the chapter, though the utility of the tool is pretty much self-evident. It does require Kumar dive into some compiler theory 101 kinds of topics (intermediate representations, static-single-assignment (SSA) form, and so on), which means now in one chapter we are getting started, seeing a new tool exploring the JIT compiler, and learning some core ideas of compilation, all in one span of text. It feels a little rushed at times, a little too compact, and a little out of order.

Chapter 5 then dives into a tutorial on building native images and running them with profile-guided optimization (PGO). This was really the first part of the book where one could get "hands on" with what was being taught, and I felt it needed to come sooner, probably right after getting GraalVM installed. These two chapters (4 and 5) were the most "muddled" in my mind, where it felt like the author was jumping around between some topics without a clear path in mind or a clear rationale for why we needed to read about "A" before we read about "B". In all fairness, though, that may be my own biases and prejudices coming to the fore--I’ve spent enough time writing books and teaching workshops that I pay careful attention to that "journey" the reader or student will take through the material.

Chapters 6 through 9 explore the polyglot nature of GraalVM, introducing the Truffle framework in greater depth, then exploring how to run different language-based code on top of the runtime. GraalVM can execute not only Java bytecode, but also NodeJS, Python, Ruby, or R (from their respective sources) and WebAssembly and C/C++ (through the use of the LLVM toolchain, which is a native compilation framework that many languages use to generate native code). To my mind, this is where the GraalVM story is the most interesting, and the author provides some simple examples, but I would’ve preferred a little more depth around each of the different languages and some of the integration challenges that arise.

Finally, in Chapter 10, Kumar gets to what most people will think is the real raison d’être of GraalVM: microservices. Accordingly, the chapter opens with a brief introduction of microservices which I found entirely unnecessary (does anybody not know what those are by now?) and then proceeds to build out an example of an application built out of microservices. The author then takes one of the services and builds it out using a variety of different microservice frameworks (Spring Boot, Micronaut, Quarkus). Frankly, I found this chapter pretty gratuitous, as if the publisher insisted that the book couldn’t be published unless it had microservices somewhere in it--it just feels that much out of place compared to the rest of the material. Still, for a lot of readers who are looking for answers to their day-job needs, it’s a reasonable demonstration of how GraalVM could be applied to a microservice-based architecture.

In summation:
For the virtual machine expert or aficionado, parts of the book will seem redundant, but for those who’ve never really spelunked below the surface of the Java (or Groovy or Scala or Kotlin or...) code they write, the book will offer just enough background to the subject to feel comfortable reading other resources, or enough to take what Kumar says through the rest of the book on faith. While Kumar’s writing is pretty clear and straightforward, the outline through the material is less so, and it feels like the book would’ve benefited strongly from a firm editorial hand to create more of a "journey" through the GraalVM landscape. Still, the words-to-concepts ratio means the material is dense enough to keep the reader from being bored or flipping through the pages too quickly, and that’s a plus. Additionally, it’s a useful guide for those who aren’t comfortable going directly to the documentation to learn how to use GraalVM.

If you’ve never heard of GraalVM or heard only rumors and want a hearty introduction into all the things GraalVM offers, this is a good book with which to get started, though I wouldn’t expect it to be one used over and over again once read cover-to-cover.

Overall verdict: 4 stars out of 5

Digital footprint updates

Tue, 22 Feb 2022 00:00:00 +0000

As I've mentioned on LinkedIn, I've been sans FTE employment since just before the holidays last year, and I'm using the time to do some re-shuffling of some of my "digital footprint" on the Internet. Let me explain a bit about what's going on here, mostly so that if you're consuming an RSS feed for one, and you're not getting what you expect, you may want to flip over to the other. (Or both!)

For starters, I've come to realize that I want two, separate "representations" of myself on the Internet for public consumption, one professional, the other personal.

The professional persona will be here, on newardassociates.com, in which I talk pretty much about what everybody expects me to talk about (if you don't know me personally): programming languages, platforms, virtual machines, management, and so on. Think of www.newardassociates.com as the pair to my LinkedIn account. That means there will be two top-level domains:

www: The professional website and "starting point" for anybody looking to view services, articles, presentations, and so on.
blogs: This is where a lot of my blog articles will end up living, and at some point I may redirect some of the links at blogs.tedneward.com over to here permanently, just to encourage professional readers to come here. The plan is to wire this up with IFTTT to pick up on a new blog post and post it to LinkedIn, in fact.

The personal persona, tedneward.com, will be a bit more free-form, and wide-ranging. I may talk about technology, but I will also feel more free to host other non-professional things, such as D&D, fiction I feel like posting, maybe some photos, and who knows what else. That means there's going to be a couple of different domains there, as well:

www: The starting point.
research: My "research garden" of links and notes for a whole bunch of things, including personal, professional, and anything else that seems interesting to me. Feel free to wander around in there if you like, but keep in mind that you're wandering around in a digital version of my study, so if the mess doesn't make sense...
blogs: This has historically been where I kept my professional blog (since 2005!), so I'll keep those links alive (some or most of them, anyway--some are just way too outdated to think anybody will care), but future posts will be all over the map. This one will wire up with IFTTT to announce new posts to Twitter, since Twitter is where I'm more comfortable showing off multiple sides of myself as an individual, rather than just branding as an "industry professional".

The key? I'm doing all of this using static site generators, Markdown, GitHub, GitHub Actions, and hosting them in Azure. This way, I can either use the GitHub web-editor to create a new post online directly (and have it auto-posted to the site within minutes), or queue it up by writing locally and shipping when I push to main. Automating all of this has been a thorn in my side for quite a while, but I wanted a reason to learn GitHub Actions, and while I could've done some or all of this using a variety of other platforms, I've been meaning to spend a little time with Azure anyway, so here we go. (And no, I didn't consider running one on Azure, one on AWS, one on Digital Ocean... because life is too short and I'd rather find a different forcing function for each of those cloud providers.)

If you're still reading, thanks for staying with, and hopefully this will reduce the barrier-to-blog that's kept me from writing more in the last few years. (It's kinda not, but let's pretend that's what it was and let me slide, OK?)

2022 Tech Predictions

Sun, 23 Jan 2022 00:00:00 +0000

By the way, as of this writing, I'm still looking for my next great adventure (ideally as an engineering leader-of-leaders), so if you find my analysis here to be interesting or intriguing--even if you disagree with it--perhaps there's a role in which I can do this kinds of strategic and executive thinking on your company's behalf? Would love to hear from you.

In 2021...

... I wrote a lot of stuff. (That always seems to happen when I do these.) Let's start. I'll include the full text of what I wrote in each bullet point first, then put the Result after it with whatever insights or comments seem relevant. (Arguably none of them are, but hey, it's my set of predictions, so....)

We began the new year with chaos and tragedy, at least here in the United States. The Capitol Insurrection of 2021, fueled by blatantly-false claims of election fraud and whipped into a frenzy by a narcissist whose continued civic freedom depends on being "untouchable" by legal recourse, meant that a lot of us had things far, far more important than technology trends to worry about. It's been a rough two months since then, as well, as we struggle to balance hopes for vaccines and coordinated federal response to this damnable virus against the mess that the year-plus of deliberate neglect from the previous US Adminstration created. It was hard on everyone, and in a lot of ways we all just want to "get back to normal" for whatever definition of "normal" you care to use. Unfortunately, I don't think we're anywhere close to "normal" yet, though the massive rollout of vaccination programs will help, but I'll get to that in a second.

Well... yeah. I didn't want to be right about that last part about not being close to normal, but sadly, it held up. 2022 started much as 2021 did, adding in vaccines but also deep hysteria and distrust from the anti-vaxx crowd whipped up by right-wing agitators. (I honestly never thought I'd write those words in a sentence about the United States of America. I really didn't.)

Thing is, it's becoming more and more apparent that the "normal" we all knew in 2019 is probably not coming back. Frankly, we should be used to this--the "normal" we all knew in the first half of 2001 never came back, either. I remember a time when you could escort your friends and family to the boarding gate, or meet them there when they stepped off the plane. I remember a time before 20 years of armed conflict in multiple nations in the Middle East. I remember a time when you could bring your own water onto an airplane. I didn't care for the TSA, I don't think it actually makes us any safer, but it's now a part of the fabric of our traveling life, and it's pretty clear that it's never going away. Sadly, I think the same will be true of many of the things we've come to embrace (by necessity) from this pandemic (and it kills me that the crowd that insisted we had to make all these changes in 2001 in the name of public safety is now telling us that all these changes in the name of public safety is government overreach).

Interesting thing, though: Not all these changes were bad. Some will have ramifications and ripple through the next decade or more, and the results might actually be things we come to embrace and enjoy. More to come.

"Conferences will begin to pick up in September. (Probability: 0.5) This is an important one to me, personally, because I miss going places and hanging out with friends and talking technology and trends and stuff. My thinking is that the vaccinations will reach a popular "critical mass" somewhere in June--which won't be yet "herd immunity" but enough so that everyone can know somebody personally who's been vaccinated and "nothing bad happened"--which would then mean that conferences will scramble to put something together as quickly as they can, but will still need a few momths' lead time. Other countries, who have been more aggressive to fight the virus, will be open sooner, but the US will lag, because we'd rather let people die while arguing "freedoms" than take the pragmatic route of eliminating the virus and then do the debates."

Result: 0 sigh Vaccines rolled out, but depending on where you lived in the US, you may or may not have been surrounded by a statistical majority (> 50%) of people who actually took them. Despite that, several conferences braved the wilds and were, in fact, held in person (lookin' at you, KCDC), apparently successfully. (KCDC, at least, seemed to go off successfully, and even more improbably, I heard zero cases of COVID infection stemming from the conference. Woohoo! Win!) -1 for the assumption that it would be more widespread, but +1 for the fact that some did, and did so effectively.
"Remote (anything) is going to be a huge focus. (0.7) In 2020, we had to learn how to survive without driving anywhere or stepping inside other buildings. UberEats, DoorDash, GrubHub, all these "remote dining" experiences surged to the forefront of our mobile devices' home screens. Retailers figured out how to do "remote shopping", enabling curbside or in-store pickups. Amazon, of course, saw a huge surge in use, but a surprising number of retailers took huge leaps to control their own remote distribution channel. Those that didn't will quickly start to figure it out on their own--and Amazon may actually lose some market share as a result, as for years the principal thing Amazon did was provide that remote distribution channel. Before, when that was just one of several options, it wasn't important enough to retailers to figure out--when it became their only option, though, they moved fast and slapped enough together to make it all work. In 2021, they'll start to tune, improve, and perfect those channels, and Amazon will really need to start aggressively fighting against those new channels."

Result: +1 To be fair, this was an easy call: Grocery stores now do curbside delivery, every restaurant that wants to do any kind of business has an account with one of those (UberEats, DoorDash, GrubHub, or some of the others less well-known), and streaming services continue to do a ton of business. Office work continues to remain entirely remote, which means most of your "office visits" (doctor, lawyer, financial, etc) continue to be entirely remote when/where they can, and frankly there's no signs of this slowing down any time soon. Amazon hasn't seemed to start "aggressively fighting against" those new channels yet, at least not with any new services, but they're definitely being much more aggressive in commercials, I think to try and reassure the public (and reaffirm) to shop there instead. Given how much Amazon analyzes numbers, I have a feeling that either the drain to their accounts isn't yet significant enough (perhaps there's less overlap than I thought), or they're preparing a major campaign for 2022. We will have to see.
"Amazon will get the Congressional eyeball. (0.5) Bezos already took a hit to his brand with the revelations of his infidelity to his wife, and Amazon's most recent union-busting efforts just added to the bad PR. Amazon will start to get more and more scrutiny as different companies start looking for reasons to pull their goods out of the Amazonian warehouses and do the distribution themselves--and that will cause Amazon to get a little desperate lest its profit margins come down too far... and that in turn will trigger some undesirable behavior, followed by calls for inquiry and oversight."

Result: -1 Nope. Lot sof "Big Tech" did, but to no real end, and with little actual movement. Amazon seems to have weathered the union-busting stories pretty well, and Bezos' infidelity sort of petered out as a story. (Which, to be honest, I like--his relationship with his wife is really his and hers to deal with, and not ours to judge.) Overall, no reasons to pull out of Amazon warehouses, no losses to the profit margins, no calls for inquiry, no calls for oversight. Probably the most complete "whiff" of the year.
"Bitcoin and other cryptocurrencies start drawing fire from unlikely places. (0.4) "Bitcoin consumes 'more electricity than Argentina', while just a year prior, it was Switzerland. And its carbon footprint is comparable to that of New Zealand. Whatever your thoughts about the viability of Bitcoin and other digitial currencies, the act of mining it is not only consuming ridiculous amounts of resources, that consumption is triggering wider ecological woes, and that's going to draw fire from a lot of different corners of the landscape--particularly because for so many people, it's just a weird geek-fan-whatever thing that doesn't have any impact on the wider world. Sooner or later, some country is going to take a hard-line stance against it, and that will start the dominoes in a negative direction."

Result: +1 Oh, my, the backlash is building. Of course, I in no way predicted the creation of NFTs, which drew a lot of the ire and focus around the whole growing collection of blockchain-based Ponzi schemes. We haven't seen many nations taking formal steps, but a few are.
"'AI' and 'machine learning' are going to run into major backlash. (0.6) Amazon ran into ethical issues with its use of machine-learned models for interviewing and hiring, because they didn't do the right amount of validation and verification of their models before putting them into production. Not a lot of companies are going to have the patience or AI-savvy to do that, either, and as a result, media backlash will start turning the shine of AI and machine learning into glare. By years' end, we may even see Congressional inquiry into the subject(s), under the umbrella of widespread investigation into "Big Tech"."

Result: -1 You know, I was surprised we didn't see more. I mean, like really surprised. There's a growing movement around better ethical governance of AI, but for the most part, the Congressional interest in "Big Tech" isn't really motivated or driven by AI or ML. That said, lots of groups--particularly those who represent or advocate for underprivileged/underrepresented groups--are coming out more vocally against the use of AI without oversight or analysis, and that's good, even if I don't consider it to be the backlash I'd thought was going to kick in.
"Any tech company that wants to compete will need to enable WFH on demand. (0.8) Look, this past year made it pretty obvious that anybody in the technology space can do their job from anyplace in the country--if not the world. Couple that with a continued insatiable demand for tech talent, and you are quickly setting up the model for a buyers' market when it comes to finding a job. "Hmm. You say your company needs me in the office, but this other company over there doesn't need that. Why should I work for you and add wear and tear to the streets, my car, and my nerves?" Not to mention that when you can work-from-home, you can realistically work-from-anywhere, including your parents' place while the kids are out building snowmen with grandma and grandpa. Some companies will fight hard to bring people back to the office, but they're going to be fighting an uphill battle, and may well lose out on some prime tech talent if they can't figure out how to be flexible. On that same note, if a company cannot or will not embrace a culture of "remote first" to allow those in the building and those outside the building to work on the same, level, playing field, that company will find itself scooping up candidates from the bottom half of the talent pool."

Result: +1 I feel kinda bad about giving myself credit here, because it really was there and obvious for all the world to see. Companies (Microsoft, Amazon, Google) have all tried to bring employees back to the office, but becaues of a combination of both COVID variants and worker resistance, have all but abandoned those plans thus far. I'll be curious to read the historical analysis on this in years to come, but I posit that companies' attempts to bring people back to the office is part of what touched off the "Great Resignation" and led to the mad shuffle of employment across the industry.
"Several voting systems to "secure the vote" will come out in the next two years. (0.7) You don't have to believe Trump's wild-eyed and clearly-debunked claims about election fraud to know that people are sensitive on the subject of voting. To the technology space, particularly the startup space, that just screams "I'm a market that demands disruption! Disrupt me!" and the VCs will happily toss billions of dollars at companies that promise to let you "vote from your smartphone" securely and safely. Most of these will crash and burn and take those billions down with them, but a few might make it out of beta and maybe even find some adoption in a few of the smaller states (Rhode Island would be my first target). Personally, I don't think there's a damn thing wrong with the system we have at the moment--but as with many things in the startup world, it's not about fact, it's about perception."

Result: -1 Well, I suppose I have another year to go before I have to call the result on this one, but for the most part we seem pretty content to just let things go as they are. If anything, one half of the political electorate seems to prefer the idea of restricting who gets to vote and how easily, rather than trying to secure better the systems of how we vote. It will be interesting to see how this continues over the next year, going into the mid-term elections, but right now it looks very much like the conservatives of the country are quite content to keep in place the same system they deeply distrusted over the last year--and, I suspect, leave them the opportunity to cry foul if and when their candidate(s) loses again next year.
"Social media spinoffs will proliferate. (0.6) Parler was just the beginning--now that we've proven beyond the shadow of a doubt that we really can't get along together on a single social media platform, many groups will start flocking to social media platforms of their own choice, further polarizing the conversations there. It will have nothing to do with technology or security--it will have everything to do with humans' basic desire to be among people "just like me". Ironically, Facebook and Instagram will be losing their stranglehold grip on the social media world just as they start feeling the ire of Congressional gaze, which will further their downward trend. These new platforms will have some space in which to grow and draw eyeballs--and unfortunately allow people to stew inside echo chambers even more."

Result: 0 There've been a few that have gotten under way, but they've really not seen a lot of uptick, it seems. We as a group are perhaps too deeply addicted to our habits of Facebook and Twitter to consider moving our social profile to a newcomer. That said, though, they're still out there, and drawing some attention, and it's important to remember that Rome (and Facebook) wasn't built in a day (or year), either.
"Languages and platforms will march on, and no one will really care. (0.8) Anybody know what version of Java is the latest? C#? JavaScript? Fact is, most of the popular languages in use are in the tapering-off point of the "value gained per new feature" curve, and most of us have stopped paying attention to the feature list of new releases. Sure, you may have some pet feature of C# you're waiting to see make it into the long-term-support (LTS) release, so your company can move to adopt that version of the language, but for the most part, whether that happens this year or next, it really makes little difference to your ship schedule. For the most part, now, knowing the latest features in these languages is about proving your knowledge and alpha-geek-ness, not about using them to solve actual business problems."

Result: +1 Java's at 18, C# is at 9 or 10 or something, and who even knows what ECMAScript 2021 looks like.
"A collegiate and university education backlash is about to begin. (0.7) When the pandemic hit, all the schools went into lockdown, including the collegiate and university campuses that had for so many years told us that the thousands of dollars students spend there was for the "college experience". When the "college experience" consists of Zoom calls from your home from professors who often couldn't figure out how to use Zoom for the first half of the semester, delivering substandard content compared to courses from Udemy or Khan Academy, and exposing professorial misconduct (like the professor who chastised a hard-of-hearing student for 'not paying attention' during remote classes) for all the world to see, it's really, really hard to understand why anyone would pay thousands of dollars per semsester for that privilege. Given how coding bootcamps can provide much of the tactical skills (but not the larger conceptual understanding--they're trade schools, not universities, and let's keep that clear) that employers are looking for in junior developers, why would I spend $50k to get that diploma from UC Riverside or Iowa State in four years? Colleges and universities are going to have to scramble to figure out how to keep from just doing "business as usual", if they want to avoid a striking loss of registration numbers that will only get worse as the pandemic continues."

Result: 0 The backlash around remote education definitely shifted to the elementary and high schools, but thus far universities and colleges aren't feeling the sting--yet. Anecdotally, I talk with college students, and they're asking these same questions, though, and thus far the universities and colleges haven't really put together any kind of message that addresses it, so the backlash may be starting quietly. Expecting to see any sort of tangible or measurable result in a single calendar year might have been a little aggressive; this one may take a decade to play out. For now, I'm still seeing people talk about it, but a lot of the energy is caught up in online-vs-in-person debates. Students and faculty simultaneously (a) don't want to expose themselves to COVID, and (b) don't want to spend their college experience online.
"Resilience and failover will be the focus in the Ops-minded world. (0.7) The huge loads on servers from people staying at home and using digital channels caught a lot of companies by surprise, and they suffered outages as their predictive load models failed them utterly. In 2021, these companies will put a bunch of engineering into making sure they can handle the load, just in time to see the load start to draw back down as people start getting out of their homes more. Once the pandemic eases up enough to allow us to leave the house and dine safely anywhere in town, we won't see that high-water mark of server load again for some time--but companies, like generals, love to prepare to fight the last war, and they will put a large investment of time, money, and thought into systems that can scale in time for the next pandemic (a hundred years from now)."

Result: +1 It definitely was the focus during my time at Rocket Mortgage, and from watching some of the articles and periodicals around the DevOps space, resiliency and failover were definitely hot-button topics. Some of that could easily be confirmation bias, though, and honestly I was expecting more, so I'm marking it a zero, Dude.

In summary:

eleven predictions,
three 0s,
and three -1s

... leaving me with five +1s, just a shade under 50%. Not bad, and about my average, it seems.

2022 Predictions

With that settled up, let's take a look at what I think will happen across calendar year 2022.

Overall, it's going to be "more of the same" with respect to the pandemic. Thanks to the intransigence of some selfish and closed-minded people, the COVID virus has plenty of people in whom it can mutate and incubate before spawing a new variant, and that will mean that COVID will, likely, remain a fact of our lives for quite some time to come. Hopefully we can get enough of a grip on it that it becomes more "comfortable" for us as a society, a la influenza, but COVID for the moment still represents a major threat to the lives and health of too many folks. For now, all indications are that there won't be a "cliff" that will indicate the end of the pandemic and a return to normal life; we will likely "taper down" slowly, masks and vaccine cards and boosters (and all the arguments that come with all of those) being a part of our lives for probably a half-decade to come.

Specifically, though, I think 2022 will show us the following:

5G rollout is going to be a lot more contentious than we expected. (Probability 0.6) 5G rollout two years ago wasn't a sure thing, but then it was because of crackpot theories about how it would affect brainwaves or somesuch nonsense. Now, the FAA has concerns, and that's a helluva lot more serious. It's pretty likely that the technical issues will be addressed, but this could very well turn into the next conspiracy-theorist farmground, which in turn will create barriers and obstacles.
Demand for full-stack developers will continue to leave companies high-and-dry. (0.8) If this isn't alreday self-evident to you, you're clearly not in a position of trying to hire engineers these days. Big Tech companies are throwing ever-greater compensation numbers at people, driving the equilibrium price for developers up, and that's before we even consider the ever-increasing numbers of tech jobs that are outpacing the numbers of people who can fill them. If a company is expecting to grow their developer staff significantly, trying to hire "full-stack" is going to run into significant hurdles, particularly if they aren't willing to move fast, take a few risks, and pay handsomely.

This is going to have some interesting second-order effects:
- Work-from-home will become normalized as part of the landscape. (0.8) Talk to that one friend you have that's a classic Keynesian economist, and they'll tell you that when supply is tight and demand is high, in addition to a high equilibrium price, the market starts to see "complementary" and "supplementary" effects kicking in. In this case, those high-demand engineers are going to get their pick of the perks, and one that many of them seem to value highly is the flexibility to work from home. That in turn will create (or enhance) pressure to let everybody on the team to work remotely--or else find that you need to hire replacements to adjust for the attrition of people leaving for places who can, still, work from home.
- Companies will start "growing" their own developer staff. (0.6) It's not something that every company can do, but those that can are going to start thinking, "If I can't buy the people I need, then I'll have to grow them." Rocket Mortgage was ahead of the game on this one, having put a program in place a few years ago already that was, for all intents and purposes, a developer bootcamp for people wishing ot make a mid-career change. Not all companies will have the resources of the nation's #1 mortgage lender, but other approaches are available: partner with existing bootcamps, join a few companies together to sponsor one, hire a bunch of juniors and apprentice them to others within the company, or even do something a little more informal like a mentoring program or internal book club or something. Whatever it is, though, those companies that start thinking aggressively about growing their next generation of developers are going to see it pay off; if you take a junior or a recent bootcamp grad and invest in them, their gratitude and loyalty will more than pay off.
- Automation tools and platforms will climb in use. (0.7) The more work we can pull off the senior full-stack developer, the more work they can get done that can't be automated. As a result, managers and execs are going to start looking for tools and platforms that can automate all the things--well beyond the whole "DevOps" suite of things. Workflows, processes, emails, the whole enchilda, it's all going to be "on the table" for anyone or anything that can cut the developer load down. This would be a good year to have an automation startup going.
  
  By the way, corollary to this: Tools to automate the use of online services--and the rampant abuse and fraud using those tools--will also climb through the roof. If you offer a website or an API (and who doesn't?!?), spend some time in 2022 figuring out how people can abuse your system. (If your answer is a confident, "They can't!", you didn't think hard enough. If your answer is a confident, "They wouldn't!" then your business is clearly not diverse or interesting enough.)
- No-code/low-code tools and platforms will surge. (0.7) Same song, different verse: low supply plus high demand equals high equilibrium price for that good/service, and supplementary (replacement) options will begin to become more attractive. For the developer ecosystem, that's low-code/no-code tools and platforms. I expect there will be a ton of chaos and movement in this space, and probably by 2023 or 2024 these platforms will start to diversify as their various features commodotize. But that'll take a while--for now, we're just going to see a sharp rise in the use of these tools, mostly by companies that have never considered themselves "tech" companies. (And if you're a consulting provider to these sorts of firms, you might want to think about how to tap into this world--and not just by selling services on top of somebody else's low-code/no-code platform, because those vendors are going to be pushing the barrier-to-entry for use of their platform as low as they can reasonably get it--which means now you're just selling labor, not expertise.)
- Developer market will grow. Slowly. (0.7) The other thing that a high equilibrium price does? Attract more players to it.
- Job descriptions will emphasize more about skills (including human skills) and less on degrees/certifications. (0.8) One interesting corollary to the tight-labor-market perspective is that companies who are finding it hard to hire are going to start paying closer attention to their job descriptions and tune them more effectively to attract talent. Five years ago, "5 years Java" wouldn't have been really a great match against a senior C# development position; today, that'll totally work, if the candidate is down to learn C#. So let's not call it "5 years Java", let's call it "Comfortable with an O-O enterprise language".
  
  The "laundry list" of "X years in Y technology" are going to start giving way to "(Feeling) in (category)" kinds of descriptors, like "Comfortable in object-oriented languages" or "Conversant with Web front-end development". Companies might have been able to let their ATS (applicant tracking system) pick and choose which resumes to view back when the supply was higher and they didn't have as many people they needed to hire; that's gone for now.
  
  Smart companies are also going to zoom in on their interview pipelines and look for ways to trim costs (elapsed time and person-hours being two metrics I'd examine), as well as look at the overall efficiency of their interview pipeline (# of candidates against # of hires) to make sure their recruiting team is giving them the biggest bang-for-recruiting-buck.
Remote tooling vendors will look to improve. (0.6) Teams, Zoom, WebEx, they all have one thing in common: We routinely lament the fact that "they all suck". (Anecdotally, which one you think sucks the most appears to be directly proportional to the amount of time you use it.) The fact that we can do this sort of virtual meeting at all is a pretty amazing thing (on the same order of, "It's a chair, in the sky!", if you ask me), but there's still some things that the meeting tools don't really handle well (shared whiteboarding/doodling, for example, which is really important to technical conversations). The various players are now well-established--so long as we have hybrid WFA/WFO in place, these tools will be a part of our lives--so now they need to start figuring out how to differentiate from one another. (Which will commodotize over time, and arguably already has, but that's another analysis for another day.)

As a corollary to this, look for lots of other tools to figure out how to embed/incorporate remote aspects into their workflow. Visual Studio Code started this, and I imagine the other IDE vendors are hard at work figuring out how to duplicate that feature or something similar to it.
"Older" tech will start to see a surge of interest and rise in adoption. (0.6) This isn't a knock on the "older generation", per se, (particularly because I'm on the cusp of being a part of it), but the rise of complexity around all these tools and platforms and applications is just getting to the point that it's becoming a mess to keep track of anything. Case in point: How many streaming services are there? How many of us have different streaming services that go to particular devices and aren't available on other devices? (Or they could, IF we spent the time to configure something correctly. Maybe.) There's only so much complexity people can handle in their lives, and they're going to start looking for ways to claw some of that cognitive budget back by simplifying by going back to simpler tools. Case in point: I really prefer straight POTS (plain old telephone service) calls these days over video meetings.
Theranos will teach us something... but we'll not be sure exactly what. (0.5) The story of Theranos may have sent some ripples through the Silicon Valley around overhyped technology and throwing huge amounts of cash into hot startups, but the lessons to be learned from the whole thing are less clear. Don't invest in wild promises? Dude, that's what the VC funds are built to do. Don't fall for first-time founders' claims of what they can make happen? The VC funds are often made up of money from first-time founders who are convinced their success has everything to do with their "methodology" and little-to-nothing to do with luck. Theranos represents everything that's wrong with the startup community... but we aren't sure what that means. Much will be written, little will be done, and we'll be having this conversation again in a few years around a different startup, because the Valley would rather lose billions than learn a lesson.
NFTs, crypto; corporate promotion of NFTs followed by a huge backlash/abandonment. (0.6) In the words of several of my schoolmates growing up, "ERmigawd, how GRODY". Blockchain/crypto-based tech systems aren't even trying to make sense anymore, they're just throwing big words at investment funds and relying on the opacity of the words to convince people they're the next hot investment. NFTs are just the latest in the Ponzi pyramid--you get to own a certificate that declares nothing more than the fact that you own a certificate. (No, you don't own the image, or the original, you literally just own the certificate; see "NFTs are like those old 'star registry' scams" for an apt example and corresponding debate.) Big companies--Twitter, for example--are starting to get into the NFT thing, but let's be honest, many of them have money to burn, and throwing a billion USD into this, against a 1% chance of it actually being something beyond just a huge pit into which you toss money, is actually a reasonable move. But average folks buying the NFTs are going to figure out the scam fairly quick, and those celebrity buyers even quicker, and that's going to create a backlash and a hue-and-cry for regulation, and people will scurry out of them almost as fast as they scurried into them.

Or not--some of those star registries are still operating to this day, so... shrug
More non-developer-sales companies will build Developer Relations teams. (0.7) This is one of my happier predictions, in that it's one of my predictions that would make me happy if it comes true. :-) Developer Relations used to be "technical sales" to developers, something that Microsoft and Google and Amazon and other companies-who-sell-tech-to-development-departments would create and support, in order to have folks who knew how to talk to their principal customers--developers--about the things companies wanted to sell. But over the past decade, the definition of "developer relations" is shifting, becoming less sales/marketing-to-developers, and more "connecting technologists--both internal and external to the company--to their corresponding communities". I've personally been a part of this twice, building DevRel teams for two companies whose principal customer base aren't developers. It takes time, and it takes dedication, but let's be clear: If your company has an API, you clearly expect developers to use it, and if you expect developers to use your API, you need a team to meet them on their turf and be there when they have questions and triage their bug reports and provide samples and so on and so on and so on. All of that falls under the general heading of "developer relations", and if you don't have one of those teams, you may as well shut down any expectations of people outside your company using your API.
Open-source backlash/flight/pressure grows. (0.7) One of the sadder things we witnessed in 2021 was the log4j security vulnerability debacle. I call it a debacle not because I blame the log4j team for having that feature in the library (although, honestly, I have long believed that diagnostic logs are way over-engineered for the problem they need to solve), but because companies' worst tendencies around open-source software came to full flower and fruition, and made it very clear that the FOSS ecosystem is inherently unsustainable in its current form.

The current ecosystem is built, realistically, on a Communist economic model: "From each (developer willing to volunteer their time and energy), according to their ability, to each (open-source project), according to its needs." In theory, the system works wonderfully well; in practice, it has no answer to actors who are willing to "coast" on the backs of the effort of those willing to volunteer their time and energy. When a given community is small, and social pressures can be used to ensure fair distribution of effort, this model can work. (Think families, or startups, or even apartment co-operatives.) When the community grows large enough that we aren't faced with the disapproving stares of our peers for coasting.... And here we are.

Open-source project maintainers, particularly those projects that aren't the most amazing/sexy thing of the year, do tremendous amounts of work, and get zero recognition when they get it right, and all the media focus and scorn when they get it wrong. That isn't going to change any time soon, and the contributors will slowly drain off, creating more pressure on the remaining contributors, until something kicks in to start replenishing the pool. (My bet is companies will eventually face up to the fact that they depend on these FOSS projects like other companies do their supply chain, and start investing in it, but it'll take a while, and even then there will still be that temptation to coast.)

All of this is going to create greater instability in this community, and cause some serious soul-searching amongst the people upon whose backs it's built.
"New programming languages are going to start the R&D cycle on languages again. (0.6) They may be "service-oriented", they may be something else, but languages like Ballerina, Jolie, and Dark are basically suggesting that there's a new level of abstraction to be sought as a linguistic first-class citizen. Frankly, if you've been doing any sort of service development (micro- or otherwise), you've felt the costs and pain (in terms of complexity and the number of moving parts you have to track) of doing so. Cloud vendors are trying to suggest that "serverless" is the way to go, but in a lot of ways it just trades the complexity off to other ares of the app; a new paradigm is what's needed, to bring the complexities back down to human-manageable levels."

I wrote that for the 2020 Predictions, and I still believe it to be true. Ballerina is starting to build its hype, but more importantly we're starting to see people exploring languages as ways of approaching low-code/no-code kinds of development. I think this is only going to increase as the year(s) move on.
Some company is going to be really happy they took me up on that hypothetical in the second paragraph. (0.8) The one about contacting me about figuring out a leadership role within the company. I build some strong teams that can do some really interesting things....

Talk to you all next year... if not sooner.

2021 Tech Predictions

Thu, 25 Feb 2021 00:00:00 +0000

In 2020...

2020 is the start of a new decade, and that usually means predictors and prophets and tech pundits and other folks who think they know a bunch of stuff try to convince you that this is the start of something entirely new. (It's those kinds of folks who predicted flying cars and hoverboards, by the way.)

Let's call out the really obvious thing NOBODY predicted: pandemic. COVID-19. Lockdown. Remote work for everybody. Absolutely nobody had that on their predictions BINGO card, and to pretend even remotely otherwise is disingenuous to say the least. This past year has been simultaneously the longest of our modern lives, and the most alone most of us have ever felt together. So many things just got wiped out by the necessary health-preservation tactics (masks, lockdown, etc) that the pandemic required.

So, with that out of the way, let's begin.

I wrote....

"Everybody is going to go ga-ga over 5G, and it won't make a whit of difference. (0.6) Apparently, we have to go learn the Eight Fallacies of Distributed Computing all over again, again. On top of which, 5G is rumored to play havoc with weather prediction, so don't expect this one to go through with the ease of the 3G or 4G upgrades. I suspect that this is going to face some major backlash, probably in 2021 or 2022, particularly if it gets deployed widely in 2020." Result: 0 Um... yeah, we want ga-ga, but not in a good way; somehow, 5G was tagged by the various conspiracy-theorist-peddling QAnon'ers as being tied to all sorts of negative and nasty things, for reasons I cannot even begin to fathom. Mobile companies still tout it, but any sort of excitement--or discussion--we might have had about it got drowned in the demands for additional bandwidth as we all went remote, and then the inane conspiracy discussions. (Really? Bill Gates is going to use the vaccine to implant chips in your blood that will enable you to be controlled by 5G? Not even the wackiest sci-fi author would dare try to make this up.) I kinda got the "ga-ga" part right, but everything else was wrong. 0 points.
"Quantum computing will be the next major hype machine. (0.6) From the "our blockchain cannot possibly be defeated by any crypto-cracking tool known to man," we whiplash over to, "our quantum computing platform can crack any modern cryptography in seconds" without missing a beat. Quantum will be the Huge New Thing, and everybody will want to take a crack at the quintillions or whatever of CPU cycles that quantum computing represents... and like most of these kinds of hype waves, really, quantum computing will be useful in a small set of niche verticals, and useless to everybody else." Result: 0 In the face of the pandemic, anything that wasn't about remote work or communication got brutally shoved off to the side. Quantum computing might be the darling of the hype in 2021, but it certainly wasn't in 2020.
"Data privacy is going to start to separate from the larger "security" set of issues. (0.5) Companies are having to take "security" seriously, but there's "serious" and then there's "if we don't do this we can't do business anymore" serious. Data privacy will fall into that latter category, given the legal penalties that will face violators of GDPR and the new California laws. This means that tools and products (likely databases and its related ilk) will start advertising and promoting their "privacy-by-design" kinds of features, as a way of making it "easy" to implement privacy in your enterprise, big or small. But they won't want to be held liable in the event of a general hack, so they'll very carefully and clearly outline differences between "privacy" and the larger field of "security"." Result: +1 At this point, any company that isn't making deep inroads into trying to corral their data and control where it goes and who has access to it is the subject of investigation, and even Congress is starting to slide its Sauron-esque eye over the "Big Tech" companies. This is not a comfortable place for many companies to be, and its only going to get worse in 2021.
"Kubernetes all the things... to whatever comes next. (0.4) Kubernetes has pretty much locked up the container world, which means that we will now start to see what peple are suggesting we need to do after Kubernetes, because the stack has to keep growing deeper, or else vendors have to start competing with something in the middle of the stack, which means having to get into a feature war or fight for standardization somehow, neither of which are proven routes to success." Result: +1 Kubernetes has won, for whatever that means, in the War for Container Management. Now.... what next? Lots of companies are standardizing on Kubernetes, still, and we're starting to see some inroads into the next big thing, but again, so much of this was derailed by a simple need to adjust to an entirely-work-from-home work shift that anything else beyond that was just postponed or buried.
"The demand for "full-stack" developer unicorns will reach its peak. (0.7) Anybody who's watching the job sites and LinkedIn listings can see that everybody wants a developer who knows a major enteprise language (Java, C#, etc), Docker, Kubernetes, SQL and NoSQL skills, JavaScript (preferably knowing Angular, React, or Vue, and possibly all three), HTML, CSS, XML, JSON, one of the major cloud platforms (AWS, Azure, or GCP, certificates from certification authorities in each of those preferred), and sometimes even more beyond that. Folks, that's not a developer, that's an entire IT department (thank you, Twitter meme), and at a certain point that bubble is going to burst. But not this year." Result: +1 It's definitely peaking, if it hasn't already, as witnessed by the success of some companies (cough Netflix cough) that won't hire anybody but full-stack developers, and senior developers at that. But the community has started to wise up to the idea that this is looking for needles in haystacks, and that you can actually get further with a full-stack team than a team of full-stacks.
"New programming languages are going to start the R&D cycle on languages again. (0.6) They may be "service-oriented", they may be something else, but languages like Ballerina, Jolie, and Dark are basically suggesting that there's a new level of abstraction to be sought as a linguistic first-class citizen. Frankly, if you've been doing any sort of service development (micro- or otherwise), you've felt the costs and pain (in terms of complexity and the number of moving parts you have to track) of doing so. Cloud vendors are trying to suggest that "serverless" is the way to go, but in a lot of ways it just trades the complexity off to other ares of the app; a new paradigm is what's needed, to bring the complexities back down to human-manageable levels." Result: 0 Dark went under, but Ballerina and Jolie are still building some interest and momentum, and other new languages also look like they're getting ready to start stepping into the light. The era of object dominance is coming to a close, probably by the end of this decade.
"Automation will take an even stronger seat at the conceptual table. (0.7) "If I can't find developers to do these things I need to get done," fumes the CEO, "I'll go out and find some tools that can do all that stuff without requiring them!" Automation, particularly that enabled by low-code or no-code tools, will begin to skyrocket as a desirable target, even though this will mean that non-developers will start doing developer-ish things, and before long, will run afoul of the same basic problems that always strike when non-developers start doing developer-ish things. (Truthfully, this is related to the programming languages issue; companies need better "technical agility" out of their IT departments, and right now, nothing is providing that. They will look for anything, no matter how outlandish or outrageous, to provide that, and in 2020 that pressure will really make itself felt.)" Result: +1 The whole "low-code" and "no-code" space is really beginning to make its presence felt, as IT departments are beginning to figure out how to use them within their larger enterprise fabric.
"Ted will write another one of these in a year. (0.99; it should be a 1.0, but there’s always the chance that the whole country will undergo some kind of apocalypse, thanks to the orange buffoon who sits in the White House, that would keep me from writing this.)" Result: -1 Actually, I'm going to give myself a -1 on this, because it's now the end of February before I got around to writing. Part of that is due to taking on a new role inside Quicken Loans in March of 2020, one which has completely absorbed every scrap of free time and attention I had. Additionally, a lot of the "writing time" I normally use was denied me--time on airplanes and in hotels. We'll see how 2021 goes, I suppose.

2021 Predictions

We began the new year with chaos and tragedy, at least here in the United States. The Capitol Insurrection of 2021, fueled by blatantly-false claims of election fraud and whipped into a frenzy by a narcissist whose continued civic freedom depends on being "untouchable" by legal recourse, meant that a lot of us had things far, far more important than technology trends to worry about. It's been a rough two months since then, as well, as we struggle to balance hopes for vaccines and coordinated federal response to this damnable virus against the mess that the year-plus of deliberate neglect from the previous US Adminstration created. It was hard on everyone, and in a lot of ways we all just want to "get back to normal" for whatever definition of "normal" you care to use. Unfortunately, I don't think we're anywhere close to "normal" yet, though the massive rollout of vaccination programs will help, but I'll get to that in a second. For now, let's look at some predictions for 2021.

Conferences will begin to pick up in September. (Probability: 0.5) This is an important one to me, personally, because I miss going places and hanging out with friends and talking technology and trends and stuff. My thinking is that the vaccinations will reach a popular "critical mass" somewhere in June--which won't be yet "herd immunity" but enough so that everyone can know somebody personally who's been vaccinated and "nothing bad happened"--which would then mean that conferences will scramble to put something together as quickly as they can, but will still need a few momths' lead time. Other countries, who have been more aggressive to fight the virus, will be open sooner, but the US will lag, because we'd rather let people die while arguing "freedoms" than take the pragmatic route of eliminating the virus and then do the debates.
Remote (anything) is going to be a huge focus. (0.7) In 2020, we had to learn how to survive without driving anywhere or stepping inside other buildings. UberEats, DoorDash, GrubHub, all these "remote dining" experiences surged to the forefront of our mobile devices' home screens. Retailers figured out how to do "remote shopping", enabling curbside or in-store pickups. Amazon, of course, saw a huge surge in use, but a surprising number of retailers took huge leaps to control their own remote distribution channel. Those that didn't will quickly start to figure it out on their own--and Amazon may actually lose some market share as a result, as for years the principal thing Amazon did was provide that remote distribution channel. Before, when that was just one of several options, it wasn't important enough to retailers to figure out--when it became their only option, though, they moved fast and slapped enough together to make it all work. In 2021, they'll start to tune, improve, and perfect those channels, and Amazon will really need to start aggressively fighting against those new channels.
Amazon will get the Congressional eyeball. (0.5) Bezos already took a hit to his brand with the revelations of his infidelity to his wife, and Amazon's most recent union-busting efforts just added to the bad PR. Amazon will start to get more and more scrutiny as different companies start looking for reasons to pull their goods out of the Amazonian warehouses and do the distribution themselves--and that will cause Amazon to get a little desperate lest its profit margins come down too far... and that in turn will trigger some undesirable behavior, followed by calls for inquiry and oversight.
Bitcoin and other cryptocurrencies start drawing fire from unlikely places. (0.4) "Bitcoin consumes 'more electricity than Argentina', while just a year prior, it was Switzerland. And its carbon footprint is comparable to that of New Zealand. Whatever your thoughts about the viability of Bitcoin and other digitial currencies, the act of mining it is not only consuming ridiculous amounts of resources, that consumption is triggering wider ecological woes, and that's going to draw fire from a lot of different corners of the landscape--particularly because for so many people, it's just a weird geek-fan-whatever thing that doesn't have any impact on the wider world. Sooner or later, some country is going to take a hard-line stance against it, and that will start the dominoes in a negative direction.
'AI' and 'machine learning' are going to run into major backlash. (0.6) Amazon ran into ethical issues with its use of machine-learned models for interviewing and hiring, because they didn't do the right amount of validation and verification of their models before putting them into production. Not a lot of companies are going to have the patience or AI-savvy to do that, either, and as a result, media backlash will start turning the shine of AI and machine learning into glare. By years' end, we may even see Congressional inquiry into the subject(s), under the umbrella of widespread investigation into "Big Tech".
Any tech company that wants to compete will need to enable WFH on demand. (0.8) Look, this past year made it pretty obvious that anybody in the technology space can do their job from anyplace in the country--if not the world. Couple that with a continued insatiable demand for tech talent, and you are quickly setting up the model for a buyers' market when it comes to finding a job. "Hmm. You say your company needs me in the office, but this other company over there doesn't need that. Why should I work for you and add wear and tear to the streets, my car, and my nerves?" Not to mention that when you can work-from-home, you can realistically work-from-anywhere, including your parents' place while the kids are out building snowmen with grandma and grandpa. Some companies will fight hard to bring people back to the office, but they're going to be fighting an uphill battle, and may well lose out on some prime tech talent if they can't figure out how to be flexible. On that same note, if a company cannot or will not embrace a culture of "remote first" to allow those in the building and those outside the building to work on the same, level, playing field, that company will find itself scooping up candidates from the bottom half of the talent pool.
Several voting systems to "secure the vote" will come out in the next two years. (0.7) You don't have to believe Trump's wild-eyed and clearly-debunked claims about election fraud to know that people are sensitive on the subject of voting. To the technology space, particularly the startup space, that just screams "I'm a market that demands disruption! Disrupt me!" and the VCs will happily toss billions of dollars at companies that promise to let you "vote from your smartphone" securely and safely. Most of these will crash and burn and take those billions down with them, but a few might make it out of beta and maybe even find some adoption in a few of the smaller states (Rhode Island would be my first target). Personally, I don't think there's a damn thing wrong with the system we have at the moment--but as with many things in the startup world, it's not about fact, it's about perception.
Social media spinoffs will proliferate. (0.6) Parler was just the beginning--now that we've proven beyond the shadow of a doubt that we really can't get along together on a single social media platform, many groups will start flocking to social media platforms of their own choice, further polarizing the conversations there. It will have nothing to do with technology or security--it will have everything to do with humans' basic desire to be among people "just like me". Ironically, Facebook and Instagram will be losing their stranglehold grip on the social media world just as they start feeling the ire of Congressional gaze, which will further their downward trend. These new platforms will have some space in which to grow and draw eyeballs--and unfortunately allow people to stew inside echo chambers even more.
Languages and platforms will march on, and no one will really care. (0.8) Anybody know what version of Java is the latest? C#? JavaScript? Fact is, most of the popular languages in use are in the tapering-off point of the "value gained per new feature" curve, and most of us have stopped paying attention to the feature list of new releases. Sure, you may have some pet feature of C# you're waiting to see make it into the long-term-support (LTS) release, so your company can move to adopt that version of the language, but for the most part, whether that happens this year or next, it really makes little difference to your ship schedule. For the most part, now, knowing the latest features in these languages is about proving your knowledge and alpha-geek-ness, not about using them to solve actual business problems.
A collegiate and university education backlash is about to begin. (0.7) When the pandemic hit, all the schools went into lockdown, including the collegiate and university campuses that had for so many years told us that the thousands of dollars students spend there was for the "college experience". When the "college experience" consists of Zoom calls from your home from professors who often couldn't figure out how to use Zoom for the first half of the semester, delivering substandard content compared to courses from Udemy or Khan Academy, and exposing professorial misconduct (like the professor who chastised a hard-of-hearing student for 'not paying attention' during remote classes) for all the world to see, it's really, really hard to understand why anyone would pay thousands of dollars per semsester for that privilege. Given how coding bootcamps can provide much of the tactical skills (but not the larger conceptual understanding--they're trade schools, not universities, and let's keep that clear) that employers are looking for in junior developers, why would I spend $50k to get that diploma from UC Riverside or Iowa State in four years? Colleges and universities are going to have to scramble to figure out how to keep from just doing "business as usual", if they want to avoid a striking loss of registration numbers that will only get worse as the pandemic continues.
Resilience and failover will be the focus in the Ops-minded world. (0.7) The huge loads on servers from people staying at home and using digital channels caught a lot of companies by surprise, and they suffered outages as their predictive load models failed them utterly. In 2021, these companies will put a bunch of engineering into making sure they can handle the load, just in time to see the load start to draw back down as people start getting out of their homes more. Once the pandemic eases up enough to allow us to leave the house and dine safely anywhere in town, we won't see that high-water mark of server load again for some time--but companies, like generals, love to prepare to fight the last war, and they will put a large investment of time, money, and thought into systems that can scale in time for the next pandemic (a hundred years from now).

There's probably a few more I could toss off, but I think that's probably enough to engender some discussion and guarantee a few sharp rocks half-heartedly tossed my way. Love to hear your thoughts in the Disqus comments at the bottom of the blog, and I wish all of you a safe and speedy recovery from the pandemic in which we find ourselves; mask up, wash up, and stick with the science, so that in time, we can all go back to living our best lives similarly to how we did it in 2019.

2020 Tech Predictions

Thu, 16 Jan 2020 00:00:00 +0000

In 2019...

... I wrote a lot of stuff. (That always seems to happen when I do these.) Let's start. I'll include the full
text of what I wrote in each bullet point first, then put the Result after it with whatever insights or
comments seem relevant. (Arguably none of them are, but hey, it's my set of predictions, so....)

Blockchain might actually start being understood. (Probability 0.4) Blockchain itself is not going away—if anything, companies seem to be putting more resources into it, not less—which suggests that enough people are seeing something of value buried deep (very deep) inside the hype. It may not be clear yet what it is, but that message is going to start trickling and filtering out and a few people might actually start to clue in on how best to use it. Result: 0 Honestly, I'm not sure how to score this one. On the one hand, we are starting to see some companies coming out with tooling and platforms that are centered around blockchains, which lends itself by definition to allowing the technology to be better understood. On the other hand, though, can anybody really, actually, describe what the hell it is? Rather than make the strong case for either point, I'm just going to call this a zero and move on.
AI and ML will begin to start separating... a little. (0.6) There’s lots of things that fall under the umbrella term “artificial intelligence”, and machine learning is only one of them. As we get more experience with ML in practical usage, we’ll start to see that the different branches of AI are useful for different things, and we’ll begin to use those terms a little more carefully. Look for “natural language” and “expert systems” and “machine learning” to start getting used independently, rather than the catch-all “AI”. Result: -1 I just can't give myself a point here. There's a few places that are starting to discern between some of the terms, but for the most part, "AI" and "ML" are still entirely synonymous, and that is just confusing the hell out of everybody.
AI will start to move out of the startup and into the enterprise. (0.7) Lots of people have kicked their AI startup off, and lots of big companies have started AI initiatives within their own companies, and they’re all finding… this stuff is hard. And what’s worse, the kind of skills required to do “AI” (whichever flavor you care to name) are rare, and those departments simply will resist scaling to the rest of the company. What usually happens after that first wave of interest, however, is that a small group of companies start to figure out how to apply AI to more mundane and practical purposes inside the big companies’ IT space, and that’s where the serious money lives. Sure, there’s going to be some flashy bits about robots and image recognition and stuff, but the real money is waiting to be made in machine learning being used to discover patterns in the data that up until this point mostly got discovered by accident. Look for a few companies to start selling AI not as a raw technology, but as part of a toolchain that a company can buy to do “something else”. Result: +1 Oh, yeah, it's happening. It's hard to point to examples, but just about everybody I know in the space is talking about how they're using AI inside of a large enterprise, in many cases by virtue of the large enterprise having bought into one of the fourteen-million different AI startups. Bunch of it will probably fail, miserably, but that's really never stopped anything on the enterprise hype curve, and frankly, is all part of the cycle.
Amazon gets hauled in front of the DOJ. (0.8) If it doesn’t happen this year, it’ll happen in 2020. Jeff Bezos has been doing a great job of staying out of the limelight when it comes to the ugly whispers of “monopoly”, but good Lord, folks, Amazon is clearly making no secret of the fact that it wants to take over the entire retail experience—and when you couple that with the fact that Amazon is now opening brick-and-mortar stores, there’s not a huge leap from “Company using its monopoly power to drive competitors out of business by taking losses on sales” to “Amazon founder Jeff Bezos was in Washington DC today to sit in front of a Congressional inquiry…“. Particularly when Amazon just bought Whole Foods, a non-technology company if ever there was one. Result: -1 Oh, but not for lack of trying---Trumpie has been doing everything short of putting Bezos in front of a Congressional impeachment trial to try and shoot the guy down, and it somewhat surprises me that putting them in front of a monoply accusation hasn't occurred to any of his followers yet. Either way, though, it didn't happen, so no point for me.
Facebook undergoes some kind of radical transformation. (0.8) Zuckerberg is on the ropes, whether he realizes it or not. People are quitting the app in droves. Milennials aren’t taking it up. Every single day, various political groups scream about Facebook’s censorship of their particular pet issue, and the shine has come off of Sheryl Sandberg’s star. Facebook’s various other branded platforms, notably Instagram, are also starting to come under fire as people begin to openly question exactly how much influence these “social media influencers” have on them—or should. Congress may or may not get involved here, but either way, what was once pretty much a lock on a market has shown some serious cracks, and I suspect several competitors will start sharpening their knives, including several startups that may or may not already exist. Zuck will need to show some determined leadership and strike off in a bold direction, or the calls will come for his head faster than they did for any of IBM’s CEOs in the 90s. Result: -1 No, Facebook seems bound and determined to just keep on keepin' on, as they say, and it's likely going to hurt them in a big way. Zuck seems completely bound to the idea that if they just make a few tweaks here and there, they can just keep on being who they are, when in fact, who they are is by very definition a vehicle for bad mojo. Look, in a lot of ways the social media world is following the same arc as early journalism (check out "yellow journalism" and see if you recognize anything), before some of the core laws and rules and conventions governing it came into being. Although, honestly, given the current political climate, we may be losing those rules and laws and conventions before too long, too.
Google will look to capitalize on Facebook’s struggles... and fail miserably. (0.8; 0.6) This is another two-fer. Right now, deep in the heart of Menlo Park, a Google executive is putting the finishing touches on a presentation to the Alphabet board around building yet another social media network app, complete with numbers about how many GMail users want to see a union of their email and social media profiles, and how that could drive additional AdWords revenue, and…. And the people to whom that presentation is shown will nod sagely, approve the effort, and…. Google will announce the beta of yet another Google social media app that will somehow be different than all the other social media Google-branded apps, because this time, it will be different… somehow…. And yet, it won’t, and it won’t, and somewhere in 2023 Google will kill it because this time, it’s replacement will be different…. Result: -1 Google has, remarkably, stayed just as clear of the social media space as it ever has, which is to say they didn't throw a new social media product into the fray, and frankly, that surprises the hell out of me.
Microsoft will, too. (0.6) Unfortunately, here in Seattle, a Microsoft exec is putting the finishing touches on almost exactly the same presentation. With any luck, Satya will toss the whole idea out the window before the presentation is even finished, but... Result: -1 And I've almost never been happier to be wrong. Satya, you did good on this one. (Of course, Microsoft probably put all that effort into Teams, so maybe this one is a wash?)
Uber is going to struggle even more. (0.8) Uber was once the darling of the Silicon Valley and Wall Street—now, it’s having a hard time getting the time of day from either group. This past year was not kind to them (a net loss of close to US$3 billion, if we leave out one-time investments), and people have figured out that Lyft and other Uber-competitors are just as convenient, but without the soul-selling that using Uber brings to mind. Uber Eats might be a saving grace here, but again, lots of competitors, and those competitors don’t have the “Uber” logo that is increasingly turning from balloon to boat anchor around their branding. Result: +1 Oh, my, yes. They're not alone, to be sure, in that Lyft and several other ride-sharing firms are expressing equal kinds of difficulties. And the whole wider collection of "unicorn companies" that were basing themselves off the idea that people would want to partake in the entirely ungoverned, uncontrolled, unmonitored, and employee-unfriendly "gig economy" (or "side hustle" if you prefer) are facing some problems. Most of which stem from the fact that these companies aren't "disrupting" anything, they're just trying to reinvent an already-existing thing by calling themselves something different and trying to eke out a profit on the narrow margins that they can find by not having to pay government oversight costs. That was, is, and never will be, sustainable.
AWS (and Azure) will only continue to add services. (0.9) There’s no end in sight! AWS already has ten database services, but is there another angle to data storage that needs to be in the cloud? Go create it! Brand it! Sell it! Turn it into a thing that Bezos can show off at re:Invent 2019! There’s no end in sight! Result: +1 Of course they did.
Calls of “enough already!” will begin to echo in the clouds. (0.9) It’s becoming far more than anyone can keep up with. AWS, Azure, Google, all of them have way too many services for anybody to reasonably keep track of, and as a result, we will start to see a new industry of “cloud advisors” begin to form—--speakers, online courses, and the like—--simply geared to guide people through the choices of which of Amazon’s dozen or so data services you might actually need to use. Result: 0 It's starting to happen, but quietly and in dark corners. Hence, I can't really cop a point for it.
C# 8.0 will ship in Visual Studio 2019. (0.9) This one’s a gimme, but I’m never one to take a free point or two. Result: +1 April 2, 2019, baby. Or late September, take your pick.
Java12 will ship. (0.9) Oracle has announced that they want to move to a six-month release cadence, so given that Java11 shipped in September of 2018, it would make sense that Java12 will follow in March of 2019. One thing that needs saying, though: it’s not clear what features would actually ship in Java12, particularly since some of the proposed features could take a year or two to ensure that they meet the high quality bar that a language and platform as widely-used as Java requires. (Note that the same thing is true of C# and .NET, but Microsoft hedges their bets some by doing major releases as part of the two-year release cadence of Visual Studio; Oracle lacks that “ship vehicle” channel, since they don’t really own a widely-used IDE—--and no, sorry, NetBeans doesn’t count there.) Result: +1 March 19, 2019, baby. Oracle has been almost nefarious in making their six-month ship cycle for Java, and whether that's a good thing or a bad thing has yet to be determined. I think the bigger question, though, is whether or not anybody actually cares about Java's ship dates anymore. I mean, I used to look at each upcoming ship date with a certain amount of excitement and attention, but now... And more to the point, there's no "Here's what's in Java 12" presentations happening at conferences much anymore, which leads me to wonder if anybody actually cares.
Android ships... Q-something. (0.8) Android definitely seems on a 1-year release cadence, but hell if I can think of a dessert that starts with “Q”. Fossbytes suggests names like “quarabiya” or “quince”, but only one of the names is actually found inside the US (“quesito”, from Puerto Rico), and all of Android’s previous code-names have been US-centric names (with one exception, the “Petit Four” release, 1.1, which was also before they started using the alphabetical order, too). OK, OK, I know, “Eclair” is technically French, but it’s widely-known and -recognized in the US, so it counts. So Android has to come up with a “Q”-based dessert that is widely-known in the US and won’t be impossible to pronounce…. or they just skip it and go to “R”. Either way, I don’t envy the person who has to make that decision. Result: -1 They gave up. THEY GAVE UP. They got to "Q", and rather than even try to shoehorn something in place, they bailed entirely and just called it "Android 10". I mean, they should've done this from the beginning, so this is wildly overdue, but damn, man, I wanted to see what they would come up with for "Q".
iOS 13 is skipped, and 14 ships. (0.4) Hey, nobody in the Western world likes the number “13”, and Apple is not one to try to convince them otherwise. Look for Apple to pull another marketing shift and try to call iOS 13 “X3” or something, just to avoid the superstitious’ wrath. Result: -1 2019 was not a good year for Ted trying to predict marketing moves. Not only did Android move to numbering, but iOS decided to stick with numbering and just ship iOS 13. sigh
Security is going to quietly become a must-have skill. (0.6) The public is tired of the bleeding. Companies are tired of the scrutiny and managing PR fallout. Developers are tired of the second- guessing and accusations. Everybody is tired of security holes biting them in the ass. Slowly, quietly, companies are going to start demanding their developers be held accountable somehow—either in interviews or in internal code reviews or in any other form they can actually use—to make sure code doesn’t get compromised. It’s become clear that the hackers are only getting more and more organized, including state-level backing, and that “thoughts and prayers” are not effective as a security strategy any more than they are as a gun strategy. Result: +1 Let's be clear, if your company isn't starting to take security seriously (particularly in the face of GDPR and its ilk), then you're really just begging to be hacked and owned and bankrupted. We haven't quiet yet reached the point where we have answers to the problems that plague us, though, and that's where the problem lies---too much expertise is required to secure bespoke applications, and right now the best answers are "outsource your authentication strategy" (to places like Auth0 or Okta), which doesn't take us very far.
More security legislation like GDPR will come. (0.7) For the same reason as the previous one, governments will start laying down regulations about data privacy and use, and many of those laws will be contradictory to one another, if not to itself. (Even GDPR has a few edge-case flaws inside of it.) What’s worse, because it will be new legislation, and the penalties will be stiff enough to merit actually doing something about it, lots of companies will find it impossible to understand what to do, obtain help from “experts”, and choose to take the most conservative route that will only make end-users’ lives harder—much like what happened with GDPR. (You think accepting the privacy-policy-to-use-cookies messages are bad now—just wait. This is only getting started.) Result: +1 Hello, California. Should've known you'd be the first. Chances are good you won't be the last, since now this forces anybody doing business anywhere in the US to have to bring themselves up to California's standards... or refuse to do business in the largest domestic economy in the country. Expect a few more states to foloow suit, and then it'll just be waiting for the rest of the bowling pins to come down. Except for Mississippi---that place is just too bass-ackwards to care. (Seriously, twelve years for a cellphone? That is some fucked-up shit right there, and I don't mind swearing that strongly about it.)
Developers start finding their voice on ethics. (0.7) It started with Microsoft and Google, but it won’t end there—more and more developers are finding that they have a certain amount of power at their fingertips, and that unlike a number of their non-technical colleagues, if they threaten to abandon work on a project, their employers will listen to them. (So long as the demand for developers exceeds the supply, this will continue to hold true.) While there will always be some numbers of developers who will not care what they work on—so long as the checks cash—a similar percentage of developers are growing more and more aware that what they do could be used for less-than-innocent purposes… and will start to become an active voice. Look for a few more attempts (successful or otherwise) at coordinated walkouts by developers and other IT staff over easily-identifiable egregiously-unethical uses of some technology or other. Result: +1 Oh, my, yes. It's happening. Google, Microsoft, other companies are finding that their developers have a voice and are growing more willing to use it. Of course, this also means that companies are starting to find ways to try and shut that voice down, which is setting up for some very interesting employer/labor fights to come.
Developers are going to start a backlash against coding-school graduates. (0.5) This is one I hope I’m wrong about… but I’m guessing I won’t be. Here’s the thinking: More and more, “coding” is being seen as the next-great-way-to-get-out-of-poverty by millions of people who have little to no idea what “coding” means. They are, however, putting thousands of dollars onto credit cards and loans in an effort to learn “coding”, so that they, too, can get at this untold paradise of billions of dollars that we technology-types have been enjoying for the past two decades. But, as any average economist will tell you, the equilibrium point depends on the intersection of supply and demand—and so long as the demand for software was high, and supply of those who could deliver it relatively low, then the price (the equilibrium point) would stay high. Floods of new developers threatens to increase the supply—shifting the supply curve to the right—and bring down the equilibrium (price) point. Salaries will fall. Developers’ power relative to the rest of the company will fall with it. And developers are analytical enough to know this, see through how the rest of that story ends, and look to take steps to solidify their “hold” that they’ve enjoyed for so long. (For the record, I believe this has already been happening for several years now—why else would we as an industry be so concerned with discerning between the true “craftsmen” and the casual practitioner of code?) Look for more and more tech-minded folks to start openly resisting the idea of hiring and mentoring junior developers as they start to figure out (or guess) that their position and high income will be threatened if they don’t. Result: +1 Ironically, though, the backlash is coming not from the people who would hire coding-school graduates, but other coding-school graduates and/or their allies, who are rallying against predatory practices on the part of said coding schools. Some of them (like the linked story) were clearly never going to fulfill on their promises, but other schools that are genuinely trying to be a force for good will likely get tarred with the same brush.

2020 Predictions

2020 is the start of a new decade, and that usually means predictors and prophets and tech pundits and other
folks who think they know a bunch of stuff try to convince you that this is the start of something entirely
new. (It's those kinds of folks who predicted flying cars and hoverboards, by the way.)

Before we get too deep into this, though, I saw an interesting Twitter meme that went something like this: In
the movie "Back to the Future", Marty McFly journeys into the past by 30 years, from 1985 to 1955. If Marty were
to make that journey today, he'd journey back to... 1990.

Shall we talk about what this wondrous new year is going to bring?

Everybody is going to go ga-ga over 5G, and it won't make a whit of difference. (0.6) Apparently, we have to go learn the Eight Fallacies of Distributed Computing all over again, again. On top of which, 5G is rumored to play havoc with weather prediction, so don't expect this one to go through with the ease of the 3G or 4G upgrades. I suspect that this is going to face some major backlash, probably in 2021 or 2022, particularly if it gets deployed widely in 2020.
Quantum computing will be the next major hype machine. (0.6) From the "our blockchain cannot possibly be defeated by any crypto-cracking tool known to man," we whiplash over to, "our quantum computing platform can crack any modern cryptography in seconds" without missing a beat. Quantum will be the Huge New Thing, and everybody will want to take a crack at the quintillions or whatever of CPU cycles that quantum computing represents... and like most of these kinds of hype waves, really, quantum computing will be useful in a small set of niche verticals, and useless to everybody else.
Data privacy is going to start to separate from the larger "security" set of issues. (0.5) Companies are having to take "security" seriously, but there's "serious" and then there's "if we don't do this we can't do business anymore" serious. Data privacy will fall into that latter category, given the legal penalties that will face violators of GDPR and the new California laws. THis means that tools and products (likely databases and its related ilk) will start advertising and promoting their "privacy-by-design" kinds of features, as a way of making it "easy" to implement privacy in your enterprise, big or small. But they won't want to be held liable in the event of a general hack, so they'll very carefully and clearly outline differences between "privacy" and the larger field of "security".
Kubernetes all the things... to whatever comes next. (0.4) Kubernetes has pretty much locked up the container world, which means that we will now start to see what peple are suggesting we need to do after Kubernetes, because the stack has to keep growing deeper, or else vendors have to start competing with something in the middle of the stack, which means having to get into a feature war or fight for standardization somehow, neither of which are proven routes to success.
The demand for "full-stack" developer unicorns will reach its peak. (0.7) Anybody who's watching the job sites and LinkedIn listings can see that everybody wants a developer who knows a major enteprise language (Java, C#, etc), Docker, Kubernetes, SQL and NoSQL skills, JavaScript (preferably knowing Angular, React, or Vue, and possibly all three), HTML, CSS, XML, JSON, one of the major cloud platforms (AWS, Azure, or GCP, certificates from certification authorities in each of those preferred), and sometimes even more beyond that. Folks, that's not a developer, that's an entire IT department (thank you, Twitter meme), and at a certain point that bubble is going to burst. But not this year.
New programming languages are going to start the R&D cycle on languages again. (0.6) They may be "service-oriented", they may be something else, but languages like Ballerina, Jolie, and Dark are basically suggesting that there's a new level of abstraction to be sought as a linguistic first-class citizen. Frankly, if you've been doing any sort of service development (micro- or otherwise), you've felt the costs and pain (in terms of complexity and the number of moving parts you have to track) of doing so. Cloud vendors are trying to suggest that "serverless" is the way to go, but in a lot of ways it just trades the complexity off to other ares of the app; a new paradigm is what's needed, to bring the complexities back down to human-manageable levels.
Automation will take an even stronger seat at the conceptual table. (0.7) "If I can't find developers to do these things I need to get done," fumes the CEO, "I'll go out and find some tools that can do all that stuff without requiring them!" Automation, particularly that enabled by low-code or no-code tools, will begin to skyrocket as a desirable target, even though this will mean that non-developers will start doing developer-ish things, and before long, will run afoul of the same basic problems that always strike when non-developers start doing developer-ish things. (Truthfully, this is related to the programming languages issue; companies need better "technical agility" out of their IT departments, and right now, nothing is providing that. They will look for anything, no matter how outlandish or outrageous, to provide that, and in 2020 that pressure will really make itself felt.)

Lastly, one more:

Ted will write another one of these in a year. (0.99; it should be a 1.0, but there’s always the chance that the whole country will undergo some kind of apocalypse, thanks to the orange buffoon who sits in the White House, that would keep me from writing this.)

Despite my best intentions, I totally abandoned the blog in 2019. With the shutdown of MSDN Magazine, and my choice to withdraw from the back-page editorials in CODE Magazine, with no drawdown in my interest in writing and no clear outlet by which to publish it, the blog should see some better love in the year to come. But... I still won't give that a 1.0 probability.

I will, however, state that this last point---the one about businesses needing to find better technical agility in their IT and technology departments---is one that I'm very, very interested in, and I now work someplace that has the resources and interest in solving some of those problems, at least internally (but hopefully in an FOSS manner). We could always use a few more talented engineers, if you're of a mind...

2019 Tech Predictions

Tue, 1 Jan 2019 00:00:00 +0000

It's that time of year again... well, actually, a few days late, but I've been busy, I swear. As has become my tradition now for nigh-on a decade, I will first go back over last years’ predictions, to see how well I called it (and keep me honest), then wax prophetic on what I think the new year has to offer us.

As per previous years, I’m giving myself either a +1 or a -1 based on a purely subjective and highly-biased evaluational criteria as to whether it actually happened (or in some cases at least started to happen before 31 Dec 2018 ended). And, to be fair, while I only posted the 2018 predictions a few days ago (I hadn't realized it, but my CI system for the blog was down, which goes to show you, it helps to verify your CI system every so often), I actually wrote it a year ago, so it's still legit.

In 2018...

At the start of 2018, I wrote, "Oh, my, it's been something of a roller-coaster for all of us, to be sure. ... [T]he tech industry has seen a huge element of chaos and uncertainty thanks to a whole slew of factors, some related and some not." And although I didn't know it at the time, that was going to be well true of 2018, as well, thanks in no small part to the huge privacy and access scandals that have rocked Facebook and other companies to their very core. Let's go back and see what I predicted, though, and see where I came out in the face of all that...

In 2018, I wrote...

... that SPECTRE and MELTDOWN are going to ripple through the hardware in a big way.
I'm sorry, spectre-who? Meltdown-what? The two hardware-based "side channel" attacks made a big splash at the time, but after that, there really wasn't a whole lot of response or concern beyond "Wait, you mean my CPUs aren't going to go as fast anymore...?" To be fair, though, much of that lack of concern was (I think) rooted in the more fundamental concerns around, "Wait, Facebook sold my data to whom, again?" or the much more bluntly-wielded, "Wait, China has slipped what additional chip onto our motherboards...?" I might have called the idea that there would be a ripple, but I got the reason way wrong. Score: 0. I get no point, but lose no point.
... that Apple will still not ship anything new. Let's look at the record: They revamped the MacBook Air, they revamped the Mac Mini, they revamped the iPad Pro, and they.... yeah, that was it. Score: +1 for me.
... that Tech unicorns are going to feel a lot more pressure. The Uber thing definitely started having some level of impact, but to be clear, it was by far more the recorded visual of Mark Zuckerberg staring into the headlights of a Congressional inquiry that really rattled a few peoples' heads, and made a number of folks in the tech sector begin (alas, only "begin") to realize that some of the stuff they are playing with is actually fire that can burn them, not just fire to drive passions or whatever. Just when GitHub got acquired by Microsoft, Airbnb started feeling heat from various lawsuits, and suddenly, the whole "tech unicorn" thing became less of a mark of success than a brand that companies were not quite so keen to burn onto their own flesh. Uber even began running commercials from their new CEO to try and reassure people that Uber wasn't made up of immoral monsters... even as they still tried to make profits off the backs of the "side hustle" and the "gig economy", which are code-words for "let's not treat people like actual employees". Score: +1.
... that Voice-controlled interface design is going to go out of control. Not yet. I was probably a few years premature with this one, but more importantly, the fascination with the voice-control doesn't seem to be expanding much---yes, companies and developers are beginning to explore what more we can do with voice, but so far, it doesn't really seem to be taking root in the populated world beyond shopping and simple home automation. Score: -1.
... that The Javascript SPA Wars will rage on, with new players entering the mix. Angular is on the downslope of the Gartner Hype Curve; VueJS is still being lifted to the top; React, meanwhile, is sitting at the top of the Hype Curve Rollercoaster, and oh, my, here comes a new challenger in Mavo. Some wars just don't know when to quit. Score: +1.
... that Android will release a new version, and it will be called Pecan. Ha! "Pie" shipped in August. Score: +1 (out of a possible 2).
... that Kotlin will become the first-class citizen for Android development. Kotlin is now included in the text of the documentation of "Build your first app", and if Java is still the language used for the majority of the samples, well, Java's been the front-and-center for Android for the previous 27 releases, so you'll have to give Google a little time to get everybody to the point where they can forget Java like the passing of a bad migraine. Google is firmly committed to Kotlin as the language of choice for Android, it seems. Score: +1.
... that Swift 5 will come out, and it will break against Swift 4. Sooooo close. Swift 5 development has preview snapshots that were out as late as December 27, but it hasn't shipped as part of Xcode and other packaging channels yet, so I don't get points. Swift 5 is trying to set a higher bar for source incompatibility with previous versions (yay), but hasn't ruled it out entirely (sigh). Details here. Still, no points for me---Swift 5 isn't expected to ship until early 2019. Score: -1.
... that Bitcoin will collapse into the obscurity from whence it came. It's been falling pretty much continuously since the end of 2017, and there's no signs it'll rebound. It may not ever reach a $0 valuation, since some fools (like the state of Ohio) have put at least some amount of faith into it (albeit probably as a way to attract additional hi-tech interest; that same article reports "In the same month, Ohio lawmakers also pitched their state as a future hub for blockchain, hoping to both attract companies in the space and blockchain talent to the jurisdiction."), and there will always be those who will always continue to support it, just like some folks still insist that OS/2 is still a real operating system. Score: +1.
... that C# will reach an 8.0 release. Nope. C# stopped at 7.3 (in Visual Studio 2017 15.7), and probably won't ship until Visual Studio 2019 ships, which will probably happen in... yeah, no points for me on this one. Score: 0.
... that Java v.Next is going to confuse the hell out of everyone. So much so that Oracle abandoned the proposed versioning scheme and went back to "Java10", "Java11", "Java12", .... Score: +1.
... that Javascript releases will start to fade into obscurity. Quick, which version of ECMAScript (which is, remember, the proper name of the language) are you using in your version of NodeJS or Chrome? ECMAScript 2016? ECMAScript 2017? ECMAScript 2018? ECMAScript 6? ECMAScript 7? Most people, I venture to guess, are running on ECMAScript v."Who-cares-so-long-as-my-code-runs". (For the record, the latest adopted standard is ECMAScript 2018, or the 9th edition of ECMA-262, and introduces "support for asynchronous iteration via the AsyncIterator protocol and async generators. ... four new regular expression features: the dotAll flag, named capture groups, Unicode property escapes, and look-behind assertions. ... [and] includes rest parameter and spread operator support for object properties." In case you were curious.) Score: +1.
... that Tim Cook rumors begin. As in, the rumors of his upcoming demise as CEO. And still, yet another year, and no real call for his resignation. I almost think the Apple community wants him to be the next incarnation of Steve Jobs that nothing short of getting out of tech completely and becoming the next Dunder-Mifflin paper producer would actually get them to want him out. Score: -1.
... that Some startup will emerge claiming to fix the "fake news" problem using AI. Seriously? It took less than a month. One of them even uses blockchain for maximum buzzwordiness. Score: +1 (I really shouldn't, since this one was so easy to predict, but....)
... that Salesforce starts making some acquisitions again. They picked up five, some of which were technology companies, and one of which (MuleSoft) definitely had some "splash" impact, valued at US$6.5 billion. Called it with a major flourish. Score: +1.
... that Docker hype starts wearing off. Well.... not really. Some of the shine is coming off of Docker, but not necessarily with Docker itself. As more and more companies (and cloud providers) are adopting it, we're all finding some pretty big limitations with it, which is why much of the conversation isn't about Docker itself, but how we can put groups of Docker images together into a larger unit---as in, in Kubernetes clusters or some of its competitors. Which means the hype around Docker isn't causing us to abandon or question it, just... work around it. Score: -1.
... that Microsoft will continue to push Amazon in the cloud world at every step. Look at the Microsoft Azure blog. Look at the list of features and products offered by Azure. How anyone can keep track of this is anyone's guess. By some measures, Azure is ahead of AWS; by others, AWS is leading. It's a tight race, by most peoples' estimation, and there's no signs of it letting up. Score: +1.
... that Amazon will start to get into the developer tools game. Last year, I wrote

Amazon can't let Microsoft continue to have a real or perceived advantage in terms of the tools and development experience for building in the cloud, so I expect within the next year or two Amazon will announce or acquire a development toolchain specifically for building cloud applications.

In some ways, I was a bit off, because Amazon announced Cloud9 in the previous year, which I missed, but strangely enough, Amazon seems to be content with leaving it there, which I don't understand. Definitely missed something here. Score: -1.
.. that Quietly, more and more companies will start looking to build "Developer Relations" departments. If this is happening, it's happening too quietly for me to see with any great confidence. I believe it to be happening, based on casual conversations I have with various folks in different parts of the country, and certainly LinkedIn shows close to 600 open positions around "developer relations", but it's still mostly with technology companies (those that sell to developers and/or the IT side of a company). It was probably folly to expect to see it "explode" within a single year, and I still have faith it will, but for now.... Score: -1.

Eleven "+1"s, to five "-1"s. Not bad, better than my average (which is usually right about a push). Can't get too upset about those numbers, really. For comparison, last year's were 15 +1s. 10 -1s, which again confirms that my skills as a psychic are definitely not going to keep me and the family fed.

But that's the past, and most people are interested in...

In 2019

As an awkwardly-named decade ("the teens"? "the 2010s"?) draws to a close, a number of interesting things are going to start shifting the industry around in some very interesting---and potentially scary---ways.

Here's what I mean:

Blockchain might actually start being understood. (Probability 0.4) Blockchain itself is not going away---if anything, companies seem to be putting more resources into it, not less---which suggests that enough people are seeing something of value buried deep (very deep) inside the hype. It may not be clear yet what it is, but that message is going to start trickling and filtering out and a few people might actually start to clue in on how best to use it.
AI and ML will begin to start separating... a little. (0.6) There's lots of things that fall under the umbrella term "artificial intelligence", and machine learning is only one of them. As we get more experience with ML in practical usage, we'll start to see that the different branches of AI are useful for different things, and we'll begin to use those terms a little more carefully. Look for "natural language" and "expert systems" and "machine learning" to start getting used independently, rather than the catch-all "AI". What's more....
AI will start to move out of the startup and into the enterprise. (0.7) Lots of people have kicked their AI startup off, and lots of big companies have started AI initiatives within their own companies, and they're all finding... this stuff is hard. And what's worse, the kind of skills required to do "AI" (whichever flavor you care to name) are rare, and those departments simply will resist scaling to the rest of the company. What usually happens after that first wave of interest, however, is that a small group of companies start to figure out how to apply AI to more mundane and practical purposes inside the big companies' IT space, and that's where the serious money lives. Sure, there's going to be some flashy bits about robots and image recognition and stuff, but the real money is waiting to be made in machine learning being used to discover patterns in the data that up until this point mostly got discovered by accident. Look for a few companies to start selling AI not as a raw technology, but as part of a toolchain that a company can buy to do "something else".
Amazon gets hauled in front of the DOJ. (0.8) If it doesn't happen this year, it'll happen in 2020. Jeff Bezos has been doing a great job of staying out of the limelight when it comes to the ugly whispers of "monopoly", but good Lord, folks, Amazon is clearly making no secret of the fact that it wants to take over the entire retail experience---and when you couple that with the fact that Amazon is now opening brick-and-mortar stores, there's not a huge leap from "Company using its monopoly power to drive competitors out of business by taking losses on sales" to "Amazon founder Jeff Bezos was in Washington DC today to sit in front of a Congressional inquiry...". Particularly when Amazon just bought Whole Foods, a non-technology company if ever there was one.
Facebook undergoes some kind of radical transformation. (0.8) Zuckerberg is on the ropes, whether he realizes it or not. People are quitting the app in droves. Milennials aren't taking it up. Every single day, various political groups scream about Facebook's censorship of their particular pet issue, and the shine has come off of Sheryl Sandberg's star. Facebook's various other branded platforms, notably Instagram, are also starting to come under fire as people begin to openly question exactly how much influence these "social media influencers" have on them---or should. Congress may or may not get involved here, but either way, what was once pretty much a lock on a market has shown some serious cracks, and I suspect several competitors will start sharpening their knives, including several startups that may or may not already exist. Zuck will need to show some determined leadership and strike off in a bold direction, or the calls will come for his head faster than they did for any of IBM's CEOs in the 90s.
Google will look to capitalize on Facebook's struggles... and fail miserably. (0.8; 0.6) This is another two-fer. Right now, deep in the heart of Menlo Park, a Google executive is putting the finishing touches on a presentation to the Alphabet board around building yet another social media network app, complete with numbers about how many GMail users want to see a union of their email and social media profiles, and how that could drive additional AdWords revenue, and.... And the people to whom that presentation is shown will nod sagely, approve the effort, and.... Google will announce the beta of yet another Google social media app that will somehow be different than all the other social media Google-branded apps, because this time, it will be different... somehow.... And yet, it won't, and it won't, and somewhere in 2023 Google will kill it because this time, it's replacement will be different....
Microsoft will, too. (0.6) Unfortunately, here in Seattle, a Microsoft exec is putting the finishing touches on almost exactly the same presentation. With any luck, Satya will toss the whole idea out the window before the presentation is even finished, but....
Uber is going to struggle even more. (0.8) Uber was once the darling of the Silicon Valley and Wall Street---now, it's having a hard time getting the time of day from either group. This past year was not kind to them (a net loss of close to US$3 billion, if we leave out one-time investments), and people have figured out that Lyft and other Uber-competitors are just as convenient, but without the soul-selling that using Uber brings to mind. Uber Eats might be a saving grace here, but again, lots of competitors, and those competitors don't have the "Uber" logo that is increasingly turning from balloon to boat anchor around their branding.
AWS (and Azure) will only continue to add services. (0.9) There's no end in sight! AWS already has ten database services, but is there another angle to data storage that needs to be in the cloud? Go create it! Brand it! Sell it! Turn it into a thing that Bezos can show off at re:Invent 2019! There's no end in sight! In fact....
Calls of "enough already!" will begin to echo in the clouds. (0.9) It's becoming far more than anyone can keep up with. AWS, Azure, Google, all of them have way too many services for anybody to reasonably keep track of, and as a result, we will start to see a new industry of "cloud advisors" begin to form---speakers, online courses, and the like---simply geared to guide people through the choices of which of Amazon's dozen or so data services you might actually need to use.
C# 8.0 will ship in Visual Studio 2019. (0.9) This one's a gimme, but I'm never one to take a free point or two.
Java12 will ship. (0.9) Oracle has announced that they want to move to a six-month release cadence, so given that Java11 shipped in September of 2018, it would make sense that Java12 will follow in March of 2019. One thing that needs saying, though: it's not clear what features would actually ship in Java12, particularly since some of the proposed features could take a year or two to ensure that they meet the high quality bar that a language and platform as widely-used as Java requires. (Note that the same thing is true of C# and .NET, but Microsoft hedges their bets some by doing major releases as part of the two-year release cadence of Visual Studio; Oracle lacks that "ship vehicle" channel, since they don't really own a widely-used IDE---and no, sorry, NetBeans doesn't count there.)
Android ships... Q-something. (0.8) Android definitely seems on a 1-year release cadence, but hell if I can think of a dessert that starts with "Q". Fossbytes suggests names like "quarabiya" or "quince", but only one of the names is actually found inside the US ("quesito", from Puerto Rico), and all of Android's previous code-names have been US-centric names (with one exception, the "Petit Four" release, 1.1, which was also before they started using the alphabetical order, too). OK, OK, I know, "Eclair" is technically French, but it's widely-known and -recognized in the US, so it counts. So Android has to come up with a "Q"-based dessert that is widely-known in the US and won't be impossible to pronounce.... or they just skip it and go to "R". Either way, I don't envy the person who has to make that decision.
iOS 13 is skipped, and 14 ships. (0.4) Hey, nobody in the Western world likes the number "13", and Apple is not one to try to convince them otherwise. Look for Apple to pull another marketing shift and try to call iOS 13 "X3" or something, just to avoid the superstitious' wrath.
Security is going to quietly become a must-have skill. (0.6) The public is tired of the bleeding. Companies are tired of the scrutiny and managing PR fallout. Developers are tired of the second- guessing and accusations. Everybody is tired of security holes biting them in the ass. Slowly, quietly, companies are going to start demanding their developers be held accountable somehow---either in interviews or in internal code reviews or in any other form they can actually use---to make sure code doesn't get compromised. It's become clear that the hackers are only getting more and more organized, including state-level backing, and that "thoughts and prayers" are not effective as a security strategy any more than they are as a gun strategy.
More security legislation like GDPR will come. (0.7) For the same reason as the previous one, governments will start laying down regulations about data privacy and use, and many of those laws will be contradictory to one another, if not to itself. (Even GDPR has a few edge-case flaws inside of it.) What's worse, because it will be new legislation, and the penalties will be stiff enough to merit actually doing something about it, lots of companies will find it impossible to understand what to do, obtain help from "experts", and choose to take the most conservative route that will only make end-users' lives harder---much like what happened with GDPR. (You think accepting the privacy-policy-to-use-cookies messages are bad now---just wait. This is only getting started.)
Developers start finding their voice on ethics. (0.7) It started with Microsoft and Google, but it won't end there---more and more developers are finding that they have a certain amount of power at their fingertips, and that unlike a number of their non-technical colleagues, if they threaten to abandon work on a project, their employers will listen to them. (So long as the demand for developers exceeds the supply, this will continue to hold true.) While there will always be some numbers of developers who will not care what they work on---so long as the checks cash---a similar percentage of developers are growing more and more aware that what they do could be used for less-than-innocent purposes... and will start to become an active voice. Look for a few more attempts (successful or otherwise) at coordinated walkouts by developers and other IT staff over easily-identifiable egregiously-unethical uses of some technology or other.
Developers are going to start a backlash against coding-school graduates. (0.5) This is one I hope I'm wrong about... but I'm guessing I won't be. Here's the thinking: More and more, "coding" is being seen as the next-great-way-to-get-out-of-poverty by millions of people who have little to no idea what "coding" means. They are, however, putting thousands of dollars onto credit cards and loans in an effort to learn "coding", so that they, too, can get at this untold paradise of billions of dollars that we technology-types have been enjoying for the past two decades. But, as any average economist will tell you, the equilibrium point depends on the intersection of supply and demand---and so long as the demand for software was high, and supply of those who could deliver it relatively low, then the price (the equilibrium point) would stay high. Floods of new developers threatens to increase the supply---shifting the supply curve to the right---and bring down the equilibrium (price) point. Salaries will fall. Developers' power relative to the rest of the company will fall with it. And developers are analytical enough to know this, see through how the rest of that story ends, and look to take steps to solidify their "hold" that they've enjoyed for so long. (For the record, I believe this has already been happening for several years now---why else would we as an industry be so concerned with discerning between the true "craftsmen" and the casual practitioner of code?) Look for more and more tech-minded folks to start openly resisting the idea of hiring and mentoring junior developers as they start to figure out (or guess) that their position and high income will be threatened if they don't.

Lastly, one more:

Ted will write another one of these in a year. (0.99; it should be a 1.0, but there's always the chance that the whole country will undergo some kind of apocalypse, thanks to the orange buffoon who sits in the White House, that would keep me from writing this.)

I was pretty terrible about blogging in 2018, largely because I was pretty well-occupied with a number of different things in my life. Some of that is getting straightened out now, and frankly I felt an "itch" every so often to write something in the blog---but couldn't thanks to the aforementioned "things", mostly due to lack of time---that I think will force itself to the surface more often in 2019. I'll probably be shuffling some of my various IT properties around a bit, too, so don't be too surprised if in 2020 this blog is split between my professional site and a more personal one (which would remain tedneward.com, but with an actual website at that domain name instead of the silly placeholder that's there now).

Thanks for reading. Have a happy, prosperous, and adventurous 2019, and feel free to drop me comments or feedback in the comments below.

2018 Tech Predictions

Tue, 2 Jan 2018 00:00:00 +0000

In 2017...

Oh, my, it's been something of a roller-coaster for all of us, to be sure. Even if we leave out
some of the political mess (and Lord has it been a mess!), the tech industry has seen a huge
element of chaos and uncertainty thanks to a whole slew of factors, some related and some not.
Untangling all of this is going to take some time, but no better time than now, so...

First, let's look at the personal predictions:

Ted will celebrate his one-year anniversary at Smartsheet in September. +1! I've now been at
Smartsheet for a year-plus, and have no real desire to change my immediate surroundings anytime
soon. The company is growing like mad, we just made our first acquisition,
and my team is starting to go out into the world on its own. Management has its downsides, but
it certainly has its upsides, too.
Ted will do less speaking this year. +1! And this will probably continue into the new year
as well. As it stands, I'm probably looking at no more than a dozen conferences, total, through
all of 2018.
Ted will not be renewed as a Microsoft MVP. +1! Despite some repeated promises from the MVP
team that they would examine my candidacy once they had done some refactoring in the process, no
new invitation to rejoin that program has come from Microsoft, and I strongly suspect that they
will continue to ignore me. Fortunately, as I said last year, most of what I care about in the
Microsoft world is already open-source, and the rest of it... meh.
Ted will look to engage with other tech companies beyond Microsoft. +1! Google and I have
circled around each other for a year or so now, yet something keeps coming up that keeps me from
being a GDE. Fortunately, like Microsoft, most (if not all) of the Google stuff I care about is
also open-source, so the lack of the "GDE" credential at the end of my email signature isn't
really all that big a deal. On a related note, though, I did become an IBM Champion of Cloud...
for a year.
Ted will continue to teach at UW. +1! Still going. Still loving it. Still looking to continue
doing it through all of 2018 and beyond.
Ted will look to publish a few mobile apps. -1! Not a single one published. Kinda frustrated
at myself over that.
Ted will continue to write for various tech 'zines. +1! My column at developerWorks dried up, but
CODE Magazine and MSDN are still going strong, and honestly, I'm OK with that---things at work have
gotten really busy as I've taken on more "management" tasks and less "technical" ones.
And finally, Ted will try to blog more. -1! It's been a busy year, and frankly I've been
just too caught up in things to really make a good go of it. Looking to change that, boss, looking
to change that.

OK, despite the +6 and -2, I'm going to reset the score back to zero, because, really, personal
predictions are just not all that interesting to anybody but me.

Let's get to the meat of it.

The calendar year 2017 is going to be a wild one for the tech industry... +1! Oh, my Lord,
was that a +1. The year opened with Trump's "Technology Summit" with Elon Musk and Jeff Bezos,
and within a month it was disbanded and completely discredited. I said "The man thrives on
being unpredictable," which was obviously spot-on, "and the techn industry... thrives on
predictability." The tech sector almost immediately distanced itself from Trump, and many
companies have since found themselves in a position where they are actually sitting across
the courtroom against his Administration.
The Congress will call for an investigation into the 'hacking' of the 2016 US election.
+1! We're all getting a little tired of the investigation, actually, but it's not wrapping up
any time soon.
Security becomes a HUGE deal for the industry. +1! If you weren't convinced of the accuracy
of this, allow me introduce you to my friends SPECTRE and MELTDOWN (which, I admit, is cheating
a little since the news about them broke just after the new year). Coupled with all the other
hacks---which, admittedly, we're starting to get a little jaded about---it's pretty clear that
life as a security consultant is a pretty prime gig. Just ask Troy Hunt.
Apple continues to plummet. +1! MacBookPro's continue to hover at 16GB max RAM. The iPhone X
has a chunk carved out of it and Apple claims that was deliberate. Facial recognition, water
resistance, and a few other "amazing!" features were all already invented and mass-produced by
Samsung.
Apple doesn't introduce any new products this year. +1! iPhone X doesn't count, since it
didn't change the form factor. No new iPad. No new MacBooks. Nothing.
PC manufacturers double their efforts to build a MacBook Pro. +1! The MBP PC clones are
becoming too legion to count---as a matter of fact, I just ordered a ThinkPad P51, 64GB RAM,
2TB SSD, and it's not much thicker than my MBP 15" from 2012. You are falling way behind,
Apple, and don't think we all don't notice it.
Apple rumors about Tim Cook's departure begin. -1. If those rumors have circled, it's not
been anywhere I could hear them.
Oracle will continue to just write Java. +1! "... releases of Java just keep coming through
both commercial and OSS channels." In fact, Oracle has committed to a new and ridiculously short
release cycle for JDK releases. There is zero evidence that they have any secret plans to screw
the entire Java-based world over by charging licenses, so let's put that one to bed already,
shall we?
Oracle Cloud emerges onto the cloud scene in a big splash. -1. Oracle has a cloud? This
was a pretty big whiff---Oracle's cloud play, if anything, appears to be falling further and
further behind the "Big Three" (Amazon, Google, Microsoft).
Salesforce makes a major database acquisition. -1. While Salesforce did eventually acquire
Attic Labs, that didn't happen until
two weeks ago,
and I would hardly call the Noms database a major database
to boot. Good luck to them, but this is hardly a +1 for me.
Salesforce releases a new programming language. -1. Salesforce was actually pretty quiet
through the calendar year 2017, probably trying to figure out what to do with all the companies
they purchased the year before.
LinkedIn Learning starts to make a serious dent in online developer training. +1! LIL is
growing their training curriculum, and while they're not to the point of buying up ridiculous
amounts of ad space online, they're definitely starting to grow their online presence.
Swift doesn't go to 4. -1. "Swift 3 held breaking changes from Swift 2, and the folks at
Apple are not stupid. Swift 4 will be far, far down the horizon for a few years yet, given that
each major version number bump has heralded incompatibilities." Well, yeah, called that one way
wrong, and what's worse, Swift 4 had breaking changes from Swift 3. Swift is a cool language,
but if they keep breaking it like this, it's going to acquire a very nasty reputation.
Microsoft ships C# 7. +1! As a matter of fact, C# 7.1 and C# 7.2 came out as well. Microsoft
seems determined to keep committed to a much shorter release cycle, so it's reasonable to assume
that C# will continue to ship once or twice each year.
No new Android version. -1. I got the code-name right (Oreo), but it shipped to the public
in August.
Twitter will continue its slide into irrelevancy. -1. Whether for good or for ill, Trump
is giving the world a view into his mindset every day through his Twitter account, and Twitter
continues to love giving him that stage. I don't know if it's good for the Twitter brand to be
so closely aligned with Trump like this, but they're clearly not "irrelevant".
The "Internet of Things" continues to draw hype, and continues to fail to deliver. +1!
Sorry, but IoT continues to hover at this stage of "Honest, it's going to be huge!" as it has
for the last three years. There's still no compelling "gotta-have-it" IoT toy or tool or app
yet, and not for lack of trying.
Tech "unicorns" will start to watch the bubble pop. -1. Uber took some serious hits in 2017,
but overall the unicorn bubble hasn't started to pop yet.
Voice-controlled fart apps will emerge. -1. I'm thinking of writing one, actually.
Facebook will find that preventing 'fake-news sites' is a lot easier said than done. +1!
As a matter of fact, Facebook's latest suggestion (as of today) is to put a "credibility rating"
on stories from various websites that's based on a crowdsourced opinion poll---in other words,
we're going to let the masses decide what's believable, because the masses are a good judge of
what's believable. (In related news, the Flat Earth Society has seen their membership numbers rise
in recent years.)
A driverless car will kill somebody. +1! -1! I wrote "The circumstances may not be the software's
fault---and in fact it's likely that it won't be, when the final analysis comes back---but the
headlines will scream, and the widespread fear of a human "not being in the loop"
will set driverless cars back by years." Well, it actually happened in 2016.
What's more, in 2017, the software was cleared of guilt in the crash.
Despite the NTSB saying Tesla had some fault
later in September, the public at large basically yawned and said, "So?" So I don't get to claim a
point for that one.
The topic of ethics and programming will begin to become fashionable. -1. The programming
industry has started having some of these conversations among itself, but that's hardly the kind of
scope I was imagining.
"The cloud" continues to grow, even as consumers wonder what the hell it is. +1! Public comprehension
of "the cloud" still remains pretty low---but what's worse, what they do comprehend is actually pretty
wrong.
"Blockchain" remains the most opaque 'thing' of the year. +1! Hello Holy Hype Machine, Batman!
Artificial intelligence will continue to remain a 'future' thing. +1! Alexa is an amazing thing
(we have several Echo Dots in the house, mostly to use as an intercom system and alarm clocks), but
it's not really doing anything but a natural language parser so far. Now, I'll admit, I'm a little
behind the times in terms of wiring up my house, but I'm also at far less risk of having my house
held hostage to a Russian hacker at the same time. AI is much more than just a voice interface,
but right now that seems to be the extent of it to the rest of the world.

Total score: 15 +1s. 10 -1s. Like most years, a mixed bag. I sincerely hope my future as a manager
is bright, because clearly I will not make for a good psychic.

2018 Tech predictions

The computing world in 2018 is going to see some interesting changes, I think. Here's what I mean:

SPECTRE and MELTDOWN are going to ripple through the hardware in a big way. (0.8 probability)
These attacks go after the speculative execution nature of modern CPUs, and frankly if the chips
can't be fixed to correct for the flaw (and there appears to be some debate about that), then this
means that an entire branch of hardware optimizations are going to go the way of the DoDo bird.
That is not going to be easy for us as an industry---accustomed as we are to "everything doubles
every eighteen months"---to get used to.
Apple will still not ship anything new. (0.6) Apple desperately needs a reputation recovery
after the whole battery fiasco, but they aren't ready to ship anything with major "pop" yet. The
new MBPs won't be ready to go this year, I'm guessing, particularly since Apple will want to see
how the SPECTRE/MELTDOWN thing plays out, and it takes a fairly long lead time to make any sort
of hardware changes before they roll out.
Tech unicorns are going to feel a lot more pressure. (0.7) After Uber's mishaps at the C-level
thanks to their toxic CEO and their questionable business practices, a lot of investors are going
to start taking a much harder look at what's going on inside the companies they've invested in.
Expect to see more than a few "unicorns" having their "bro" CEOs replaced by actual adults who
have actual business models.
Voice-controlled interface design is going to go out of control. (0.8) Like it or not, the voice
thing is here to stay, and the "or not" part of that statement comes when everything is voice-controlled.
Why is this an issue? Think about this---when working with an application on your desktop or a
tablet or phone, the application that currently has focus---the one in front of you, selected,
responding to your mouse or touch---is the only one that gets access to the input that you provide,
whatever that may be. But voice applications, they can't tell when they have focus---as a matter
of fact, the whole point of them (as evidenced by the commercials) is that you can be doing something
entirely unrelated, and just toss off an "Alexa, drop in on Master Bedroom..." and suddenly you can
be doing something on the Internet or talking with somebody else a whole house away. So what happens
when you have two or three devices all listening simultaneously, and what happens if you refer to
them by name during casual conversation? More than once, my family and I have been talking about our
Echo devices, and suddenly Alexa pipes up with, "I didn't know you felt that way" or "I can't find
camel sushi on the Internet." (I have no idea, either.) The proliferation of voice devices is going
to vastly exceed our ability to regulate between them, and that's going to create some serious chaos
in the household and the workplace.
The Javascript SPA Wars will rage on, with new players entering the mix. (0.7) In the beginning,
it was AngularJS. Then React. Then Angular. Then React countered with ReactNative. Now VueJS is tossing
its hat into the ring, and the Javascript developers of the world are still fighting about which is
the best single-page application framework to use. They will probably be fighting about it long after
we're all dead, too, by the way, if the Java Web framework wars are any kind of history by which to
guide us. Honestly, VueJS shouldn't even be considered a competitor to these, since it's not really
a SPA framework but more of a "page" framework, but does that stop people from lumping them all
together and demanding a casualty count? Nope.
Android will release a new version, and it will be called Pecan. (0.8, 0.4) I'm giving myself
two probabilities here, one that a new version will come out, and one that it will be called Pecan.
Android seems to be trying to aim for a yearly cadence, and for the most part they seem to be reaching
it, but again, whether a new Android release comes out and whether a new Android release is made
available for your device are two very different (and seemingly unrelated) stories. Oreo will
probably just be reaching general availability by this time next year, it seems like.
Kotlin will become the first-class citizen for Android development. (1.0) This is a no-brainer
to me; between Google's choice of JetBrains as the backplane for Android DevStudio, and the fact that
they really want to be rid of the whole Oracle lawsuit baggage as quickly as possible, it seems to
me that it's a no-brainer that Google will basically deprecate all of the Java toolchain they can,
get Kotlin to produce ART bytecode natively out of the compiler, "hide" the existing Java libraries
behind Kotlin standard libraries, suggest that Kotlin's Java compatibility provides a "legacy access"
feature for "legacy" libraries, and essentially move the whole developer ecosystem over to a Kotlin-
based one for all future Android development. Note that this process will take years (if not a full
decade) to complete, but 2018 is the year Google starts making that move really hard to ignore.
Swift 5 will come out, and it will break against Swift 4. (0.6) It's the dumbest thing Apple and
the Swift community could possibly do at this point, but that hasn't stopped them from doing it three
times already (from 1 to 2, from 2 to 3, and from 3 to 4), so why buck the historical trend? Sure,
just break all that code---it's not like people have legacy code bases in the mobile world, we're
all just writing everything new and from scratch anyway, right? (Incidentally, I think this is part
of why Swift is having such a hard time spinning up a huge component/package ecosystem the way Ruby,
Node, Java and .NET have all managed to do. Yes, Cocoapods is a thing, but it's a lot smaller and
less active in many ways than its peers in the other ecosystems are.)
Bitcoin will collapse into the obscurity from whence it came. (8.4) Bitcoin is a currency that
is based on nothing more than the faith of the people who buy into it, and that's an extremely weak
basis upon which to build a currency. Supporters of bitcoins will say that all currencies are based
on faith, but that's not entirely true---faith in a currency is based upon faith in the strength of
the government that backs that currency. A US dollar is worth anything at all because we trust the
strength and faith of the US government. This has been true of every currency ever issued by any
nation (or other non-national organization, such as the Confederate States of America) throughout
history, and simply wanting it to be different won't make it so. People put trust and faith into a
government; they don't (or simply can't) put that faith into cryptography, particularly since it's
not obvious to them how the whole crypto/bitcoin thing works in the first place. Coupled with the
increasing costs (in power and heat, both of which are starting to have environmental impact) of
generating bitcoin and the huge time sink involved in transferring bitcoin from one party to another,
this whole thing is basically a scam, a sham, and a Ponzi scheme. The sooner it dies the better.
Now---all of that said, the reason this is a 0.4 probability is that people have (literally) bet
their life savings on this, and those who have invested that heavily will not want to see it go
down without a fight, and will be struggling like mad to keep the house of cards upright. Couple
that with the back that the bitcoin exchange sponsors are making out like bandits, and you have a
ripe situation for exploitation and outright criminal behavior---which is precisely why governments
are starting to take notice and take action.
C# will reach an 8.0 release. (0.4) It's possible, but not highly likely; the smaller, more
incremental releases of C# actually work against this, since they can release (literally) one
new feature at a time, and thus avoid the need to do a major-version release. If C# doesn't reach
8.0, it'll probably end the year on a 7.3 or 7.4 release, however.
Java10 will ship. (0.0) Ha! Fooled you. No, the real prediction is....
Java v.Next is going to confuse the hell out of everyone. (0.9) Oracle has said that there will be no
more numbered releases of Java, and that they will move to a YY.M versioning scheme of Java. This
means that in March, Java 18.3 will ship, and in September, 18.9. The goal is the same as
Microsoft's---frequent incremental releases---and I think the reaction will be much the same:
confusion, chaos and frustration from their principal user base as everybody tries to figure out
what the hell it is they're running and what the hell it is they need to run in order to use some
particular tool or library.
Javascript releases will start to fade into obscurity. (0.5) Not the language itself, but the
different release versions of Javascript. Or, rather, ECMAScript, since that's its proper name.
Proof: which version of the ECMAScript standard does Node 8.current implement? 2015? 2016? 2017?
2018? Answer: none of the above---it's a mix of each. Which then suggests the question, why have
"bundled" language releases described in documents like the ECMAScript Specification documents at
all? Before the end of 2018, the general Javascript population's concern for ECMAScript specs,
if there ever was any, will be almost entirely gone.
Tim Cook rumors begin. I still think this is going to happen soon.
Some startup will emerge claiming to fix the "fake news" problem using AI. (0.7) It will be
big news, it will make a huge splash, it will get snapped up by Facebook, and it will turn out to
be entirely worthless in the long run.
Salesforce starts making some acquisitions again. (0.7) I suspect that Salesforce will start
looking around for some larger companies to acquire to bring them into some fields that they currently
don't have much play or reputation in. Some will be technology, some might be product/platform
vendors. But Benioff's going to do something in 2018, probably starting around April or May.
Docker hype starts wearing off. (0.6) Look, it's the natural cycle of technology, and now that
Docker has made its impact on the world, people will start using it for things that Docker was
never meant to do, fail miserably and publicly, and claim it's Docker's fault. People who've ever
been burned by it will jump onto that train, and Docker will take a fairly large PR hit, the same
way MongoDB did in 2016 and 2017.
Microsoft will continue to push Amazon in the cloud world at every step. (0.6) Azure has, despite
all predictions close to a decade ago when it got started, managed to close the gap with AWS in terms
of both features, size and profit, and Satya doesn't show any signs of cooling off on it. Microsoft
is going to meet AWS feature-for-feature, and since Microsoft has a few things that Amazon itself
lacks (such as a development toolchain, among other things), Azure is going to start drawing some
attention away from AWS so long as it doesn't do something stupid. For that reason....
Amazon will start to get into the developer tools game. (0.4) Amazon can't let Microsoft continue
to have a real or perceived advantage in terms of the tools and development experience for building
in the cloud, so I expect within the next year or two Amazon will announce or acquire a development
toolchain specifically for building cloud applications. Were I a betting man, I would guess they will
announce something entirely new and put a ton of marketing and branding behind it, and that it will
be some kind of combination of object- and functional, with perhaps some security features built in,
but it will have "cloud" (and therefore "services") as a key component laced within it. This will be
Amazon's "Apple/WWDC announcing Swift" moment, and they will play it as a signal that Amazon is ready
to go toe-to-toe against Microsoft in developer tools.
Quietly, more and more companies will start looking to build "Developer Relations" departments. (0.9)
With the acceptance that HTTP-based APIs are here to stay, more and more companies are going to start
putting together APIs to access and/or interact with the data they store. If "the new oil is data",
as many economic periodicals are now claiming, then the new wells will be APIs, and those wells will
need to be advertised and explained by people who are familiar with talking to developers, even if
the company doesn't sell to the developer crowd. Think Ford, or Netflix, or even Zeek's Pizza, a
Seattle-local pizza chain that now has a website that will allow you to order pizzas online. If
Zeek's had an API, we could actually automate the ordering of pizza for user groups, which would save
my team at Smartsheet a ton of work---but somebody would have to be able to explain those APIs to the
developer community, serve as some level of technical support on those APIs, and so on. With the
rise of "software platforms" behind these APIs will come the need for people who can talk to the
developers (and their bosses), hence the need for more DevRel departments.

Some of this is stuff I plan to blog on more in the coming calendar year. For now, thanks for reading,
thanks for staying tuned, and talk to you more in 2018.

On the Google-bro Memo

Tue, 29 Aug 2017 00:00:00 +0000

tl;dr By now, everybody has heard of the memo that was passed around by the Google-bro, claiming that Google should reduce its efforts at explicit diversity hiring and how his message was unwelcome within Google's halls; my reaction is that he had a small point, but it was drowned in the inanity of his larger, and far more incorrect, message.

First, let's set a few facts out into the world:

I read the memo. Front to back, top to bottom.
I spent a significant amount of my college years analyzing papers like these.
My politics tend to stem from left of center principles, although I have for years believed that the nation is best served by a healthy balance between right and left. (For example, I mourn the loss of a conservative party in the US whose principal concern is reducing government spending.)
I do not now work for Google, nor have I ever.

Having established that, let's begin.

Summary of the memo

Effectively, my read on the memo was simple: Google-bro sought (despite all the caveats and disclaimers) to effectively posit the argument that women are inferior to men as Google employees, and that therefore the efforts Google puts in to hire them are efforts that could be better used elsewhere. (Or, put another way, the gender gap in tech is explainable not by systemic bias in the industry, but by other factors that Google fails to recognize, and that programs that seek to reduce that gap are wasteful and potentially creating greater problems.) He also states that this argument is not welcome inside of Google because of the monocultural echo chamber that Google has built that makes people of right-of-center beliefs feel unwelcome, unaccepted, and "shamed" for their beliefs.

Reaction from the outside

Reactions of course spanned a gamut of emotion. Much of it centered around the notions of free speech and reverse discrimination, or in some cases reacting to Google's reaction to terminate the Google-bro's employment.

Many were outraged at the argument about women being inferior in technology, but for the most part, as is common in this industry around the critical points, everybody was basically arguing past each other: "Misogyny!" "Free speech!" "Misogyny!" "Free speech!" ad infinitum.

Some chose to try to co-mingle the two points, including the essay written by one well-known industry talking head who chose to frame his discussion in terms of an elevator or some such. Alas, that well-known industry pundit's piece entitled "Thought Police" did nothing to really shed any light on the subject, and his follow-up piece lamenting the inability to discuss things rationally similarly failed to offer up anything resembling a discussion. Or rational thought. He then later offered up a piece about his experience working with women in technology which, as is common for white dudes who want to continue to believe that there's no real gender issue in the technical world, because he hadn't really run across it. He cites his horrible mistake at a keynote that actually offended a ton of people (calling it "stumbling upon an issue", italics included), but never really apologizes for actually having, you know, offended anybody. (His take is that "I thought about the complaints and issued appropriate apologies if I thought they had a reasonable point." Because only the rational and reasonable are due an apology when a mistake is made.)

(In the same post, he goes on to talk about "character assassination" conducted against him, which I personally find amusing in a post entitled "Women in Tech". Yes, Uncle Bob, I'm referring to you; no, I'm not seeking to assassinate your character; yes, I'm calling you out for having made multiple mistakes on this issue; and no, I'm not expecting that you will come to hear anything I say here as a reasonable or rational discussion. I have tremendous respect for your professional skill, but my respect for your personal morals and ethics has taken a huge hit, nor am I in any way under the illusion that your stance on this issue will be affected an iota by my analysis, commentary or opinion. What you choose to do from here is, as always, entirely up to you.)

Unfortunately, numerous people have made similar mistakes, conflating the two issues, until it got all well out of hand and impossible to discuss in any way, shape or form even remotely resembling fair, calm, or open-minded exchange. I chose to wait for a while before throwing my own comments out into the world. I don't expect that this will do much to provide additional clarity, but the more I was chewing it over in my head, the more I wanted to get all these thoughts out.

Disclaimer

I don't mean to suggest that these views represent anybody's viewpoint but my own.

In point of fact, my company recently posted a statement on their position on the whole thing, which I thought was neatly summarized by this single sentence:

In short, our position is this: If we didn’t think each and every one of our employees had smart things to say, they wouldn’t be here. We will fight to ensure that every employee’s voice is heard, so long as it does no harm to anyone else.

You can stop right here if you want to---that's basically the tl;dr version of my own viewpoints on the matter. What follows is my dissection of what I thought was a ridiculously poorly-written position paper.

My take

As I said above, I noted two basic points to the memo, but those points comprise some pretty wide territory, so I will seek to address them in manageable pieces.

Biodeterminism

Google-bro's main argument, that women are somehow not as well-suited for positions at Google as men are, is an argument that reappears every decade or two, and its more widely known as "biodeterminism", that is, that your biology has a deterministic effect on your success at a particular "thing". It was widely used, for example, as an argument in the years leading up to the Civil War against the Negro race being able to survive in a modern world.

Among our neighbors of Central and Southern America, we see the Caucasian mingled with the Indian and the African. They have the forms of free government, because they have copied them. To its benefits they have not attained, because that standard of civilization is above their race. Jefferson Davis, Speech, 1858

We recognize the fact of the inferiority stamped upon that race of men [African-American] by the Creator, and from the cradle to the grave, our Government, as a civil institution, marks that inferiority. Reply in the Senate to William H. Seward (29 February 1860), Senate Chamber, U.S. Capitol. As quoted in The Papers of Jefferson Davis, Volume 6, pp. 277–84. Transcribed from the Congressional Globe, 36th Congress, 1st Session, pp. 916–18.

Biodeterminism (also known as "scientific racism" has been used in all sorts of arguments, including those which have suggested that black atheletes are naturally more athletic than their white counterparts, or that Asians are intrinsically better studiers, and so on. In fact, it goes back centuries, including but not limited to the development of the IQ test, as well as the study of "craniometry", or that the shape of your skull is a highly determining factor in your intelligence.

In 1981, Stephen Jay Gould wrote the definitive takedown of biodeterminism as an argument, in response to much of the biodeterministic argument taking place at the time. (He wrote a revised edition in response to the same argument that cropped up a decade or so later. He's since passed away, or I suspect he'd be hard at work on a third edition.) A paleontologist by training, Gould effectively sought to demonstrate that much of the evidence being submitted as supportive of a theory of biological determination of intelligence was in effect bad science and confirmation bias of the researchers' pre-determined opinion. (His work was criticized in turn for succumbing to bad science, but much of that was done by those who sought to push the biodeterministic argument forward---his science has since been cleared, though his prose is argued to be a bit "overgeneralized" and "broad" in places, which is a fair critique.)

More to the point, Google-bro's work in the memo was shoddy and largely unsubstantiated. For example, he listed a number of ways men and women differ biologically, such as "The underlying traits are highly heritable", a wonderfully vague statement that fails to list the traits of which he speaks, how they are heritable, which traits aren't heritable, and by what scientific basis he makes the claim. (We won't even begin to dissect the point "[The biological differences are] exactly what we would predict from an evolutionary psychology perspective.")

His attempts at making use of gender stereotypes as assumed psychological behavior of all (or at least a stastically significant majority) men and women involved in technology is cherry-picked at best. For example, he cites the Wikipedia article "Sex differences in psychology" for his "differences in personality traits", and yet fails to note the very first line of that page, which reads:

Sex differences in psychology or gender differences are differences in the mental functions and behaviors of the sexes, and are due to a complex interplay of biological, developmental, and cultural factors. Differences have been found in a variety of fields such as mental health, cognitive abilities, personality, and tendency towards aggression. Such variation may be both innate or learned and is often very difficult to distinguish. Modern research attempts to distinguish between such differences, and to analyze any ethical concerns raised. Since behavior is a result of interactions between nature and nurture researchers are interested in investigating how biology and environment interact to produce such differences,[1][2][3] although this is often not possible.[4]

In particular, let's take note of the sentence "Such variation may be both innate or learned and is often very difficult to distinguish", which sort of blew right past him when he read the page.

Also of particular interest is the last sentence of the opening section:

Differences in socialization of males and females may decrease or increase the size of sex differences.

Considering that much of the criticism leveled at the industry from women in the field centers around the social aspects---and the fact that they are routinely subjected to a degree of criticism and discrimination (accidental or otherwise)---including but not limited to the assumption made at very early ages that "boys are better at math and science", it seems that at the very least, Google-bro's biodeterminstic argument is on flawed and very, very shaky ground. Lacking any sort of empirical evidence or experiments specific to this industry, it is reasonable to conclude that his argument is fundamentally flawed, particularly considering the overwhelming amount of anecdotal (and some scientific) evidence to suggest that yes, there is in fact a structural bias weighing in against women in technology.

And frankly, that's not the point I really want to discuss.

Free speech

The right of free speech is guaranteed by the Consitution of the United States, such that the government is not permitted to muzzle its citizens with regard to the topics they wish to discuss in public. You may demonstrate, you may assemble, and you may criticize your government and/or its leaders all without fear of reprisal by said government or any of its executive/enforcement bodies.

That said, however, a private company is not a government. Google has every right to terminate the employment of any of its employees "for cause" so long as that cause is not explicitly protected by law. You may not terminate somebody because of the color of their skin, the plumbing of their genitalia, the invisible entity they choose to worship, or the number of years they've spent on the planet; these are all protected by law.

The right to assemble, the right to speak your political mind, or the right to be ridiculously stupid are not protected rights within a private organization. (Neither is the length of hair, by the way, nor the kind of shoes you choose to wear.) If an employee walks into the company and states, "I believe that Adolf Hitler was a great man" in a very loud voice, that company has every right to terminate that person's employment every bit as much as if the same individual were to walk in and start proclaiming his allegiance to Stalin. Or to start chanting phrases from the Necronomicon.

The cause here is not based on the words coming out of their mouth, but the impact those words are having on the people around them---frankly, walking into your place of employment and spending the day trying to convert everybody over to your form of religion is equally grounds for termination, because you are impeding their work product by making them uncomfortable. It has nothing to do with the content of your words, but everything to do with the impact it is having on the people around you. Naturally, the more extreme the viewpoint, the more uncomfortable people will become, particularly if you come in wearing clothing and accessories that support that viewpoint, if only because they're thrust into a situation where they aren't sure how to react. Even if that viewpoint is "The US Army is an amazing place to be", if you're decked out in fatigues and carrying a mock rifle.

(True story: I contracted for a firm where one of the project managers was a recently-retired US Army vet. He was methodical, disciplined, and very effective as a project manager. However, over a span of four months, he grew increasingly irritable, came to work in his fatigues, and openly discussed killing techniques. Several complaints were filed, and the company finally took action when he came to work openly wearing his survival knife on his belt, since bringing a weapon onto company grounds was a clear violation of the company handbook. Prior to that, however, the company had already started to assemble the necessary documentation to demonstrate that his actions were creating a hostile workplace environment, so barring any major shift in his personality back to a more workplace-friendly manner, his termination was essentially assured. In retrospect, it was pretty clear he was suffering from some sort of PTSD-ish affliction, but any suggestions that he might want to look into some form of counseling or therapy were basically met with, "Are you saying I'm unable to do my job? Are you calling me incompetent?" and a very threatening look. Most people stopped trying to help him after that, unfortunately. More to the point, however, even had he not brought the knife, his manner, his dress and his choice of topics made many people very concerned about coming in to work---he would today be easily labled as a "disgruntled employee risk"---and the company had every right, if not outright requirement, to terminate his employment. Their hesitation was driven by the fact that HR needed to make sure they had the necessary documentation in case of lawsuit.)

Employer resources

Google has been criticized by both sides: from the left, for not firing Google-bro right away, and from the right, for firing him at all. "Unsafe!" screams the left, and "Thought police!" screams the right. Google-bro filed a wrongful termination lawsuit immediately after his firing, which will make interesting fodder for the legal pages once it finally reaches a judge, if ever---it's actually in Google's best interest to make the case quietly disappear, most likely by trading a non-defamation clause and gag order in exchange for a cash payout.

Again, regardless of the content, the bigger question here is, did the individual make use of company resources---his Google email account, his Google-issued equipment, the Google-supplied network or the Google-paid time---to engage in any of this? If so, he is clearly subject to the terms and conditions of his employment agreement, which usually state that such resources may not be used for personal purposes. Writing said memo, particularly one that was not submitted through any sort of HR channel or to his supervisors anywhere along the hierarchy, pretty much qualifies as personal purposes, every bit as much as using those resources to write a work of fiction would.

If Google-bro made use of company resources to write and/or distribute his memo, he's basically in violation of the company's terms of service, and that's pretty much all she wrote there. If he wrote the memo on his personal Google Docs account (not his work one), on his own time, on his own laptop, and used his personal GMail account to send it to others at Google on their personal accounts (not their work accounts), then he can claim that he never used Google-corporate resources, and that this was every bit a "personal" exercise. Chances of this being the case? Pretty close to nil, from what little evidence I can see.

If Google-bro can prove that this memo was, in fact, something he was expected and/or asked to do as part of his assigned duties, this becomes a very different scenario. Even then, however, we run into the next point.

Employer damage

If an employee does damage to the company in some tangible form, such as engaging in activites that harm the company's reputation (which I would argue would be the case here), then the employer absolutely has cause to terminate. Nowhere in the employee handbook at most programming positions does it say that stealing an entire Office Depot's worth of stationery supplies is grounds for termination, nor does it say that standing outside the company waving a sign that says "This company is run by a neo-Nazi" is grounds for termination. In fact, there's a lot of things that a company doesn't explicit say are grounds for termination, because anything that can be demonstrated to cause harm can be used as sufficient grounds.

Consider the sample corporate e-mail and Internet use policy here; it states that

All technology provided by [Company Name], including computer systems, communications networks, company-related work records and other information stored electronically, is the property of the company and not the employee. In general, use of the company’s technology systems and electronic communications should be job-related and not for personal convenience.

It's entirely likely that if you work in this industry, you signed something similar at your place of business, and if you're using your company laptop to write code for a side project, you're exposing that side project to a potential IP-ownership claim from your company to that side project. Most companies choose to look the other way on most personal- or personal-ish use, largely because it's hard to police and because they want to allow employees the freedom to use those resources in exchange for employees' willingness to work a little extra beyond the standard 40 hours, but make no mistake---companies can, and will, assert that clause when it's apparent that what's being done harms them.

Such as this case.

Free speech (redux)

But the issue Google-bro raises about free speech does highlight something that every company needs to be aware of: psychological safety. Google, as is also true of many other Silicon Valley companies, is quite left-leaning in their internal political views. For those of us who are of similar minds, that's comforting and feels natural. However, national statistics don't lie; only about half of us actually feel that way.

Just as a left-leaning individual will feel a little out of place in a company that leans right, a right-leaning individual will feel similarly misunderstood in a company that leans left. And it should be obvious that there's limits even in the case where one leans in the same direction as the company---if executives at Google started advocating the violent overthrow of capitalist governments in order to usher in a glorious new age of the proletariat, a significant number of Google employees would feel really uncomfortable.

This is perhaps the only place where Google needs to consider its actions terminating the Google-bro a little more carefully. Did they, by their action, lead those who "lean right" to feel more uncomfortable and unwelcome at work? If so, they've bought themselves a much deeper and more problematic issue than just what Google-bro's misguided essay brought to their doorstep.

His point that "People generally have good intentions, but we all have biases which are invisible to us" is a reasonable one, actually, and one confirmed by numerous psychological experiments over the past several decades. He goes on to say

At Google, we talk so much about unconscious bias as it applies to race and gender, but we rarely discuss our moral biases.

Again, I do not work at Google so I cannot comment on the truth of this statement as it applies at Google, but it is a fair point to suggest that companies often do not openly discuss the "moral biases" (although that term is pretty ambiguous and never defined within the scope of his memo) amongst the employees. That seems a fair statement to make.

Political orientation is actually a result of deep moral preferences and thus biases.

This is where things start to go off the rails---political orientation is most often a combination of a large number of factors, and to suggest that they stem entirely (or even mostly) from "deep moral preferences" is fallacious. (Evidence: How many of the Christian Crusade voted for Donald Trump, a man who is divorced three times, each time to the woman he was cheating on his previous wife with? If we use that---or any of a number of other political examples---as the means by which to examine the statement, either the Christian Crusade clearly does not care about sexual fidelity in its elected leaders despite their protests to the contrary during the Clinton Administration, or the statement simply does not hold.)

Considering that the overwhelming majority of the social sciences,
media, and Google lean left, we should critically examine these prejudices.

And this is where things really fall apart.

Much of the social sciences, while it does demonstrate strong political leanings, does not overly lean left. (Let's also leave alone the idea that the "social sciences" is another wonderfully vague and ambiguous noun that never gets defined.) Because much social science research is conducted on college campuses, it is easy to assume that anyone teaching a sociology class is immediately a leftist, but in fact, if we extend the definition of "the social sciences" to include the practice of social science, such as sales, marketing or human resources, it's pretty clear to see that there is no overt leaning anywhere. If, however, we are to assume that he means only the social science research, then it should also be an apples-to-apples comparison with media research and Google Research, which is obviously a touch nonsensical.

(Let's also leave alone the idea that political philosophy---one of those "liberal arts" that I think he implies by the use of the term "social sciences"---is what's being discussed every day on various websites and Twitter, and just as obviously not leaning in any particular direction.)

Neither does the media lean strongly in one direction---in fact, it's long been known among those who've studied the political bias in newspapers over the last century that many/most reporters tend to lean left, while many/most editors tend to lean right. This was widely considered positive, as it leant a degree of balance to the newspapers' reporting. Recent developments (over the last decade) in "fake news" and clearly-defined political bias (FOX News, MSNBC) have clearly upset that particular apple cart, but for the most part, traditionally high-ethic journalistic channels (New York Times, Washington Post, etc) have sought to maintain that balance, their Op-Ed sections notwithstanding.

If we broaden the term "media" to include "anybody who can stand up a website", which obviously could potentially include sites like Breitbart and Stormfront, then it should be patently obvious that the political leaning of "the media" is actually not leaning in any one particular direction at all.

Falsification Doctrine

One of the best ways in which one can test one's own hypothesis is to conduct what Karl Popper called "falsificationism", whereby one actively seeks to disprove one's assertion. To Popper, this was the critical distinction between science and pseudoscience:

Popper stresses the problem of demarcation—--distinguishing the scientific from the unscientific—--and makes falsifiability the demarcation criterion, such that what is unfalsifiable is classified as unscientific, and the practice of declaring an unfalsifiable theory to be scientifically true is pseudoscience. Falsifiability

For example, upon observing a white swan, an amateur at logic may suggest that "That swan is white". That is a statement of fact, what Popper called a "singular existential statement", since it makes a statement about the universe that can be verified by examining the evidence suggested. Upon observing a second and third and fourth swan, all of which are white, it may lead the amateur to conclude "All swans are white". However, this is a statement that now suggests that there is no such thing as a black swan---proven incorrect as soon as a black swan is observed, which can only happen by continued observation and research. As a matter of fact, the only way that statement could ever be verified true is by finding all swans in the universe (including those not on planet Earth), observing their color, and documenting that they are, in fact, white. This is obviously a great deal of work, which most amateurs refuse to conduct. "I've observed four swans, that seems like a sufficient set of data points from which to conclude, so clearly my observations are reflective of the universe." (Most statisticians would probably refute that point quite strongly.)

Instead of trying to observe all swans, taking the opposite approach---embracing a doctrine of falsification and seeking to observe or discover evidence that would render the statement incorrect---is often far more effective and efficient.

Conclusions

Google-bro's memo is a perfect example of an opportunity to refuse to throw the baby out with the bathwater. Discarding the ridiculous elements of his memo, we are still left with the very relevant and important question of, "Do people at Google of right-leaning political viewpoints feel psychologically safe in airing their views out at company events and forums?" Note the deliberate phrasing of my question: if we assume that Google is a left-leaning company (which seems reasonable) and that the left-leaning individuals are thus more comfortable airing their views out, the question should be whether those of the opposite side of the political spectrum feel the same way.

This question, by the way, is one that should be asked at every company.

In this particular case, if Google wants to assert that their company culture is "psychologically safe", then the right question for Google to ask is, "How many people here feel unsafe rendering their views in an open forum?" Note that this doesn't mean that anybody at Google should feel open about sharing views that are openly hostile or threatening; the company bears a direct responsibility to ensure that all of its employees are physically safe before considering their psychological safety. But they have a responsibility to ensure that the safety extends to both sides of the political spectrum.

And, bear in mind, that does not mean Google now has an obligation to ensure "equal air time" or "equal support" of all causes. As the executors of the company, Google executives are permitted to choose to pour company resources into whatever charitable or political causes they feel best benefit the company. In many cases, that means that firms will make campaign contributions to the electoral campaigns of candidates from both sides simultaneously, for example.

2017 Tech Predictions

Mon, 2 Jan 2017 00:00:00 +0000

Bear with me for a moment, though. This is just too good.

In 2015...

... I said:

Microsoft acquires Xamarin.

Oh, baby. Off by a year. I'm should go back and give myself a +1 for this one. It was really surprising that they hadn't. As a matter of fact, if Microsoft had listened to me and done it in 2015, they'd probably have saved themselves a TON of money compared to what they actually paid for Xamarin in 2016. But they made the acquisition, Xamarin is now part of the Microsoft family, and (finally!) .NET developers have access to the Xamarin toolchain and can build native iOS and Android apps without having to shell out additional cash to do so. 'Bout time, Microsoft. (I suspect this had everything to do with Satya, to be honest.)

OK, gloat over.

In 2016...

... I said:

Microsoft will continue to roll out features on Azure, and start closing the gap between it and AWS. Calling this one a +1; it doesn't take much research to see this has definitely been happening in 2016. However, it's not necessarily a true statement that they've been closing the gap; Amazon keeps adding stuff as well, and the feature-parity lists are starting to get ridiculous. Whether these features are actually of use, however, is an important distinction, and something for the second half of this post.
(X)-as-a-Service providers will continue to proliferate. Oh my, yes, Ted gets another +1 for this. When running a gaming convention has an (X)-as-a-Service for it (seriously, here) then you know the proliferation is in full swing. PaaS providers are exploding everywhere, and while a few have disappeared (farewell, Parse!), it's clear that this was the gold rush of 2016.
Apple will put out two, maybe three new products, and they'll all be "meh" at best. I should've broken this into two predictions: one about Apple's "meh" products, and one about wearables. If I'd done that, I'd have scored two +1's for it, because not only have wearables not really gone very far (show me somebody wearing a smart watch, and I'll show you a geek with too much time on their hands and not enough "discrimination" in their discriminatory income), but Apple's product releases have been... "meh"! I'm looking at you, iPhone 7, and I'm really looking at you, MacBook Pro. (When Consumer Reports doesn't give the MBP its top rating, you know the luster has failed.) More on Apple in the second half.
iOS 10 will be called iOSX. Dangit. Such an opportunity wasted. -1
Android N will be code-named "Nougat". Why, hello there, Android 7.0 Nougat. So pleased to make your acquaintance. +1
Java9 will ship. As I noted last year, @olivergierke pointed out that Java9 had already slipped to 2017, so this one was already a -1. Sigh. And I called it a "no duh" event, too---I'm going to let this one cancel out the extra +1 I'd have given myself for the Apple/wearables thing, just to keep the math safe (and my ego relatively sized). The article he cited says that Oracle "blamed the delay on complexities in developing modularization", a la Project Jigsaw.
Facebook will start looking for other things to do. Welllllllll, it'd be really tempting to say that Facebook's now "things to do" was "Be the deciding factor in who gets elected by passively encouraging the widespread dissemination of fake news and outright falsehoods!", but seriously, who would've believed that even if I had predicted it? Which I didn't. -1
Google will continue to quietly just sort of lay there. A year ago, I wrote, "Google, for all that they are on the top of everybody's minds since that's the search engine most of us use, hasn't really done much by way of software product invention recently. ... I suspect the same will be true of 2016--they will continue to do lots of innovative things, but it'll all be "big" and "visionary" stuff, like the Google Car, that won't have immediate impact or be something we can use in 2016 (or 2017)." And.... yeah. +1 More emphasis around the existing products they've built, but as a company, they've clearly spent most of 2016 on the Alphabet/Google restructure (which accomplished... what, exactly?), and anything new has been either way quiet or way removed from the business.
Oracle will quietly continue to work on Java. A year ago, I wrote, "[Oracle is not] going to kill it, but there's really not a whole lot of need to go around preaching its message, either. So they let the evangelists go, and they'll just keep on keepin' on." Score a +1 for the long-haired geek in Seattle; they just keep posting new code.
C# 7 will be a confused morass. If we permit me the freedom to call it ".NET Core" instead of just "C# 7", then wow do I get a +1 on this one. Even if I just constrain my prediction to C# 7/Roslyn, I still score one, but once you throw in the CoreCLR and "dotnetcore" and the different profiles and.... Holy spaghetti web browser history, Batman! The demarcation lines of the different project teams working on this whole thing are starting to become really clear as the different OSS projects each look really consistent within themselves, but then, when you get to the borders, things just.... fall apart.
Another version of Visual Basic will ship, and nobody will really notice. Alas, there was no new version of Visual Basic, since it would be in lockstep with the release of C# 7 (which didn't ship), but nobody really noticed. Or cared. Still, nothing shipped, so -1.
Apple will now learn the "joys" of growing a language in the public as well. First there was Swift 2, which was itself source-incompatible with Swift 1, and then during the summer, Apple shipped Swift 3, which was... source-incompatible with Swift 2, owing to some language changes that the community effectively decided was necessary. +1. (And thanks for that, by the way---made teaching iOS this Fall a royal PITA.)
Ted will continue to layer in a few features into the blog engine. You've got comments! And I'll take that +1, thank you very much.

Nine up (ten, if we count my Xamarin prediction from 2015), four down. Not bad. But now, we move on to the more interesting part of the post: 2017.

2017 Predictions

The calendar year 2017 is going to be a wild one for the tech industry, largely owing to the rather large orange elephant in the room---Donald Trump's election to President of the United States is a huge wildcard whose randomness simply cannot be understated. The man thrives on being unpredictable, and like most industries, the tech industry (for all that it cherishes "innovation" and "disruption") thrives on predictability. His collection of "tech titans" at Trump Tower last month yielded absolutely zero positive traction that I can see, and I suspect that the various corporate tech leaders (Nadella, Bezos, Cook, etc) are all looking at him right now the way humans do a rogue elephant---he could be good for them, so long as he doesn't go wild and start tramping everything in his path out of spite, anger, fear, or any other of a half- dozen emotions. There's no prediction here, though---just a "wow, this is an X-factor" that in turn makes predictions that much harder.

But on that note....

The Congress will call for an investigation into the 'hacking' of the 2016 US election. (0.8 probability) To be honest, I'm not sure if anybody knows exactly what we mean when we say "the Russians 'hacked' the US election" in casual conversation. There's no clear evidence that the voter machines themselves were cracked or tampered with, but it's fairly easy to see a correlation with the DNC hacks and Wikileaks disclosures and Trump's corresponding favorability gains in the polls. That said, though, the five-hundred or so US politicians that make up the Congress (and excluding Trump himself and his transition team) are not comfortable with the idea that somebody outside the US engaged in some kind of manipulation of the election, and they are going to want answers. Just yesterday or the day before, though, Trump made the comment that hacking is "extremely hard to prove", and he's right about that---without some kind of "smoking gun" found in a Russian government employee's possession, it's going to remain a major point of contention in the coming year, and investigation or not, it's not going to go away regardless of what the investigation finds.
Security becomes a HUGE deal for the industry. (0.8) The election is just the tip of the iceberg; consumers may have gotten used to (and complacent about) corporate security disclosures, but the idea that the election could be hacked is sending shivers down the collective spines of anyone who does anything online. The downside is that it's such a complex topic, it's hard for anyone who's not a computer security expert to really understand what to do; even among experts, there's a fair amount of disagreement, even on simple issues like scope (how widespread is it) or actual facts vs hype. Pair that with the paranoia that is inherent in any security professional (if you think computer security types are paranoid, try talking to physical security professionals for a while), and you have an industry that's ripe for a lot of snake oil and hyperbole. My prediction, then, is that the industry starts to see the first set of "security snake oil" products somewhere within the calendar year 2017. And by that, I mean products that claim to provide security for your interactions online but in fact do nothing of the sort. (Late-night infomercials about downloadable web pages that can "clean your system" of viruses and malware, move over---it's time for late-night infomercials about downloadable web page that can "secure you against even the most determined attacker"!)
Apple continues to plummet. (0.7) Their products this year were merely slightly-enhanced copies of the previous line of products (iPhone 7 vs iPhone 6) or containing gimmicky "enhancements" while the core of the product remained essentially unchanged from prior generations (MacBook Pro). Sorry, folks, the TouchBar does not qualify as "disruption" or "innovation"; it's a strip of touch-sensitive glass from an iPad designed to start prepping you for the idea that Apple can remove the keyboard entirely, replace it with a touchpad, and then put a hinge in between two iPad Pros and call it a "MacBookPad Pro" and charge you $10k for it. (And by the way, if you're thinking about one of the new MacBook Pro machines, make sure you go into an Apple Store and try it out--the keyboard is definitely not the same as its been for years. It feels like they took about half of the keys' "press depth" away, and it totally changes the "touch" on the keyboard. I imagine somebody could get used to it in time, but... ugh.)
Apple doesn't introduce any new products this year. (0.6) And by new, I mean something that's not an incremental improvement on what they've already got. Heck, I'll even go so far as to say that this means that there's no new form factors to the existing product line. (Meaning, no new-sized iPad or iPhone or laptop.)
PC manufacturers double their efforts to build a MacBook Pro. (0.8) The MBP is vulnerable, for the first time in a half-decade, and PC manufacturers are going to look for ways to capitalize on that. Somebody is going to put out a similarly-sized, similarly-weighted non-touch-screen Windows 10 laptop with 32GB of RAM and a 1 or 2 TB SSD, the usual collection of ports, and price it around the same as MacBook Pro ($2k to $4k), and developers will start buying them. (Bonus points to that manufacturer if they offer Linux as an out-of-the-box option.) I know I will....
Apple rumors about Tim Cook's departure begin. (0.6) Cook has proven that he's no Steve Jobs; in fact, the comparisons between his and Steve Ballmer's reign at Microsoft are proving eerily and entirely similar. Both basically took companies that were defining the marketplace and shepherded them into a position of trying to manage the cost structures and find better price-points, and in doing so, killed off much of the mojo that drove both firms. Ballmer took close to a decade to be run out of Microsoft (and even then, it took BillG's intervention behind the scenes, from what I can tell), but I don't think the Apple Board is going to wait that long---I think by the end of 2017 we're going to start hearing serious rumors about Cook being offered a golden parachute to give up the center chair and let somebody else in to run the show.
Oracle will continue to just write Java. (0.7) Oracle, despite the best efforts of media and journalists everywhere, just refuses to get drawn into "techno-drama". Java hasn't been the Trojan Horse into corporate pocketbooks that all the Java-doomsdayers were predicting back when Oracle acquired Sun, and releases of Java just keep coming through both commercial and OSS channels. There's really no reason at this point to doubt that Oracle is going to do anything but continue down that path. Make no mistake, I'm sure they're looking for ways to monetize Java in some way so that they can try to earn back the cash they spent to buy Sun, but I don't think it's going to be through selling or charging for the JDK or JRE anytime soon.
Oracle Cloud emerges onto the cloud scene in a big splash. (0.6) IBM now has Bluemix and Watson, and they were really the last of the "big-iron" holdouts around the cloud. (What I mean by that is that all companies have been quietly flirting with cloud, but some push it loud and clear, a la Microsoft or Google, and some were playing it very quietly for a while.) With IBM acquiring Loopback (a NodeJS server-side stack) last year, it's clear that IBM is going to push JavaScript as their main cloud development play, essentially ceding the Java-cloud development ground to somebody else. Amazon has historically been the place that Java developers have gone to run their Java code in the cloud, but if Oracle can build a compelling offering (particularly with a free tier that AWS currently lacks), this could be a relatively big splash. Between Oracle's reputation in the database world, if they have a solid "stack" offering that basically makes a Java-based back-end a snap to start up, Oracle could essentially claim the Java-favored cloud play from Amazon. (Yes, Heroku is out there and holds a fair amount of Java and Scala love, but now that they're owned by Salesforce I suspect the Java-leaning flavor of Heroku to wane a bit.)
Salesforce makes a major database acquisition. (0.5) Salesforce is growing, and they're clearly interested in expanding their cloud to be more than just the CRM. With Heroku, they have a Platform that developers can feel comfortable on, but they don't have a big-name database (relational or otherwise) that complements that play. They currently are sitting on a ton of cash, and last year's crop of acquisitions didn't include a big database storage name. There's not a ton of players left out there, but I could see them making a strong push to get something like Cassandra or Couchbase. (Yes, they have Data.com, but that doesn't seem to be making much headway in the developer mindset space.)
Salesforce releases a new programming language. (0.4) Let's call the spade a spade: Apex is a Java knock-off, and it shows a lot of warts, particularly since it hasn't really kept up with what few improvements Java-the-language has made in recent years. The last company to be in this position---a red-hot platform but a language feeling a little creaky at the corners and just plain "old" everyhwere else---was Apple right before they released Swift. Salesforce has the engineering power, they are looking to command more of the developer mindshare, and they have a ton of cash to blow, so.... Whether this happens this year, next year, or 2019, I'm not sure, but if it doesn't happen this year, the odds go up each year after that.
LinkedIn Learning starts to make a serious dent in online developer training. (0.5) Between the fact that LinkedIn Learning (formerly Lynda.com) is growing out its library to a pretty respectable degree, and the fact that Microsoft now owns LinkedIn, it's pretty reasonable to assume that Microsoft is going to start making this available to its developer community in various ways. This may happen in 2018, though, depending on how swiftly Microsoft moves to incporate LinkedIn assets across the rest of the firm; if they bought LinkedIn solely for the CRM data to go with Dynamics, for example, then this probably won't happen for a few years.
Swift doesn't go to 4. (0.7) Swift 3 held breaking changes from Swift 2, and the folks at Apple are not stupid. Swift 4 will be far, far down the horizon for a few years yet, given that each major version number bump has heralded incompatibilities. Apple will not want to call anything "Swift 4" and dredge up memories of incompatibilities in their customers' minds for a while. Swift might get a 3.1 in the summer, but that's as far as it'll go.
Microsoft ships C# 7. (0.8) Roslyn needs to ship in 2017 if Microsoft is going to be able to call this open-source process a success. Otherwise it'll start a lot of people grumbling. (Yes, a new version of Visual Basic will come with it, and it will make basically no news.)
No new Android version. (0.4) Android-N is still slowly making its way through the networks, and while we'll probably start hearing rumors of what Android-8 (Oreo?) will include, with a targeted ship date of 2018, probably 1Q or 2Q.
Twitter will continue its slide into irrelevancy. (0.5) Let's face it, Twitter's days are numbered. If you're holding Twitter stock, now's a good time to sell---when Twitter was left out of Trump's "tech summit" last month, the stated reason was that it was "too small". Put that into your brain-pan and circulate for a while---the service that invented microblogging and is one of the core founders of "social media" was "too small" for the PEOTUS' time. Twitter hasn't really done anything "new" or "interesting", but simply continued to be the 140-character microblogging platform it's always been. It's reaching commodity status, in fact. That's not a good sign for a company that wants to be more than it is. I suspect Jack Dorsey gets tossed on his can, the company starts looking for a new CEO, and the "new vision" will start to take shape by the end of the year (2017), and then in 2018 we find out that the "new vision" is terrible, takes them out of their "core business", and the slide accelerates. But nobody buys them this year, not yet.
The "Internet of Things" continues to draw hype, and continues to fail to deliver. (0.6) It's been how many years we've heard about IoT now, and how it will revolutionize our lives, and all we've really seen thus far is the wide variety of Internet-enabled devices being subverted for a widespread DDoS attack. Wearables, "smart refrigerators" and other IP-enabled devices are proliferating, but---to perhaps everybody's surprise but mine---nobody's quite sure what to DO with these things once you have them. Your thermostat is online; terrific. Does it have an API that will let me query meter usage? No, that's a different thing, and a different API, and a different connection endpoint, and.... Oh, and be careful, somebody could remote-hack your thermostat and hold your house hostage. Because that's worth the risk.
Tech "unicorns" will start to watch the bubble pop. (0.3) Uber, Lyft, all these companies that are valued at double-digit billions with zero profits, major losses, and no real assets to sell in the event of a bankruptcy.... All of this is going to start to make some investors nervous, particularly when they look around and realize that the tech sector has been carrying the country's economy through its "recovery" (yes, we've been in a recovery for the last half-decade!). All it takes is a few small stones to start the avalanche.
Voice-controlled fart apps will emerge. (0.6) Seriously. As Alexa and Siri and these other voice-activated systems start to move into stationary devices in your home, and as the SDKs for these systems start to become more widespread, the first thing developers will do is build some kind of ridiculously silly app (it would be a kindness to call it a game) that will somehow sweep everybody's sense of humor into the toilet. (Seriously. Imagine it. "Alexa, did you have beans for dinner?" "Yes, I did, and-- BRAAAAAAAAAAP!" It's exactly the kind of thing that would get people giggling for hours on end, particularly in a weed-induced state. Did I mention I live in Seattle?)
Facebook will find that preventing 'fake-news sites' is a lot easier said than done. (0.8) As a result, they'll put some kind of "AI" filter on linked sites, declare a victory, and try to get out of the political game entirely. It's a lose-lose scenario for them: one man's "fake news" site is another man's "revolutionary take" backed by the First Amendment, and Facebook does not want to be anywhere near a court trying to justify their actions against Free Speech. (Old-timers like me will remember Prodigy, an online service that started censoring content, which started its slide into doom.) Zuckerberg doesn't want to be held responsible for swaying important political events one way or another, but neither does he want to be the target of numerous political activist lawsuits (from all directions). As Joshua (the AI in the WOPR, back in the 80s movies that every geek my age openly worshipped) learned, Zuck will discover that sometimes "the only winning move is not to play".
A driverless car will kill somebody. (0.5) It's only a matter of time. The circumstances may not be the software's fault---and in fact it's likely that it won't be, when the final analysis comes back---but the headlines will scream, and the widespread fear of a human "not being in the loop" will set driverless cars back by years. Expert testimony and repeated demonstrations will do nothing to shake the public's fear that a computer-driven car could "hit a bug and kill me".
The topic of ethics and programming will begin to become fashionable. (0.3) Somewhere alongside the driverless car's first fatality, people will start asking how the car's programming makes decisions that most humans make in a split-second without even thinking about it. Case in point: the car detects that a motorcycle rider has had a problem and the rider has laid the bike down in the road right in front of the car. (For discussion purposes, there is no room left to brake; the rider is too close.) The car can either swerve to the side to avoid the now-helpless rider, potentially causing a major accident involving multiple people; or the car can simply continue forward, running over (and very likely killing) the motorcycle rider but avoiding the possibility of multiple fatalities from a larger accident. Most humans would swerve---but is that the "right" decision? More to the point, what should the software be programmed to do? Once the public gets wind of these kinds of decisions being made by geeks behind flat-screen LCDs, it's going to cause a major outcry. (And yes, these kinds of decisions are going to be encoded in the software, somewhere.)
"The cloud" continues to grow, even as consumers wonder what the hell it is. (0.7) Let's be clear---as of right now, the cloud is basically a developer thing. My parents really don't "get" the cloud, largely because there's really nothing they get from it. Sure, one can argue that GMail is the world's most popular cloud email service.... but your email is just stored on a server that Google owns, as opposed to a server that your ISP owns. (If that's your definition of "cloud", then pretty much all client-server computing is "cloud" in your world.) People are looking at more online services for things like bill payment, true, but those are basically services being offered by vendors with whom these people are already doing business--again, that's not "cloud". Cloud offerings have basically found a home in the developer world, but general-purpose cloud, the way that cloud was first being sold, is losing its window of opportunity to get hold of general consumers' minds. (I lose this prediction if my parents are suddenly smitten with a product that stores or computes for them and isn't a vendor they already have a relationship with.)
"Blockchain" remains the most opaque 'thing' of the year. (0.8) Everybody will go on and on about its huge technical advantages and obvious benefits, while never actually describing what it is or how it could work to change the world it's so clearly destined to change. It's the ultimate hype machine, and it will show no signs of slowing down until maybe the end of the year. By that time, something will emerge out of it (the way blockchain emerged out of bitcoins and cryptocurrency) that will carry forward the legacy of "changing the world" without actually changing anything.
Artificial intelligence will continue to remain a 'future' thing. (0.8) Part of the reason I say this is because AI is like magic---if you can understand it, it's not interesting anymore and it's just an implementation detail. We've had rules engines and natural language processing for years. When Amazon started doing "predictive analysis" of what you would like to buy, we pulled "data science" and "behavioral analytics" out of the "AI" world and into its own category. When AI figured out how to make the spoken word make sense, we called it "speech-to-text" and it was a feature on Android already back in the v2 days. (Marry speech-to-text up with a natural language parser, and you have Siri---which, remember, was its own company before Apple acquired them.) No, Alexa is not going to revolutionize the world any more than Siri did---the act of talking to a machine is not particularly new, and it's only as good as the services that sit behind the parser and can "hook in" to the parsed text. "Cortana, fire up StarCraft 2" is easy to parse and start an application; "Cortana, fire up StarCraft 2, and find me a random Hard co-op match as Artanis" requires not just firing up an application, but also "hooking" inside the application to know how to carry out the rest of the request. That requires an API platform that all applications can hook into, provide, and describe (in natural-text terms) to the voice-control system. That is not going to be easy to define, adopt, or test.

On a personal note, several predictions come to mind:

Ted will celebrate his one-year anniversary at Smartsheet in September. I'm optimistic about these guys, and the things we can do together. I'm looking forward to taking them into the developer limelight in a variety of different ways.
Ted will do less speaking this year. My new role actually encourages me to help develop new talent for my employer to go out and do the actual speaking, so while I'm definitely down for doing a few conferences this year, it's not going to be more than 12, total, for the calendar year. I enjoy speaking, but I'm looking to be a lot more careful about where I speak now.
Ted will not be renewed as a Microsoft MVP. Actually, this appears to be fact, not a prediction. MVP renewals for the January cycle went out already, and I didn't receive one. Fortunately, most of the stuff I care about in the Microsoft world is all open-source (or moving that way) anyway, and while it's been nice being on the MVP mailing lists, there's really been nothing there that's been all that insightful or amazing. (And, fortunately, living in Redmond makes it trivially easy to get together with anybody on a product team if I really want or need to, and I am privileged to call many of the people on those teams "friend".) It would've been 14 years, but as we Stoics say, "All good things, in time, must come to an end."
Ted will look to engage with other tech companies beyond Microsoft. Google just started a new MVP-like program, and I've been teaching Android and Angular and some Google Cloud Platform stuff for a while, so perhaps they'll welcome me into their fold.
Ted will continue to teach at UW. I've been guest-lecturing at UW for the past three years now, and I'm loving it. The students are bright, eager, and a helluvalot smarter than I was at that age. They're an incredible joy to teach.
Ted will look to publish a few mobile apps. I've had a few ideas floating around for a while, but just never really made the time to do it. Even if they never turn a dime in profit, I'm long overdue for having a few apps in the respective mobile stores.
Ted will continue to write for various tech 'zines. I love having the back-page editorial at CODE Magazine, the column in MSDN, and the various series on developerWorks, among others. I fully intend to keep all that going at full speed. (And I'm always looking for new outlets, if anybody has any leads on paid technical content gigs!)
And finally, Ted will try to blog more. The perennial projection. I've got much to blog about, including the patterns series, as well as some interesting themes and ideas floating around the ol' brain pan.

Happy Holidays, and thanks for reading!

Revisiting Rotor

Thu, 13 Oct 2016 00:00:00 +0000

tl;dr As part of preparing for a workshop next week in Poland, I've been diving back into the CLR source code---which takes me back to my old friend, Rotor.

For those of you who came to the CLR late, back in 2002 Microsoft offered up an open-source version of the CLR called the Shared Source CLI. ("CLI", for those who don't know, is the official ECMA Specification describing the virtual machine that .NET uses as its runtime. The "CLR" is the commercial name for Microsoft's implementation of the Common Language Infrastructure specification.) I can still remember the feeling of shock and awe when I heard that Microsoft was actually going to release the source---albeit with some of the core parts simplified for easier research purposes---to what it considered its flagship technology for the coming decade. Wow.

And then I got to write a book on it. Double wow.

And then Microsoft decided to pay me to follow up on the first edition with a second edition, which they would give away for free. Which, by the way, you can get here.

Triple wow.

But I want to call out something I wrote eight years ago, in the Prologue to the second edition:

"[T]he book also marks a turning point, as well: with the release of the FCL source to the wider world of the development community and the lack of significant changes to the execution engine since v2, the Rotor distribution has effectively been "cut loose" by its original creators, to stand on its own within the community, as every open source project must do at some point. This is not a cause for alarm or concern—-the Mono project continues full force, and Microsoft‘s growing comfort with the open-source community leads to the distinct possibility that the commercial CLR source will, one day, stand where Rotor once stood."

It's probably bad form to smirk as hard as I am right now. But I'm still doing it.

Thank you, David Stutz, for blazing the trail that Rotor could go down, and in doing so start the avalanche that would eventually become Microsoft's engagement in the larger world of open source. We in the Microsoft community owe you a pretty big debt.

The Fallacies of Enterprise Computing

Wed, 24 Aug 2016 00:00:00 +0000

More than a decade ago, I published Effective Enterprise Java, and in the opening chapter I talked about the Ten Fallacies of Enterprise Computing, essentially an extension/add-on to Peter Deutsch's Fallacies of Distributed Computing. But in the ten-plus years since, I've had time to think about it, and now I'm convinced that Enterprise Fallacies are a different list. Now, with the rise of cloud computing stepping in to complement, supplment or replace entirely the on-premise enterprise data center, it seemed reasonable to get back to it.

I'll expand on the items in the list over future blog posts, I imagine, but without further ado, here's the Fallacies of Enterprise Computing.

New technology is always better than old technology
Enterprise systems are not "distributed systems"
Business logic can and should be centralized
Data, object or any other kind of model can be centralized
The system is monolithic
The system is finished
Vendors can make problems go away
Enterprise architecture is the same everywhere
Developers need only worry about development problems

As Deutsch said, "Essentally everyone, when they first build a [enterprise] system, makes the following [nine] assumptions. All prove to be false in the long run and all cause big trouble and painful learning experiences."

Naturally, I welcome discussion around these, and I may edit and/or append to this list as time goes by, but this is where the past decade has led me.

New technology is always better than old technology

After building IT systems for more than sixty years, one would think we as an industry would have learned that "newer is not always better". Unfortunately, this is a highly youth-centric industry, and the young have this tendency to assume that anything new to them is also new to everybody else. And if it's new, it's exciting, and if it's exciting, it must be good, right? And therefore, we must throw away all the old, and replace it with the new.

This cannot be emphasized enough: This is fallacious, idiotic, stupid, and brain-dead.

This fallacy is an extension of the old economic "limited market" fallacy: The more gains one entity makes in a market, the more that other entities lose. (Essentially, it suggests that the market is intrinsically a zero-sum game, despite obvious evidence that markets have grown substantially even in just the last hundred years since we started tracking economics as a science.) Thus, for example, if the cloud is new, and it has some advantages over its "competitors", then every "win" for the cloud must mean an equal "loss" for the alternatives (such as on-prem computing). Never mind that the cloud solves different problems than on-prem computing, or that not everything can be solved using the cloud (such as computing when connections to the Internet are spotty, nonexistent, or worse, extremely slow).

Now, for those of you who have been engaged in the industry for more than just the past half-decade, here's the $65,535 question for you: How is "the cloud" any different from "the mainframe", albeit much, much faster and with much, much greater storage?

Those who cannot remember the past are condemned to repeat it. --George Santanyana, Historian

I've seen this play out over and over again, starting with my own entry into the IT universe with C++ (which was the "new" over C), and participated in a few system rewrites to C++ from other things (Visual Basic being one, C being another, sometimes some specific vertical stuff as well). Then I saw it again when Java came around, and companies immediately started rewriting some of their C++ systems into Java. This time around, I started to ask, "Why?", and more often than not, answers of "We don't want to fall too far behind" or "We need to modernize our software" were the fairly vague answers. (When pressed as to why "falling behind" was bad, or why software needed to be modernized, I was usually shushed and told not to worry about it.)

In the years since, I keep thinking that companies have started to get this message more thoroughly, but then something comes along and completely disrupts any and all lessons we might have learned. After Java, it was Ruby. Or, for those companies that didn't bite on the Java apple, it was .NET. Now NodeJS. Or NoSQL. Or "cloud". Or functional programming. Or take your pick of any of another
half-dozen things.

Unfortunately, as much as I wish I could believe that "it's different this time" and we as an industry have learned our way through this, I keep seeing signs that no, unfortunately, that's too much to hope for. The easy way to mitigate this fallacy is to force those advocating new technology to enumerate the benefits in concrete terms---monetary and/or temporal benefits, ideally, backed by examples and objective analysis of pros and cons.

By the way, for those who aren't sure if they can spot the fallacy, the easy way to tell if somebody is falling into this fallacious trap is to see if their analysis contains both positive and negative consequences. No technology is never without its negatives, and a practical and objective analysis will point it out. If it's you doing the analysis, then force yourself to ask the question, "When would I not use this? What circumstances would lead me away from it? When is using this going to lead to more pain than it's worth?"

Enterprise systems are not "distributed systems"

This means, simply, that any enterprise system is subject to the same fallacies as any other distributed system. Reliability, latency, bandwidth, security, the whole nine yards (or the whole eight fallacies, if you prefer) are all in play with any enterprise system.

If you're not familiar with the Eight Fallacies of Distributed Systems, take some time to make yourself familiar with them and some of the mitigation strategies.

Business logic and should be centralized

(Note: I wrote this up a long time ago in a blog post as the "Eleventh Fallacy of Distributed Systems", but it feels vastly more relevant as an Enterprise Fallacy.)

The reason this is a fallacy is because the term "business logic" is way too nebulous to nail down correctly, and because business logic tends to stretch out across client-, middle- and server- tiers, as well as across the presentation and data access/storage layers.

This is a hard one to swallow, I’ll grant. Consider, for a moment, a simple business rule: a given person’s name can be no longer than 40 characters. It’s a fairly simple rule, and as such should have a fairly simple answer to the question: Where do we enforce this particular rule? Obviously we have a database schema behind the scenes where the data will be stored, and while we could use tables with every column set to be variable-length strings of up to 2000 characters or so (to allow for maximum flexibility in our storage), most developers choose not to. They’ll cite a whole number of different reasons, but the most obvious one is also the most important–by using relational database constraints, the database can act as an automatic enforcer of business rules, such as the one that requires that names be no longer than 40 characters. Any violation of that rule will result in an error from the database.

Right here, right now, we have a violation of the "centralized business logic" rule. Even if the length of a person’s name isn’t what you consider a business rule, what about the rule stating that a person can have zero to one spouses as part of a family unit? That’s obviously a more complicated rule, and usually results in a foreign key constraint on the database in turn. Another business rule enforced within the database.

Perhaps the rules simply need to stay out of the presentation layer, then. But even here we run into problems–--how many of you have used a website application where all validation of form data entry happens on the server (instead of in the browser using script), usually one field at a time? This is the main drawback of enforcing presentation-related business rules at the middle- or server-tiers, in that it requires round trips back and forth to carry out. This hurts both performance and scalability of the system over time, yielding a poorer system as a result.

So where, exactly, did we get this fallacy in the first place? We get it from the old-style client/server applications and systems, where all the rules were sort of jumbled together, typically in the code that ran on the client tier. Then, when business logic code needed to change, it required a complete redeploy of the client-side application that ended up costing a fortune in both time and energy, assuming the change could even be done at all–the worst part was when certain elements of code were replicated multiple times all over the system. Changing one meant having to hunt down every place else a particular rule was–or worse, wasn’t–being implemented.

This isn’t to say that trying to make business logic maintainable over time isn’t a good idea–--far from it. But much of the driving force behind "centralize your business logic" was really a shrouded cry for "The Once and Only Once Rule" or the "Don’t Repeat Yourself" principle. In of themselves, they're good rules of thumb. The problem is that we just lost sight of the forest for the trees, and ended up trying to obey the letter of the law, rather than its spirit and intentions. Where possible, centralize, but don't take additional costs beyond the benefits of doing so.

By the way, one place where the "centralize only if it's convenient" rule has to be set aside is around validating inputs from foreign locations---in other words, any data which is passed across the wire or comes in from outside the local codebase. In order to avoid security vulnerabilities, data should always be verified as soon as it reaches your own shores, even if that means duplicating it in every foreign-accessible interface.

Models can be centralized

As tempting as it is to create "one domain model to rule them all", particularly given all the love for Domain-Driven Design in the past ten years or so. A similar corollary to the "one domain model" is the "one database model"---at some point in the enterprise IT manager's tenure, somebody (usually a data architect or consultant) will suggest that massive savings (of one form or another) can be had for the taking if the company takes the time to create a unified database. In other words, bring all the different scattered databases together under one roof, centralized in one model, and all the data-integration problems (data feeds into databases, ETL processes, and so on) will be a thing of the past as every single codebase now accesses the Grand Unified Data Model.

I have never seen one of these projects ever actually ship. Other architects have told me that they've had them ship, but when I follow up with people who've been at said companies, the universal story I hear is that once built, the resulting model was so complex and unwieldy that within a short period of time (usually measured in months) it was abandoned and/or fractured into smaller pieces so as to be usable.

The problem here is that different parts of the enterprise care about different aspects of a given "entity". Consider the ubiquitous "Person" type, which is almost always one of the first built in the unified model. Sales cares about the Person's sales history, Marketing cares about their demographic data (age, sex, location, etc), HR cares about their company-related information (position, department, salary, benefits status, etc), and Fulfillment (the department that ships your order once purchased) cares about address, credit card information, and the actual order placed.

Now, obviously, trying to keep all of this in one Person entity (the so-called "fat" entity, since it has everything that any possible department could want from it) is going to be problematic over time---if nothing else, fetching a list of all of the Persons from the system for a dropdown will result in downloading orders of magnitude more data than actually required. (This also runs afoul of the "Bandwidth is inifite" and "Latency is zero" and "Transport cost is zero" fallacies of Distributed Systems.) Clients will quickly start caching off only the parts they care about, and the centralized data model is essentially decentralized again.

The next reasonable step is to split Person up into "derived" models, usually (in the relational sense) by creating subsidiary tables for each of the specific parts. This is reasonable, assuming that the cost of doing joins (in the relational sense) across the tables is acceptable. Unfortunately, these sorts of centralized data models are usually supposed to hold the entirety of the enterprise's data in one database, so the costs of doing joins across millions of rows in multiple tables is often prohibitive. But let's leave that alone for a moment.

Where things really start to go awry is that enterprise systems are never monolithic (see the next fallacy), and the code that accesses the centralized data model often needs to be modified in response to "local" concerns; for example, HR may suddenly require that "names" (which are common to the Person core table) be able to support internationalization, but Marketing is right in the middle of an important campaign, and any system downtime or changes to their codebase are totally unacceptable. Suddenly we have a political tug-of-war between two departments over who "owns" the schedule for updates, and at this point, the problem is no longer a technical problem whatsoever. (This is the same problem that sank most centralized distributed systems, too---any changes to the shared IDL or WSDL or Schema have to ratified and "bought off" by all parties involved.)

Where this falls apart for domain models is right at the edge of the language barrier---a domain model in the traditional DDD sense simply cannot be shared across language boundaries, no matter how anemic. Classes written in C# are not accessible to Java except through tools that will do some form of language translation for local compilation, and these will almost always lose any behavior along the way---only the data types of the fields will be brought along. Which sort of defeats half the point of a Rich Domain Model.

The system is monolithic

While this may have been true in older systems (like, around the mainframe era), often the whole point of an enterprise system is to integrate with other systems in some way, even if just accessing the same database. Particularly today, with different parts of the system being revised at different times (presentation changes but business logic remains the same, or vice versa), it's more important than ever to recognize the different parts of the system will need to deploy, version, and in many cases be developed independently of one another.

This fallacy is often what drives the logic behind building microservice-based systems, so that each microservice can be managed (deployed, versioning, developed, etc) independently. However, despite the fact that many enterprise IT departments are building microservices, they then undo all that good work by then implicitly creating dependencies between the microservices with no mitigating strategy to deal with one or more of those microservices being down or out. This means that instead of explicit dependencies (which might force the department or developers to deal with the problem explicitly), developers will lose track of this possibility until it actually happens in Production---which usually doesn't end well for anybody.

The system is finished

The enterprise is a constantly shifting, constantly changing environment. Just when you think you've finished something, the business experts come back with some new requirements or some changes to what you've done already. It's the driving reason behind a lot of the fallacies of both distributed systems and enterprise systems, but more importantly, it's the underlying impetus behind most, if not all, enterprise software development. Enterprise developers can either embrace this, and recognize that systems need to be able to evolve effectively over time---or look for work in other industries.

This means, then, that anything that gets built here should (dare I say "must") be built with an eye towards constant-modification and incessant updates. This is partly why agile methodologies have taken the enterprise space with such gusto---as a construction approach, by the fact that agile embraces the idea that everything is constantly in flux, it deals far more easily with the idea that the system is never finished.

Vendors can make problems go away

Alternatively, we can phrase this as "Vendors can make problem 'X' a vendor problem", where 'X' is one of scalability, security, maintainability, flexibility, and just about any other "ility" you care to name. As much as vendors have been trying to make this their problem, for the better part of two or three decades, they've never been able to do so except in some very narrow vertical circumstances. Even in today's cloud-crazed environment, companies that try to take their existing enterprise systems and move them to the cloud as-is (the classic "lift and shift" strategy) are finding that the cloud has nothing magical in it that makes things scale automagically, secure them, or even make them vastly more manageable than they were before. You can derive great benefits from the cloud, but in most cases you have to meet the cloud halfway---which then means that the vendor didn't make the problem go away, they just re-cast the problem in terms that make it easier for them to sell you things. (And even then, they can only make a few of those probems go away, often at the expense of making other problems more difficult. As an example of how deployments and dependency management got burned, for example, see "npm-Gate".)

Enterprise architecture is the same everywhere

Somehow, there seems to be this pervasive belief that if you've done enterprise architecture at company X, you can take those exact same lessons and apply them to your experience at company Y. This might be true if every company had exactly the same requirements, but ask any consultant who's been engaged with clients for more than a few years, and you'll find out that the Venn diagram of requirements between any two companies overlaps about 80% or so. But here's the ugly truth of that secret: if we look at the Venn diagram of all the companies, they aren't overlapping on the same 80%---it's always a different 80% between themselves and any other company. Which means, collectively, that the sum total of all companies overlaps across maybe 5%. (All accounting systems agree on what credits and debits are, but from there, the business rules tend to diverge.)

Given that enterprise architecture is highly context-sensitive to the enterprises for which it is being developed, it would stand to reason that enterprise architecture will differ from one company to the next. No matter what the vendor/influencer tries to tell you, no matter how desirable it is to believe, there is no such thing as a "universal enterprise architecture"; not MVC, not n-tier, not client-server, not microservices, not REST, not containers, and not whatever-comes-next.

Developers need only worry about development problems

Enterprise systems come with much higher criticality concerns than the average consumer software product. Consider, for a moment, the average iOS or Android application---if it crashes mid-use, the user is obviously annoyed, and if it happens too often, they might uninstall the application entirely, but no signficant monetary loss is incurred to the company. If, on the other hand, the company's e-commerce system crashes, literally thousands of dollars are potentially being lost per minute (or second, if the scale is that of an Amazon or other large-scale e-tailer) until that system gets back on its feet and can start processing transactions again. And that's not counting the cost of potential customer service costs or even lawsuits if an order is lost because the system went down mid-transaction and put the data into a corrupted or unrecoverable state. Nor does that consider the intangible costs that come into play when Ars Technica or Forbes or---worst of all---the Wall Street Journal covers the outage in their latest report.

Enterprise systems, by definition, have much higher reliability and recoverability concerns. That means, practically speaking, that any enterprise system must pay much greater attention to how the system is administered, deployed, monitored, managed, and so on. Thanks to the emphasis on the whole "DevOps" thing, this is becoming less of an argument with most developers, but even within companies that don't subscribe to all of the "DevOps" philosophy, developers will need to spend time thinking (and coding) about how operations staff will do all of the things they need to do to the system after its deployment.

For example, one such concern is that of error management and handling. But first, please repeat after me: "It is never acceptable to find out about an enterprise system outage from your users." Why this is not an accepted truth is well beyond me, but countless enterprise systems seem to feel it perfectly acceptable to show their users stack traces when things go wrong, or that it's perfectly acceptable to only worry about restarting the system when a user complaint informs Operations it's down.

Yes, vendors can often provide certain kinds of management software to look at the system from the outside---keeping track of processes to make sure they're still running and such---but on the whole, it's going to be up to developers building the enterprise system to make sure that Operations staff can peer inside the system to make sure everything is running, running smoothly, and can make the changes necessary (such as adding users, changing users' authorized capabilities, adding new types of things into the system, and so on) without requiring a restart or editing cryptic text files. Management, monitoring, deployment, restarting the system after a failure---these, and more, are all developer responsibilities until the developers provide those capabilities to Operations staff to actually use.

On Finding Learning

Fri, 22 Jul 2016 00:00:00 +0000

tl;dr In a recent blog post, a commenter asked some questions that I felt were a bit more easily answered in the main blog format than in comments. Specifically, he asked two of the more common "How do I..." questions---finding motivation, and finding time.

In the "The Value of Failure", Greg asked two questions:

Finding motivation

First off:

How to motivate myself to grow and learn, when I'm the only senior java developer in my company. I'm surrounded with people who don't care about gro wing [sic], and learning.
Let's say I can change my "corporation".

For this one, I'm going to channel some Susan Fowler and suggest that motivation comes in two basic forms: "extrinsic", whereby motivation comes in external forms (external rewards, such as bonuses or raises, or external threats such as being reprimanded, shamed, or fired), or "intrinsic", whereby motivation comes from within.

Intrinsic motivation vastly trumps extrinsic motivation. Psychology has confirmed this through a variety of studies. Anecdotal evidence will usually affirm it as well, if you happen to be in a position to observe both.

The real message is what Fowler says in the very beginning of the book: It's not a question of motivating you. You're already motivated; the real quetion is what you are motivated to do. And whether you're motivated to do the things that you think you want to do. (It's not always easy to answer that, and it's not always as obvious as we might think.)

For example, you're already motivated to learn, Greg. You're asking questions, you're reading blogs and watching videos, and you're looking for new input. This isn't a question of motivation for you. At least, not yet, anyway.

What you may be feeling is frustration, given that you're surrounded by people who don't feel similarly as you do. And that's where I'd turn the tables on you, put on my philosopher's and psychologist's hat, and ask, why does their motivation matter to you? It's the only obstacle you cite in your question, so it's clearly important to you, but why?

In fact, you're sort of arguing that they're not motivated. I would disagree---as Fowler points out, they, too, are motivated, but they are motivated in a different direction than you are. It might be an interesting exercise to find out what they're motivated to do, rather than blame them or implicitly shame them by suggesting that their motivations are somehow inferior or less professional than yours. (And I'm not suggesting that you're making a big deal out of it; I'm merely saying that there is a strong strain of "Everyone working in this industry should be constantly growing and learning" that is common to the Software Craftsmanship movement, and I find it to be a little (or a lot) elitist.)

Getting back to the practical, I think your question is not so much about motivation as it is about facilitation---in other words, how do you find new things when it's not coming to you through the channels of your current job? And for that, I think you can probably guess the answers: user groups, hackathons, and so on. You've already hinted at the idea of leaving your current work situation, which is obviously one approach, but I wouldn't be too quick to pull the trigger on that just yet. Yes, it's always cool to (as Chad Fowler once put it) "Be the worst musician in any band you're in", so that you're always surrounded by people who will challenge you, but those positions can be hard to find, and they still don't always give you everything you want or need. One thing that I do, which you may find one avenue to approach, is to teach---I currently guest-lecture at the University of Washington on mobile development, and frankly, I find that I've learned more about mobile development for both iOS and Android from doing that than I ever did "just" writing code. (Students are amazingly good at challenging assumptions and asking questions you never think to ask yourself....)

Finding time

Next,

What is the most efficient balance between learning and working? I have two kids and no time after work to learn. The only time I can find is during my working day. Unfortunately project I'm working on, don't give me any new experience...

As a father of two boys-grown-to-men (23 and 16 as of this writing), I have total sympathy with you. It's not easy, trying to juggle family life and professional life outside of the usual forty hours a week. And while it would certainly be nice if work would help out by allowing you some dedicated time each week towards study and/or other kinds of advancement, the key thing is to remember that most companies are a little short-sighted on this point---they believe they are paying you for your time spent delivering direct value to the company, and most don't do a great job of tracking the indirect value that you can bring through taking time to train. We can rant and wail about how that should be different, but given the tight margins under which many companies operate, it's an established---and not soon-to-change---fait accompli.

And the fact is, I can't really tell you how to balance your life, other than to tell you to balance your work/life balance deliberately.

People will often speak of "finding" time. Time is not "found". We all of us have the same amount of time per day, and the only variable here is how we choose to spend it.

Most of the time, people will utter statements like the above with the intent to persuade you to spend more time with your family. I won't do that; in fact, I'll even go so far as to tell you to spend more time at work, not because I think you need to do so in order to be a solid "Craftsman", but because if you are the principal breadwinner in your family, then you need to make sure you are doing your due diligence to your family not just for today, but for the next eighteen years (or more). You do that by maximizing your opportunities to bring in revenue to the family, so that your family can have the financial freedom and flexibility to live the kind of life you want them to be able to live.

But, paired with that, is the realization that your children will only be young once. They will be looking at you all of the time, and they will be learning from you how to be an adult, even when you think they're busy playing with their toys or nose-deep in their phones.

Yes, I'm speaking out of both sides of my mouth here, because this decision---how much time to spend in your off-hours improving yourself both as a programmer and as a father---is one that is intensely personal, and not one that should be made for you by anybody other than yourself.

Thus we get to what I mean by the phrase "make it deliberately": examine the things you do in your life on a daily basis. Look for ways to cut things out of your life that aren't really things that you think will bring real value. For example, none of my family watches TV together. (As a matter of fact, we don't really watch much TV at all, though my 16-year-old likes to binge-watch anime from time to time.) Cutting TV out actually creates a ton of extra time for reading or coding or cruising through blogs and such.

When I tell this to people, I sometimes hear "But in some families, TV time is the only time the family really spends together, so if you cut that out...." In my family, that's not the only time we spend together; about ten or so years ago, one of the popular board-game companies (Milton-Bradley, I think) started an ad campaign about "Family Game Night". Each week, one night is reserved for the family to play a board game together. My family started doing that, but before long we found that after making dinner and doing dishes, we were kind of burned out about cleaning up the table to set up a game.

So we adopted a new rule: We took the games with us out to restaurants, and we played over dinner, there, in the restaurant. Lots of different games: Settlers of Catan, Magic: The Gathering, Carcassone, Ticket to Ride, Smallworld, and others. Wait staff at restaurants soon recognized us and (because we tended to tip a little extra by way of thanks for letting us consume a table for two or even three hours, sometimes) greeted us by name and took us to the larger tables so that we would have room to play.

What does this have to do with making time to learn programming? Simple: It's easy to feel less guilty about telling the kids, "Hey, I have to go out and get some work done" when the kids know that they will get time with you later, and it will be "quality time", playing a game and tossing trash talk back and forth. (Games in my family tend towards the competitive end of the spectrum.)

Mindfulness

This is where I get a little philosophical again, and start talking about that weirdly Zen thing called "mindfulness". It's really not all that deep of a Zen thing, not like "the sound of one hand clapping" or anything; mindfulness is simply the idea that if you're going to be doing X, be doing X, and not any of the two hundred other things that tend to distract and weaken your presence.

If you're in a meeting with somebody, and they're constantly checking their phone, looking at their laptop, jumping up every so often to take a phone call or send a text message, and so on, it's generally not a good feeling. You can't be sure that you've got their full attention. Any decisions that come out of that meeting are always going to carry with them the stain of "But was Fred really listening to me when he said that?"

If you're going to spend time with the kids, then focus on them. Play with them. Talk to them, listen to them, create new stories and games with them, and so on. Be "present" in that moment with them, without the distractions and without the constant interruptions (to the best of your ability; there's always going to be exceptions, obviously).

But the other interesting thing about family is that sometimes, you can actually spend time with them and be doing things that help your career simultaneously.

Case in point: As I write this blog post, I'm sitting in a restaurant with my wife. She is heads-down on her laptop, as am I. We're just sitting here, not talking to one another, each of us doing our own thing and in our own world, yet this will, in some indescribable way, count as "together time". I've done this with the kids, too: they bring homework, I bring the laptop, and we work together at a restaurant (have I mentioned my rather serious addiction to Diet Coke before now?). I'm there if they have questions (though now at their age, I'm generally not of much help other than to try and work through to the answers with them), and the rest of the time, I'm heads-down writing code or working on an article or whatever.

Wrapping up

Keeping up with all of the things going on in this industry is tricky and hard. Do so deliberately---mindfully---by deciding very specifically what you want to study next. (Turns out it won't really matter what you choose, only that you choose something to study.)

Decide very specifically how and when you want to spend the time with your family, and don't be afraid to carve out some time for yourself to do that study.

Look for ways to claim some time back---if you currently mow your own lawn, look at the cost of hiring a gardener to do the work for you, and use the time you've saved to study instead.

Create some forcing functions---user group presentations, articles for publications, anything with a deadline---as extrinsic motivation to make sure you get that study time in. (Deadlines also help provide "justifications" to the family for why Daddy needs to work, and can't help with the dishes, by the way. ;-) )

Lastly, do all of this deliberately and consciously. Think about spending your time the same way you would your money---sit down with your spouse or significant other and discuss how you need to invest time into your career during the off-hours. Budget. Carve out some time for yourself to study whatever it is that sounds good. Take Sunday afternoons for yourself, and block out any sort of social activities with the family for that time, if you must.

The first book(s) I ever wrote, I would do so at night. I would come home from work, do dinner and after-dinner with my wife and kids, then about 9pm or so, after the kids were asleep, I'd take a laptop down to a 24-hour restaurant in town, and work on the book there until about midnight or 1 (or later, sometimes). Head home, slip into bed, wake up and go back to work the next day. Didn't leave me much time for doing much socializing, but that was OK with me---my friends understood that this book was a big deal, and from time to time, I'd put the book aside and hang out with them for a while, just to recharge the batteries. Weekends, Saturdays were writing time, but Sundays were family time.

You don't have to commit that kind of time. You don't have to give up anything you don't want to give up. But you do have to understand that all of life is about priorities and budgeting time, and that if you're doing it right, the time you're spending on activities matches the priorities that you've set for yourself; if they're not, then it's time to do a thorough audit of your "time budget", and get them back into line with one another.

The rest, as they say, is up to you. Good luck.

(EDIT: My wife read this after it was published, and found a few typos and places where things weren't clear, so I tried to clean it up a touch. Release early, release often, as they say...)

Intellectual Honesty

Thu, 7 Jul 2016 00:00:00 +0000

tl;dr At last night's Seattle Languages meeting, I was reminded of what intellectually-honest debate does and does not look like; then, as part of the discussions and argument around the tragic deaths of several black men at the hands of police, I was presented with a link to a page entitled "Ten Signs of Intellectual Honesty". This is good material.

First off, the original link: Ten Signs of Intellectual Honesty. I'm going to be quoting from it liberally, however, in case you don't want to click through. (But please do so at least once sometime, so the author gets their just kudos for posting such awesomeness.)

With no further ado....

Do not overstate the power of your argument.

"One's sense of conviction should be in proportion to the level of clear evidence assessable by most. ... Intellectual honesty is most often associated with humilty, not arrogance."

The humility thing really strikes a chord with me. Many years ago, I had the chance to conduct a one-on-one CNN-style interview with Bjarne Stroustrup (the creator of C++, if you're not familiar with the name), and one of the first things that struck me was how quick he was to disavow the things he doesn't know. There is no braggadocio, no "look at the cool shit I've created", nothing. Brian Goetz (the Java language architect at Oracle) is similarly quite ready to admit what's outside his wheelhouse. Venkat Subramaniam is one of the most open minds I've ever met. And so on and so on and so on; the bigger the "name" in Computer Science, the more humble they tend to be. (With a few exceptions, many of whom have now "outed" themselves over social media.) I have since taken this to mean that the louder you crow about yourself, the less others are willing to crow about you, which is probably because there's not all that much to crow about.

"If someone portrays their opponents as being either stupid or dishonest for disagreeing, intellectual dishonesty is probably in play." Alas, it's not always that easy; too often, genuinely-interested people are arguing over the facts, because we're too busy clinging to the facts that we like and ignoring the ones we don't.

Show a willingness to publicly acknowledge that reasonable alternative viewpoints exist.

"The alternative views do not have to be treated as equally valid or powerful, but rarely is it the case that one and only one viewpoint has a complete monopoly on reason and evidence."

This is even more true in areas where clear objective evidence is simply lacking. I will freely admit that people can build non-trivial systems in Perl; that doesn't change the fact that I dislike the language. Nor does it change the fact that it can be useful. I have my reasons for why I don't like it, and we can debate as to whether those reasons are legitimate or pure opinion, but that doesn't mean it isn't useful to people who've used it in the past.

Be willing to publicly acknowledge and question one's own assumptions and biases.

"All of us rely on assumptions when applying our world view to make sense of the data about the world. And all of us bring various biases to the table."

Another quote from Stroustrup: "The more I know, the more I know I don't know." If the guy who's forgotten more about programming languages than I will ever know can say that, then there's no room for me to stand up and insist that somehow I am never wrong. I think I know things, but there's always the chance that I got the information wrong, the information I got was wrong when I got it, or the situation has changed since I got that information.

Be willing to publicly acknowledge where your argument is weak.

"Almost all arguments have weak spots, but those who are trying to sell an ideology will have great difficulty with this point and would rather obscure or downplay any weak points."

(Yeah, I really don't have a whole lot more to add to that.)

Be willing to publicly acknowledge when you are wrong.

"Those selling an ideology likewise have great difficulty admitting to being wrong, as this undercuts the rhetoric and image that is being sold."

This is probably the hardest thing in the world to do, and it's why so often we do so with these weasel-word qualifiers. "If I offended anyone, I am sorry that they were offended." Or "I was perhaps mistaken about the degree of truth in the statements you and I exchanged during that conversation." These are not admissions of incorrectness. These are attempts to salvage the ego and balm the uncomfortable feeling that comes with being incorrect about something.

Make the resolution right now: Admit when you are wrong at least once a day. That's mine.

Demonstrate consistency.

"A clear sign of intellectual dishonesty is when someone extensively relies on double standards. Typically, an excessively high standard is applied to the perceived opponent(s), while a very low standard is applied to the ideologues' allies."

It's hard sometimes to see, but this is part of why I think we engage in some of these endeavors---not to convince anybody of anything, but to find out where my own thinking is not being consistent and/or "fair".

Address the argument instead of attacking the person making the argument.

"Ad hominem arguments are a clear sign of intellectual dishonesty. However, often times, the dishonesty is more subtle. For example, someone might make a token effort at debunking an argument and then turn significant attention to the person making the argument, relying on stereotypes, guilt-by-association, and innocent-sounding gotcha questions."

Yeah, this starts to sound like just about every "flame war" I've ever seen go on between people who don't have respect for one another. Including a few I've been in. (And apologies, by the way, to anyone I made ad hominem attacks against in the past, intentionally or accidentally. I don't recall any, but I'm sure there's more than a few across my past.)

When addressing an argument, do not misrepresent it.

"A common tactic of the intellectually dishonest is to portray their opponent’s argument in straw man terms. ... Typically, such tactics eschew quoting the person in context, but instead rely heavily on out-of-context quotes, paraphrasing and impression. When addressing an argument, one should shows signs of having made a serious effort to first understand the argument and then accurately represent it in its strongest form."

Sometimes, unfortunately, in the tone-less medium of the Internet, it's too easy to interpret genuine questions for clarification or additional information around an argument as a precursor to a strawman, which is why as a general rule, if I'm seeking additional information, I try to just keep it to the question and nothing else---no interpretation and no "spin". (I will also be the first to admit that it's a favorite tactic of mine, to do exactly the opposite: Ask the question, anticipate the answer, and provide the "devastating" counterargument to what I think the answer will be. This is my commitment to try and break that habit.)

Show a commitment to critical thinking.

Want to start? Pick up a book on philosophy. Seriously. This is a subject that is all about doing nothing but critical thinking---there's little in the way of repeatable, observable experiments that can be done in philosophy to arrive at an empirical truth. (In fact, one could argue that as soon as we can do that in a subject, it becomes "science" instead of philosophy.)

Best thing you can do for your career, honestly.

The original post contains a quick link to this, which contains the following workable list:

gather complete information – more than one source
understand and define terms (make others define terms, too)
question the methods by which results were derived
question the conclusion: do the facts support it? is there evidence of bias? remember correlation does not equal causation.
uncover assumptions and biases
question the source of information
don’t expect all the answers
examine the big picture
look for multiple cause and effect
watch for thought stopping sensationalism
understand your own biases and values

From Human Biology: Health, Homeostasis, and The Environment, 3rd Edition, by Daniel D. Chiras.

Again, all of these are things that philosophical thinking tends to drum out of you. It begins with asking questions, and then questions about the questions.

Be willing to publicly acknowledge when a point or criticism is good.

"If someone is unable or unwilling to admit when their opponent raises a good point or makes a good criticism, it demonstrates an unwillingness to participate in the give-and-take that characterizes an honest exchange."

More importantly, it demonstrates that you can separate the argument from the arguer---and gets the egos out of the way. (By the way, this holds for any criticism---if you're being criticized over something, including things like a code review, finding at least one thing in the critique that you can agree with in any way can help to separate your own ego out of the discussion.)

This whole blog is good, if you're a fan of science (with an emphasis on biology, it seems).

Speaking Tips: James Ward's Suggestions on Abstracts

Thu, 7 Jul 2016 00:00:00 +0000

James Ward also wrote an article on how to write abstracts, and I realized after I published that I forgot to call out to his article as well. Mea culpa.

First off, James' post is here, and for the most part, I think we agree on most things.

Where he says "Make the title clear, concise and enaging", I would word that differently, simply because too many readers will see "engaging" and think "cute", which I have come to dislike. Go with simple, straightforward, and clear. Go with boring. (The body of that list item makes it more clear, I think, that he and I actually agree on this.)

James then says "Start the abstract with a problem or controversial statement". Staring with the problem is basically the "pain" part of my abstract. I dislike the idea of starting with a controversial statement, however, because I think that tends to attract the wrong kind of attendee to your session. Specifically, it goes back to the whole, "You get what you go looking for" mantra that parents have been trying to teach their kids---if you go out there looking for a fight, you'll find one soon enough. In an abstract, if you say, "Object-orientation is dead; functional programming is here to rule the world", several things will happen:

Some of the people who will show up will show up just to argue with you. You are threatening their understanding of the universe, and in many cases you are directly threatening the tools they use to make a living, and as such, you are threatenting them.
Some of the people who will show up will be functional programmers who are zealots like you profess to be, and they will either (a) turn on you if your session isn't as zealous as your abstract suggests, or (b) "help" you during the session by yelling or shouting at would-be object-oriented hecklers. What's worse, their behavior will often kick in even on questions asked by people who really don't have an opinion, thus ruining the experience for them.
Most importantly, the people you really want to reach will probably not show up at all. Think about it---would you attend a session feeling safe and comfortable if the abstract read, "All computer programmers are a drain on our society, and in this session, we'll talk about how regular folks can throw off the chains of tyranny imposed on us by these pasty-white nerds and their mysterious boxes of electricity"? Probably not.

Of course, if the intent of the session is to encourage that kind of argumentation, which is common for a speaker panel or some other audience-participation-driven session, then by all means, go for it. But remember, your abstract is your contract with the audience---if you don't make good on the contract (including the salacious "Jer-ry" parts, if that's what you promise), then you've failed and you're probably going to get hammered in the evals.

Next: **"Make the abstract about the content and attendee, not you". This one, I didn't talk about, and he's right. In all fairness, I think this is true of much more than the abstract, and something I think is just a general approach/attitude about all speaking, but it deserves mention and I'm glad James throught to bring that out.

Here we might disagree a little bit more strongly: "Clearly describe what attendees will take away from the session". My concern hinges on the word "clearly"; many reades might take that to mean "in great detail", and as I said earlier, an abstract is not a detailed play-by-play of what you're going to do in the talk, but a high-level overview. The body of his tip suggests that he and I are more on the same page, though, when he says, "Make sure the abstract clearly identifies what the attendee has to gain from going to the session". In other words, make sure the session has the Promise to go along with the Pain.

But when he says, "Although this could have been improved by more clearly saying that attendees will not just walk away with the 'why' but the 'how'", I'm not sure I agree. It depends on the focus of the session---for a higher-level session, such as a discussion of the Fallacies of Distributed Computing, for example, the "how" isn't as important. The "why" is essentially the point of the session. If the session is supposed to be technical and demonstrative, however, then the "how" needs to come out, and it's a good idea to say that in the abstract, but without giving away which "how" points you're going to discuss. (Again, you as the speaker want to maintain some "pivot room" within the talk, even after it's been submitted.)

Lastly, James rounds it out with "Like a resume, make it professional". People, that shouldn't even have to be mentioned. If you want to be taken seriously, you have to take your communication skills seriously in turn.

(Sidebar: I've heard some folks complain that this unfairly biases the selection process against those for whom English is not a native language. I'm sympathetic to that point, but you have to see it from the organizers' perspective---if the conference is in English, and the abstract doesn't demonstrate a strong grasp of the English language, what faith do the organizers have that the presentation itself will have a strong grasp of the English language, and thus be something the attendees will be able to understand? And let me be the first to say that if I were preparing to deliver a session in France, speaking in French, I would fully expect my session to be rejected out-of-hand, because my French simply isn't up to the task, written or spoken. This isn't about discrimination, it's about making sure the attendees get what they paid for: something they can understand and use in their day jobs. It's for this same reason that American speakers need to be very sensitive to American cultural references when speaking outside of the US, something I already alluded to in the previous article. Clarity must triumph.)

Honestly, if you're not a professional writer, then it behooves you to take your abstract to a copyeditor or your favorite family "GrammarNazi" and ask them to do a pass on it. It may not be fun---being corrected rarely is---but if it makes the difference between a "one look and to the rubbish bin" and "a second look and maybe a third and acceptance",isn't it worth it?

It is too possible

Tue, 21 Jun 2016 00:00:00 +0000

tl;dr Once again I find myself in the position of needing to call BS on a blog post and deconstruct it: Yes, it is possible to be a good .NET developer, and here's why.

First, as always, if you've not read the original, check it out. Again, I'll be quoting from it, so you needn't remember all of it, but read it once just to get a good idea of what's there.

Now, let's deconstruct.

Why you can't be a good .NET developer

Mr Ashton seems to be basing his argument on a couple of things:

Developers are leaving .NET.
The reason developers are leaving .NET isn't because of "self-loathing", but because they work in shops that cater to the lowest-common-denominator.
... There is no third reason. Just lots of use of the perjorative word "derpy". Because that will make the point far better than any logic could.

Let's break those down:

Developers are leaving .NET

Mr Ashton opens with:

The reason why people "leave" .NET is ---

This one will be hard to invalidate, because there's two forms of evidence that we can pull into play: empirical evidence, which is hard to gather objectively, and anecdotal evidence, which is easy to gather (for each of us individually, anyway), hard to refute, and yet hardly industry-indicative.

Here's what I mean. Assume that I have observed that when the door slams, people jump. As a matter of fact, I can go around and slam doors, and watch that everybody within close proximity of the door jumps. That means that, anecdotally, 100% of the population will jump when I slam the door. Yay science!

Of course, I probably should mention that none of the people being observed are deaf, and none of them have been through this particular experiment before. (As it turns out, you can get used to anything, including the noise of slamming doors, and thus not flinch, despite what anecdotal evidence may tell you.)

But perhaps this is a tangent, becaues Mr Ashton never comes back to this point of developers leaving---he prefers instead to focus on the rationale, which is:

Lowest-common-denominator enterprise shops

To work in a development shop with a team is to continually cater for the lowest common denominator of that team and the vast majority of software shops using .NET have a whole lot of lowest common denominator to choose their bad development decisions for. ... It'll not happen because as long as you're working on a platform that is primarily used by derpy enterprise shops, you will continually be held back because those derpy enteprise shops are continually be held back by the derpy enterprise developers that work in the derpy enterprise shops.

In case you weren't quite clear on his position, he goes on to say:

It isn't self loathing, it's self preservation and an eventual realisation that you can't actually progress so long as you're being held back by bad decisions made to cater for the slow and the stupid. Self loathing is just an intermediate stage that people go through while they still believe they can make an impact on the environment around them by caring and shouting into the void to enact tiny changes that help nobody.

Sounds to me like Mr Ashton has some angst he needs to work out in therapy.

Perhaps he has some strong empirical evidence to back this position?

He cites two pieces of anecdotal evidence, and then makes some broad sweeping generalizations:

Tangible examples? I remember well the insistence of one boss that we use TFS because some developers would find it hard to use git.

Or, perhaps, because the boss had some other good reasons not to use Git. I
can imagine a couple:

Git often implies GitHub, which can be costly for businesses if the source needs to remain closed-source. By all means, there are alternatives to GitHub, one of my favorites being BitBucket, but developers frothing at the mouth over Git often can't comprehend the idea of using it without GitHub.
Managers often need their reports. TFS, for all of its flaws, has a better integration story in the ALM cloud than GitHub does, and management that is used to (as in, built processes around) some of these reports will find it is a non-trivial cost to move to something other than TFS.
Why does which source control system you use matter all that much? Developers often get all worked up over this, but frankly, teams spent years working with non-distributed systems like Subversion and CVS before Git came along. As a matter of fact, unless your team spends significant amounts of time offline from the source-control server, there's a strong argument to be made that you don't need to use Git or Mercurial at all.

I remember the steadfast committal to ASP.NET web forms because the "new concepts" in ASP.NET MVC were going to take too long for the team to become productive in.

And that's a reasonable concern. Look, developers love to chase the bright new shiny, whatever that happens to be, and frankly they're more than willing to learn at their employers' expense at times, making mistake after mistake after mistake until they've mastered the new shiny in question---at which point they start chasing after the next new bright shiny. Quite often, these bright shiny chasedowns don't yield any practical benefit to the end-user---in fact, they achieve the exact opposite. Working with new languages, platforms, and tools often leads to more bugs in the early days, and if there's no commensurate improvement to the end users (in the long term, forget about the short term), then there's absolutely no business reason to go down this path.

I actually agree that WebForms is vastly inferior to MVC. My former company had a client that was heavily invested in WebForms, and refused all interest in moving to MVC, and their code was literally impossible to automate testing around because of it. As a long-term strategic decision, it was a poor one. But as a short-term tactical one, it made sense.

And that's the bigger picture: sometimes management needs to make short-term tactical decisions that aren't optimal in the long-term. Eventually, if the company has strong management, that course gets corrected. But that's not a flaw in the technology, it's a flaw in the management structure using it. I can (and have) see Java shops making the same decisions. C++ shops are hardly immune to this. Nor are COBOL shops.

The answer to this, of course, is to make sure that the team is spending some time coming up to speed on the bright shiny things that strategically are important to the company; my preference would be that they spend at least two hours a week doing some guided study around those topics. But that's another topic for another day.

There is now this furore [sic] over .NET core and the new thing in the tiny 0.001% of people that care are whether they persist in using Windows or switch to more productive environments.

Of course we assume that "more productive environments" is a shadowy reference to the Mac or a Linux environment, but frankly, I find that I'm just as productive in a Windows-based world as I am in a Mac-based world. The tools are different, and the approach is different, but there's a reason why I still make use of both environments on a regular basis.

And, by the way, .NET Core is an as-yet-unreleased environment. Trying to point to that as a failure of the .NET platform as a technology is like pointing to a half-finished supercarrier sinking. Of course it sank---it wasn't finished! Funny thing is, most half-finished environments or ships will also sink. Shall we compare the Ruby 0.9 release against the .NET 4.6 release and see which one wins, while we're at it?

Of course @aliostad gets it right here and points out that the primary "Important Thing" should be a focus on functional programming in languages like F# and of course the reason it doesn't happen is because "it's too hard for most people in our team".

I love the unattributed quote---it's so easy to pretend that you're quoting somebody without having to actually name anybody or any particular context, thus making it hard to refute. It's "almost as if you can't really find anybody to substantiate your arguments with actual fact".

But in this particular case, I've heard it from several development managers, speaking to both F# and Scala. And I've challenged them on several occasions, to let me spend a few hours going over the concept with their team, and see what the team thinks afterwards. In each of those situations, the team has "gotten it", and although they're not functional programmers ready to embrace Haskell as the One True Programming Language (which is ludicrous anyway), they understand the core concepts and can see where and how it might be applied. Or not applied, as the case may be. (Because, quite honestly, there are a number of cases where a functional mindset doesn't really fit the problem at hand. And this is why both of those languages---F# and Scala---are actually object/function hybrid languages, and not strictly functional.)

The larger issue here, though, is that managers understand---in ways that software developers don't seem to, as Mr Ashton emblemizes---that learning new syntax and concepts and approaches takes time. Absent any formal time set aside for the team to learn those new things, that time will most often come at the expense of the project on which these developers are working. And while sometimes that means that the project will come in early (because learning the new thing helped shorten the overall release cycle), far more often than not, it means the project gets pushed back. Which leads management to a pretty inescapable conclusion:

Project + Tool(new) = delays

Perhaps management isn't the derp here, after all.

But why is this .NET's fault?

This the part that doesn't seem to be elucidated in Mr Ashton's rant; he cites his two anecdotal examples (only one of which is actually .NET specific; one could see the TFS/git argument taking place in a Java shop just as easily as in a .NET shop), then makes his two sweeping generalizations, but without any actual evidence that this is somehow localized to the .NET segment of the industry. News flash, Mr Ashton: This exact same argument happens in Java shops ("Why can't we use Groovy/Scala/Clojure/Kotlin?") and, as companies invest more deeply into Ruby and/or NodeJS (assuming they ever do---Ruby seems to be fading fast from the enterprise landscape, for whatever reasons), they will make the same kinds of decisions around those platforms, as well.

Because, Mr Ashton, it turns out that this isn't a question of the tool or the language, it's a question of the environment in which they are being used. And, as any "derpy" enterprise development manager knows, the bleeding edge is called the bleeding edge for a pretty good reason.

Well, that means it's the derpy co-workers' fault

Let's be clear, fundamentally Mr Ashton's arguments really aren't with .NET, whether he realizes it or not. He's actually angry at his industry colleagues at these "derpy" enterprise shops, because (in his not-so-subtle opinion) they're the ones that management is trying to accomodate with their "derpy" decisions. Again, just to be clear:

It isn't self loathing, it's self preservation and an eventual realisation that you can't actually progress so long as you're being held back by bad decisions made to cater for the slow and the stupid.

It's pretty clear that Mr Ashton considers most enterprise software developers to be "slow and stupid". This is elemental logic: a manager doesn't make a decision for a team based on the consequences for just one member of the team, but on the basis of what the majority of the team has or lacks. Therefore, in Mr Ashton's view, the majority of enterprise developers are "derps" who are simply too slow and too stupid to understand the new things. What magnificent benefits could be had, if only they would stop plodding down the lane like oxen, and instead gallavant and run through the forests, wind whipping through their hair as they leap over obstacles like the rest of the stallions do!

Except that oxen never break their ankles attempting to leap an obstacle that is too high for them. Oxen will never dump their riders on the ground because they ran under a low-hanging branch that was too low for the human on their back. And oxen will never decide on their own to simply refuse to make a particular jump because, frankly, they just don't want to.

Roads were created to help eliminate the obstacles, and make it easier for farmers to get their goods to market where, you know, they can sell them and stuff. Which is used to buy other things, like food for the oxen. And a roof over their heads in the winter. And the brush the farmer uses to comb out the snarls in their fur. And so on.

(OK, all analogies break down eventually.)

Truth is, there's a lot of good reasons to stay to the road. Roads are something that anyone can navigate, whether oxen, stallion or simple plain ol' human. Roads also have this funny way of making it clear what is interesting in the world---if a particular town starts to grow, more people will start traveling to it, and the roads will---organically---stablize and get smoother and easier to travel. In fact, funny thing---if the road gets too much traffic on it, it will start to degrade, at which point people will step in and pave the thing. And put up signs to regulate traffic on it. And so on, each time reducing the obstacles on the road itself and making it easier to travel.

Don't be hating the "derpy" enterprise developer because they don't want to go crashing through the forests like stallions do. If you don't want to be one of those "derpy" enterprise developers, then don't be one. But considering those folks are the ones that wrote (and continue to maintain) the code that cashes your paycheck every month, maybe you'd be better off thanking them, instead of insulting them.

The Value of Failure

Sun, 29 May 2016 00:00:00 +0000

tl;dr Celebrating success is always a welcome thing. But in a lot of ways, the people we should be celebrating are the ones who failed, and then learned from it. As a matter of fact, there's a reasonable correlation to be drawn here---that those who are truly successful are the ones who failed first.

This is actually a post that I've been holding on to for a while now, having originally been inspired by a Harvard Business Review article that goes into this more deeply. In it, the author states:

One intriguing feature of these enormous success stories is that so many of them are little-known companies in ordinary, sometimes downright boring industries: railroads, health insurance, back-office automation. ... But the more important part of the story... is that every one of these star performers faced at least one “near-death experience” during the course of its long-term success. I don’t mean a few quarters of sluggish growth or a one-time product flop, but a radical shift in its market, a major technology disruption, or a disastrous strategic bet that threatened the company’s very existence.

Many, many years ago, when I was contracting for this guy Mike Cohn, he mentioned to a good friend of mine (who was working there with me) that "Ted needs to fail. Twice." Needless to say, Mike didn't say it to my face. I wouldn't have heard it for the advice that it was---I'd have seen it (as many people do, when I tell this story) as a desire to see me brought low or something. In truth, it was anything but that. Mike also was the guy who told me that, "You have so much book knowledge, people forget that you don't have a lot of experience, and expect a lot more out of you than you're capable of delivering. It's like being 6-foot-4-inches at 13; you look like an adult, but you're not one, yet."

In other words, Mike was channeling the old adage: "Good decisions come from wisdom. And wisdom comes from bad decisions."

(And, sure enough, once I had failed, twice, it was then that I started making real progress in my career as a programmer/consultant/architect. Mike had that one absolutely spot-on.)

The HBR author talks about how companies trying to avoid this near-death experience just somehow don't seem to "get it":

In an effort to understand the new logic of change and the emerging rules of success, we convened a conference around the theme “How Do You Overthrow a Successful Company?” .... It was a gathering of executives, strategists, and change agents from illustrious big companies who sensed that there were massive shifts on the horizon and who were determined to reckon with those shifts and embrace a new generation of business models, a new era of technology and communications, a new level of customer expectations and sophistication. ... In other words, they were leaders who wanted their companies to win big in fast-moving times without a near-death experience. It was a great idea for a conference, yet it amounted to a hill of beans.

He goes on to point out that all of the named companies in attendance, and how they were struggling or had met with nothing but struggles and difficulties---clearly the conference didn't really help them all that much.

So what is it about failure that is necessary for success?

Failure forces retrospection

First of all, far more than any success, failure forces players to examine what's going on and why they're not being successful. We see it all the time in sports: the first-round draft pick gets to the big leagues, and doesn't really seem to take the whole thing seriously, and (not surprisingly) performs disastrously. They're accustomed to success, so they don't bother with the kind of internal reflection and retrospection that is necessary to get better.

This happens with companies, too. When a company is successful, the investigation as to the mechanics of that success are limited, or else captured as part of the company's CEO tossing off some sound bites. "We are a company of passionate individuals who are committed to our firm's success" seems to be the phrase du jour. Success has everything to do with the company's effort, and nothing to do with any sort of external factors that might have affected the results.

When you fail, though, suddenly people want answers, and the usual raft of platitudes don't hold up anymore.

Failure creates desperation

More importantly, though, when the company's on the brink, there's a sense of focus and desperation that says "Well, we've got nothing left to lose, so let's go all-in on this and see what happens." This isn't a new idea; Cortez used it in his conquest of the Aztecs. By "burning the boats" (though history actually records that he scuttled the ships, rather than burned them), Cortez created a "We have no road back, we have to go forward, or die" mentality among his men. Their options were thus narrowed down to exactly two: conquest or death. Robert Greene, in his book "The 33 Strategies of War", calls this "The Death-Ground Stragegy".

Quite often we feel somewhat lost in our actions. We could do this or that--we have many options, but none of them seem quite necessary. Our freedom is a burden--what do we do today, where do we go? Our daily patterns and routines help us to avoid feeling directionless, but there is always the niggling thought that we could accomplish so much more. We waste so much time. ... Over two thousand years ago, the Chinese strategist Sun-tzu came to believe that listening to speeches, no matter how rousing, was too passive an experience to have an enduring effect. Instead Sun-tzu talked of a "death ground"--a place where an army is backed up against some geographical feature like a mountain, a river, or a forest and has no escape route. Without a way to retreat, Sun-tzu argued, an army fights with double or triple the spirit it would have on open terrain, because death is viscerally present. Sun-tzu advocated deliberately stationing soldiers on death ground to give them the desperate edge that makes men fight like the devil. ... The world is ruled by necessity: People change their behavior only if they have to. They will feel urgency only if their lives depend on it.

Obviously, we'd prefer not to hold death to peoples' faces in order to get them to better react at work, but sometimes, when the chips are down, the best thing to do is to simply state, "This is the only path forward, we either march, or we die."

So....

... what's the takeaway here?

Actually, it's pretty simple: Do you really want to succeed? The Death-Ground Strategy says that you should basically put yourself into a position where you either succeed, or die. If you're contemplating becoming a technical speaker, quit your job and throw "all-in" around speaking at technical conferences and what-not. Either you will speak at the user group, or you will starve.

Yeah, I'm not a huge fan of that. Try it if you like, but that seems like huge downside for very small upside. Maybe for an army trapped in a foreign land whose only hope for survival is to fight its way back home, it works, but for most of the rest of us, who are nowhere close to trapped behind enemy lines, much less staring death in the face.... yeah, not so much.

But there is value in failure, if only because it forces us to engage in that reflective and introspective process that helps us emerge stronger and better over time. Which then suggests to me that practically speaking, the takeaway here is to Put yourself into failure-potential situations. Take those risks. Put your career and credibility on the line. Engage in things that you aren't sure you can do, and because nobody likes to fail (including you!), you'll put forth much more effort into the exercise.

"33 Strategies" offers the following:

Stake everything on a single throw.
Act before you are ready.
Enter new waters.
Make it "you against the world."
Keep yourself restless and unsatisfied.

So, for example, if you want to be a technical speaker, stand up and volunteer to give a talk at the local user group. Matter of fact, volunteer to do four talks over the span of the next year. ("Four talks?! Are you out of your mind?!? That's like one every three months!") Don't do just the same talk four times---do four completely different, separate talks, which will then force you to figure out how to write good talks in a compressed period of time.

Or, if your desire is to get a startup off the ground, commit to having an MVP ready by the end of the calendar year 2016 without leaving your job yet. Because let's face it, if you can't get the MVP done in six months while holding down a regular full-time job, you won't be able to handle working the 80 hours a week that a startup will demand of you during the early days. And you'll need that MVP before you can get any kind of angel or seed funding anyway.

And when (not if!) you fall, figure out why, get back up, and keep going.

Practice, practice, practice

Fri, 8 Apr 2016 00:00:00 +0000

tl;dr Recently the Harvard Business Review ran an article on how readers could prepare for difficult business situations, using the analogy of coaches preparing their teams for different eventualities by simulating those eventualities on the practice field. There's lessons to be learned here for both programming and speaking.

First, go check out the HBR article: “Practice for Tough Situations as You’d Practice a Sport” by Andy Molinsky. It's a relatively short reads (as all HBR's web articles are; they're usually summaries of longer books, either theirs or something closely related to business). His focus:

To learn soft skills in a way that truly prepares us for what we’ll face when it really matters, we can take a few lessons from a different arena where learning, development, and performance are essential: professional sports.

Belichick: The Legend

My hometown football team is the Seattle Seahawks. Two years ago, we were defeated in the Super Bowl when a Patriots cornerback leaped in front of a sure-thing short slant pass for the game-winning touchdown on what was almost the final play of the game.

(And yes, critics, that pass is a sure-thing; in the 168 times it was executed all across the season, by any team, including both the Seahawks and the Patriots, as well as all 30 other teams in the league, only once was it ever intercepted---that fateful play in the Super Bowl.)

Bill Belichick is the coach of the New England Patriots. He is quite a character.

One key tenet of professional sports coaching, for example, is to prepare people in the most realistic contexts possible. When professional football teams prepare for their next opponent, they’ll take into account the likely conditions they’ll face. If the stadium the team is playing in is going to be noisy, coaches like Bill Belichick of the New England Patriots will play extremely loud music at practice to mimic game-time conditions. Belichick has even been known to pour water on practice balls to prepare the team for wet game-day weather.

Bill Belichick is one of the most intense, concrete, no-holds-barred anything-to-win coaches you will ever see in this lifetime. He oozes ruthlessness out of every pore, but it's a laid-back ruthlessness---he will never get in your face and be angry, but players have said that if he goes very quiet on you, you're in deep yogurt. Point is, Belichick has a strong reputation for success (more championship rings as a coach than I think any other coach in the history of the game), and much of that success is that nothing fazes the guy. Absolutely nothing. Team's down by 42 going into the final two minutes of the game? Belichick has been there before, and the team feels it. Knows it. If anybody's going to figure out how to score a metric crap-ton of points in an impossible amount of time, it's Belichick. Nobody relaxes playing against a Belichick-coached team, no matter how good your lead is.

When Malcolm Butler, the cornerback, was talking about that interception at the goal line, he pointed out that he had recognized that formation and that play from game film that they had studied of the Seahawks' game plan. He knew---from the bottom of his heart---what the play was going to be, and if you look at the replay, he simply muscles his way into the exact right spot where Russell Wilson threw the ball. He didn't have time to track where the ball was going---if I remember correctly, he was already muscling into the spot before Wilson let go of the ball. Butler knew where that pass was going. He had seen it before. They had studied it before. Which is to say, his coaches had seen it, pointed it out to their players, and the players had spent time practicing what they would do if and when that situation emerged.

And it paid off for them, in spades.

Belichick isn't, perhaps, the smartest coach in the league. But he's always got an answer for anything the other team throws at him, and more often than not, the other teams can't match whatever scheme he's cooked up. His teams win through preparation, not talent.

And that's the point: it's often the preparation---not the talent---that creates success.

Preparation

When I interviewed at DevelopMentor, all those many years ago, I was required to conduct a "test teach" down at the company's LA (home) office. I prepared a talk, of about 35 minutes in length, and I was asked to deliver the talk in one of the classrooms while the students were out having lunch. Luc (who would later become the "instructor liaison", but for that moment was basically the "man Friday" for most of the company) took orders ahead of time, and ran out to a nearby restaurant to order and bring back the food.

I set up my slides (which I'd actually written as plain HTML files, since I was very much a newbie at PowerPoint), and as I glanced up, I saw about 30 people in the room. Some were standing, most were sitting, a number were typing away at keyboards, a few were chatting in the back. Most of them really weren't paying any attention at all to me. I waited for a few minutes to see if they would notice I was ready to go. I'd been told, "You only have about 35 minutes, and we can't hold up class, so make sure you stay within time".

Fortunately, I'd done some Toastmasters in the past. When a crowd doesn't notice you're ready to start, you might do something to get their attention, then go ahead and start. Some speakers will clear their throat, some will be timid and say, "I'd like to get going now, please", but me... well, I've never really been wired that way.

In a small room of about 30 people, I tossed off a "HELLO!" that was about two volume levels too high. It was pretty loud---a few people actually jumped. And everybody looked at me, completely shocked. And without missing a beat, I said in a much more appropriate tone of voice, "My name is Ted Neward, and I'm here to talk to you about Design Patterns today."

See, at that point they had a choice---they could admit rudeness and go back to what they were doing (which most crowds won't do, since most of us are relatively polite people), or they could pay attention to the talk, but once I had their attention, it was really on them.

And the only reason I knew this? Because I'd practiced this before, as part of Toastmasters.

I found out later, from Luc, that the room was quite deliberately coached to be a little "hard" on me. As a matter of fact, about ten minutes into my talk, Luc showed up with lunch, and stage-whispered his way around the room delivering lunch to everybody in it. All. Thirty. People.

Because you know what? If I can't lecture through a distraction or two, I'm not going to last long as a professional speaker. They had the same idea as Toastmasters has had: the more you practice dealing with a situation, the more comfortable you are at dealing with it when it eventually does happen.

Professional sports players practice every scenario they can think off---physically and/or mentally---so that when the moment arises, they're up to the challenge.

How do you prepare?

Practical

Molinsky writes,

For example, you might work on rehearsing your pitch to potential VCs in front of a crowd of colleagues you’ve coached to pepper you with difficult questions. You might create situations where a VC is late to the meeting — or rushing you to finish your pitch in half the time you had planned. You might also do the session in a setting that mimics what you’ll likely encounter in the real world, whether that’s a noisy coffee shop or an overheated conference room.

Many speakers often "practice" their presentations in front of a sympathetic crowd---friends, family, people who genuinely love this person, and want their feelings to be spared any sort of discomfort or pain.

If you love me, and I ask you to help me practice, you won't spare me discomfort. You won't try to shield me from pain. I need that discomfort, so that I know what it feels like. You actually hurt me by not making the situation realistic. Or worse than realistic.

When I was mentoring a young speaker before her first big debut presentation, I brought her down to the conference room where her talk would be to practice. We found the maintenance staff setting up chairs in there. I asked if we could use the room, and when they said, "Well, we need to be in here to set up the chairs", I told them that was fine. It was more than fine---it was perfect. I made her go to the front of the room and deliver her talk to an audience of five or so; myself, and the maintenance staff that could've cared less about her talk while they were setting up. I spent most of the time fiddling with my phone; no eye contact, no nodding, no real feedback of any kind. She was essentially giving her talk into a vacuum. (Which, as any speaker can tell you, is an incredibly draining experience.)

Eventually the staff finished, and after a few more minutes, where I deliberately kept moving around the room (partly to judge her voice volume while she was speaking, to be able to tell her whether she needed to go louder or softer), we wrapped up and went to go get dinner.

It was perhaps a bit harsh, but the point remains the same: people will get up and leave your talk for reasons that have nothing to do with your talk. Distractions will happen. Cell phones will ring. Tornado sirens will go off. Water will start pouring out of the ceiling for no reason whatsoever. (All of these have happened to be at one point or another.) You cannot control the circumstances surrounding your talk.

As an soccer coach, I routinely told my teams, "We are only as good as we practice." As a volunteer football coach for my son's football team, I routinely said the same thing. As a manager, I used that same logic to justify the time spent setting up infrastructure for our own internal IT, seeking to mimic the same tools, technology and process that we would use for any client.

An organization will only be as good as how it practices. How well do you deal with outages? How do you know? If you've never actually practiced an outage, you have no idea. You may have a plan, but if you've never practiced it, you've never actually gone through it, and therefore you really don't know if any of this is going to work. How well do you deal with bug triage? How well do you deal with peformance problems? How well do you deal with refactoring?

How well do you practice any of these things? If you don't, then you suck at it. I guarantee it.

The old joke says that a musician was stopped on the streets one day and asked, "How do you get to Carnegie Hall?" He looked at the questioner, and without a hint of irony, said, "Practice, my boy, practice."

Oh, and her talk? She killed it.

Reclaiming Design Patterns (20 Years Later)

Fri, 25 Mar 2016 00:00:00 +0000

tl;dr 20 years ago, the "Gang of Four" published the seminal work on design patterns. Written to the languages of its time (C++ and Smalltalk), and written using the design philosophies of the time (stressing inheritance, for example), it nevertheless spawned a huge "movement" within the industry. Which, as history has shown us, was already the hallmark of its doom---anything that has ever become a "movement" within this industry eventually disappoints and is burned at the public-relations stake when it fails to deliver on the overhyped promises that it never actually made. It's time to go back, re-examine the 23 patterns (and, possibly, a few variants) with a fresh set of eyes, match them up against languages which have had 20 years to mature, and see what emerges. (Spoiler alert: all of the original 23 hold up pretty well, and there's a lot of nuance that I think we missed the first time around.)

Patterns: What were they for, exactly?

In the early days of the patterns "movement", when the patterns were new and fresh, we didn't spend much time really discussing this all that much. They just were; those of us who read the book nodded at a few of them, having experienced them before in code "in the wild", and we went on about our day. Intuitively, it seemed, we realized that there were places where we could apply this knowledge, we appreciated the new dimensions the patterns opened inside our heads, and... yeah, cool.

The Gang of Four (GOF) seemed to realize from the beginning that this was a subtle art/science; in the last chapter of the book (which nobody ever seemed to read, unfortunately), they said:

It's possible to argue that this book hasn't accomplished much. After all, it doesn't present any algorithms or programming techniques that haven't been used before. It doesn't give a rigorous method for designing systems, nor does it develop a new theory of design—--it just documents existing designs. You could conclude that it makes a reasonable tutorial, perhaps, but it certainly can't offer much to an experienced object-oriented designer.

And in truth, that was the case: it hadn't accomplished much. One guy with whom I taught at DevelopMentor called Design Patterns "23 ways to use a pointer". What was the big deal?

And similarly, when management/senior team leads/architects threw a copy of the book at junior developers, expecting that they could read the book and suddenly "level up", they were profoundly disappointed.

The benefits, it seemed, were more subtle:

We hope you think differently. Cataloging design patterns is important. It gives us standard names and definitions for the techniques we use. If we don't study design patterns in software, we won't be able to improve them, and it'll be harder to come up with new ones.

Certainly, the GOF book spawned a movement, and the patterns movement spawned a whole catalog of patterns beyond the original 23 that the GOF came up with. But then things started getting even more abstract and high-level; patterns became "pattern languages" and that in turn spawned "meta-patterns". Then people started documenting the negative, calling them "anti-patterns".

Once people started realizing that there was money to be made, the writing was on the wall.

Various book vendors started publishing "patterns" books that barely touched on the GOF's original model. Patterns books became synonymous with "reusable code" (instead of "reusable elements of design"). IDE vendors started looking for ways to incorporate patterns as code generators. Patterns somehow also became the provice of UML and other design notations, and the goal at one point was to figure out how to create reusable design templates in UML that corresponded to patterns.

Like so many things, patterns became trendy and attractive to people who had no idea what they were for. How could they ever have actually met those peoples' expectations?

By the mid-2000s, patterns became a bad word, and speakers started essentially "trashing" patterns, suggesting that somehow they were an artifact of "bad languages" and "primitive thnking" and "subsumed into good languages".

Patterns were clearly useless.

And yet... they keep appearing. We keep using their terms and lingo. Why? The GOF actually (in that same chapter at the back of the book) called it back in 1995:

"Design patterns provide a common vocabulary for designers to use to communicate, document, and explore design alternatives. Design patterns make a system seem less complex by letting you talk about it at a higher level ofabstraction than that of a design notation or programming language. Design patterns raise the level at which you design and discuss design with your colleagues." (p389)

and

"Knowing the design patterns in this book makes it easier to understand existing systems. ... People learning object-oriented programming often complain that the systems they're working with use inheritance in convoluted ways and that it's difficult to follow the flow of control. In large part this is because they do not understand the design patterns in the system. Learning these design patterns will help you understand existing object-oriented systems." (p389)

and

"Design patterns provide a way to describe more of the "why" of a design and not just record the results of your decisions. The Applicability, Consequences, and Implementation sections of the design patterns help guide you in the decisions you have to make."

and

"One of the problems in developing reusable software is that it often has to be reorganized or refactored [OJ90]. Design patterns help you determine how to reorganize a design, and theycan reduce the amount of refactoring you need to do later."

(Yep, they were talking about refactoring long before it became hip.)

Most of all, they actually predicted the very problem that would be the downfall of patterns as a whole:

"It's easiest to see a pattern as a solution, as a technique that can be adapted and reused. It's harder to see when it is appropriate—--to characterize the problems it solves and the context in which it's the best solution. In general, it's easier to see what someone is doing than to know why, and the "why" for a pattern is the problem it solves. Knowing the purpose of a pattern is important too, because it helps us choose patterns to apply. It also helps us understand the design of existing systems. A pattern author must determine and characterize the problemthat the pattern solves, even if you have to do it after you'vediscovered its solution." (p393)

Too many short-sighted people saw the "solution", and in their rush to find reusable code, they forgot (or chose not) to learn the "why", and as a result ended up making some very stupid---and easily preventable---mistakes.

What is a pattern?

There's been a lot of discussion on this over the yeas, but I'm going to keep this down to a single sentence:

A pattern is a solution to a problem within a certain context that has a set of predictable consequences.

That's all it is. There's nothing magical, mysterious, or super-academic about it. If you can describe all four parts of that tuple, you have a pattern. Write it down. Publish it somewhere. When we start seeing some similarities between us all, we can start to see what the commonality is and then give it a nice well-chosen name, and it can join the general lexicon.

The Gang of Four published 23 of them. There's a lot more out there. Most are actually pretty useful. But particularly the first 23, because they stretch across a ton of different languages. (The patterns community later came to the terminology of an "idiom", which was something that was language-specific. Thus, the "Resource Acquisition Is Initialization" (RAII) idea from C++ was more-or-less tied to C++ as an idiom, and didn't really qualify as a pattern, per se.)

Why are we arguing again?

The last sentence, however, brings up another important point: Too much of the patterns community spent too much time arguing about them, and whether a given thing was a pattern or not, or whether a particular code snippet was an implementation of this pattern or that pattern, or....

There's several thoughts at work here:

The exercise of philosophical debate around a pattern's composition is important to the benefit they provide. This was why the patterns community said things like "It's not a pattern until it's been discovered three times" and "A pattern isn't really a pattern unless it's been workshopped at a patterns conference". The point wasn't to add some kind of mystery or intrigue or make them solely the provice of those "in the know"; the point was to debate them in a philosophical matter to refine the pattern to its essence, and keep the quality high. But once people start saying, "Pfffft" and avoided those critical steps, then suddenly everything became a pattern. If suddenly we can start calling local variables a "pattern", then the whole point of designing a higher-level toolbox of terms becomes lost. If anybody can call anything a "pattern", then we have everybody calling everything a pattern, and patterns lose their efficacy in conversation. If patterns are somehow localized to the team they were 'born" on, then now I cannot converse with anybody outside of my team. Science has a rigor to it (in particular, the peer-review portions) for a reason, and that rigor was what the patterns community tried to bring.
Which pattern happens to be in use here is a subject of philosophical debate. LOTS of people got hung up on arguing with other members of the team over what kind of pattern some code was. I cannot imagine a more useless activity, unless the discussion led to new insights (a new application of it, or a new "angle" to it) around said pattern. We burned a lot of goodwill by arguing with people over this.
Patterns were not the be-all/end-all of good design. You can have a good design that doesn't fit an established pattern. Pattern snobs needed to stop putting a qualitative judgment around those who weren't pattern snobs.

In the end, we patterns snobs probably brought the downfall upon our own heads, but we had help.

Time to take them back

So I figure it's time, 20 years later, to start the discussion all over again. I don't imagine that I can rebuild the entire movement on my own, but at the very least, I can take the old prose, dust it off, and look to bring it into the 21st Century and the languages that we use. There's also probably a few more patterns that we've found along the way, and where I think they fit, I'll take a stab at a few and put them up for people to consider and workshop. (Or, rather, more of a "mob-shop", since a weblog isn't really a workshop setting.)

Over the next n months, I'm going to put up the original 23 patterns re-cast for the "modern world". I'll do a quick recap of the pattern, but cast into the form I prefer (Problem/Solution/Context/Consequences). I deliberately won't spend a whole lot of time trying to re-describe all the prose from the book---that would be copyright infringement, in my mind, and I not only want to steer clear of that, but I really do want to encourage people to buy a copy of it for themselves, do their own reading, and debate whether my interpretation is legitimate and/or reasonable. Then, I'm going show how a few languages can implement them, and I'm going to range pretty freely over a bunch of different languages: C++, C#, Java, Swift, F#, Scala, JavaScript, and a few others. I encourage you to comment and post suggestions/amendments/corrections. I'm definitely not the smartest person in the world, and certainly not the last word on how to apply certain language idioms to a particular problem. Plus, I'd love to see us collectively flesh out some implementations around these patterns across all the popular (and maybe a few less popular, just for educational purposes!) languages.

It'll take time, so bear with me. I'll post another blog entry with the details soon, but in the meantime, if you haven't done so in a while, take down your copy of GOF, dust it off, and crack open to the first chapter.

When Interviews Fail

Wed, 23 Mar 2016 00:00:00 +0000

tl;dr Peter Verhas asks a seemingly innocent question during a technical interview, and gets an answer that is not wrong, but doesn't really fit. He then claims that "Sometimes I also meet candidates who not only simply do not know the answer but give the wrong answer. To know something wrong is worse than not knowing. Out of these very few even insists and tries to explain how I should have interpreted their answer. That is already a personality problem and definitely a no-go in an interview." I claim that Peter is not only wrong, but that in addition to doing his company a complete disservice with this kind of interview, I personally would never want to work for a company that takes this attitude.

Let's begin with his original article, here. Go have a look before you read any further---it's actually not that long.

Now, having familiarized yourself with the material, let's deconstruct it entirely, shall we?

Asking the wrong questions

For starters, the whole process begins, in my opinion, entirely on the wrong foot:

There are questions on a Java technical interview that even the most entry level junior is expected to give the right answer for. Since I am facing candidates who are not that junior I do not even bother most of the times to ask those questions. I assume that the candidate knows the correct answer. Sometimes, however, there are some candidates who I feel from the start they are juniors and to cut the interview short not wasting his/her and my time I ask some of those simple questions. The answers usually reveal the real level of knowledge and we can get to an agreement in a short time about the assessed level.

It begins with the phrase "There are questions....". Folks, let me be very clear about this: If you are conducting a technical interview, then you need to be asking them to write code, not answer questions. Unless their role in the position is to be a question-answerer of programming questions (in which case, you are interviewing for a teacher, not an actual programmer), then you need to be asking them to demonstrate their technical skills, not their knowledge of the terminology.

The reasoning for this should be pretty clear, but in case it's not, I'll argue it from logic, example, and analogy.

Logic: Not every programmer you interview will be classically-trained. They may not know all the preferred terminology. Are they "getters and setters", or "automatically-defined properties", or "accessors and mutators"? It really sort of depends on what language you grew up with (C++, for example, preferred the latter for quite some time). It depends on which books you read. It depends on whether you even had to discuss this with other people---perhaps the candidate actually learned everything from a book and reading stuff on the Internet. (StackOverflow's recent poll suggests that around a third or more of the developers in the candidate pool are self-identified "self-taught" developers.) Do you really want to be screening out perfectly-qualified candidates because they don't have the right words? And this doesn't even begin to address the pressure-cooker situation that most candidates feel they're in when they interview, causing them to flub even simple answers. Which brings me to...

Example: A developer who worked for me for two years was a quite capable C# developer. This is a guy who led teams, mentored some of the more junior developers, and came up with some quite capable designs. And then, when asked during a meeting by a prospective client, to explain what a static method is, he flubbed it completely, and started talking about constructors and what-not. I sat there looking at him for a few minutes, with a total "Dude, WTF?!?" look on my face, before he realized what he was doing. By Peter's criteria, he'd failed the interview. And yet, he served as the team lead for that client for nine months after that meeting, with nary a complaint about his skills, his abilities, or his answers on questions about static methods (which, ironically, never came up!) ever again. In other words, once we got out of the pressure cooker, he did fine, and his work showed it. Which brings us to...

Analogy: If you're hiring a band to play your wedding, do you really care about their ability to explain musical theory and composition? Or do you care more about their ability to play your favorite dance tunes, to play the song that your spouse has chosen to be "your song", and to get Grandma and Grandpa on to the dance floor with a rendition of "Funky Chicken"? Most bands (dare I say all of them?) get a gig based on their body of work and/or their demo tape, not their ability to answer questions.

Expecting the wrong answers

Continuing, Peter says,

To know something wrong is worse than not knowing. Out of these very few even insists and tries to explain how I should have interpreted their answer. That is already a personality problem and definitely a no-go in an interview.

Oh, hubris, thy name is "programming interviewer". Let's see what I mean:

One such simple question is: Can a static method in a class call a non-static method of the same class? If you know Java a little bit you know the answer: no, it can not. A static method belongs to the class and not the instance. It can even be executed using the name of the class directly without any instance of the class. It can even run when there is not even a single instance of the class in the whole JVM. How could it invoke a normal method that runs attached to an instance?

Oh, hubris. There is no reason that a static method cannot invoke an instance method. The thing that Peter is hinging on here is the fact that the static method lacks a reference to a particular object (which normally is the "this" reference), which is his justification for his answer: "No this, no method call".

And yet:

But then again: the answer from one candidate this time was: yes. And he even started to explain that it may happen that the static method has access to an instance. It may get an instance as a method argument and through that reference, it can call an instance method. That person was right.

And yet:

It did not, however, change the fact that he did not know Java well enough, but as a matter of fact in this very specific question, she was right.

So let me get this straight: the answer to the question was right, when you've said all along it was wrong, and yet "this candidate didn't know Java well enough"? I'd say that the candidate understood Java well enough to be able to find an answer that was not the one you expected, and yet was actually correct.

See, here's what's going on: the interviewer, convinced of their own technical superiority, is walking into an interview with a predetermiend set of answers to questions they're going to ask, and when a candidate doesn't follow that predetermined script, they're "not smart enough".

Here's a personal example. I was interviewing with a company years ago for a C++ position, and was asked, "Can a private field be accessed from outside of the class?" The normal candidate's answer is supposed to be "No, private creates an encapsulation barrier hiding the field from the rest of the world."

#include <iostream>
#include <string>
using namespace std;

class Person
{
public:
  Person(const char* fn, const char* ln, int a)
    : first_name(fn), last_name(ln), age(a)
  { }
  
  string description() {
    return first_name + " " + last_name + " is " + to_string(age) + " years old";
  }
private:
  string first_name;
  string last_name;
  int age;
};

int main() {
  Person ted("Ted", "Neward", 45);
  cout << ted.description() << endl;
}

The "age" field is, by design, unavailable to the rest of the world, right?

My answer: "Of course it can. You just have to cast the object instance over to a void*, calculate the offset from the start of the object, and then access it."

int main() {
  Person ted("Ted", "Neward", 45);
  cout << ted.description() << endl;
  
  void* pTed = (void*)&ted;
  int offset = sizeof(string) + sizeof(string);
  char* pTedAge = (static_cast<char *>(pTed) + offset);
  cout << static_cast<int>(*pTedAge) << endl; // prints 45
}

I even showed them how you could generalize this into a template (I called it "THackOMatic", and consider it one of my finest creations in the language.)

Now, you can take this one of three ways:

Wow, never thought you could do that. That's interesting. I wonder...
Well, yes, you can do this, but it's totally not a good idea.
This is totally not in the spirit of the question. You are still wrong.

And If you're in the first two camps, I'm kind of there with you. It's a cool hack, but it's a hack nevertheless, and hacks are usually a sign that you're doing something wrong, except in very narrow circumstances where there's simply no other way around it and you accept that you're the only one who will be touching that code from now on.

But if you're in the third camp, you're missing the point. The point being, the candidate figured out a way around a supposed stumbling block in the language. If you can't recognize that, then I submit that the fault lies with you, not with the candidate.

Which brings me to my last point.

You hire what you interview for

Right or wrong, the candidate took your question, thought their way through it, and came up with a novel answer to the question. And by focusing on the answer, you missed the important part--they found a way around it.

Part of this process is supposed to be finding those candidates who meet a certain technical bar, but part of the process is also supposed to be finding those people who will find ways around obstacles. Bugs, production outages, design flaws, whatever, you're supposed to be finding the candidates who won't accept the status quo as a concrete law of the universe.

A candidate did exactly that, and you shot them down.

You wanted a plain vanilla, boring-as-shit, everybody-get-in-line-and-do-what-the-supervisor-tells-you kind of answer, and they gave out an "out of the box", creative, hammer-throwing-at-the-screen one.

Do you claim you hire "only the best"? When you're doing this, you're hiring only the middle part of the bell curve, at best. Those who are on the far right-hand-side of the curve will be the out-of-the-box thinkers, who understand that rules are made to be broken sometimes, and under specific circumstances.

Do you really think they will want to work for you at this point? I wouldn't.

Wrapping up

So my challenge here is this: If you're an interviewer, what are you interviewing for?

By the way, that company I interviewed at all those years ago? The interviewer's response was classic: "Well, the right answer was supposed to be 'no', but I can see what you're doing here. You're the first person to ever give me that as an answer." They hired me shortly thereafter, and before I left the firm, I used a couple more language tricks to help cut the size of their codebase down pretty significantly in a couple of situations.

Hire the fired

Wed, 24 Feb 2016 00:00:00 +0000

tl;dr A Forbes.com article Q/A recently stated that job-seekers need to hide the fact that they've ever been fired from a position, because of the stigma associated with such an action. I couldn't disagree more.

First off, as is common for my posts, read the original, so we all know what I'm aruging against. But if you just want the short summary, a reader, "Madelyn", writes to "Liz" (presumably the Forbes equivalent of a Dear Abbey or something), saying

"You advise people to withhold the fact that they've been fired and to make it sound like they left their last job on their own. How can you advise people to be so unprofessional? If someone got fired from a job, they need to say so and they need to explain it."

"Madelyn" goes on:

"Personally, I would never hire anyone who had been fired from a past job. ... Maybe some employers are okay with that, but I wouldn't take a chance on that person. If the person I hired had problems again, I would have to acknowledge that I knowingly hired someone who had already been fired once!"

Wow.

Why Madelyn is wrong

First of all, it's clear that "Madelyn" is making the fundamental assumption that being fired is a negative thing (a "black mark") on your career history. I really don't agree---the assumption here is that you were fired "for cause", meaning that your manager had a Really Good Reason to terminate you. While that sometimes might be true, that's hardly always the case. Too many times, people are fired (terminated, separated, "let go", or whatever other non-"fired" word HR cares to use) because of personality conflicts with their boss (which are not always the fault of the individual under consideration, that is to say, you), because of personality conflicts with their colleagues (which are not always the fault of the individual), or even because the company simply doesn't need that particular department or the excess capacity in that position (a la a merger or acquisition).

Granted, much of the time, that latter scenario is called being "laid off", and "Madelyn" allows for that as an out: "Maybe they get laid off, but that is a different scenario." But let's be really clear about something---in a classic layoff, a company owes some form of severance pay, and most companies are quite loathe to pay that out, so they will simply terminate instead of "lay off". Case in point: when I worked for Neudesic, I was terminated without severance because the company had taken a huge hit in the form of a client contract termination. There was no "cause", per se---they simply couldn't afford to keep me on given that they'd taken this huge blow to their client portfolio. Should they have paid me severance? Probably, but it was likely a non-starter as a discussion point, and I didn't really feel like arguing with them about it.

In the case that I think "Madelyn" is trying to use as her stalking horse, a company will fire an employee for making a mistake of some monumental proportions. In some scenarios, it's because an employee just keeps making the same mistakes over and over again. In this case, the termination is understandable, and justifiable. (Even then, that doesn't mean that the employee is entirely at risk--if they weren't trained, yet kept being tossed into the pool to drown over and over again, it's hard to blame the employee.)

But in some scenarios, it's because an employee does something that the company doesn't want to be potentially liable for---the case of the fast food employee who chooses to give leftover food to homeless, for example, runs the risk of violating local FDA or health department rules. In order to avoid liability (or accountability) the company terminates that employee--even though in this case it's for doing something that most of us consider to be a good act--"with cause", because it potentially could be termed "theft". If you're that employee, this is really something to be ashamed of? This is really a reason not to hire a candidate---because they chose to act in accordance with a high moral standard (and didn't realize that doing so was creating problems)?

Why it shouldn't matter

But perhaps even more to the point, even if the company calls your previous employer to verify your employment, typically your employer will say nothing more than "He/she worked here from this date to that date" and hang up the phone. They won't say more than this in order to avoid any sort of legal liability. In fact, most employers of any size will demand that any reference check go to their HR department, who will then verify the dates, and if asked a reason, choose not to disclose the reasons for the end of the engagment. (Granted, I suppose it's always possible that your manager could choose to say something "off the record", but such things have a nasty way of somehow making it onto the record, which then leaves the company vulnerable to legal liability again; for that reason, most managers are taught pretty strictly not to say anything.)

Which then brings up the point: Why on earth would you bother telling them?

Seriously, beyond the bare facts of your employment history, you need not disclose facts about your employment past: your manager's name, your colleagues' names, your relationship with either employers or colleagues, your previous hits or misses, your previous salary or benefits, none of it is the interviewing manager's business! They of course may ask, but you are under no legal obligation (that I'm aware of, anyway) to disclose any of it.

And, given "Madelyn"'s rather medieval opinion on how to judge people, I think it's in your best interests not to volunteer anything. After all, the interviewer is expecting you to come into the position with a "clean slate", and not hold any of your previous managers' mistakes against the company---shouldn't you be granted the same "clean slate" in turn?

Why you should hire the fired

Most of all, though, those who have been fired have something valuable that their new employer could really use: experience. It is an old adage that says, "Good decisions come from experience; experience comes from bad decisions." For most people, being fired is not a positive thing, and they will replay the event over and over in their head, looking to learn from it. Personally, I would want the benefit of that experience. What did they learn from it? What can they bring to this position from that prior situation that we can learn from?

If you ever interview with me, let me make it clear: You need not tell me why you were released from that position. (If I forget, and ask that question, you may remind me that you are under no obligation to tell me; quite frankly, you should remind me.) If you choose to tell me that you were fired, I'll of course be interested to know what you learned from the experience, and how we can apply it to here.

But, of course, I may ask you the "what did you learn" question anyway, regardless of whether you were ever fired. Because, again, how you left your last position really shouldn't matter a damn.

"Liz"'s response

... is great. (Yes, she agrees with me.) Some excerpts:

"I definitely advise job-seekers to steer clear of managers who have firm and unshakable stereotypes in their heads, including "Never hire someone who's been fired from a job! Such managers are not typically people who can inspire others to greatness."
"Sometimes speaking up or bucking the status quo gets amazing people fired from their jobs."
"You can decide you don't want to hire people who are too outspoken, creative or non-linear for your taste. That is your privilege."
"Getting fired is simply waiting an instant too long to say 'I quit.'"

Never tell your prospective employer why you left. Because, again, how you left shouldn't really matter a damn to anyone.

2022 Edit: Caveat

If you were fired for reasons stemming from criminal violations and/or imprisonable offenses that required arrest and/or incarceration, you are (I believe) under legal requirement to disclose that on your employment application. If you're one of those people and you're reading this blog, chances are pretty good you've paid your debt to society and are looking for the clean slate. I, personally, think you are owed the chance to start over, but many big companies have different policies on that. I think your best bet is to identify those companies that have a heart and are willing to give you that chance--but, as I'm sure you're aware, that is really going to be a hit-or-miss proposition. If you're interviewing with me, I'll do what I can, but my hands are likely to be tied by HR policy in that case.

On Apple and encryption

Wed, 17 Feb 2016 00:00:00 +0000

tl;dr By now, everybody has heard that the FBI has issued a request (which is now being backed by a court order to comply) to Apple to provide software to unlock the iPhone 5c of one of the San Bernardino shooters. This is a massive request, with huge implications for everybody (Apple, Americans, foreigners, I really mean everybody). And most of those implications, I believe, are bad.

First of all, let's get some basic facts in place:

The San Bernardino shooters killed 14 people and seriously injured 22. However, "They were not directed by [foreign terrorist] groups and were not part of any terrorist cell or network."
The FBI obtained an iPhone belonging to one of the shooters. Because it was locked (like so many of us do), the FBI issued a request for technical assistance to unlock the phone.
Apple has refused, and issued a statement why.
A federal court judge has issued a court order demanding Apple's compliance with the FBI's request.

Whether Apple will comply with the court order remains to be seen. (UPDATE 2021: From all available evidence, they did not.)

I find myself a whole mess of different thoughts and reactions, and I'm just going to lay all this out in no particular order. Naturally, the disclaimer "I am not a lwyer" applies to all of this, and I'd love for those with training in the law and having passed the bar exam to weigh in with their expert opinion.

Legal issues

Let's start with the legal stuff.

Why does the FBI need this?

I'm confused. If the FBI issued a statement saying that they do not believe these people were part of an established foreign terror group or network, then what does the phone give them? In fact, what does the phone give them, period? Certainly there's the possibility that there's some kind of Twitter history (their favorites) or Facebook saved pages, but in both of those cases, they could go to the respective social media companies and get the information there. Even a history of SMS messages and phone log could be pulled from Verizon (the shooter's carrier).

I simply don't understand what information the FBI expects to remove from the phone that they couldn't get elsewhere. There's only two scenarios I can imagine that require this step:

there is something stored locally to the device that isn't stored in the cloud, or
if there is information stored to the cloud that they cannot access, probably because the cloud provider in question is not subject to US legal injunction/demands.

Suddenly, the whole "Safe Harbor" argument of a few months ago takes on a wholly new light. For those who weren't following the crisis/discussion then, the US government's willingness to demand companies surrender consumer information, in violation of EU Data Protection Directives, caused the EU to declare that the US was no longer a "safe harbor" for its data, and therefore not a safe place to store EU consumer data.

This is a perfect case where this can come into play---if the shooters' phone was using a system that stores its data in data centers on European soil, the US government has no jurisdiction over those servers, and therefore cannot strong-arm the company into complying with an order to surrender that data. This is exactly why the EU was concerned. And, apparenty, with good reason.

You can argue that in this case, the US should have the right to be able to demand that data. After all, this is not only a criminal investigation, but an investigation that could potentially have international implications. This is not unusual, nor new. Which leads me to my next thought....

Why isn't a warrant sufficient?

Normally, when the police want to obtain data about an individual (living or dead) held by a third party, they must obtain a warrant from a judge, ensuring that the request is a legal one and within the boundaries of jurisprudence. We see this all the time in legal discussions and on TV shows (for crying out loud). But there's no discussion of any warrant being issued in this particular case, nor of Apple being given an opportunity to challenge the legality of the warrant, which leads me to wonder, was no warrant issued, and if so, why not?

The device was actually owned by the county

This story takes on a slightly different twist because the device apparently wasn't owned by the shooters themselves, but by the county (for whom they worked, apparently?). Thus, the legal steps necessary to obtain the right legal framework by which to obtain the data was easily navigated---the FBI asked, the county granted.

This raises some interesting lessons for IT departments around the use of mobile devices, as well as for those who use mobile devices as part of their work day:

Never mix personal and professional on one device.

Look, I know it's really nice to be able to get your email on your phone, but there's all kinds of weird legal gray areas that come into play when you do so. Does the company have the right to wipe your phone remotely in the event that you lose it? Will the company pay you to replace the phone if some software they have you install on it somehow bricks the device? Will you be liable for violations of company privacy if your phone is lost with company email on it? (And the answer to that last one is a resounding yes, by the way.)

But if you're an IT department, you have to worry about all of these things and more, because now, technically, if you allow a BYOD ("Bring Your Own Device") policy, and you can't get on the device (such as is the case here), you're in for a world of PR nightmares at a minimum, and outright legal problems in the more common case. IT departments are, by definition, responsible for the data that resides on their systems, and if systems include the hardware on which the software resides, if it's not something the company owns or has access to, you are basically walking a very ugly line.

If you're a CTO or VP of IT, you need to be rethinking and re-examining your mobile policy, right now. If you want employees to be able to use mobile devices as part of their job, provide them (and have a very straightforward and strict policy regarding what software can be installed and what networks can be accessed in place). If you have employees using personal mobile devices as part of their day job, make sure you have a memo of understanding in place that lists the specific conditions you (the company) demand their agreement on in order to do so. If you don't have that memo of understanding, sock away a little extra from the profit line into a savings account marked "Legal Fund", because you'll need it someday, I guarantee it.

Apple's customer note covers more than just this one case

Apple notes (correctly) that

[Your] information needs to be protected from hackers and criminals who want to access it, steal it, and use it without our knowledge or permission.

This is, I think, without argument.

What they are not saying, and should, is that the government needs to be included in that collection of parties from which our information is protected. This is a fundamental right of any US citizen, and arguably anyone in the world. "Innocent until proven guilty" and all that.

Compromising the security of our personal information can ultimately put our personal safety at risk.

This is also absolutely true, for some more directly than others. Ask any of the battered women whose husbands are under court orders, ask anyone in Witness Protection, but more importantly, ask any woman who has ever had a stalker. But it doesn't even have to be that exotic---look at your browser history, the bank application on your phone, your email, your Facebook message history, and so on. There's so much information on these devices that losing one is not just a hassle, in some cases, it's downright life-paralyzing.

The same spirit that keeps the US government off of our phone lines (without a warrant), out of our personal mail (without a warrant) and out of our homes (without a warrant) needs to keep them out of our phones (without a warrant).

European and other visitors could have their privacy waived

Let's be clear: If the US wins this case, it means that any foreign national traveling in the US would be subject to the same kind of intrusion, regardless of what their government thinks and regardless of where their data is stored. And the various governments that govern those citizens would be really, really pissed off about that. This feels like an escalation of the Safe Harbor incident from last year, and I imagine that other governments would feel much the same---but this is so new that I doubt they've had time to formulate a response. Expect to see those statements and responses come in the months to come if this goes through.

Apple complying with this one case sets precedent

Suppose that, in the face of everything, Apple agrees to do this for this one case. Can you possibly imagine how many requests they will receive from every police department all over the country seeking their technical assistance in doing so for iPhones removed from arrested suspects for everything ranging from burglary to financial malfeasance? It would be a huge nightmare for them.

On top of that, you can well imagine that Google, Microsoft, Samsung and other manufacturers are watching this whole thing with very keen interest, because if the government can compel Apple to do this, it opens the doors to demanding it of any hardware manufacturer within the US. And none of them want any part of this any more than Apple does.

Apple complying with this one case hurts them

"Hi, I'm a Mac. (And I'm a PC.) Hey, PC, can you secure your data against government intrusion? (Yup.) Oh."

Android devices, particularly those manufactured overseas, would suddenly become a really hot commodity. Apple simply cannot be seen to back down on this, or they will get crucified in the extremely hot shooting war of words that they have with their competitors, Samsung in particular. This could cost them millions, if not billions, in sales. While the various "patriots" will claim that this is somehow just the price of freedom, I don't see the US government stepping up to provide adequate compensation in the event that this is the case.

Technical feasibility

Here's where we get technical. Bear in mind, I write iOS apps, and I teach writing iOS apps to a variety of people, but I've never worked for Apple, have no access to any insider information or source code, and have never actually tried to hack an iOS device, so I may be missing some details here.

As noted here, there's three things that stand in the way of the FBI (or any other attacker) bypassing the lock screen on an iPhone to obtain access:

iOS may completely wipe the user’s data after too many incorrect PINs entries
PINs must be entered by hand on the physical device, one at a time
iOS introduces a delay after every incorrect PIN entry

(By the way, the relevant iOS security documentation describes all of this in much greater detail. I will periodically refer back to this document as we go.)

Thus, according to the court order, the FBI requested Apple to:

... bypass or disable the auto-erase function whether or not it has been enabled;
... enable the FBI to submit passcodes to the SUBJECT DEVICE for testing electronically via the physical device port, Bluetooth, Wi-Fi, or other protocol available on the SUBJECT DEVICE; and
... ensure that when the FBI submits passcodes to the SUBJECT DEVICE, software running on the device will not purposefully introduce any additional delay between passcode attempts beyond what is incurred by Apple hardware.

Let's take these one at a time.

Bypassing the auto-erase function may or may not be feasible, depending on how it's implemented. If it's an API call that Apple invokes under certain conditions, then whatever this "bypass" is either has to prevent that condition ("n failed login attempts") from ever being triggered, or it has to somehow replace the code that does the erasing with something more benign. Depending on where and how the device OS code is stored, this can either be trivial or require a custom set of ROMs be plugged in to the device.

Tapping the PINs on the device screen is the pain, and the FBI sought a way around that by asking that the device accept passcodes via "device port, Bluetooth, Wi-Fi, or other protocol available". In other words, by any means supported by the device for input. Doing so through the device port requires the device be in the hands of the attacker/FBI, but Bluetooth and Wi-Fi are both wireless protocols---this could be done remotely from a distance.

"Not purposefully introduce any additional delay between passcode attempts" refers to the delay that iOS deliberately puts in, as described in the Security Guide (p12):

"To further discourage brute-force passcode attacks, there are escalating time delays after the entry of an invalid passcode at the Lock screen.

That same paragraph is where Apple describes the setting to wipe the phone in the event of too many failed login attempts:

If Settings > Touch ID & Passcode > Erase Data is turned on, the device will automatically wipe after 10 consecutive incorrect attempts to enter the passcode."

So again, this basically says that the cracking tool would have to somehow either prevent the conditions (tracking the number of failed logins) or replace the code that wipes the device.

Could this be done? Potentially, if the device were "flashed" with a new, hacked, operating system:

[Provide] the FBI with a signed iPhone Software file, recovery bundle, or other Software Image File (“SIF”) that can be loaded onto the SUBJECT DEVICE. The SIF will load and run from Random Access Memory (“RAM”) and will not modify the iOS on the actual phone, the user data partition or system partition on the device’s flash memory.

This is not rocket science. A SIF is what gets loaded onto your phone every time you do an update of the phone.

Now, if this were the end of the paragraph, Apple's concerns about this somehow becoming a "master key" for all iOS devices becomes a huge issue:

They have asked us to build a backdoor to the iPhone. ...

Specifically, the FBI wants us to make a new version of the iPhone operating system, circumventing several important security features, and install it on an iPhone recovered during the investigation. In the wrong hands, this software — which does not exist today — would have the potential to unlock any iPhone in someone’s physical possession.

The FBI may use different words to describe this tool, but make no mistake: Building a version of iOS that bypasses security in this way would undeniably create a backdoor. And while the government may argue that its use would be limited to this case, there is no way to guarantee such control.

The court order does try, at least on the surface, to make sure this doesn't become the "master key" for all devices that Apple warns against:

The SIF will be coded by Apple with a unique identifier of the phone so that the SIF would only load and execute on the SUBJECT DEVICE.

But Apple points out:

Some would argue that building a backdoor for just one iPhone is a simple, clean-cut solution. But it ignores both the basics of digital security and the significance of what the government is demanding in this case.

In today’s digital world, the “key” to an encrypted system is a piece of information that unlocks the data, and it is only as secure as the protections around it. Once the information is known, or a way to bypass the code is revealed, the encryption can be defeated by anyone with that knowledge.

The government suggests this tool could only be used once, on one phone. But that’s simply not true. Once created, the technique could be used over and over again, on any number of devices. In the physical world, it would be the equivalent of a master key, capable of opening hundreds of millions of locks — from restaurants and banks to stores and homes. No reasonable person would find that acceptable.

Here's what is under discussion, again from the Security Guide. First of all, as described on p6/7, each device has a "unique device ID" (ECID) and, in a particular chip on the device called the Secure Enclave, a "unique ID" (UID). These are then used as part of the encryption process to secure various things on the device, including files (which we'll talk about later). More importantly, these aren't values that can be picked up from casual use of the device (p10, my emphasis):

Every iOS device has a dedicated AES 256 crypto engine built into the DMA path between the flash storage and main system memory, making file encryption highly efficient.

The device’s unique ID (UID) and a device group ID (GID) are AES 256-bit keys fused (UID) or compiled (GID) into the application processor and Secure Enclave during manufacturing. No software or firmware can read them directly; they can see only the results of encryption or decryption operations performed by dedicated AES engines implemented in silicon using the UID or GID as a key. Additionally, the Secure Enclave’s UID and GID can only be used by the AES engine dedicated to the Secure Enclave. The UIDs are unique to each device and are not recorded by Apple or any of its suppliers. The GIDs are common to all processors in a class of devices (for example, all devices using the Apple A8 processor), and are used for non security-critical tasks such as when delivering system software during installation and restore. Integrating these keys into the silicon helps prevent them from being tampered with or bypassed, or accessed outside the AES engine. The UIDs and GIDs are also not available via JTAG or other debugging interfaces.

The UID allows data to be cryptographically tied to a particular device. For example, the key hierarchy protecting the file system includes the UID, so if the memory chips are physically moved from one device to another, the files are inaccessible. The UID is not related to any other identifier on the device.

In other words, the UID being used as keys into AES means that we are now talking about the strength of encryption algorithms somehow being broken without having the key. (For those of you who aren't crypto-savvy, this is basically the whole point of a crypto algorithm: to prevent enciphered text being reversed to plaintext without the use of the key used to encipher it.)

Note that the Security Guide points out that these keys are used as part of file encryption (p11):

In addition to the hardware encryption features built into iOS devices, Apple uses a technology called Data Protection to further protect data stored in flash memory on the device. Data Protection allows the device to respond to common events such as incoming phone calls, but also enables a high level of encryption for user data. Key system apps, such as Messages, Mail, Calendar, Contacts, Photos, and Health data values use Data Protection by default, and third-party apps installed on iOS 7 or later receive this protection automatically. Data Protection is implemented by constructing and managing a hierarchy of keys, and builds on the hardware encryption technologies built into each iOS device. Data Protection is controlled on a per-file basis by assigning each file to a class; accessibility is determined by whether the class keys have been unlocked.

And goes on (p12):

Every time a file on the data partition is created, Data Protection creates a new 256-bit key (the “per-file” key) and gives it to the hardware AES engine, which uses the key to encrypt the file as it is written to flash memory using AES CBC mode.

... The per-file key is wrapped with one of several class keys, depending on the circumstances under which the file should be accessible. Like all other wrappings, this is performed using NIST AES key wrapping, per RFC 3394. The wrapped per-file key is stored in the file’s metadata.

When a file is opened, its metadata is decrypted with the file system key, revealing the wrapped per-file key and a notation on which class protects it. The per-file key is unwrapped with the class key, then supplied to the hardware AES engine, which decrypts the file as it is read from flash memory. All wrapped file key handling occurs in the Secure Enclave; the file key is never directly exposed to the application processor.

... The content of a file is encrypted with a per-file key, which is wrapped with a class key and stored in a file’s metadata, which is in turn encrypted with the file system key. The class key is protected with the hardware UID and, for some classes, the user’s passcode.

In other words:

You have to have the per-file key in order to read the file. This is stored in metadata.
You have to have the file system key in order to read the metadata.
The file system key is derived from the UID and the user's passcode.

What this means, then, is that Apple, in order to comply with the FBI's order, must figure out a way to defeat all this key-based encryption. Since it lacks the keys in question, it would be discovering a way to bypass or defeat AES encryption. Doing that would essentially invaldiate AES for use across all applications, and considering that banks and other high-security folks use AES largely because it hasn't been broken, doing so (if it's even technically feasible, and right now most cryptanalysts don't believe it is, or we wouldn't be using it) would send a shock wave of enormous proportions across the entirety of the online world.

And no, that couldn't be limited to just one device.

Summary

Frankly, the idea that Apple could somehow be compelled to do this is a disturbing one to me. It raises all kinds of bad precedents for other companies, it weakens consumer privacy protection, and frankly it's not clear to me why the FBI wants or needs this information in the first place.

This story is not done. Not for a single moment. I will be watching this with some interest, and no small amount of concern.

Parse shutdown lessons

Sat, 30 Jan 2016 00:00:00 +0000

tl;dr Facebook/Parse announced that they are shutting down the popular Back-end-as-a-Service. While opinions are certainly going to vary as to why, I thought it an interesting situation to examine and, upon reflection, comment.

As I read the post, a couple of things immediately came to mind:

"That's a very restrained blog post." Seriously, it says more by what it doesn't say than what it actually does say:

"We have a difficult announcement to make. Beginning today we’re winding down the Parse service, and Parse will be fully retired after a year-long period ending on January 28, 2017. We’re proud that we’ve been able to help so many of you build great mobile apps, but we need to focus our resources elsewhere."

Quite honestly, this has all the sentiment and emotion of a "Dear John" letter. "We understand that this is difficult, but really, it's not you, it's me. Please try to move on with your life and in time, you will find that it's better this way for all of us." The only real explanation given is that "we need to focus our resources elsewhere."

Facebook is how big, and somehow they can't keep the Parse team focused on doing what they were doing? Sorry, no, I don't believe it. This isn't about needing to re-route critical resources (people? servers? office desks?) to other parts of the company, not by a long shot.

The Parse acquisition was always a weird one, to me. Quite frankly, I couldn't really figure out what Facebook wanted with Parse in the first place. Facebook is not really a developer-facing company, when you get right down to it. Yes, they're a software comapny, and yes, they've recently been tossing some interesting open-source projects out the door---but Facebook, unlike Microsoft, Amazon, or Google, doesn't really have a business model that incorporates developers at the core. They make money off of people who use Facebook, and buy things from ads displayed in Facebook, and towards that end Facebook certainly wants developers to build applications that make use of Facebook, but Parse really didn't do any of those things.

The Parse guys had to know this was a possibility. Frankly, it's always a possibility when there's a buyout---the acquiring company either changes their strategy, their opinion, or finally reveals that their main goal was to (a) remove you from the playing field, or (b) redirect your "assets" (which usually means the people working for you) to other parts of the company. Sure, it was all wonderfully believable that Facebook admired the plucky startup and wanted to see them continue to support other plucky startups. Also listed in the wonderfully believable category: Santa Claus, the Free-Market Fairy, and the idea that Bill Cosby is just misunderstood.

The companies that bet on Parse on their back end must have mixed feelings right now. To be sure, the Parse guys are bending over backwards to try and make this a smooth transition: not only are they providing a database migration tool, but they've also open-sourced a Parse Server implementation, which (on the surface, anyway) claims to be API-compatible with the original Parse Server. So, sure, there's a migration strategy in place, and they've given basically a years' notice to people, to give them plenty of time to bail out. But right now, every Parse naysayer is standing up in the conference room and shouting, "SEE?!? I TOLD YOU SO!!", even if their main objections to Parse were along a completely different axis. Meanwhile, every company that was considering any kind of cloud system is probably having something of a re-think, because if it can happen to Parse, why can't it happen to Azure, Google Cloud Platform, Bluemix, and so on?

(And for the record, it's a stupid comparison, but it does, I think, highlight the difference between betting on a company that provides the cloud hosting directly---a la the people who own the hardware and the dirt it rests on---and those companies that use said hosting to provide some kind of value-add on top of it, which of course includes all the various "aaS" members like Heroku, Mongolab, and a whole gaggle of others. Azure, AWS and Google Cloud are not going away; the massive investment in the physical infrastructure alone means that those companies are in it for the long haul. Anything else, though, I can't say with absolute certainty---"easy up" can just as quickly become "easy down" when the investment doesn't seem to be paying off the way upper management thought it should.)

The OSS "cloud standards" guys are going to become very annoying, very quickly. In the same vein as the Parse naysayer, every OSS project that tries to hide the underlying cloud platform differences away from the developer is going to be using this as the perfect reason/example for why you should work against their "cloud-standard API" and not trust the actual cloud provider itself. It's the Java "Write Once Run Anywhere" argument all over again, and frankly, I think it's a tired old retread that really just needs to die. Write Once, Cloud Anywhere is never going to yield the kind of benefits that its proponents hold up, if you ask me, in the same way that Write Once, Run Anywhere didn't---it won't be any one large reason that shuts it down, but a whole host of little pinpricks that ultimately just lead the developer to say, "Ah, screw it" and just work against their cloud provider of choice directly. Yes, in the event that a cloud provider does end up shutting down, a rewrite will be necessary---but the JDBC API argued the exact same thing against the case when you had to change databases mid-stream during development, and in fifteen years of talking to people about JDBC, not to mention all the JDBC-based projects I was part of personally, I never found one project where that actually happened.

My UW students are going to be very sad. Each quarter that I've taught mobile development at UW, I've had the students spend three or so weeks on a final project to show off their skills. Most of those projects have required some kind of back-end service for data storage and retrieval, and by far and away the most popular service of choice has been Parse. It'll be interesting to see what they choose to use once Parse finishes shutting down.

This is going to ripple through the BaaS market. There's a number of BaaS offerings available, and Parse shutting down means that (a) all those soon-to-be-ex-Parse customers are going to be looking for a new hosting home, (b) all current and future BaaS customers are going to be looking a LOT more carefully at the terms of service and, particularly, looking for language around what they can expect if the service shuts down, and (c) a new surge in "cloud vs private cloud" articles are about to hit the Internet.

I wonder how long before a Parse look-alike spins up. In many ways, Parse wasn't just about the API that they offered, but the fairly quick-and-easy ability to stand up a server/service and store data/user information/etc to it. Given that now the data storage component (MongoDB, for all practical intents and purposes) and the server API component (the aforementioned Parse Server) are now entirely in the open-source domain, how long will it take somebody with cloud space to spare to spin up a "Parse-like" service and offer it as an alternative to those customers who don't really want (or can't?) host the service themselves?

I wonder how long it will take Google, Amazon, Microsoft and any other cloud provider to jump all over this. Seriously, if I'm running the Microsoft Azure team, I'm telling somebody in my organization to start working on a "How to Transition from Parse to Azure Mobile Services" as of yesterday. Ditto for every other cloud provider on the planet. (And if it takes your team too long to put said guidance out the door, perhaps you need to rethink your mobile back-end offerings.)

I wonder how many Parse employees are going to survive the purge. If the Parse service as we've always known it is shutting down, then there's likely no longer a need for many of the people who worked at Parse.

I wonder how Google is going to reassure the Firebase team. Firebase is, without a doubt, one of the more interesting BaaS providers out there, and Google acquired them around the same time that Parse was acquired by Facebook. If I'm on the Firebase team, I'm probably a little nervous that somebody at Google is thinking the same thing (whatever that was) as the Facebook folks, and I'm wondering if my job is safe. (And just to be clear, I think shutting down Firebase would be a Really Bad Idea(tm), for a whole slew of different reasons.)

I kinda wish I'd gotten around to making an app with Parse before now. You know, just to be able to say, "I was there when Parse was still a thing." Because, you know, hipster.

I wonder how much money the Parse founders made. I imagine this isn't how they imagined their service coming to an end, but having millions in the bank probably helps offset the "sting" of having to write that "Dear John Customer" email above.

How do you learn?

Sun, 24 Jan 2016 00:00:00 +0000

tl;dr I've been asked a number of times over the years how, exactly, I approach learning new stuff, whether that be a new programming language, a new platform, whatever. This is obviously a highly personal (meaning specific to the individual offering the answer) subject, so my approach may or may not work for you; regardless, I'd suggest to anyone that they give it a shot and if it works, coolness.

The first thing to understand about how I approach learning new stuff is that I agree whole-heartedly with our former Secretary of Defense, Donald Rumsfeld, when he says (said) that there are "three kinds of knowledge":

That which you know you know. In other words, the stuff that you already know. You have empircal knowledge of the subject. I know C++: I know most of the keywords, I know pretty much all of the syntax, I understand the mechanics of how the language compiles and what it compiles to, and so on. Personally, I consider myself to "know" a subject if I've made use of it, in some fashion, for a moderately complex task. "Hello world" doesn't count (if we're speaking of a programming language), but doing one of the code katas can, if it's got some good "range" to it (doing some I/O, a little algorithmic exercise, and so on).
That which you know you do not know. I am aware of the subject, but I have not spent much, if any, time on it. For myself, however, the key distinction between this category and the following one is, "Do I understand what this subject's purpose is? Do I know it's "two-sentence story", and can I tell that story to somebody else?" (I'll talk more about the two-sentence story in a second.) This usually implies having done a small bit of reading on the subject, or doing that "hello world"/getting-started kind of tutorial. I know enough to have a good idea of where this thing fits in the world, but not enough to get started in it without having to do more reading and research.
That which you do not know. I have no awareness of this subject; I don't even know it exists. Obviously, this is hard for me to hold up examples, because if I knew of any, I'd be moving them into the second category already. One such possibility is Kali Linux, which (from the context of a few articles/books I've seen touch on the subject, as well as referencing the website to get its address just now) is apparently a Linux distribution tailor-made for doing security attacks and penetration testing. I have no idea what makes that distribution somehow better than any other, so right now I can't really formulate a two-sentence story around it other than what you just read. (And even then, one might argue that since I know of it, I am now aware that I don't know anything about it, and therefore it goes into category 2 as opposed to this one.)

Confused yet? It's not a hard breakdown, but it turns on the notion of knowledge vs ignorance---I can know a thing, or I can be aware of a thing and not know anything about it, or I can be ignorant of a thing. (And, of course, others will interpret the distinctions differently, and that's OK, too. This is just how I apply this to the topics that I need to know for work-related discussions/engagements/etc.)

So, assuming that somebody or something brings a new thing to my attention, here's (roughly) the process by which I go learn something new:

Formulate a two-sentence story on it

The term "two-sentence story" dates back to my time at DevelopMentor. When we were working out new class materials, the common refrain was, "What's the two-sentence story for this thing?" In other words, if you could only describe the thing in two sentences, what would they say? Originally, the TSS was geared more around the class itself---"What's the two-sentence story for this class?"---which is a little different than coming up with a TSS for a technology as a whole, but I found it a useful mechanism to help identify and isolate the aspects of a new thing that I care about for a given technology. Some examples:

AspectJ: Aspect-oriented programming is a flavor of "metaobject programming", a code mechanism that seeks to capture first-class constructs that "cut across" the traditional object-oriented hierarchy, around fields and/or methods, and apply those constructs reusably in multiple cases. AspectJ is a language-based implementation of AOP against the Java language running on the JVM.
Groovy: Ruby and Java meet up one night in a bar and have a night of passion. Groovy is that love child, exhibiting the linguistic tendencies of Ruby, but with the underlying structure and (more or less) common language heritage of Java.
Parrot: A register-oriented virtual machine that seeks to be for dynamic languages what the JVM or CLR were for statically-typed object languages (Java and C#). Tied closely at first to the next generation of the Perl language, and as a result more or less died with it.
MongoDB: A schemaless document-oriented database written to run without any platform dependencies (statically-linked executables). Mongo looks to provide some of the traditional server-based database experience that is familiar and comfortable to those who are used to traditional RDBMS environments, while also channeling the "free-form" experience that those from the NoSQL crowd are expecting, including the ability to "scale up" to large datasets without having to significantly reprogram the code that accesses it.
Boo: A Python-based language for the CLR that opens up the compiler pipeline for plugins to view and/or modify the AST of the code before execution. Interprets and/or compiles on the CLR.
CouchDB: A schemaless document-oriented database that embraces HTTP and REST to the n-th degree, simultaneously embracing the same sort of "distributed non-centralized" model that is favored by distributed version control systems. In other words, Couch doesn't have a single source of truth, but collectively determines what that truth might be through replication and optimistic updates.
Lua: A dynamically-typed tuple-centric scripting language designed to be easily embedded into native platform environments, most notably from C/C++ code. It is widely used in the gaming industry, and has a very well-designed FFI (foreign function interface) specifically to enable easy interoperability between the "high-level" Lua code and the native implementation underneath it.

As you can tell, not all of these are wrapped up in academic language, because in some cases, they are specifically tied to my own understanding of other things. For example, Groovy being "the love child of that one-night stand between Ruby and Java" meme, that tells me a whole lot of things---namely, that Groovy runs on the JVM, and can be seen as "just another Java", but has lots of genetic influence imparted upon it from Ruby. (Not surprising, considering that James Strachan, when he first started working on the language, wanted very specifically to bring over some of the things he saw in Ruby.)

The key here is that the two-sentence story is my own story; it's somewhat personal to me, because it keys off of what other things I know pretty well. I don't expect that my two-sentence story for any given subject, when viewed alone or outside of my head like this, will actually impart much of anything in terms of actual understanding to anybody else. It's purely for me.

But obviously, getting to the point of being able to formulate a TSS on it requires some research.

What is this thing like?

Like most people, I don't have a ton of time, so I need to come to grips with a thing fairly quickly. So when I'm first looking at a subject, the goal is to find something I already know to which it is similar and/or from which it descends or is based. Having made that initial connection, though, doesn't mean that I have enough knowledge on the thing yet---next, I have to start looking for ways that it's different from the anchor: how is "A" not like "B"? No two things "A" and "B" are ever perfectly alike, so where are the differences? How profound are those differences? And does that mean I need a new anchor upon which to rest my understanding of the new thing?

For example, when I was first learning Ruby, the obvious thing it was similar to was other object-oriented languages. It's tempting to see it thus as "another Java" or "another C#". However, Ruby is a dynamic O-O, whereas Java and C# are statically-typed O-O, so that connection is weak. Even weaker, C++ compiles to native code, whereas Java compiles to bytecode and runs on a virtual machine; Ruby does neither of these things. It is a straight interpreted language, so that's even less of a comparison. Yes, Java/C# and Ruby both have this notion of "objects" and "fields" and "methods", but that's like suggesting that a Ferrari and a Mack truck are similar, in that they both have "engines" and "wheels" and "drivers".

As I started hearing more about Ruby from the Rubyists around me, a more accurate connection started to form between Ruby and Smalltalk---both are object languages, but both are dynamically-typed, and both support some very similar kinds of "meta-programming", most notably the idea of "method_missing", where an attempt to invoke a method on an object that doesn't have said method won't result in a compile-time or runtime error, but instead potentially be handled by routing the request through a well-defined "trap" that can then examine the method call at runtime to determine what to do next. (This is how ActiveRecord works, by the way.)

Oh, and the syntax of Ruby was originally pretty Perl-based, but has since mellowed out significantly, to the point where most Rubyists of reputable fame avoid the Perl-isms, so that it reads a little more "English-like" now. (With all apologies to those who aren't native English speakers.)

Now, obviously, this isn't a 100% accurate summation of Ruby. JRuby, for example, is an implementation of Ruby that runs on top of the JVM, and as such, would sort of defy what I said earlier about it being interpreted. Fair enough. But at that level of nuance, I'm pretty clearly out of category 2 and into category 1.

What is this thing used for?

One of the best ways to begin to understand a thing is to see what it is used for. For most of us, Ruby came to prominence because of Rails. (Several people I knew who were talking a lot about Ruby and RoR at the time were fond of saying, "Ruby-on-Rails is a domain-specific language for producing web applications.") The approach was obvious: Ruby was about building web apps quickly. No wories about fine-tuning, no concerns over performance, the goal was to crank out a new web app quickly.

Rightly or wrongly, that paints Ruby as a higher-level tool, a paint spray-gun, if you will, that will blast out a ton of stuff quickly, without much concern for the degree of control at the edges. (And that's OK, by the way, even if it is perhaps a tad misguided in terms of the interpretation of Ruby itself.) That positioning deliberately put it opposite tools like Java and certainly way far away from C++.

But it also (again, perhaps wrongly) mischaracterized Ruby as "just" a web tool, and as a result Ruby missed out on the "cross-platform" nature that it represents, something which the JavaScript/Node community is trying very hard to cash in on.

Use it!

Assuming that I now have a better grip on the thing, the next step, to take it out of "known unknown" and bring it into the "known known" is to try and use it on something.

Contrary to the popular myth that "everybody learns differently", actually using a thing remains one of the best ways to actually grab hold of a subject and keep it. In this particular case, I don't mean just going through an existing tutorial and repeating the steps, but creating your own project and slogging your way through it from start to finish. Sometimes, it's helpful to do the same project over and over again, a la a Code Kata, but sometimes I want to try something new, partly because I want to explore an idea or a concept.

Case in point: Fifteen years ago, I wanted a blog, and at the time, all of the existing ones were just not what I was looking for. (I don't remember why---I may have just told myself that because I wanted to write my own, too. I just don't remember.) More to the point, I had this idea about self-modifying JSP pages: What would the practical implications be, from a coding perspective, if I had a blog implementation that, like a Wiki, would be modifying itself? In other words, creating a new blog post would be writing a new JSP page to disk, and subsequent requests to that page would be simply compiling-on-the-fly that JSP, as per any other JSP page? So I built that implementation (while on vacation with the family at DisneyWorld, actually), and cool! It worked. There were a few issues with it, though---for example, I had to get a little creative with servlet filters and use those as the "controllers" in a traditional MVC model---but that taught me that filters could be just as good, if not better, than servlets as the controller in an MVC model. (Which is in turn what made NodeJS "middleware" make so much sense to me, fifteen years later.)

Another example: A year or two ago, a buddy of mine came to me and wanted to know how hard it would be to build a web app that would allow volunteers to track their hours and make it easier to generate the email reports he had to send on. (He was a section leader, responsible for the pacific Northwest section of a volunteer organization, and the owning company wanted monthly emails about which people were doing what work at which locations. He was having to chase after people, then hand-assimilate their emails into a larger email and send that on to the company, which was costing him hours each month that he could be better using.) At the time, I was learning AngularJS, so I thought, "Sure, why not?" I built out a rough prototype for him, AngularJS on the front-end, but (in a daring architectural move!), I decided to try going "no-server", and instead use the REST API feature offered by MongoLab to a MongoDB instance in the cloud. It worked, but I fought with AngularJS all the damn time, and thank God he never really started using it in any kind of Production-like capacity, because that code was an absolute mess. It wasn't until the very end of working on it that I started to see "the Angular Way", and by then, I'd have had to rip it all up and start over. (I kept trying to use it as a kind of bastardized jQuery, and that didn't work out so well.)

These "forcing function" projects are generally projects that aren't anywhere close to production requirements, and usually focused around my own interest and/or issues. The recent "DevOpsing the Blog" projects are a good example of that: I had a problem, I decided to explore how to use TeamCity to fix it, and lo, the results. (That said, a commenter suggested what I think may be a much better solution to the automated posting to Twitter and LinkedIn, and I'm exploring that now, too. If you got to this post from my Facebook feed, then the Facebook integration---which I'm not sure if I'll keep, we'll see---works, and I just have to decide which approach I want to keep.)

"Forcing function" projects ideally will be projects I can throw away and walk away from, because sometimes, the attempt to use a particular tool or a particular approach will just fail spectacularly, and the resulting mess will be something nobody, particularly me, wants to support. So it's always better NOT to use a project for which I am getting paid as a forcing function project. (The sole caveat: If the client knows that this is a project that I am using as a "forcing function" project, and they verbally and contractually agree to pay me to explore it, that's a different story. Anything less is, in my mind, a violation of professional ethics.) But the project will hopefully be non-trivial (no CRUD apps), and should be something a bit larger than what I see in the tool's tutorial section.

And in some cases, it can be helpful to reuse a forcing function---for example, the blog. I might re-do that buddy's volunteer system in something like MeteorJS, just to see how well it would work and to compare the two approaches.

In any event, doing a forcing-function project like this is usually sufficient to take something from category 2 into category 1, and from there, I tend to leave it be.

Summary

There's probably a few other tricks that I'm forgetting about right this second (or may not even realize I'm doing, until I catch myself doing it), and if something comes up, I'll append/update the post. But for now, that's basically where it goes.

Functional Programming, for the Uninitiated (using Java)

Mon, 18 Jan 2016 00:00:00 +0000

tl;dr It's been a few years since I did this particular routine for the NFJS shows, but I found a sequence of demos/explanations that really demonstrated clearly why Java (and other classic O-O) developers should learn a little functional programming style, even if they never pick up an actual functional language. The keystone to that sequence of demos? "Collections are the gateway drug to functional programming."

I have concluded over the last decade or so that the best way to demonstrate a new concept is to "root" it (anchor it) cognitively against something developers already know how to do. With objects, that often entails "business objects" or "domain objects". I looked for ways to somehow relate functional programming to these business objects through collections (the Java Collections in particular, since Java was the platform of choice for the No Fluff Just Stuff conferences at which I honed a lot of this message, but the story works pretty equally well for .NET---which admittedly has LINQ but for a long time wasn't recognized as a functional language within it---or C++.)

The Domain Type

Imagine we have a collection of Person types:

public class Person {
  public Person(String fn, String ln, int a) {
    this.firstName = fn; this.lastName = ln; this.age = a;
  }
  
  public String getFirstName() { return firstName; }
  public String getLastName() { return lastName; }
  public int getAge() { return age; }
  
  public String toString() {
    return "[Person: " +
      "firstName:" + firstName + " " + 
      "lastName:" + lastName + " " +
      "age:" + age + "]";
  }
  
  private String firstName;
  private String lastName;
  private int age;
}

Yawn. Pretty boring stuff. Note that my Person type has no mutators/setters; this is both because this demo simply won't need them, and because in general it's better to try to work with immutable types anyway.

Groups of Persons

It's not uncommon that we will have a group of Persons with which we want to do various domain-ish kinds of things. The typical way to do this is with an array:

Person[] attendees = new Person[] {
  new Person("Ted", "Neward", 43),
  new Person("Charlotte", "Neward", 39), // and she will ALWAYS be 39
  new Person("Michael", "Neward", 22),
  new Person("Matthew", "Neward", 16),
  new Person("Carter", "Wickstrom", 45),
  new Person("David", "Dennison", 41),
  new Person("Kristy", "Overton", 20),
  new Person("Hilary", "Clinton", 79),
  new Person("Bernie", "Sanders", 129)
};

A motley crew if ever there was one.

Frequently, we find that we need to do various things with a subset of this crowd; for example, it's often happy hour, and we want to give everybody a nice cold one to top off the day:

for (Person p : attendees)
  System.out.println("Have a beer, " + p.getFirstName());

Spot anything particularly wrong with this code?

Of course you do---you're like my wife, who has this crazy notion that my 16-year-old shouldn't actully be drinking. (Actually, neither should Kristy, for that matter.) So, when this bug is pointed out, the natural thing to do is put a guard clause of some form in the "for" loop:

for (Person p : attendees)
  if (p.getAge() >= 21)
    System.out.println("Have a beer, " + p.getFirstName());

And, we're done, right?

Not really; the next time we want to do something across the entire group, we need to write almost exactly the same code again:

for (Person p : attendees)
  if (p.getAge() >= 65)
    System.out.println("Here's your retirement check, " + p.getFirstName());

or:

for (Person p : attendees)
  if (p.getAge() >= 16)
    System.out.println("Here's your driver's license, " + p.getFirstName());

or even:

for (Person p : attendees)
  if (p.getAge() >= 18 && p.getAge() <= 65)
    System.out.println("Have you registered for the draft, " + p.getFirstName() + "?");

When viewed this way, it's easy to see that for loops violate the DRY principle.

An aside: arrays suck

By the way, arrays in Java suck. They're not quite as bad in .NET, but they really REALLY suck in C++. Basically, the closer your language is to thinking of an array as just a pointer with some sexy pointer math on top of it, the more they suck. It's far better to use the collection classes that are (now) standard with every language.

In Java, that usually means using List and ArrayList, and to a lot of developers, that means initializing that set of Persons to start with gets really cumbersome:

List<Person> attendees = new ArrayList<Person>();
attendees.add(new Person("Ted", "Neward", 43));
attendees.add(new Person("Charlotte", "Neward", 39));
attendees.add(new Person("Michael", "Neward", 22));
attendees.add(new Person("Matthew", "Neward", 16));
attendees.add(new Person("Carter", "Wickstrom", 45));
attendees.add(new Person("David", "Dennison", 41));
attendees.add(new Person("Kristy", "Overton", 20));
attendees.add(new Person("Hilary", "Clinton", 79));
attendees.add(new Person("Bernie", "Sanders", 129));

But as it turns out, the Java Collections library has a better approach to this; they refer to them as algorithms, but in essence, these are black-box methods that do some pretty simple things:

List<Person> attendees = Arrays.asList(
  new Person("Ted", "Neward", 43),
  new Person("Charlotte", "Neward", 39), // and she will ALWAYS be 39
  new Person("Michael", "Neward", 22),
  new Person("Matthew", "Neward", 16),
  new Person("Carter", "Wickstrom", 45),
  new Person("David", "Dennison", 41),
  new Person("Kristy", "Overton", 20),
  new Person("Hilary", "Clinton", 79),
  new Person("Bernie", "Sanders", 129)
);

The Arrays class is part of java.util, and it's a set of static methods that are "just used". No class initialization, no objects, just a good-ol' method call. It's like 1985 called, screamed "Happy Birthday!" into the phone, and left a brand-new laser disc on our doorstep along with the receipt from Blockbuster.

Still, it works, and there's a lot to be said for the simplicity and the ease by which we convert from the array form to the Collection form using asList up there. But while we're on the subject, let's not call these "I don't really do anything on an object" things "methods"; let's get used to the idea of calling them "functions", since they modify no object state, don't live on an instance, and really only use the "class" around them as a lexical scoping mechanism (and because Java has this really annoying rule about everything having to be part of a class because, you know, "globals are bad". Unless they're called "singletons", of course.)

Reusing code

Stepping back, look at the various loops above; it's fairly easy to generalize what we're doing; in each case, we're testing to see if somebody meets an age criteria, and if so, taking some kind of action. We like reusable code, so let's do that:

class Utils {
  public static void happyHour(List<Person> persons) {
    for (Person p : persons)
      if (p.getAge() >= 21)
        System.out.println("Have a beer, " + p.getFirstName());
  }
  public static void retirement(List<Person> persons) {
    for (Person p : persons)
      if (p.getAge() >= 65)
        System.out.println("Here's your check, " + p.getFirstName());
  }
}

We can actually do better than this pretty quickly, or so we think:

class Utils {
  public static testGTAndPrint(List<Person> persons, int age, String message) {
    for (Person p : persons)
      if (p.getAge() > age)
        System.out.println(message + p.getFirstName())
  }
  public static void happyHour(List<Person> persons) { 
    testGTAndPrint(21, "Have a beer, ");
  }
  public static void retirement(List<Person> persons) {
    testGTAndPrint(65, "Here's your check, ");
  }
}

But the flaws come out pretty quickly---we're not always going to want to test if the age is greater than the passed value (sometimes it will want to be less-than, like when seeing if anybody qualifies for the kids' menu), and sometimes we will want to do something other than print a message to the console.

The larger pattern here is obvious, though: test, then act. How do we generalize this?

Strategy pattern, of course!

Everybody knows Strategy

Java has long had a rich history with patterns, particularly those defined in the "Gang of Four" book, and Strategy is one of the ones that has been with us from the beginning. Most of the time we're familiar with it from wanting to compare two objects to see if one is "greater" or "lesser" than the other, using the Comparable interface: one simply implements it on the class that we want to compare, and provide an implementation of it.

Unless, of course, we want to sort by a different criteria. Then things get a little tricky.

See, part of the magic of Strategy is that its implementation is supposed to be replaceable at runtime for another implementation, and we can't do that if we inherit the interface and define the implementaiton as a method. This is not an unreasonable thing to want for comparisons; sometimes we want to arrange people by last name, sometimes by first name, sometimes by age, sometimes by height, and so on. We want to change the implementation used to do the comparison at runtime, so we use a Comparator; for a Person, that often looks something like:

class AgeComparator<Person> implements Comparator<Person> {
  // return negative for "less than", 0 for "equal", or 
  // positive for "greater than"; easiest way to do that is
  // to just subtract one from the other
  //
  public int compare(Person lhs, Person rhs) {
    return lhs.getAge() - rhs.getAge();
  }
}

Then, when sorting (perhaps using a SortedSet or the Collections.max or Collections.min methods), we simply instantiate and pass in the Comparator of choice.

By the way, though, I never liked seeing these guys float around as top-level classes; the AgeComparator there is specific to Person, so it really should be captured inside the class, if you ask me:

class Person {
  public class AgeComparator<Person> implements Comparator<Person> {
    // return negative for "less than", 0 for "equal", or 
    // positive for "greater than"; easiest way to do that is
    // to just subtract one from the other
    //
    public int compare(Person lhs, Person rhs) {
      return lhs.getAge() - rhs.getAge();
    }
  }
}

And now that you mention it, we really don't ever need more than one of them at a time, so we can just instantiate one, make it static (and make it final while we're at it, since we probably don't want anybody outside of Person to change what it means to compare two Persons by age):

class Person {
  public static final Comparator<Person> BY_AGE = new Comparator<Person>() {
    public int compare(Person lhs, Person rhs) {
      // Notice that now, since we're inside the class, we can go directly at the
      // fields, which may be faster, depending on how complicatd the internal state
      // of the class is
      //
      return lhs.age - rhs.age;
    }
  };
}

All boned up on Strategy? Cool.

By the way, this is where things are going to take a bit of a cognitive leap, so if you need to, you know, grab a coffee or go to the bathroom or something, now's a good time to do it. Fortunately, this isn't a conference, so I'll just stop here and wait for you until you get back. Totally cool---I have a few emails to do anyway.

(Hums and fires up email client)

Ready? Let's keep going.

Strategy to the reusable rescue

We have two Strategies at work here, actually: one to do the test, and the other to do the action, so let's create two interfaces, one for each:

interface Test {
  public boolean test(Person p);
}

interface Act {
  public void do(Person p);
}

Now we can rewrite Utils a little bit more... reusably.

class Utils {
  public static testAndAct(List<Person> persons, Test tester, Act actor) {
    for (Person p : persons)
      if (tester.test(p))
        actor.do(p);
  }
  
  public static class DrinkableAge : Test {
    public boolean test(Person p) { return p.getAge() > 21; }
  }
  public static class RetiredAge : Test {
    public boolean test(Person p) { return p.getAge() > 65; }
  }
  
  public static class HaveABeer : Actor {
    public void do(Person p) { System.out.println("Have a beer, " + p.getFirstName()); }
  }
  public static class HeresYourCheck : Actor {
    public void do(Person p) { System.out.println("Here's your check, " + p.getFirstName()); }
  }
  
  public static void happyHour(List<Person> persons) {
    testAndAct(persons, new DrinkableAge(), new HaveABeer());
  }
  public static void retirement(List<Person> persons) {
    testAndAct(persons, new RetiredAge(), new HeresYourCheck());
  }
}

You know what, though? Every time we call either of those methods, we're creating new objects, when they don't really need to be new; we can use the same one, over and over again (even across multiple threads, since they don't hold any state). Let's make them Singletons:

class Utils {
  public static testAndAct(List<Person> persons, Test tester, Act actor) {
    for (Person p : persons)
      if (tester.test(p))
        actor.do(p);
  }
  
  public static final Test DRINKER = new Test() {
    public boolean test(Person p) { return p.getAge() > 21; }
  };
  public static final Test RETIRED = new Test() {
    public boolean test(Person p) { return p.getAge() > 65; }
  };
}

... and in fact, we can generalize the "printers" a little more by getting a little creative:

class Utils {
  public static testAndAct(List<Person> persons, Test tester, Act actor) {
    for (Person p : persons)
      if (tester.test(p))
        actor.do(p);
  }
  
  public static Actor createPrinterActor(String message) {
    return new Actor() {
      public void do(Person p) {
        System.out.println(message + p.getFirstName());
      }
    };
  }
  
  public static final Actor HAVE_A_BEER = createPrinterActor("Have a beer, ");
  public static final Actor HERES_CHECK = createPrinterActor("Here's your check, ");
}

... which then makes the other methods even more interesting:

class Utils {
  public static testAndAct(List<Person> persons, Test tester, Act actor) {
    for (Person p : persons)
      if (tester.test(p))
        actor.do(p);
  }
  
  public static void happyHour(List<Person> persons) {
    testAndAct(persons, DRINKER, HAVE_A_BEER);
  }
  public static void retirement(List<Person> persons) {
    testAndAct(persons, RETIRED, HERES_CHECK);
  }
}

Interesting, wot?

Utilicizing Utils

Have a look back at the testAndAct method again; is there anything really all that specific to the Person type inside of it? Not a lick, which implies that it can actually be genericized entirely away from Person itself:

class Utils {
  // will not compile yet
  public static <T> testAndAct(List<T> items, Test tester, Act actor) {
    for (T t : items)
      if (tester.test(t))
        actor.do(t);
  }
}

... but in order for that to work, Test and Act have to be made generics, as well. (Note that the method is generic, not the entire Utils class, which necessitates the "generic type argument" at the front of the method signature.)

Now, it's time for a bit of an admission: technically, testAndAct is actually two methods in one. One is a method designed to filter out items that don't fit a particular criteria, and the other is to perform a specific action against each element in the collection. So let's rename the interfaces accordigly, and break the two methods out:

interface FilterFn<T> {
  public boolean apply(T it);
}
interface ActionFn<T> {
  public void apply(T it);
}

class Utils {
  public static <T> List<T> filter(List<T> src, FilterFn<T> criteria) {
    List<T> results = new ArrayList<T>();
    for (T it : src)
      if (criteria.apply(it))
        results.add(it);
    return results;
  }
  public static <T> void map(List<T> src, ActionFn<T> action) {
    for (T it : src)
      action.apply(it);
  }
}

I've made a little terminology switch here, too. Each of the methods inside the interface is called "apply", for historical reasons. (That's what functional guys call doing something with a method on an object, "applying" it.) And "test" became "filter", and "action" became "map". (That last one deserves explanation---the functional crowd says that when you apply a function across each element of a collection, you're "mapping" the function to each element. Why that particular verb, though, I honestly don't know.)

(And yes, all you classic functional programmers, I know that you may disagree with my use of the terminology; hold your criticism for now, please. I've heard "map" used several different ways in several different languages, and I don't want to get into the nuances right now.)

And while we could go back and rewrite the happyHour and retirement methods on Utils, it actually is becoming more clear that Utils is about a lot more than just people. As a matter of fact, it's arguable that those are actually things that should be defined on the Person class itself, since they relate more to Person-hood than anything else:

public class Person {
  public static final FilterFn<Person> DRINKER = new FilterFn<Person>() {
    public boolean apply(Person it) { return it.getAge() >= 21; }
  };
  public static final FilterFn<Person> RETIRED = new FilterFn<Person>() {
    public boolean apply(Person it) { return it.getAge() >= 65; }
  };
}

And now we can go back to just using them inline, right?:

Utils.map(
  Utils.filter(attendees, Person.DRINKER),
  new ActionFn<Person>() {
    public void apply(Person it) {
      System.out.println("Have a beer, " + it.getFirstName());
    }
  });

... because this is so much clearer!

Sanity, O Sanity, wherefore art thou, Sanity?

It's usually at about this point in the discussion that even the most open-minded soul starts looking at me a little quizzically, because this really doesn't seem to have a whole lot of upside to it---it's longer, it's much more indirect than a traditional iteration loop, and frankly, using this inline just feels hideous. Part of this is the awkward pre-Java8 syntax. But part of it is that this just doesn't seem to have much advantage to it.

Until you get to writing tests.

Here's the challenge: how many tests do you need to write each time you write a for loop in your code? Typically, there's going to need to be a test to exercise every possible combination through that loop, for each for loop in the code. And let's talk about how many different permutations of collections you need to construct in order to test all of those permutations. This is going to spiral out of control pretty quickly.

But in my new version, how many tests do you need to write? I count:

Test that Utils.filter() works
Test that Utils.map() works
Test that DRINKERS works
Test that RETIRED works
Test that the anonymous ActionFn works

And, really, look back at DRINKERS and RETIRED; do we really need a test to determine that a simple getter works, or that Java's math works? I'd argue that we can strike those out completely. And as for the ActionFn, when's the last time that System.out.println failed you? Or String concatenation?

So we're down to two methods to test. (And I would argue that they're also so ridiculously simple, they don't need testing, either, but maybe that's pushing it for you, I dunno.) But here's a nice parting gift for our contestants today: Once those two methods are tested to your satisfaction, they're tested for everything that might use them---whether it's a Person being passed in, or a Pet or a CheckingAccount doesn't matter.

Hunh. That's not a bad tradeoff, actually.

And if we use the static import to pull in the members on Utils (a la import static Utils.*;), then map and filter become top-level symbols, so we can leave off the "Utils." prefix. And heck, if you rename "map" to "for" (oh, if only we could, curse you keywords!), you end up with something like this:

map(
  filter(attendees, Person.DRINKER),
  new ActionFn<Person>() {
    public void apply(Person it) {
      System.out.println("Have a beer, " + it.getFirstName());
    }
  });

Now, all it really takes is to create one of those "printerActor" objects somewhere as a static, and this can simplify down to:

map(
  filter(attendees, Person.DRINKER),
  HAVE_A_BEER
);

It sure as hell doesn't look like Java much anymore, but is there any question left in your mind as to what is going on here? It almost reads like pseudocode!

Now do this a million times...

But here's the kicker: What if there aren't just ten attendees, but ten million? The filter and map implementations in Utils are totally inappropriate for that; they should be parallelized, probably using the ForkJoin framework from inside Java7, or at least breaking up the work into smaller chunks and handing each chunk out to a separate thread for processing, gathering those results up into a final collection, and and and

And how often do you want to write that?

In the earlier "for" loop case, you'd have to write that every single time you wanted to do anything with this 10-million-item collection. Every. Single. Time.

In my Utils-based version, though, the FilterFn and ActionFn implementations remain exactly as they are; we just need a new version of Utils that does all that parallelization stuff, and now the code can take advantage of parallelization by going from this:

map(
  filter(attendees, Person.DRINKER),
  HAVE_A_BEER
);

to this:

map(
  filter(attendees, Person.DRINKER),
  HAVE_A_BEER
);

See the difference?

Actually, I lied; we actually need to do it like this:

import static Utils.*;

// ...

map(
  filter(attendees, Person.DRINKER),
  HAVE_A_BEER
);

import static ParallelUtils.*;

// ...

map(
  filter(attendees, Person.DRINKER),
  HAVE_A_BEER
);

That's not bad. Change one lie of code, and voila! You have a thread pooled set of operations. Twice. Without changing an actual implementation line of code.

But wait---act now, and we'll throw in a set of Ginsu knives as well. Or, maybe not. But something just as cool, I promise.

Folding universes

The functional world is not done with us yet. It's got another surprise or two up its sleeve.

First of all, there's two more generic "function" types that they like to define; one is a "transformation" function that takes an A and turns it into a B, like so:

interface TransformerFn<R,T>
{
    public R apply(T it);
}

class Utils {
  public static <R,T> List<R> xform(List<T> src, TransformerFn<R,T> transformer)
  {
    List<R> results = new ArrayList<R>();
    for (T it : src)
      results.add(transformer.apply(it));
    return results;
  }
}

where "R" means "result" and "T" is the traditional input type.

Using this one is pretty easy to understand---the Person has several of these kinds of "transformation" methods already defined, in the sense that you can think about getFirstName as a method that "transforms" a Person into a String that happens to hold that person's first name as its value. So it's possible to take our list of attendees, and "transform" them into a list of last names (perhaps for sorting reasons), by using a particular Utils method that basically looks like all of the others except for the type signature:

List<String> lastNames = 
  Utils.xform(attendees, 
    new TransformerFn<String, Person>() {
      public String apply(Person it) { return p.getLastName(); }
    });

The other is an "accumulation" function, where we want to accumulate a value while doing something to each element of a collection. Doing this is called "folding", because it takes a collection of stuff and effectively "folds" it into one value, one element at a time. (These are sometimes called "folding left" or "folding right", depending on whether it starts at the beginning or the end of the collection.) Like so:

interface FoldFn<R,T>   // F2<R,T,R>
{
    public R apply(R accum, T it);
}

class Utils {
  public static <R,T> R foldLeft(R seed, List<T> src, FoldFn<R,T> folder)
  {
    R accum = seed;
    for (T it : src)
      accum = folder.apply(accum, it);
    return accum;
  }
}

The easist example to see here is to "fold" a series of numbers together; perhaps we want to find the average age of all the attendees, so we sum them all up, and then divide by the number of attendees:

// In other words, instead of doing this:
//
int accum = 0;
for (Person it : attendees)
  accum = accum + it.getAge();
System.out.println("total age = " + accum);
System.out.println("average age = " + (accum / attendees.size());

// Now we would do this:
//
Integer totalAge = Utils.foldLeft(0, people, new FoldFn<Integer,Person>() {
  public Integer apply(Integer accum, Person it) {
    return accum + it.getAge();
  }
});
System.out.println("total age = " + totalAge);
System.out.println("average age = " + (totalAge / attendees.size());

Seems kinda pointless, ya?

The XML factor

Oh, by the way, the customer called; they want to see all of those attendees in a SOAP call by Monday. Time to break out your JAXB, or other Reflection-based Java-to-XML library of choice, right?

While you do that, I'm just going to transform this entire collection of Persons into XML with only the use of these two functions:

String xml = Utils.foldLeft("<people>", people, new FoldFn<String,Person>() {
  public String apply(String accum, Person it) {
    return accum + "<person>" + it.getFirstName() + "</person>";
  }
}) + "</people>";
System.out.println(xml);

And you know what's even more interesting? Doing it this way, I can actually change the format of how the Person renders into XML without having to chagne a thing inside the Person.

Oh, and did I mention that this is---still---all ridiculously easy to test? Because all of this code is, when you look at it, made up of entirely idiotically-simple primitives that really require no testing whatsoever. Point to one aspect of any of these functions that isn't drop-dead, absolutely-the-most-primitive-thing-the-language-can-do kinds of operations. For crying out loud, the FoldFn is just doing String concatenation and a simple getter call; what the hell is there to test?!?

Intrigued yet?

Wrapping up

Don't get me wrong; there's a lot more to functional programming than just what I've demonstrated here, and Java8's new labmda support (which, by the way, I'm going to leave to you to use to refactor this code to Java8 syntax, because it's a good exercise and you need the practice anyway) most certainly does not do all the things that one would expect a modern "functional" language to do. (Talk to me when you know what "currying" is.)

Oh, and now, is it clear why I think collections are the gateway drug to functional programming?

Technical Debt: A Definition

Mon, 18 Jan 2016 00:00:00 +0000

tl;dr A recent post on medium.com addresses the topic of technical debt; I had an intuitive disagreement with the thrust of the post, and wrote this as a way of clarifying my own thoughts on the matter. It raises some interesting questions about what technical debt actually is---and if we can't define it, how can we possibly understand how to avoid it or remove it, as opposed to our current practice of using it as a "get-out-of-this-codebase-by-blowing-it-all-up" card?

@kellan wrote:

Technical debt exists. But it’s relatively rare. When you start arguing with someone about technical debt, you’ll generally encounter a definition like: Technical debt is the choices we made in our code, intentionally, to speed up development today, knowing we’d have to change them later. Hard coding a variable because currently there is no plan to change it is a common example of technical debt. Similarly not modularizing a function.

This is a fairly clear, succinct, and easy to reason about definition, that describes a phenomena that exists relatively rarely. Relatively rare compared to what? Compared to the amount of technical debt we ascribe to the codebases we work on. How then do we explain the overwhelming prevalence of technical debt we encounter when we talk to people about code?

The term is being abused, or at least dangerously overloaded.

He's certainly right about that! But just prior to that, he says, "Technical debt doesn't exist", and sort of wanders around that idea for a bit.

Here's the rub: He then tries to define what technical debt actually is:

"Maintenance work."
"Features of the codebase that resist change."
"Operability choices that resist change."
"Code choices that suck the will to live."
"Dependencies that resist upgrading."

I'll leave you to read his descriptions of each.

Critique

Unfortunately, a lot of the definitions he raises there are highly subjective and extremely difficult to understand, except at a base, emotional, almost visceral level. I mean, when you explicitly use the phrase "suck the will to live" as one of your definitions, it's hard to really put a concrete discussion around that.

Consider, for example, that particular point: "A significant percentage of what gets referred to as technical debt are the decisions that don’t so much discourage change but rather discourage us from even wanting to look at the code. ... We often describe this code with the suck-the-will-to-live quality as messy (spaghetti), unmaintainable, or amateurish. Often what we’re describing under the surface is fear and confusion. We don’t understand the code, and when we don’t understand things, as human, we tend to get scared, tired, and angry."

I'm sure every single person reading this has an immediate reaction, akin to the screams through the Force that Obi-Wan felt when Alderaan was destroyed. Everybody remembers That One Project, or That One Class, or That One File.... Nobody wanted to touch it, it was a mess, and people would look for every reason under the sun to avoid opening it, as if there was some kind of icky black ichor that could ooze out of the screen and keyboard and infect us with its ugliness.

And yet, if we compare the stories, we will all have very different concrete-terms descriptions of what that thing was. And I'll even bet that if you cast the net wide enough, and we spend enough time comparing stories, we'll even find that one man's "suck the will to live" is another man's "Whoa, man, that's actually kind of a cool hack."

Case in point: in the earliest days of my career, I was a contractor working on some C/Win16 code at Intuit. A really cool 3-month gig (and in those days, it was way cool to have Intuit on your resume). I was working as part of the "Slickrock" team, which was the code-name for Intuit's nascent electronic banking section of Quicken 5 for Windows. It was some cool stuff.

Except....

Well, first of all, everything was written in C. Not C++, as was the leading-edge of the day, but using Intuit's home-grown C/Windows library that they'd put together since the earliest days of the product. At the time, I was kinda bleah on the whole idea. (In retrospect, hey, if it still works, you know?)

And there was this one dialog box to which I was assigned, which had a bunch of bugs in it that needed fixing, that nobody else on the team wanted to touch. Eager to prove to all these grizzled veterans that I was capable of handling the toughest stuff, I leapt at the chance to get into this thing. (If you get this picture of the eager young Private fresh from boot camp, volunteering to go out on that mission that the grizzled old Sargeant knows will just crush the life out of him, you're probably not too far off the mark.)

And here's what I found: this dialog box code was one, giant, four-page-long function, where three-and-three-quarters of it was wrapped in one giant-ass do-while loop. But not just any do-while loop; no, this one was the most bizarre thing I'd ever seen. It looked something like this:

do {
  /* do one thing */
  
  /* do another thing */
  
  /* check that thing */
  
  /* what about the thing over there */
} while (0);

It was my own private "WTF?!?" moment. No wonder everybody wanted to stay clear of this thing! This was the craziest code I'd ever seen, and clearly it was because they weren't using C++!

(Yeah, I kinda was that stupid back then.)

But when I showed this to one of the other engineers and said the 90's equivalent of "Dude, seriously?", he pointed out that I'd missed an important part of the whole thing:

int result = -1; /* Not OK! */
do {
  /* do one thing */
  if (!thing-worked)
    break;
  
  /* do another thing */
  if (!another-thing-worked)
    break;
  
  /* check that thing */
  if (!thing-checked)
    break;
  
  /* what about the thing over there */
  if (!thing-over-there-checked)
    break;
    
  result = 0; /* OK! */
} while (0);
return result;

In other words, this incredibly idiotic thing actually served a useful purpose: it obeyed the old C rule of "single entry, single exit", and more importantly, it was rather elegantly obeying the fail-fast principle. (Why bother doing all these other checks if you've already failed at the first step?)

Now, I grant you, this could've been solved using C++ using exceptions; instead of the (not-really-a-)loop, he could just have done a "try", and then each step could've thrown their own new exception type, and there'd have been either a single "catch" to return the appropriate error code (since this block was returning either -1 or 0, depending on success), or even maybe a separate "catch" block to handle each different error condition, and---

But you know what? Today, looking back at it, I don't know if that would've been much clearer, or much shorter, or what-have-you.

Is this still life-sucking-code? Or is this an elegant hack? I'll be honest, I'm not sure anymore, of either position.

Technical Debt: A Definition

I don't have one.

Seriously.

Not one I particularly like, anyway. Google it, and you get:

Technical debt (also known as design debt or code debt) is a metaphor referring to the eventual consequences of any system design, software architecture or software development within a codebase.

"eventual consequences"? You mean, like "it works"? Seriously, consequences are not always bad, which is why the Gang-of-Four used that same word to describe the results of a particular solution applied to a particular problem within a certain context. Consequences can be positive, and they can also be negative. The use of the Strategy pattern can allow for varying an algorithm at runtime---but with it comes an added cost in complexity of determining which Strategy to load, for example, or the additional cognitive load of having to realize that now the Strategy being executed may be nowhere local to the code actually executing it (which would at some level seem to violate the principle of locality, depending on the situation).

Wikipedia goes on to say:

The debt can be thought of as work that needs to be done before a particular job can be considered complete or proper.

Now that's interesting, because that certainly doesn't jibe with what @kellan was alluding to earlier---this sounds like things like documentation and tests and such. And yes, that definitely could create a problem, if a company/team/programmer goes off and writes a whole bunch of untested, undocumented code; I'd call that indebted code, probably, sure.

Unless, you know, it doesn't really need documentation. Or tests. Like, for example, a module composed of much smaller functions, each of which are effectively small primitives that really don't need testing, a la:

def calcuate(lhn, op, rhn)
  return op(lhn, rhg)
end

def add(lhn, rhn)
  return lhn + rhn
end

def sub(lhn, rhn)
  return lhn - rhn
end

puts calculate(1, add, 2)

Do these really require comments? Tests? Wouldn't it actually add to the technical debt to put those into place, since now they must be maintained and kept up to date should something change in here?

I'm obviously reaching here, but I don't think the point is entirely invalidated by the simplicity of my example---after all, well-written methods are supposed to be small and focused, and we prefer classes not to be large, and so on, for precisely these kinds of reasons.

Technical Debt: It's a metaphor, stupid

Go back to Wikipedia for a second; there, they finish the definition's first paragraph with this:

If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on. Unaddressed technical debt increases software entropy.

See, this is the heart of the matter: technical debt is a metaphor. That's it. That's all it is. It's a literary mechanism designed to help people who are not programmers understand that there are decisions made during the development process of a project, decisions which are deliberate choices to take a shortcut or avoid a more generic solution in the interests of getting past the obstacle quickly.

Except that nothing ever remains "just" a metaphor in our industry. Inevitably, we have to dissect it, treat as if it were a real, in-the-room-with-us kind of thing, and start crusades "for" or "against" it. Because REASONS.

And I admit, I'm not immune to this tendency myself. Because in examining a metaphor, so long as we recognize it's a metaphor and therefore bound to fail at some point (the model is not reality), we can actually find some interesting edge-cases that may or may not apply, and that leads us to some interesting conversations about the concept, even if it doesn't fit the metaphor anymore.

Technical Debt: A Fowlerian Definition

Martin Fowler has gone into great detail about the different kinds of technical debt in the form of a debt quadrant, arranged along two axes of "Deliberate vs Inadvertent" against "Reckless vs Prudent".

I like this, simply by virtue of the fact that it captures the mindset of the developer or the team at the time they were making that decision.

But I don't like it because, well, because who cares what they were thinking, or why? Isn't technical debt just technical debt? I mean, that $50 purchase on your credit card, was it a measured and thoughtful purchase, perhaps some tools at the local hardware store, so that you can perform some home repais, or was it a reckless and idiotic purchase, perhaps some tools at the local hardware store, so that you can pretend to yourself that you're actually going to take this weekend and perform some home repairs, but deep down you know you're just fooling yourself, you'll never get it done, and the tools will now be left to rust in a quiet corner of the garage (or worse, you'll leave them out in the backyard and it rains and and and)?

Seriously: the guy who wrote that do-while loop at Intuit? I have no idea what he was thinking---and did that intent really make a helluvalot of difference to me (or the rest of the team) as I (we) tried to pick it up and extend/modify/debug it? I won't speak for the rest of the team, but to me, it made not a single whit of difference.

But here's the vastly more important thing to realize about debt: At the end of the day, you still owe $50.

Wherever that debt appears on Fowler's quadrant, you still have to pay it off. Or it will gather interest, and eventually (if you leave it long enough) bankrupt you.

Granted, this is perhaps where that metaphor starts to wear a little thin. In a codebase, where we perhaps deliberately chose not to use a Strategy pattern, but instead just coded the algorithm by hand directly into the code, because we don't really see any need for any greater degree of flexibility, we have potentially amassed some (perhaps small) amount of technical debt. The $50 hammer, if you will.

In a traditional credit card scenario, that $50, compounded at 5% or 10% interest, will, without exception, eventually turn into a monstrous pile of debt that you cannot pay off, assuming you leave it unpaid for that long.

But that non-Strategized algorithm? So long as the client requirements don't change, there's not a thing wrong with it. It can continue to run, and run, and run, until the heat death of the universe, and nothing happens.

This suggests to me that technical debt isn't just about what the developers on the team at the time were thinking about. This suggests that technical debt has two more components to it:

The thoughts of the developer(s) who have now inherited the code.
The requirements (or lack thereof) of the project for which the code was written.

See, if the client never changes their requirements, there is no technical debt. So long as the code continues to run, without problem, then what the code looks like is entirely irrelevant. It's only when the client says, "OK, now we need to do this other thing with this codebase" that it becomes a problem.

Although, now that I write this, I realize that's not entirely accurate, either.

If the client's requirements haven't changed, but the code doesn't run, or runs into errors while running? Those are bugs, and the code needs to change (to remove the bug). And that's the other case where now, a developer struggling to understand the code in which the bug may (or may not) live will be running into difficulty. Enter technical debt again.

Which now suggests that technical debt is essentially "a developer's cognitive difficulty in understanding and/or modifying a codebase". Nothing to do with the decisions made at the time of the codebase's creation, and everything to do with the developer who is attempting to understand what the code is trying to do or how to modify it.

Technical Debt: Moving on

So does @kellan (and Peter Norvig) have it right, that "code is a liability"?

On the surface of it, maybe: if I write code, it is potential technical debt.

See, it's not technical debt yet, because it won't actually be a debt until we trigger the "understanding and/or modifying" clause of the above. The Ruby script I hacked together to transform my old blog's XML over to Markdown files for the new system, I won't know whether that code is technical debt until I (or anybody else who wants to use or modify it) go back into it again and face the cognitive load of understanding it or modifying it. So it's like the infamous cat in the box, neither debt nor not, until the box is opened.

Logging Hours

Mon, 18 Jan 2016 00:00:00 +0000

tl;dr A recent DZone post lamented how logging hours makes the author "die a little inside each time". I used to feel the same way. Then I grew up and got over it.

As is typical with my posts, I ask you to go read the original (or in this case, the DZone copy of the original) before proceeding much further; I'll quote it, but it's always better to have the source freshly in mind when reading the rebuttal.

And it is a rebuttal; just not necessarily aimed at him.

The act

The author, Matthew Casperson, opens with:

Friday afternoon rolls around, and the obligatory call for weekly hours goes out. “Make sure your hours are up to date!” And I die a little inside. In truth, assigning hours to tickets is not actually that much work. As I have been told many times over, logging hours can’t take much more than a few minutes. So why does this simple act of corporate obedience feel like I have just been beaten over the head?

He then goes on to cite four main reasons why this "simple act of corporate obedience" makes him feel "beaten over the head":

"Because it feels so pointless."
"Because it feels like a lack of trust."
"Because I don't know where these numbers go."
"Because it feels like a lazy management technique."

He goes on to explain each of those four points in some detail, but I think the points pretty well stand for themselves in summary form.

Truth be told, I have a hard time arguing with him. Or, rather, the version of me from about a decade ago has a hard time arguing with him. Filling out timesheets felt so pointless and stupid, why bother? What if I just totally randomized the numbers and sent that in? Would anybody actually notice? (Nobody ever did, by the way.)

But the 2016 version of me will, actually, argue this with him. Part of it is simply professionalism---if the client or your employer says to do this thing, you do it, because it's simply the professional thing to do. (And I have every reason to believe that Matthew is being a professional and doing it, lest there be any doubt there.)

More importantly, the 2016 version of me will also turn around and argue this for him, because everything he says screams out to me that his management is Failing him.

Utterly, completely, and totally Failing him.

With a capital "F".

Management Failures

Let's start with the simple one first: When an employee talks about an act as being a "lack of trust", usually that suggests that other factors are in play. Matthew writes, "... all this work and effort feels completely undermined by management asking for an hour by hour account of my day. Am I that bad at my job that you really feel the need to know where every hour of it was spent?"

This is a smell that suggests that there is more at work here than just the logging of hours; employees do not feel distrusted simply because management wants a time accounting. (There's good reasons for wanting this, which I'll get to in a second.)

This is a smell that suggests that the employees don't trust management. It's really that simple; trust, like most relationships, operates on a "bank account" kind of principle, and if employees are feeling like the trust account is a little low, then it's really up to managers to (a) recognize that, and (b) work to put some trust back into the account. (Of course, that does go both ways, which is what makes relationships hard.)

This is emphasized by his last point, about logging hours as being a "lazy management technique". Listen to what he says next: "Meaningless numbers being interpreted in secret ways by management that never speaks to me...." Wow. Matthew's management has a serious problem on their hands. "Meaningless", "secret", "never speaks to me", those are three red flags in just the first half of a single sentence. This is a statement of pure opinion, despite the allusion to factual studies about employee disillusionment that follows in that paragraph, but the fact is, employee feelings are just as legit as fact. The fact that an employee feels disrespected is every bit as important as whether or not they actually ever were. Emotions are real things, and need to be treated as such.

I think a lot of the management Failure here stems from his third point.

When an employee sees an act that "feels so pointless", it's because they don't see any value within the act. If it's an act that doesn't generate value to them, then management has Failed in a pretty simple way:

Management has not explained what these numbers are for.

Honestly, when I was managing consultants working for the consulting company, it was painfully obvious where the numbers were going---these were billable consultants, and so their hourly reports were going directly into the invoices that we were sending to the client for payment. When we asked them to put a little explanation around their time, it was almost always met with nods of recognition, because they could easily understand why those might be needed.

But suppose you're not working for a consulting company---why track the numbers?

Velocity

As a manager, I can certainly see a desire for those numbers---without those numbers and how they relate to story points or features, I can't really effectively track my team's ratio of estimation to actual.

And that ratio is actually very, very important.

Because when I'm called into a meeting with other departments (or other clients), and they start tossing off what they want us to work on next, I'm often asked to provide a ballpark estimate for how long this thing might take. "Oh, we know, it's just a ballpark estimate/SWAG, we won't hold you to it," they always say, but truth is, once a number gets named, it tends to "stick" and become the measuring stick against which all other things are discussed.

At that moment, my goal as a manager is to protect my team from an unachievable schedule---that is, quite frankly, the most important thing I will do all day that day. I can't just toss off a "oh, sure, whatever, let's say.... six months?" with a cavalier wave of my hand, because chances are I'm totally underestimating the work on so many different levels. My first reaction must be to stall, distract, and delay, until I can get a reasonable estimate to them.

(By the way, it's not unreasonable for them to want an estimate---contrary to the beliefs of the "No Estimate" movement that is currently making the rounds. Modern budgetary accounting requires some idea of what it will cost up front, before you commit to the purchase. Think about it this way: How many of you would buy a car with no idea of the price, agreeing to send a certain amount of money to the dealership every month until the dealer told you that was enough? It's a complete and deliberate "head in the sand" rejection of how 99.9% of the world's corporate budgeting and internal accounting works.)

Now this doesn't mean I need to create estimates down to the final story---a SWAG is still appropriate here. But if I have that velocity metric, I can take a couple of hours, do a rough pass over what they want, come up with a number of stories or feature points that are within the same order of magnitude (hopefully) of the actual result, and then multiply that times our velocity to have a first-pass SWAG that at least puts it in the right ballpark.

But that's just me.

Pulse

In some cases, managers use these numbers as a sanity-check, a way of feeling out how the team is doing, without having to tear them away from their actual work with one-on-one meetings. Think about it this way: which is less intrusive to you as a programmer, to spend a few minutes at the end of the week to fill out a simple report that will give me a high-level view of what you're doing (and by the way, why are you spending so much time debugging? Didn't we get that super-nasty bug fixed last week? I think I need to follow up with you), or would you rather pull yourself away from your desk for an hour a week so you and I can go over all of that in person, when you will probably forget half of the details that would show up in a report?

Personally, I don't like the reporting approach---I would much rather have the one-on-ones, because then we can talk about a bunch of things that wouldn't normally go into a status report. (Actually, I tend to do both---at the consulting company, I looked at the submitted timesheets, and had one-on-ones with the team leads, so that I could both passively get a feel for what was going on, and then validate---or not---that same feel via an active conversation.)

Is this an act of lazy management? Maybe. But what developers need to realize is that developing software is not like constructing a building or a house, because 98% of what they do is entirely ephemeral and very difficult for anybody else to see, even other software developers. (Try it sometime---observe what another developer is doing and the progress they are making without talking to them about it directly, and then try to describe what progress you've seen them make or not make. Even with rigorous source control practices and team-wide discipline around check-ins and merges, it's really, really hard.) If you're not a software developer, trying to understand what your team did today can be almost downright impossible without asking them.

So they ask them, in the form of a report.

Other reasons

Perhaps Matthew's manager isn't in the same position that I am, or isn't asked (and held) to estimates that early in the process. Why, then, might the numbers be necessary?

I could posit a few theories (Accounting needs them in order to know how to charge time against internal budgeting reports, or HR needs them to feed into the payroll system so that they can keep track of your vacation time, or the CTO wants a roll-up report each year of how much time was spent on different activities within his organization because he is convinced they are spending way too much time in debugging and maintenance and he needs those numbers so he can go to the CEO and the board and ask for a larger budget to accomodate their requests for the next fiscal year, or...), but any theory I come up with is entirely just a flight of fancy, and entirely irrelevant; the point is not why I think the numbers are important, but that Matthew's management has not said why they think the numbers are important.

Of course, the other possibility is that the management Failure occurs much, much higher up, and that this time-tracking was established by the previous (or the previous previous) CTO, and right now those numbers just go into a report that nobody ever reads. Which is its own management Failure.

History

Napoleon Bonaparte, one of history's greatest generals, was less fond of the precise marching orders than the various generals (particularly the Prussians) that he faced. His approach was far "looser" (and arguably much more agile); he would split his army into pieces, and to each of his Field Marshals he would give certain tasks.

But before he turned them loose, he would also make sure that each of the Field Marshals also understood what the overall goal of the campaign was: "We must take that town there before we move on the city, so that we have a place by which to set up the cannons to bombard the walls.", for example. In this way, when he turned them all loose, they all understood the larger picture. Then, when one saw an opportunity to further the overall strategic goal, they could act on it quickly, without having to send a messenger back to Napoleon himself to find out if that was something they should do. If the town in question lay open and undefended according to their scouting reports, they could dispatch a battalion or two to seize it and hold it. Then, when the rest of the French moved into their various positions, the town was already seized, and Napoleon could advance his plans that much more quickly.

Management that fails to explain the "Why" of a particular policy is effectively falling into that most heinous of management Failures, that of treating their employees like cogs. No, employees can't (and shouldn't) know everything that a manager knows---certain things, for legal and privacy reasons, must remain confidential. Lots of data about the company must remain in the hands of only a certain few.

But the hours policy is very, very likely not to be one, and it's even more highly likely that even if there are places where these numbers go that the employees can't/shouldn't know about, there's other reasons that we can talk about, to give them a sense of "why", and let them see that those few minutes spent every Friday to fill out the damn timesheet actually serves a useful purpose, even if it's not useful to them personally.

Because being told to do something that serves no useful purpose to anyone simply undermines your credibility as a manager. And that's probably the worst management Failure of them all.

Postscript

Just after I posted this, there was an email from the Harvard Business Review entitled, "The Best Bosses Follow These 5 Rules", and they read like a complete counterpoint to everything Matthew described:

Manage individuals, not just teams. When you’re under pressure, you can forget that employees have varying interests, abilities, goals, and styles of learning. But it’s important to understand what makes each person tick so that you can customize your interactions with them.
Go big on meaning. Inspire people with a vision, set challenging goals, and articulate a clear purpose. Don’t rely on incentives like bonuses, stock options, or raises.
Focus on feedback. Use regular (at least weekly) one-on-one conversations for coaching. Make the feedback clear, honest, and constructive.
Don’t just talk — listen. Pose problems and challenges, and then ask questions to enlist the entire team in generating solutions.
Be consistent. Be open to new ideas in your management style, vision, expectations, and feedback. If change becomes necessary, acknowledge it quickly.

If Matthew's management adopted these five, particularly #1, #2 and #3, they would immediately seek to manage him as an individual, which would lead them to finding out more about what makes him tick, and that would lead them to following up on the time reports. They would seek to provide a sense of meaning around those silly reports, and his feedback could go into perhaps even making the reports better, depending on what the reports are actually used for.

Most of all, if they were good bosses, they'd be embracing #5, and we probably wouldn't have had to have this little conversation.

Good chat.

Maybe is Selfish

Fri, 15 Jan 2016 00:00:00 +0000

tl;dr Don't hedge your answers when somebody is asking you for a commitment; "Do, or do not. There is no try." (Yoda) Saying "maybe" is, at best, your way of preserving your ego, and at worst, your way of trying to avoid a commitment.

Michele Leroux Bustamente tweeted her fondness for this article, and while I agree with the author's premise, I disagree with some of the reasoning and rationale, and with some of the proposed recommendations on how to proceed.

Critique #1: 'Maybe' means ...

To start, the author says (complete with snowy-dog meme picture that doesn't seem to have anything to do with the actual text), "You can’t execute on maybe. You say maybe because you falsely believe it’s helpful. In reality, you need to be liked. ... You say maybe because you want to feel included and important without encumbering yourself."

Actually, my experience with people who waffle on these answers suggests some far more subtle things are at stake here:

Isn't it kinder? I don't want to say "no" out loud because, well, "no" just has such a bad reputation. I mean, "no" is the equivalent of rejection, and who really likes rejection, right? Isn't it kinder to give them an ambiguous answer so that they can walk away with their pride intact?
I don't know if I can. Wow, your request challenges me, and I'm genuinely not sure if I can do it. It's potentially something I could do, but there's obstacles, including but not limited to: my time, my skills, and/or my faith that I could really do it.
I KNOW I can't do it, but I don't want you to think less of me. My ego's on the line here, somehow, and while I don't want to agree, and have you hold me accountable when I fail later, I also don't want to disagree and then have to explain why I'm saying no. (A lot of software developers fall into this one, it seems.)
I don't believe in what you're doing. Yeah, that experiment you're running that will generate all kinds of money playing the stock market? Yeah, apparently the only one who doesn't know you're full of shit is you, and I don't really want to be the one to tell you.
I don't think you'll actually execute. You've had all these requests before, and they never actually amounted to anything, and rather than argue with you about it, it's easier to just toss off a "maybe" and remain ambiguous.
I really don't want to be responsible for this. What you are proposing is a thing that might backfire, and if it does, I don't want to be in the backdraft when it blows up.

I'm sure there's other reasons, too. Those are what leaped to mind first.

Critique #2: 'Maybe' is harmful

He continues: "Every time you say maybe, you paralyze others. Should we find someone to do the job, or will you? Will you feel offended if you show up and someone else has filled your role?"

Partial agreement; in some cases, you're paralyzing the other with your maybe, but in most cases, it's a far different problem.

Interlude: 50 Shades of 'Maybe'

By the way, there's more than one way to say "maybe"; the author points out four, but there's literally thousands:

"If I can."
"I’ll do my best."
"I’d like to, but I’m not sure..."
"If I don’t show up, move forward without me."
"It's distinctly possible."
"I can see both sides."
"That's an interesting idea, but..."
"Have you gotten approval for this yet?"

... and so on. One of my favorites, of course, is the classic "Go ask your mother." Or its corollary, "If it's OK with (somebody else), it's OK with me."

Problem: Not committing creates miscommunication

It's been my experience that a "maybe"'s biggest problem is the fact that people will hear whatever they want to hear, so if you come back with a "maybe", those who want to hear "yes" will hear it, and those who want to hear "no" will hear it. We learn this very quickly as parents:

Kid: "Mommy, can I have a pony?"
Mom: "Maybe someday, dear..."
Kid: thinking That's not a no! That means yes! Hooray!

What's even better about it, "maybe", when interpreted this way as an answer, has the pleasing short-term effect to us of making the questioner go away! Success! Our answer actually accomplished more than what we thought it would do, so internally we reinforce the logic we used from the earlier list above (kinder/don't know/don't believe/whatever), and we will start going to it even more often.

At least, until it blows up on us.

Problem: The time bomb of percevied commitment

Because that's the lurking danger: "Maybe" is a ticking time bomb of perceived commitment, and a lurking miscommunication waiting to happen, which will usually end in bad emotional juju between these two people, and potentially a lot more (depending on the situation):

Employee: "Hey, boss, can I have that promotion?"
Boss: "Maybe somebody, Bob..."
Employee: thinking Wow! He thinks I could do it! I mean, he didn't say "yes", but he can't really commit to it right now since there's all this HR nonsense he has to go through, but if he didn't think I was right for it, he'd have just said so, right?
Employee: (later that night at home) "Honey, guess what? My boss thinks I'm good for that promotion! Go ahead and book the expensive vacation to Hawaii, we're going to celebrate!"

Then, when the miscommunication becomes clear, Really Bad Things come to roost. I don't have to draw an example---I'm sure each reader has their own story they can draw on.

Problem: Honesty and reputation

"Maybe" has another nasty side effect---when that time bomb finally does go off, and you are presented with the need to clarify your "maybe", usually in the opposite result than what the requestor assumed, you will develop a reputation in their mind for being dishonest.

Employee: "Hey, boss, can I have that promotion?"
Boss: "Maybe somebody, Bob..."
(Three months pass...)
Boss: "And I'm proud to announce that Janet has received the promotion."
Employee: thinking THAT DICK! He promised it to ME! OK, boss-man, you want to play it that way? Fine, two can play at that game, you back-stabbing liar.

If the conversation ever goes to a verbal one, you will be on the defensive immediately, and you will have to remember that you said "maybe", first of all, and then, be placed in the unenviable position of having to explain---now, while emotions are running high---what you "really meant" by that "maybe".

These conversations generally don't end well.

Solution: Commit

Frankly, the answer to the problem lies in the realization that not committing is not acceptable. And "committing" here doesn't mean "yes"; yes, there's a group of people out there who believe you should say "yes" to everything because then your life is far more interesting and challenging than if you didn't (and they even made a move out of it), and frankly, yes, you should take a few chances here and there, I agree.

But what I'm talking about here is more fundamental: commit to either a "yes" or a "no". It's harder, but it's more honest, and yes, sometimes it requires you to face that argument up front that might get pushed off to later if you just do a "maybe":

Kid: "Mommy, can I have a pony?"
Mom: "No, dear. Horses are expensive, you're too young, and we live in an apartment."
Kid: "WAAAAAH!"

Having said that, though, sometimes we can actually get to a better result by being honest this way:

Kid: "Mommy, can I have a pony?"
Mom: "No, dear. Horses are expensive, you're too young, and we live in an apartment."
Kid: "WAAAAAH! You think I'm not old enough to have a pet!"
Mom: "Oh, dear, is that what this is about? Tell you what---let's start with a goldfish, and if you take good care of it for a year, we can talk about getting a kitten or puppy after that, OK?"
Kid: sniffling "But why can't I have a kitten or puppy NOW?"
Mom: "I need you to show me you're responsible enough to remember your chores before I can feel good about getting you something that will depend on you more."

Now we're getting somewhere. The communication is clear, unambiguous, and direct. And what's better, you don't have the uncomfortable situation of being forced to explain a miscommunication later.

Employee: "Hey, boss, can I have that promotion?"
Boss: "Wow, Bob, I didn't know you were interested. Hmm. Here's the thing: you're missing some of the key skills that promotion would require, and I'm not certain you'd have a chance to demonstrate those skills before the promotion date closes. As it stands right now, no, I wouldn't give it to you."
Employee: "Oh. Wow."
Boss: "That said, Bob, I like that you're interested in growing your career, and that's part of what I'm here for, so let's you and I start talking about how we can get you those skills so that you're primed for the next opportunity that arises. Would that work for you?"
Employee: "Oh! Wow! Yeah, that's great!"

Net result: positive progress, and crystal-clear perspective.

Solution: How to Commit

The original author offers a list of very New-Age-ish "Follow your energy" stuff that I don't find particularly helpful. (And I find his "Execute on compassion" to be particularly dangerous, since---as mentioned above---most people think the compassionate thing to do is to say "maybe" rather than "no".)

Here's my list of suggestions:

Don't answer right away. These questions often come out of nowhere, and most of us don't wake up in the morning and contemplate all the answers to the questions we might be asked today. So give yourself a moment to think about it first. "Wow Bob, I didn't know you were intersted. Hmmm." Then, just shut up and think.
What does your intuition tell you? Our intuition is pretty often accurate, so at least take a moment to reflect on what it's telling you. It's not the complete answer, but it's a good sign that this is where your heart wants to go.
Spend a few minutes weighing pro and con. What would be the result if you said yes? What would be the result if you said no? How does this decision actually affect anything? If we discover that the answer is pretty trivial, then pffft who cares? Go with yes. (Why yes? Because of that earlier "always answer yes" thing---it turns out that if there's not much downside, your life will probably be more interesting if you go with "yes" than "no". Carpe diem, man.)
Express your commitment clearly. The words "yes" or "no" should show up there somewhere. Forcing yourself to say "yes" or "no" makes your communication that much more clear, and that much less ambiguous. It will not always go over well, but it's far better to just take the bitter medicine now than let it fester, ferment, and get that much more bitter (and that much larger!) in the future.
Be willing to express your rationale. Notice that I say "rationale" there, instead of "reasons". "Reason" is something that implies logic and fact, and sometimes, you just simply don't want to do this. That's OK! Sometimes, we don't want to do things. I don't want to eat broccoli. My kids don't want to do homework. Heck, I don't want to do homework. And those are legitimate feelings, so long as they're taken into consideration appropriately. "Honey, Daddy just hates horses. Can't stand them. If you want to get a horse when you move out of the house, that's up to you, but that's just not something I'm going to let in while I'm living here." They may disagree with it, and want to claim that your feelings are misplaced or that you should be able to set those aside for the good-of-whatever, but now at least we're being clear about what's going on, and that allows the conversation to move to a new level of communication.

'Maybe's are for emotional wimps

In the end, really, the use of a "maybe" is your way of wimping out of something you don't want to do, and not only are you trading off pain now for pain later, but you're also not being honest with anybody---not the questioner, and not with yourself. You're selling your own feelings and decision-making process short, and you're setting yourself up for a harder process later.

"Do, or do not. There is no try."
---Yoda

"Do, or do not. There is no maybe."
---Me

Just commit.

A Good TechBlog Read

Thu, 14 Jan 2016 00:00:00 +0000

tl;dr I've found a new blog that I'm enjoying reading so far, and thought readers might want to browser-bookmark for future consumption.

Twitter sometimes dumps some good stuff into my face, and in this case, this comes to me via James Ward over Facebook. (I think he posted it to Twitter and it echoed to his Facebook feed, but details, details. I saw it through Facebook, so Facebook gets the credit.)

Meet TechBlog. I have no idea who they are other than what I see on the website---my guess is that they're an online real-estate system of some fashion, but whether that's as a competitor to Redfin or something, I really can't say. But their blog is a very nice mix of technical topics, and the writing is quite well-done. Consider this list of posts that I flagged just browsing the first three pages of the site:

"How We Used Category Theory To Solve A Problem In Java"
"Business-Friendly Functional Programming Part 1: Asynchronous Operations"
"Painless Javascript Testing? Surely You Jest"
"Static Analysis: What's It Good For?"
"DevOpsDays2015 As a Graduate". This one deserves a little more explanation as to why I flagged it, and that explanation comes right away in the second paragraph, which I thought was a great "catch": "So what is DevOps? Nobody really knows yet, not even the engineers supposedly belonging to this group can concisely explain what it is. The scope of the word has grown unwieldy, despite its state of infancy. In fact, the crowd seem to hate the very use of the word. Everyone pauses and takes time to punctuate each mention of the phrase DevOps with a facetious “air-quotes” gesture."
"Minimising Build Time With Recursive Containers"

In particular, James linked to the first post (the category theory one), and it's a good read. I still don't claim to completely understand the mojo and mystery around monads, monoids, category theory, endofunctors and all that other jazz, but the author does a great job of setting up the real-world business problem, then talking about the abstract stuff (the basics of monoids and functors), then talking about how this can be used to solve their problem, complete with Java interface examples, and builds up from there. It's a very well-written piece, and maybe finally I begin to understand the ideas here better.

I don't know who these folks are, but damn, I'm impressed. Not that the random musings of a longhaired geek in Seattle makes all that much difference to them, I imagine, but well done, folks, and I'll be looking to consume more of your posts over the next several months, so keep 'em coming!

Death to Technical Monoculture

Wed, 13 Jan 2016 00:00:00 +0000

It's really starting to appear like the "technical monoculture" that so pervaded the 90's and 00's is finally starting to die the long-deserved ugly death it was supposed to. And I couldn't be happier.

I was sitting with a buddy of mine in a sports bar.

(Yes, it was Canyons. For those who don't know me too well, it's one of my favorites here in town---not that I really use it for a sports bar, per se, but because they are a good group of folks who have nice large wall-mounted TVs that we can remote into using GoogleCast from a laptop, which means, combined with their WiFi, copious outlets and a nice space in the back where we're more or less isolated from the rest of the crowd, I have a perfect place to host the SeaLang meetups, yay. What's better than learning about new programming languages? Why, learning about new programming languages with beers, nom-noms, and Diet Coke!)

He happens to work for Microsoft, and we were commenting on how Microsoft really seems to be getting their mojo back, and a lot of it stems from the fact that Microsoft internally is really starting to embrace things outside of the traditional Microsoft sphere of influence. Case in point: Azure. If you're running on Azure, you have your choice of just about anything you can name as your programming language of choice (C# and Visual Basic, of course, but also Java---which will include pretty much anything that compiles to JVM bytecode, so throw in Scala, Groovy, Clojure, Kotlin, and a few more---and PHP and NodeJS, and ...) to use. They're embracing different data stores, too. And---and be honest, how many of you ever thought you'd see this---you can run Linux VMs on Azure to host all of this stuff if you want, so anything that Azure doesn't host "natively" can always be running inside of a VM.

The interesting thing about this conversation was that it really became apparent to us that the traditional "Microsoft shop" (of which, not surprisingly, Microsoft has long been one) is essentially an endangered species. No longer will it really be acceptable for a developer to "just" know Windows, C#, and SQL Server. You're going to need to know a whole lot more than that, because frankly, there's a lot of tools out there that are actually better than anything Microsoft offers natively. Redis, for example; there's no equivalent that I've found to Redis in the "Microsoft world", and frankly, why should I have to settle for an equivalent? If I like Redis, and it seems useful, why in Heaven's name shouldn't I be able to use it? Just host it in Azure in a Linux VM, and keep going.

But this street isn't entirely one-way. When the Pandora's box on the JVM got blown off (thanks to AspectJ, which not only suggested that "objects are everything" was a fallacy, but also that one could run multiple languages on top of this virtual machine, and it could be a Good Thing), it opened a lot of Java developers' eyes to the fact that Java wasn't, perhaps, the last programming language they would ever use. And now, with the CoreCLR migrating to other platforms, coupled with the fact that the C# language is actually kind of a nice language to use....

Couple that with Xamarin being able to cross-compile to different mobile platforms, or RubyMotion being able to do the same, and....

Oh, and toss in that recent announcement about Apple and Android for good measure...

Yeah. It's something of a new world.

As someone who's spent a decade and a half talking about interoperability across platforms, it's somewhat gratifying to see all of this starting to take shape. Scary, too, I'll grant you, if you're one of those people who's always felt that they could just learn the one thing and then go out and be emplyable forever.

But this is coming, and you have essentially only a few choices: Fight it (and essentially lose, since there's no signs that this is ever going to change), ignore it (and essentially become obsolete over time), or embrace it.

I think it fairly clear which path I choose. And to the best of my ability, I will help you if you choose it, too.

Happy People Still Do Quit

Fri, 8 Jan 2016 00:00:00 +0000

Twitter leads me to some interesting blog entries sometimes, and this time, it led to me to @rands's entry entitledShields Down, which appears to have the subtitle "Happy People Don't Leave the Jobs They Love". In it, he's got some good discussion about being a manager and the realization that by the time of the exit interview, it's already way too late for him to "save" his employee from leaving. Or that, in fact, it starts much, much earlier than that.

He begins by essentially pointing out that when you get that email from your friend, who says, "Hey, dude, you should totally come check out this place where I'm working right now" and you agree to it, your "shields are down". This is his code-phrase for "you are now considering leaving your place of employment".

Umm.... yeah.

Shields down! Oh noz!

Here's my first disagreement with him:

Technology employees are always considering leaving their place of employment.

Seriously. If you're in this industry for any length of time, you've done the math, and realized that there's far more jobs out there than people to fill them, particularly if you're a programmer and have just a few precious years of experience under your belt--those with zero years have a harder time of it, for sure, but it's still a lot easier than people with zero years of experience and an Ancient Languages degree, even if it's a Master's or something.

This is not a hard rationale to follow: These are highly analytical people we're talking about, and they are fully capable of looking around and seeing their co-workers jump out of one thing and land into another. Couple with that the fact that most programmers are paid far better than anyone of equivalent experience in other industries--all the lawyers who are making six digits immediately within two years of graduation of law school, please raise your hand?--and you begin to realize that for the average tech employee, their shields were already down.

All that email from their buddy did is make them realize that there was maybe an option a little closer to home than they might have thought otherwise; all that means is that they are simply one step closer to a new gig than they thought they'd be, and they don't even have to engage with a recruiter if they don't want to.

(By the way, how incredibly arrogant is it of us as an industry to be all up in arms over recruiters calling us--dare I say pestering us--about finding a new job? Seriously, folks, sure, it's a PITA when they're spamming your Inbox, but it's really no worse than Viagra ads, and FFS, quit being such primma donnas. You really want to be in an industry where there aren't recruiters constantly chasing you? Go take up a job working in Ancient Languages for a while, or maybe Astronomy or some other academically-dominated field, where the ratio of jobs to candidates is like 1-to-10. You're like the hot girl in high school who is constantly complaining to all of her friends how all these cute boys just won't leave her a-LONE, for crying out loud. Cry me a river.)

The Algorithm of Job Switching +3, vorpal blade

Rands goes on to suggest that upon shields-down status, people start going through a pretty complex set of questions:

When you are indirectly asked to lower your shields, you immediately parse, place a value, and aggregate your opinions on the following:
* Am I happy with my job?
* Do I like my manager? My team?
* Is this project I’m working on fulfilling?
* Am I learning?
* Am I respected?
* Am I growing?
* Do I feel fairly compensated?
* Is this company/team going anywhere?
* Do I believe in the vision?
* Do I trust the leaders?

... you use that blend and ask yourself one final question as you consider lowering your shields. What has happened recently or in the past that either supports or detracts from what I value?

The answer to that question determines whether your shields stay up or go down.

Here's the thing: that is a hell of a list to end up with an all-yes set of answers. Honestly, I don't think there's ever been a job, ever, that would yield a 100% "yes" rate. Particularly when we start with "Am I happy with my job?"; that's sort of the roll-up question that is the sum total of all of the other answers. You're happy with your job if you like your team, your manager, you're doing fulfilling work, and so on.

Were I to take this checklist to anyone--and I do mean anyone--I suspect the answers will inevitably come to some percentage of "yes", and some percentage of "no". (And that's assuming all of the answers are intrinsically binary, which I don't really believe to be the case.) It's like trying to go up against a windmill with a lance, or an orc trying to win a battle against a warrior (of any level) wielding a +3 longsword, vorpal blade.

When faced with such staggering odds, the answer is a foregone conclusion. So, again, employees' shields are already down.

The Interrogation

He goes on to do the human thing: "Each time [a co-worker leaves] I work to understand two things: Why are they leaving? When did their shields go down?"

He points out that #1 is the smooth, practiced answers that they use to convince themselves that this is the right thing for them to do. But he wants to drill into the successor: When did their shields go down? What was the critical moment?

I can answer #2 for you. Matter of fact, I already have: their shields were always down.

What follows, though, is best left in his own words:

To find and understand this shields-down moment, I ask, “When did you start looking?” Often the answers are a vague, “It kind’a just happened. I wasn’t really looking. I’m really happy here.”

Bullshit.

If I’m sitting here talking with you it means two things: I don’t want you to leave and, to the best of my knowledge, you didn’t want to leave either but here you are leaving. It didn’t just happen. You chose. Maybe you weren’t looking, but once your shields dropped, you started looking. Happy people don’t leave jobs they love.

He's spot-on with this assessment: you made a choice to interview somewhere else, you made a choice to evaluate their offer, you made a choice to accept the offer and all the "transitions" that are involved (changing your daily routine, changing your work friends, working to discover what the new lunch options are, and so on), so it didn't "just kinda happen" any more than that Hollywood starlet "just kinda happened" to be naked in bed with her co-star while her boyfriend was a continent away. Choices are choices.

What sort of concerns me the most, though, is his conclusion:

The reason this reads cranky is because I, the leader of the humans, screwed up. Something in the construction of the team or the company nudged you at a critical moment. When that mail arrived gently asking you about coffee, you didn’t answer the way you answered the prior five similar mails with a brief, "Really happy here. Let’s get a drink some time!" You think you thought Hmmm… what the hell. It can’t hurt.

What you actually thought or realized was:
* You know, I have no idea when I’m going to be a tech lead here.
* Getting yelled at two days ago still stings.
* I don’t believe a single thing senior leadership says.

Often you’ve forgotten this original thought in your subsequent intense job deliberations, but when I ask, when I dig, I usually find a basic values violation that dug in, stuck, and festered. Sometimes it’s a major values violation from months ago. Sometimes it’s a small violation that occurred at the worst possible time. In either case, your expectations of your company and your job were not met and when faced with opportunity elsewhere, you engaged.

Rands, I'm going to pop a bubble here for a second: It's not you, man. (Or, maybe, perhaps, it's just just you, man.)

The Contract

When somebody comes to work for me, there's an implicit--and I try to make it explicit--contract that we establish: You tell me what you want, and I will do everything in my power, within the boundaries of my authority and what's practical for the company, to help you get it.

That first part is where things need to be more clear: you need to tell me. You want that tech lead promotion? OK, let's talk about what that means, what skills you need to have in order to take that job, and what of those skills you currently lack or need to brush up on. More importantly, if we don't have room for a new tech lead, no matter how badly you wnt it, no matter how skilled you are at being a tech lead, I can't give you that promotion.

Shields down? Of course. So what? Employees' shields are always down, remember? They were down yesterday, and the day before that.

(Don't believe me? Quick, all you technical types: hands up if you've NEVER thought, "Damn, that guy from Vietnam made like a million dollars making Flappy Bird, and I'm still here writing code for the TPS reports. I wonder....")

So your shields are down? Well, it is a Friday. So what?

Rands seems to treat this moment with fear, as something to be avoided.

I don't fear that moment. Matter of fact, I personally embrace it.

Managing a shields-down team

I admit, I'm fairly new to this management thing. But there's a few things I'm pretty sure about, and so far (2 years, and running) it's worked out pretty well.

If the place for which I work doesn't currently have the opportunities for you, then I think it's my job as your manager--an individual who should be responsible for not just making sure you get your work done on time, but is also interested and responsible in your job and career growth--to help you find the challenges you want and/or need.

Yes, this means it's my job as your manager to help you find something outside of the current workplace sometimes. Or, at the very least, not stand in your way.

Because, to be really blunt about it, I can't do the kinds of constant scrutiny that Rands holds himself accountable for:

I’ve been here for three years and I’m looking for a change of scenery. It happens. Two months ago, someone told them their project was likely to be canceled. It wasn’t.

I have no idea what I as their manager was supposed to do: cancel the project anyway? The business makes decisions based on the business' needs, not the needs of the individual contributors on those projects. Any business that didn't make decisions that way would very quickly be out of business.

You know, I have no idea when I’m going to be a tech lead here. At the end of last month, she heard via the grapevine that she wasn’t going to be promoted. When she got the promotion she deserved, it was too late.

Did she ask about the promotion? Did she come to me and say, "Hey, Ted, I'm really interested in taking that tech lead role?" Did we discuss her current qualifications and how she could put herself into the best possible position to take it? Was I responsible for choosing who received that position? And did I communicate to her why she didn't receive it, if she didn't get it?

I'm in the enviable position right now of having a dozen people working for me, most of whom are easily tech leads. They're sharp, they're self-directed, they can see the forest for the trees, and I don't have enough teams on which to make them all leads. We've rotated a few around, and we're trying to grow the firm, but if one of them comes to me and says, "Dude, I am really ready to be a tech lead, and I won't take no for an answer", then the only option is to help them find that role, outside of the firm if need be.

Getting yelled at two days ago still stings.

You know what? If I yelled at you, then yeah, that means something really went south. Don't get me wrong, I'll hold you accountable if you screw up, but there's really no reason for any one adult human being to be yelling at another one unless somebody's life is on the line. And if I didn't come to you afterwards to apologize and clear the elephant out of the room, then that's on me. I would like if you would come to me and say, "Hey, Ted? The other day, when things got a little heated, I thought you were a little harsh and I didn't appreciate it" so we can sort that out, but that's only going to happen if you're comfortable doing so, and if you're not, then our work relationship was already in jeopardy.

I don’t believe a single thing senior leadership says. At the last All Hands, I blew off a question with a terse answer because I didn’t want to dignify gossip. I forgot there is signal even in gossip.

Rands, give yourself a pass here. Employees don't reach that level of distrust based on one blow-off. (Not unless they were significantly cynical to begin with, anyway, which is its own problem.) If you have a continuous history of blowing people off, that's another problem entirely. And they may trust you, but not your management--and there's not a damn thing in the universe you can do about that.

More importantly, you can't dignify gossip, but you can't stop it, either. When I do a one-on-one with my folks, I even explicitly ask them, "Are there any myths, legends, lore or gossip you want me to confirm or deny?", because employees talk, and I want to give them a chance to air that stuff out and get an answer from me. But, again, I'll have my limits--sometimes I don't know the answer, and sometimes I can't say the answer. Case in point:

"Are we going to land that new client?" Frankly, I don't know. Things may look good (or they may look bad, or they may not look like anything at all), but nothing's a given until paper is signed. I'll let you and everybody else know when I do. If you want my personal opinion, though, it sounds relatively positive (or negative, or the conversation went way south when they started talking bill rates in the pennies per hour, or...), but that's just my opinion. [UPDATE 2021: Note that this does not take into account that there are always some topics I simply cannot speak to, for legal and/or liability reasons. I usually try to make it clear when we've hit one of those topics by saying, "I can't speak to that because of legal and/or liability reasons" and I've never had an employee disblieve me when I do.]
"What happened to Bob? Why isn't he here anymore?" Ah, that's an HR thing, and I can't really comment on it, sorry. Suffice it to say we had an issue, and we (the leadership) felt it necessary to take that step. (Assuming Bob was fired, of course, and didn't leave of his own accord.)

I'm a manager, not God.

The Contract, Redux

If you work for me, it's simple: You tell me what makes you happy, and if I can make it happen for you, I will make it happen. But:

... you have to tell me. Ask my wife, I am not a very good mind reader.
... you have to accept that I am not a miracle worker. Giving you what you want has to be within the company's and my boundaries to grant. I can't let you work from home on a regular basis if we have needs that require you to be in the office (whatever those may be). I can't get you onto a project using F# (no matter how much I may want to!) if we don't have any projects that are using F#. I can't get you a pony. I have my limitations.
... you have to keep up your end of the deal. We have Pluralsight subscriptions for all of our employees--you can't tell me you don't feel like you're not getting enough learning in your job if you're not taking advantage of that. Sorry.

And in return for all that, it is my job to:

Give you autonomy. I hired you because you're a professional, and you'll get the job done. So it's my job to stay out of your way, until you need me to get into the fray and clear out an obstacle before you that you can't clear out yourself (or would cost exponentially more to handle than if I did it).
Grow your competency. You're not a fixed "resource" (Lord how I hate that word), you're a human being, and no matter how competent you are currently, you can be better. Part of my job is to help you grow your own competency, even if it far surpasses my own.
Help you relate. You're not an island, and you're not a cog. You are part of a team of people working on a project, even if you're working alone for a while. If you "fit" within the team, and the team is "connected", that's going to be a huge factor in your willingness to stay and contribute. If you don't fit, then it's my job to do what I can to help find a way for you to fit, whether that's helping you adjust to the team, or helping the team adjust to you.

If I'm not doing these things, then I've broken my end of the contract, and you should be out looking for other places to work.

UPDATE 2021: Netflix recently has come out saying that they tell their employees that if a recruiter calls, take the call, so that Netflix can use that as a way to judge what the market looks like. I like it, and I've adopted it. It's the best way to make sure your employees are comfortable, by making it clear that you trust our relationship enough to not only freak out about them interviewing--but encourage it.

And honestly, if they find a place that's ultimately going to be better for them, I wouldn't want to stand in their way of taking it up. Would you want to work for somebody who would?

Clausewitz on Policy (Software Craftsmanship)

Wed, 6 Jan 2016 00:00:00 +0000

As many readers know, I didn't spend my collegiate years studying algorithms; instead, I obtained my degree from UCDavis in International Relations with an emphasis on military strategy and history. That meant a tremendous amount of study in history, pyschology, a little philosophy, and military affairs.

One of the great books on strategy, Strategy, written shortly after the end of World War Two, was widely considered to be Hart's best work on the subject, though he had plenty to say before the book's publication; the great German general Guderian was quoted as having studied Hart prior to the war's outbreak. He's a pretty easy read, too, by the way, as historians and military advisory goes.

Recently (for a variety of reasons) I picked it back up again. The first 75% of the book is about war throughout history, as a means of establishing Hart's backdrop for the theory that he advocates in the latter quarter of the book. The closer he gets to WW2, the more detailed his analysis and history, so it's not really a comprehensive study of military history by a long shot, but it's still more than most students of American education have ever received.

The last quarter, though, is where he expounds on "The Theory of Strategy", and in the opening paragraphs, he makes a very salient point about war, strategy, and one of its finest writers (Clausewitz) and how a misinterpretation of writing can lead to all kinds of problems.

Let us first be clear as to what is strategy. Clausewitz, in his monumental work, On War, defined it as 'the art of the employment of battles as a means to gain the object of war. In other words strategy forms the plan of the war, maps out the proposed course of the different campaigns which compose the war, and regulates the battles to be fought in each.'

One defect of this definition is that intrudes on the sphere of policy, or the higher conduct of the war, which must necessarily be the responsibility of the government and not of the military leaders it employs as its agents in the executive control of operations. Another defect is that it narrows the meaning of 'strategy' to the pure utilization of battle, thus conveying the idea that battle is the only means to the strategical end. It was an easy step for Clausewitz's less profound disciples to confuse the means with the end, and to reach the conclusion that in war every other consideration should be subordinated to the aim of fighting a decisive battle. (Strategy, p. 319)

It's been a while since I read Hart, but as soon as I read it, my mind did a double-take. Substitute "information technology strategy" for "policy", "software project" for "war", "sprints" for "battles", "executive team" for "government", "software craftsmen" for "military leaders".

Let us first be clear as to what is strategy. Clausewitz, in his monumental work, On Software Projects, defined it as 'the art of the employment of battles as a means to gain the object of a software project. In other words strategy forms the plan of the software project, maps out the proposed course of the different campaigns which compose the software project, and regulates the sprints to be fought in each.'

One defect of this definition is that intrudes on the sphere of information technology strategy, or the higher conduct of the software project, which must necessarily be the responsibility of the executive team and not of the software craftsmen it employs as its agents in the executive control of operations. Another defect is that it narrows the meaning of 'IT strategy' to the pure utilization of sprints, thus conveying the idea that sprints are the only means to the IT strategical end. It was an easy step for Clausewitz's less profound disciples to confuse the means with the end, and to reach the conclusion that in war every other consideration should be subordinated to the aim of fighting a decisive sprint. (Strategy, p. 319)

Get where I'm going with this? The construction of software is simply a tool--and not, by any stretch, the only tool--by which a company can achieve its ends. Yes, bespoke software is often a powerful and useful tool for doing so, but so is a US Army Mobile Division. Sometimes, "sending in the troops" is an appropriate tool for use in a political situation, and sometimes, you have to hold back and try something else.

Similarly, not all IT effort is around custom software development. There can be much utility in a small shell script that accomplishes a goal as there is in an "enterprise-scale" software project that comes complete with unit tests, continuous integration, and automated deployment.

Certainly, the analogy isn't perfect--the insertion of a small SEAL team behind enemy territory requires a metric crap-ton of planning in the same way that loading up the US 5th Infantry and shipping them off to Korea requires. But, relatively speaking, it's a lot easier to dispatch SEAL Team Six to a hotspot to engage in a sharp, focused mission, than to dispatch even just one US regiment; the logistics are simply night-and-day different.

During the Korean war, Gen MacArthur was vehement and loud that the US Army, after its incredibly successful amphibious landing at Inch'on. MacArthur very clearly wanted to prosecute the war deep into Chinese territory, since that was the only way to win the war, and claimed that refusing to engage in this wider war was "appeasement".

As President Truman looked for a way to prevent war with the Chinese, MacArthur did all he could to provoke it. Finally, in March 1951, he sent a letter to Joseph Martin, a House Republican leader who shared MacArthur’s support for declaring all-out war on China–and who could be counted upon to leak the letter to the press. “There is,” MacArthur wrote, “no substitute for victory” against international communism.

For Truman, this letter was the last straw. On April 11, the president fired the general for insubordination.

This is an important moment in American history; there was serious concern that MacArthur would return home and stage a coup d'etat, seeking to replace Truman and secure his "no substitute". Instead, MacArthur did what any professional does: he accepted the termination as Truman's right, and delivered one of the most moving speeches in American history.

The point here was that MacArthur was one of Clausewitz's "less profound disciples": he failed to understand that Truman's policy was NOT to seek a wider war with the communists, since that meant potentially taking on both China and the USSR, with a nation that had just emerged from a half-dozen years of armed conflict around the world, with not much in the way of allied support to do so. It would very likely have proven to be conclusively a massive disaster.

There were very good reasons Truman and his Adminstration had that policy in place; MacArthur, however, sought to be the craftsman at his profession, and demanded that "if we're going to do this, we're going to do it right".

Again, I laud and applaud the goals of the Software Craftsmanship movement, with its emphasis on quality and focus on "doing the right thing". But then some of its leadership says things like "Always remember that you, as a software developer, are also a stakeholder. You have a stake that you need to safeguard."

I'm sorry, but no. You are not a stakeholder. You may have a vested interest, but those are different terms. You do not own the software you are creating. You are producing this for the use of people who are not you. You will walk away from this at some point, and never worry about this again. A stakeholder, on the other hand, according to Dictionary.com, is "the holder of the stakes of a wager".

Unless that is your money you're paying yourself with, you are not a stakeholder.

Now, I'm pretty sure the Craftsmanship movement is going to jump on the second definition there, "a person or group that has an investment, share, or interest in something, as a business or industry", and key in on that word "interest", and say "We have an interest! That's OUR blood, sweat and tears, and that gives us an investment into it!"

And by that definition, every person who works as a janitor in the Trump Towers has an investment in the building. So therefore they can sell off their share of that investment, right?

Sorry, your blood, sweat, and tears were paid for. Your employer, your client, whomever it was that paid you to write this thing, they have the investment, not you.

You may have a sense of professional pride, and that's a good thing.

But never conflate your professional pride with what it means to be the actual stakeholder in the project.

The Story of the Chimps

Tue, 5 Jan 2016 00:00:00 +0000

Or, why passwords must be eight characters.

This just crossed my Facebook feed:

Take five chimpanzees. Put them in a big cage. Suspend some bananas from the roof of the cage. Provide the chimpanzees with a stepladder. BUT also add a proximity detector to the bananas, so that when a chimp goes near the banana, water hoses are triggered and the whole cage is thoroughly soaked.

Soon, the chimps learn that the bananas and the stepladder are best ignored.

Now, remove one chimp, and replace it with a fresh one. That chimp knows nothing of the hoses. He sees the banana, notices the stepladder, and because he is a smart primate, he envisions himself stepping on the stepladder to reach the bananas. He then deftly grabs the stepladder... and the four other chimps spring on him and beat him squarely. He soon learns to ignore the stepladder.

Then, remove another chimp and replace it with a fresh one. The scenario occurs again; when he grabs the stepladder, he gets mauled by the four other chimps -- yes, including the previous "fresh" chimp. He has integrated the notion of "thou shallt not touch the stepladder".

Iterate. After some operations, you have five chimps who are ready to punch any chimp who would dare touch the stepladder -- and none of them knows why.

Originally, some developer, somewhere, was working on an old Unix system from the previous century, which used the old DES-based "crypt", actually a password hashing function derived from the DES block cipher. In that hashing function, only the first eight characters of the password are used (and only the low 7 bits of each character, as well). Subsequent characters are ignored. That's the banana.

The Internet is full of chimpanzees.

I first heard this story from Dave Thomas ("RubyDave") at a NFJS show.

Thing of it is, it applies to soooo much more than just the passwords-must-be-eight-characters rule. Dave used it to illustrate why "if" statements in so many languages require parentheses around the boolean condition they are testing -- until Ruby came along, anyway. Turns out it was a parser issue with an ancient predecessor to C, and nobody asked "why" ever since.

The unwillingness to ask "why" certain things are the way that they are -- or perhaps the "shut up and sit down" reaction from the rest of the dev team when the question is asked -- is what leads to situations like this.

It reminds me of another of my favorite stories:

A young couple are just married, and the starry-eyed husband, being in a delightfully pensive mood, decides to simply watch his new bride fix dinner one night. As she is preparing the roast, she slices off about an inch from either side of the roast and throws the perfectly good meat away.

A little taken aback, the husband asks, "Honey, why do you do that?"

She replies, "Because it makes the roast taste better, and besides, my mother always did it that way."

Intrigued, the husband calls his new mother-in-law and asks her if she does the same thing. When she says yes, he asks why, and she replies, "Because it makes the roast taste better, and besides, my mother always did it that way."

Now really intrigued, he calls his grandmother-in-law, and asks her. When she says yes, he asks why.

"Beacuse my pan is too small."

Sorta makes you wonder: what does your team do because "it's always been done that way"?

UPDATE(2021): Subsequent iterations of telling this story have led me to understand that, as with most great fairy tales, this didn't quite happen as written. "A quick search revealed there was a real experiment that seems to be the basis of the story, though that research didn't prove much. It was a small study in 1966 titled “Cultural acquisition of a specific learned response among rhesus monkeys” by Stephenson et al." (from https://www.linkedin.com/pulse/five-monkeys-experiment-john-stepper/ ) In fact, this article suggests that it's not even close to true:

This story has been doing the rounds since 1996, and it has never been verified. It seems to have first appeared in a book called Competing For The Future by Gary Hamel and C. K. Prahalad, and by “appeared” I mean it was just made up. The authors never provided a source. None of the authors who have referred to the experiment in the past eighteen years have provided a source either. None of the appealing memes or infographics that describe the story now provide a source. Suffice to say, there is no source, because the experiment never happened.

... and even quotes the first article in this paragraph as being partly responsible for the spread of false information:

A blogger by the name of John Stepper writes about how amazing the Talk was and how Eddie [Obeng] was able to bring this untrue story to life [in a TED talk]. He then asks if it really happened, and says: "A quick search reveals it did happen though the details are quite different." ... This is perfectly true, if by “quite different” he really means “not the same at all, in any way.”

Sadly, the absence of anybody citing another study where this was done, conclusively, leads me to agree, this experiment never, in fact, happened. Which is both heartening and sad--sad, because it's a great story and tugs at the heartstrings, and heartening, because it suggests that the nature of "herd mentality" that we all fear so much may not be quite so much to fear.

2016 Tech Predictions

Mon, 4 Jan 2016 00:00:00 +0000

As has become my tradition now for nigh-on a decade, I will first go back over last years' predictions, to see how well I called it (and keep me honest), then wax prophetic on what I think the new year has to offer us.

In 2015...

In 2015, I said:

"Big data", "Big data", "Big data". You will get sick of this phrase. I can't speak for everybody, but I can tell that the end is near for the term, because suddenly everybody is using the term, and they're using it to mean anything and everything. "Big data" is not just about doing deep data science analysis on petabytes of data; it's about any analysis (even simple reporting) on any collection of data (no matter how large or small) for any reason. +1
"Internet of Things". You will get sick of this phrase, too. This hasn't quite happened yet, but we're close. IoT is also starting to fray at the edges as a definition, and when that happens, it's immediately ripe for abuse and marketing. More importantly, though, lots of people are starting to realize that IoT is neither the huge "automatic win" that we all sort of assumed it would become. +1
"Internet of Medicine" or "Big Med". Well, nobody's started using the term yet, but certainly they're spending a lot of time in this space. They just don't like my term yet. I'm pouting over it, but it's still a -1.
"Tech bubble" becomes a "thing". Oh, this one came down to the very wire, but as December of 2015 rolled in, concerns over the actual valuations of the so-called "unicorns" were starting to show up, and lots of people were beginning to openly wonder if Silicon Valley and Wall Street were experiencing a falling-out. It took a hail-mary pass to do it, but I'm claiming my +1.
C# and Java will both make big announcements. C#6 shipped, but Java9 didn't, leaving me sort of confused as to how to score this one. However, I did say, "Those who care will take note, those who don’t, won’t. Really, we’re kind of past the point where either of those languages are going to be interesting to anyone who’s not already in that space", and frankly, if you weren't a C# or Java developer, you probably didn't even hear a whisper about either one (pro/shipping or con/not-shipping), either way. +1
Go is going to either take off, or crash and burn. The point of this one was that Go was reaching an inflection point, and while I think it's gathering some momentum (including with me personally--this new blog is using Go to do the site generation), I can't really tell if it reached an inflection point, so -1 to me.
Microsoft acquires Xamarin. Oh, as much as I thought (and still think) that it would be a great story for both sides, it didn't happen, and probably never will. sigh -1
Amazon just quietly keeps churning. I dunno how I would measure this one, but in some ways, as long as Amazon just keeps churning out new feature after new feature on AWS, and keeps making money selling stuff on their main web property--which they continue to do--then I think pretty much anything here qualifies as a +1. But it was kind of a lame prediction to begin with, now that I re-read it.
Google continues to throw sh*t against the wall, looking for their Next Big Thing. I believe the exact phrase I used was, "Expect a lot of announcements, a lot of "beta"s, and none of it with any kind of realistic or even well-planned business model behind it--including the Google Car." And, sure enough, we've heard a ton about the Google Car, among other initiatives, but nothing has stepped up as a product yet to even come close to acting as a second line of income for the firm. I call it an easy +1.
Web use on mobile devices decreases in favor of apps. In particular, I said,

This is going to happen whether the public wants it or not, because companies have figured out that it behooves them to have you "trapped" inside their app (where they can control all the content) rather than on their website. More and more websites are going to try and redirect you to inside their app, rather than allow you to casually browse on their site, because then they think they "own" your eyeballs. The only way this changes is if/when some firm gets crushed in the court of public opinion by doing something really stupid... and that won't happen in 2015. Wait for it in 2016.

Various "clickbait" sites were the ones I was thinking about in particular, and while some of them (I'm looking at you, Uberfacts) have floated a mobile app out there, the apps themselves don't seem to be lighting any fires in the mobile marketplaces. I'll talk more about this in a bit, but for now, I'm giving myself a -1.
Hipster "Uber for X" apps will be all the rage. Have you been to San Francisco recently? Talked to anybody on the street there? This one was a slam-dunk +1.
Mark Zuckerberg grows up a little. Zuckerberg will never admit it, but now that he's married and starting a family and all, he's starting to grow up. His paternity leave step was a big one, and signals that maybe he's finally ready to "adult" now. If so, he's in good company--it took Bill Gates getting married and having kids to come in out of the rain, too, and ever since that time, Bill's become a philanthropist of the highest order. +1
Larry Ellison buys a sports team. Didn't happen yet. -1
Perl makes one final gasp at relevancy, fails, and begins to decompose. Oh, this one is funny; how could I be so right, and so wrong at the same time? Wrong, because Perl 6 actually
finally shipped. And yet, so right because... well, how many people do you know using it? Or were even paying attention when it came out? Or.... Yeah. +1

Nine up, four down. Not bad.

That was the easy part. Now, on to the....

2016 Predictions

In no particular order:

Microsoft will continue to roll out features on Azure, and start closing the gap between it and AWS. This one is not hard to imagine. Microsoft is committed to making Azure a core part of their company success and survival, and Amazon has a list of features that Azure lacks, so it really boils down to "take one down, cross it off the list, lather, rinse, repeat".
(X)-as-a-Service providers will continue to proliferate. We're seeing a huge surge in these various companies that are providing some vertical thing as a service, and for most of those they're tech-related (such as Database-as-a-Service, Container-as-a-Service, and so on). Part of that is because if it's one thing software developer geeks know what to do, it's what they wish they had when they were working that last project and really wished they could have when they were working it. This coming year will mark the high-water mark of companies that provide *aaS products to the developer community, and then they'll all start cannibalizing each other and some shutdowns, acquisitions and partnerships will kick in.
Apple will put out two, maybe three new products, and they'll all be "meh" at best. Let's be frank, folks, the luster is off the shiny Apple logo on the site of the building. Tim Cook is no Steve Jobs, and Apple of 2015 was not the Apple of 2005 or 2010. The Apple Watch is interesting, but it certainly hasn't taken off. No watch seems to have, in fact, become the "it" thing. I don't see many of them (or their Android competitors, to be fair) at tech conferences, and casually glancing around the airport doesn't show a ton of them in use. I don't think this is going to change any time soon, either. For most people, the wearable just hasn't really offered up that compelling reason yet, and I don't think 2016 is going to see one, either.
iOS 10 will be called iOSX. Just because they can, and because it would confuse the hell out of people, and because Steve Jobs is not here anymore to tell that VP of Marketing to sit down, shut up, and let the grown ups do this.
Android N will be code-named "Nougat". They might go with "Nutella", but that would involve copyright and trademark issues, which I imagine they'd want to avoid.
Java9 will ship. This is a "no-duh" prediction, but I'm not above claiming a few of those. Really, the bigger question there will be what will ship that Oracle calls "Java9", and my personal feeling is that modules/Jigsaw/whatever-we're-calling-them-now won't be in it. Slamming a module mechanism on top of a platform that's a decade old, millions of programmers wide and a billion lines of code high is not easy, and I don't think Oracle really has the energy, motivation or need to push them through the morass of headaches that will stem from imposing a module system into place. UPDATE: @olivergierke points out that Java9 had already slipped to 2017, so this one is automatically going to be a "miss" next January. The article he cites says that Oracle "blamed the delay on complexities in developing modularization", a la Project Jigsaw. Honestly, I'm going to stand by this prediction, because it would not surprise me in the slightest if Oracle comes back at some point in 2016 and says, "You know what? Fuck it." and ships Java9 without modularization in place--I don't really think Java9 needs it at this point, and I'm not entirely sure that shipping with it will make Java all that much better. Time will tell...
Facebook will start looking for other things to do. Yes, Facebook has been ridiculously successful to date; it claims more population than most nations on Earth, in fact. But the company is led by a classic Type-A personality, and the softening of his character by the birth of his firstborn notwithstanding, this is when Zuckerberg comes back from leave and says, "OK, boys and girls, it's time to take us down a new path!" and charges off into who-the-Hell-knows-what. I won't hinge the prediction on what that would be, I just think it'll be something outside of the social media realm (or tied to it just a little bit).
Google will continue to quietly just sort of lay there. Google, for all that they are on the top of everybody's minds since that's the search engine most of us use, hasn't really done much by way of software product invention recently. Google+, Google Hangouts, yeah, sure, that was so 2013, but what have you done for us lately? And honestly, what have they done recently, in 2015? Casting back through my memory (and setting Android off to the side, since I consider that more or less an independent effort in a lot of ways), I came up with nothing. I suspect the same will be true of 2016--they will continue to do lots of innovative things, but it'll all be "big" and "visionary" stuff, like the Google Car, that won't have immediate impact or be something we can use in 2016 (or 2017).
Oracle will quietly continue to work on Java. Oracle took a bit of a PR hit this year when they fired/let go a number of "Java evangelists", and that set the newsstands aflame with hints and rumors that Oracle was getting ready to abandon Java. Frankly, if I'm Larry Ellison (or the VP that has Java under my umbrella), I'm asking a very fundamental question: What the hell does Java need with evangelists at this point? Everybody more or less knows what it is already, there's nothing to sell in of itself, and that money could probably be put to better use hiring people to work on the codebase itself, or putting the cash back into the rest of the firm to hire a few more Oracle Database salespeople. Oracle didn't acquire Java because they saw it as a way to inflict the Oracle Database upon the world--quite the opposite. Oracle acquired Java because they use Java, all over the place, and this way they had control over a technology that they had "bet the farm" on in a variety of ways. They're not going to kill it, but there's really not a whole lot of need to go around preaching its message, either. So they let the evangelists go, and they'll just keep on keepin' on.
C# 7 will be a confused morass. Microsoft is now striding boldly into that open-source world it timidly courted just a few years ago. But in a lot of ways, this is highly uncharted territory for the software giant, and for the OSS world as well. Sure, Linus has been releasing Linux kernel after Linux kernel for years, but with himself as the autocrat in charge of it all. Microsoft wants to make use of the open source man-hours to help advance the cause of C# 7, but whether they've smoothed out what that process will look like and/or how they will deal with the inevitable conflicts between committers and company isn't yet clear. (Oracle is in this same boat, in a lot of ways, and there's a lot of people who think that Java is too much Oracle, not enough OSS, so to speak.) I think the C# 7 release will be one of the first that the world gets to see take shape in a purely public forum, and they will be a bit confused and surprised at how chaotic a product release can really be. (Yes, C# 6 was sort of in that same boat, but only a handful of folks were really paying attention.)
Another version of Visual Basic will ship, and nobody will really notice. Actually, that already happened--remember when C# 6 shipped? They shipped a new version of VB then, too. Alas, the ship has sailed on VB, and frankly, at this point, it's really just a husk of its former self. Most of the VB luminaries are all speaking and/or writing in C# these days, and only staunch loyalty to their fond memories of the language is what keeps it at all in the conversation anymore. Sad, but... Oh, well.
Apple will now learn the "joys" of growing a language in the public as well. Swift is now open-source, and that will bring with it the same pains as what Oracle and Microsoft are feeling. Enjoy, guys!
Ted will continue to layer in a few features into the blog engine. For example, right now I have no comments feature, and I suspect people will want to start telling me how incredibly wrong I am about so many of these. So, on the docket already, Disqus or Discourse or some other JavaScript-based comment-engine integration. Plus, I want to tweak the template I'm using for the blog's look and feel a little (although keeping it way simple, especially compared to what I had before), so there's likely to be more than a few tweaks here and there. (Again, not really a hard prediction to make, but I always like to close on a prediction that I have a relatively .9 probability chance of hitting.)

Happy Holidays, and thanks for reading!

Your Job is Not to Write Code

Mon, 12 Jan 2015 00:00:00 +0000

An open letter to software developers all across the world.

Dear Software Consultants,

I hate to be the one to break this to you, but you were lied to when we hired you. Your job isn't about code at all.

The interview process was kind of a train wreck (and, yeah, about that, sorry, but we really don't know how to hire folks, so we kind of tested you on what we thought was easily testable--you know, binary trees and all that stuff--even though your day-to-day is about moving an image on a web page over a few pixels), but you're here now, sitting in front of the customer, talking about writing code, so it's pretty easy to understand how you'd be mixed up.

See, the thing of it is this: if you get right down to it, I (your boss) don't really care if you write code or not. The clients don't really care if you write code or not. (They say they do, but hear me out on this.) And the clients' customers and users, they don't really care if you write code or not.

What they care about is the result: Did you make their life better today somehow?

In a consulting gig, we get paid to solve clients' problems. That's it. Nothing else. Frequently, writing code solves those problems, and when that's the case, then boo-yeah, write, test, ship! Everybody is happy.

But when writing code gets in the way of solving those problems, then we have a problem. Which is to say, the clients have a problem, which makes it our problem, which makes it... you guessed it... your problem.

Here's some of what we mean.

You have a MacBookPro with 16GB of RAM and a full tera of SSD disk space, tied to a phat pipe that streams full gigs of data down to the office. That's awesome. That makes you productive.

But that can mask certain problems--if your code thrashes the network for some reason, making dozens or hundreds of round trips to the server, when it all runs on your laptop, that's going to seem totally acceptable. When it goes out into production in a cloud data center that's only as close as the nearest mountain range (which kinda sucks for those living in the Midwest), those minimal trips back and forth on your laptop suddenly turn into long-haul calls across the country. Not such a great experience.

And you're using Google Chrome, with all the latest updates and security patches. Perfect! Except that the clients' users are running Firefox, IE, Safari, and Opera. And probably versions dating a few years back, too, because consumers just don't upgrade their software nearly as often as developers do.

And we--remember this, because this is important--we are in the 'solutions' business. So no matter how amazing your code is, if it doesn't solve the clients' customers' problem, then it is a waste.

You heard about similar kinds of situations in college; the term "gold plating" refers to the effort that engineers will sometimes put in on parts that nobody but themselves will ever know exist. But this is a slightly different problem: You are putting in effort building something that doesn't actually solve the client's customers' problems. So, in many respects, it's a waste.

This is going to require some adjustments.

First off, the question around any particular feature or story is simple: Does this make the client's life better? In many cases, the answer will be, "Yes, but only because it makes their customers' lives better", which is a totally acceptable answer. If the answer is, "I don't know", then your next task is to find out the answer to that. Bear in mind, the client often doesn't know the answer to that. Which is part of the solution we are supposed to be bringing to them, when you think about it.

Second, you need to ensure that whatever you build, it meets the client's customer halfway: Does it run on their target environment? If you don't know what their target environment is, by the way, you have just identified something you need to know, stat.

Third, is there better value you can bring in another way? Is there a simpler change in business process that would obviate the need for this code? Suggest it. Is the client doing something "stupid" that you think should be fixed? Bring it up. Is the client trying to put square peg in round hole? Point it out. But....

Fourth, remember that at all times, the client's right is to ignore us completely. Yes, it's frustrating, but you would feel the same way were it you talking to the doctor or to a mechanic about changes to your life or repairs made to your car. "You're charging me $15,000 for an engine overhaul that YOU thought needed to be done, without checking with me first?!?" is not a great way to collect repeat business.

Look, consultants, it's not all that hard to understand: Most of the time, you will deliver value to our clients by writing code. But that means that your ultimate product here is not the code, but the value.

Your job is not to write code.

Your job is to deliver value. In whatever form that may take.

Which reminds me, while we're here, remember to embrace the the Oath of the Conscientious Programmer:

I swear to fulfill, to the best of my ability and judgment, this covenant:

I will respect the hard-won scientific gains of those programmers and researchers in whose steps I walk, and gladly share such knowledge as is mine with those who are to follow. That includes respect for both those who prefer to keep their work to themselves, as well as those who seek improvement through the open community.

I will apply, for the benefit of the customer, all measures [that] are required, avoiding those twin traps of gold-plating and computing nihilism.

I will remember that there is humanity to programming as well as science, and that warmth, sympathy, and understanding will far outweigh the programmer's editor or the vendor's tool.

I will not be ashamed to say "I know not," nor will I fail to call in my colleagues when the skills of another are needed for a system's development, nor will I hold in lower estimation those colleagues who ask of my opinions or skills.

I will respect the privacy of my customers, for their problems are not disclosed to me that the world may know. Most especially must I tread with care in matters of life and death, or of customers' perceptions of the same. If it is given me to save a project or a company, all thanks. But it may also be within my power to kill a project, for the company's greater good; this awesome responsibility must be faced with great humbleness and awareness of my own frailty. Above all, I must not play at God, and remain open to others' ideas or opinions.

I will remember that I do not create a report, or a data entry screen, but tools for human beings, whose problems may affect the person's family and economic stability. My responsibility includes these related problems, if I am to care adequately for those who are technologically impaired.

I will actively seek to avoid problems that are time-locked, for I know that software written today will still be running long after I was told it would be replaced.

I will remember that I remain a member of society, both our own and of the one surrounding all of us, with special obligations to all my fellow human beings, those sound of mind and body as well as the clueless.

If I do not violate this oath, may I enjoy life and art, respected while I live and remembered with affection thereafter. May I always act so as to preserve the finest traditions of my calling and may I long experience the joy of the thanks and praise from those who seek my help.

You're smart. You'll figure it out. Just remember, code is not the goal. It is merely the means to the end, which is to "make the (client's) pain go away".

(UPDATE: The original author of "Your Job is Not To Write Code" was not happy at my unauthorized derivative work of her post--it violates her no-reprint policy--and has asked that I remove it. I liked the original intent, however, so I am rewriting/editing this post to reflect the same sentiment but without the derivation. My apologies.)

2015 Tech Predictions

Mon, 5 Jan 2015 00:00:00 +0000

It's that time of year again....

First, as always, we revisit the predictions I made last year, and see how well they stacked up. (Because, as I've said before, anybody can make predictions without going back to measure your accuracy; I believe in accountability, even to my own silly blog predictions.)

On 3 January, 2014, I said:

iOS, Android and Windows8 start to move into your car. Ehhhh.... Yeah, sort of, but not nearly as much as I thought they would. There's a few folks experimenting with this, but frankly, after a big splash announcement, the idea pretty much went nowhere. I don't think the idea is "done", per se, but I do think we're looking at a three-to-five year quiescience while the industry figures out what to do with this (and do it in such a way that it doesn't kill somebody). (And no, by the way, I don't count Google's self-driving car as a point for me under this item--that's a wholly different thing.)
Wearable tech hypes up (with little to no actual adoption or innovation). I said, "There'll be absolutely zero reason for anyone to get one for calendar year 2014", and frankly, I stand by that. The Samsung smart watch, the Google Glass, all of the third-party wearables... none of them have really "wow"ed anybody with their raison d'etre. Sure, get text messages on your watch, or place a call (on your phone) from the watch.... But nobody's doing that. I see hardly any of the wearables out in the wild, with the sole exception of those who are wearing a FitBit or something similar. (The Microsoft Band looked like a cool device, but again... nobody's using them in the wild for anything but exercise stat tracking. That's hardly a "gotta have it" kind of reason.)
Apple's gadgets will be more of the same. Basically, I said, "iPhone, iPad, iPod, MacBook, they're all going to be incremental refinements on what we see already." And sure enough, the iPhone got a little bigger, there was no real MacBookPro refresh, and the iWatch is still pretty much vapor at this point (and it's not clear what compelling reason it would have even if it did ship). Sure, the image is there on the Apple website, slated for "early 2015", but frankly... it's a watch. One that I'll have to recharge every night or two. Yawn. Not impressed, so far. I said, "Apple is clearly the market leader, and they are clearly in the grips of the Innovator's Dilemma, and they have no clear challenger (yet) that threatens to dethrone them, leaving them with no reason to shake up the status quo", and I stand behind that 100%.
Android market consolidates further around Samsung and Motorola. I'm sorry, Moto-who? They've all but disappeared from the Android marketplace in a lot of ways, leaving it to be a two-horse race: Samsung/Android vs Apple/iOS. In fact, I think Google is spinning Motorola back off again into its own entity, and the industry collectively... Meh'ed. Given how much angst there was when Google acquired Motorola, I would've thought this would've had a bigger impact, but I think Google figured out fairly quickly that they're not really a hardware company, and don't want to be.
We'll see one iOS release, two minor Android releases, and maybe two Windows8 minor releases. "I can't see the mobile market getting any kind of major surprise this year." Yep. One iOS release (8), two minor Android releases... well, Android 5 isn't a minor release, but until it ships on hardware, it's hard to call it a "major" release, either. I'd almost call it vaporware, except it's the opposite--it's there, anybody can build for it, you just can't... run... it anywhere.
Windows 8/8.1/9/whatever gains a little respect, but not market share gains. Surface 3 is an awesome device--not because it runs Windows8/9/whatever, but in spite of it, for a lot of people. But beyond the Surface, are any tablets/mobiles running Win8/9/whatever getting much traction? I don't see it. Sadly, Windows remains a laptop-bound OS, though the traditional definition of "laptop" is clearly getting a makeover as PC manufacturers try to muscle in on the iPad space from the other direction.
UI/UX emphasis is going to start moving to "alternate" input streams. Yeah, would've been nice, but it's not there yet. Kinect has kind of faded into a whimper. Minority Report-style interfaces are still the stuff of science fiction for the time being, and probably will be, again, for the rest of the decade.
Java-the-language starts to see a resurgence of "mojo". Oh, my, yes. Devoxx this year was quite all abuzz about Java8, and it really feels (to many in the Java community) that Java's got its "buzz" back, at least at the language level. JavaEE is basically shot/fragmented to all hell, but JavaSe is definitely turning a few peoples' eyes in the Java space. Language creators for the JVM that just want to create "a better Java" by taking basic C/C++-style syntax and adding anonymous functions are basically consigning themselves to "this is a fun side research project that I expect to go nowhere" status.
Meanwhile, this will be a quiet year for C#. Yeah... kind of. It's hard to call this one; on the one hand, nothing officially shipped, but on the other hand, Roslyn and its ilk got open-sourced, which is kind of a big deal... if you like dabbling with langauges and/or using language tools directly as part of your toolchain... which really is just a small percentage of developers. I dunno--hard to say.
Functional languages will remain "hipster" tools that most people can't use. Yeah... sigh. I wish this weren't the case, but 2014 was not the "year of the FP'er", by any stretch. Scala is making strides and getting some folks really excited... but those folks were the ones already excited. I don't see it making its way "down" the "programmer hipster ladder", so to speak, to reach "regular" line-of-business developers, and tools like sbt are not helping. (sbt's syntax is really something only its mother could love... and she'd only admit it if it were in the hospital or something.) I get that it's cool that you can use operators like ~>>= in your language, guys, but in a build tool? Seriously? That is not lowering the barrier to entry.
Dynamic languages will see continued growth and success. Well, Node got forked. Dunno what that means. Ruby seemed to actually lose a little steam, and some cynicism and frustration seems to be setting in, particularly among the different "breeds" of Ruby developers. Python still just sort of... hangs out there... with a dedicated community that seems to be getting no larger and no smaller. And Apple's Swift... well, it's not really a dynamic language, per se, but it's not exactly a statically-typed one, either... Oh, hell, I give up. You tell me whether this is continued growth or success. On the other hand, I did say, "I'm becoming more and more convinced that having a language that supports both static and dynamic typing capabilities represents the best compromise between those two poles of software development languages", and boy does that seem to be happening, with Swift just being the largest-splash of the examples so far.
HTML 5 "fragmentation" will start to echo in the industry. Not happening, unless you call the different browsers' different levels of support for HTML 5 "fragmentation". Wait--that's exactly what that looks like. So... is it any worse than before? Folks, any time a website tells you "This site best viewed in..." or "This page optimized for...", that's exactly what that looks like. And it's happening now: Chrome is the most egregious of the lot, with most developers choosing to optimize for it, but the fact that anybody's doing it for any browser tells you that the "standard" really isn't.
"Mobile browsers" become just "browsers". Open your phone browser. Browse to a page. Aside from the smaller screen, is there anything the browser doesn't support that your desktop does? Didn't think so. As a matter of fact, Android finally just up and admitted it, and now ships with Chrome as an app on their Samsung/Android devices. Let the "mobile browser" be done, and it's eulogy written here: "Farewell, mobile
browser, and take your crappy disadvantages with you; we missed you not at all."
"Mobile web" starts a slow, steady slide into irrelevancy. ... sites optimized for "mobile" browsing experiences--which represents a non-trivial development effort in most cases--will start to drop away, mostly due to neglect. Yep.
"Responsive web" becomes the new black. ... we'll see web sites using CSS frameworks (among other tools) to build user interfaces that adjust themselves to the physical viewsizes and input capabilities of the target browser. Yep. This is becoming a necessary skill for web designers, and any new Web-facing application that ships without some level of responsiveness may as well just write at least half of its potential market off.
Microsoft fails to name a Ballmer successor. Helloooooo, Satya Nadella. I'm not a Microsoft Insider, I had no idea who this guy was when he got announced. But so far, I'm pleased with what he's done. Granted, he hasn't been perfect, but he's doing the right things from a company perspective--embracing other languages and platforms, and providing clear direction for the company as a whole... for the most part. Jury's still out, and will be for at least the next half-decade, but he came out clearly wanting to make his mark, and he did. Now let's see how far Microsoft employees will follow him and his guidance, and then let's see how the market does. The first is a lot easier than the second, but the second doesn't happen without the first.
"Programmable Web" becomes an even bigger thing, leading companies to develop APIs that make no sense to anybody. Have you seen Zapier? It's like the DOS batch file language of the Internet. And if that's too much, try IFTTT. It's the... I dunno, what's higher-level than DOS batch files? Seriously, Web APIs are becoming so expected that any day now I expect to see a Web API that allows you to create a Web API on demand. (And we're actually closer than you think...)
Five new single-page JavaScript MVC application frameworks will ship and gather interest. You count--I'm getting tired. AngularJS 2.0 is going to take a different direction from 1.x, which almost makes it a new framework. Famo.us is getting close to ship. Ionic. And that's just off the top of my head. The JavaScript ecosystem is rapidly spinning into that open-source freefall that the Java-Web world turned into, and didn't climb out of for more than a half-decade.
Apple's MacPro machine inspires dozens of knock-off clones. Nope. Not a peep. Oh, well.
Desktop machine sales creep slightly higher. I don't have the numbers handy, and it's 1AM as I write this, so I'm not looking. Even if it's not happening in 2014, I think over the next half-dozen years we're going to see that 80/20 mobile/desktop split, in time.
Dozens of new "cloud" platforms will be introduced, and most of them will remain entirely irrelevant behind the "Big Three". Well, dozens are coming out, depending on which flavor of "*aaS" you consider to be "Cloud", and what seems to be happening is that the Big Three are picking them up and incorporating them as part of their offering (a la what happened with FireBase). This is probably only going to rise--Heroku got picked up by SalesForce somewhere along the way, for example, and I fully expect Parse to do the same before long. As for the big players (Oracle, HP, etc) that aren't one of the Big Three... yeah, they'll quietly fade and die.
We will never see any kind of official announcement, much less actual working prototypes, around Amazon's "Drone Delivery" program ever again. Anybody? Anybody? Lots of discussion around drones, but nothing from Amazon, and certainly nothing shipped.

Overall.... shrug About par for the course, 50/50 by my count.

So what do I see for 2015? Here goes:

"Big data", "Big data", "Big data". You will get sick of this phrase. What I didn't say last year was anything about big data, and frankly, the hype machine is not even remotely close to being done with this. EVERYTHING will be made into a "big data" play before 2015 is over, to the point where nobody knows what "big data" really means anymore... except for the people quietly doing it.
"Internet of Things". You will get sick of this phrase, too. Look, let's just call it right now: IoT is clearly in the overestimate-in-5-years/underestimate-in-10-years Bill Gates quote territory. We have no idea what IoT will end up looking like, practically speaking, and yet billions will be spent on people who claim to know exactly that. Save your money. Or, if you really want to give a smart-sounding consulting some cash to tell you what you already know, send it my way, and I'll show up at your place of business, look wise and distinguished, and tell your boss, "It depends" in fifty pages and with two-hundred PowerPoint slides. Everybody wins.
"Internet of Medicine" or "Big Med". I'm calling this one right here: with all the health-related big data plays going on right now, it's only a matter of time before some chirpy journalist or press agent coins this. It's a fascinating opportunity--there's a lot of really interesting and cool ideas around the intersection of big data-about-your-health, the wearables world, medical AI/data-analysis, and mobile devices, and just about every entrepreneur and VC is going after it. So within the next year or two, many of those are going to ship in some form, and then... I have no idea, but it'll be big/catastrophic, whatever happens.
"Tech bubble" becomes a "thing". Internet- and technology-related firms are getting ridiculous valuations, and those of us who were in the industry in the years leading up to 2000 know what this means. I'm sorry, but when a technology-training company gets valued at $1 billion, which is more than some Fortune-5000 firms that've been around for a half-century or more.... this is clearly "bubble" territory. Trust me, I don't want this bubble to pop, particularly since I'm supposed to be releasing a few courses through them, but I remember when everybody was certain that "Pets.com" was a multi-million Web property, too... and I remember when I worked for a company that thought technology training was a "recession-proof" industry. These numbers are ridiculous, and it's only going to take one high-value collapse to start pulling the rest of the house of cards down. Before 2020, you mark my words. In 2015, though, the warnings and arguments against those warnings are only going to intensify.
C# and Java will both make big announcements. C# 6 will probably ship somewhere in 2015 (they called the product "Visual Studio 2015", after all), and Java9 will probably start to make some news towards the end of 2015, though I don't expect it to ship. Those who care will take note, those who don't, won't. Really, we're kind of past the point where either of those languages are going to be interesting to anyone who's not already in that space.
Go is going to either take off, or crash and burn. Go seems to be approaching an inflection point, and while I'm not smart enough to know what's going to happen there, I think this will be a year of decision for them.
Microsoft acquires Xamarin. Microsoft seems to be coyly flirting with the idea, and really, it makes a ton of sense for them to do so: it gives them a way to be relevant on the two major mobile platforms without abandoning .NET or their own platform. Really, the only question that remains is what kind of cash it would take to do the buyout, and does Microsoft have that money available to it after its rather (IMHO) ill-advised acquisitions of the past (cough Nokia cough).
Amazon just quietly keeps churning. Really, this seemed like a pretty quiet
year for Microsoft's cross-Sound neighbor, and I don't know that'll change in 2015. Kindle seems pretty successful, but the Fire doesn't seem like it's gone far as a full-fledged mobile device. Amazon is fighting some brush fires on its borders over licensing and royalties, and that's probably distracting them in some major ways. (They're also developing a reputation as a firm that burns employees out fast, which also isn't helping, I imagine.) I don't think 2015 is a year of huge developments for them, but in a lot of ways, they don't need them--AWS is, as near as I can tell, the gold standard of "cloud", and as long as they just keep pace with whatever Azure comes out with, they're in good shape there.
Google continues to throw sh*t against the wall, looking for their Next Big Thing. What I mean by that is this: Among wine snobs, there is a story (possibly apocryphal) of a wine-tasting expert who had a wall in his house painted a very particular shade of white. Then, when tasting a red wine, he would throw the contents of the glass against that wall, to observe the color and the way the wine beaded and ran down the wall as part of his evaluation. Frankly, that kind of feels like what Google is doing: throwing idea after idea "against the wall" of public opinion, trying to see what stands out and looks great and garners interest. If I didn't know better, I'd say they're desperately trying to find a new line of revenue beyond AdWords, just as Microsoft a decade ago was desperately trying to find a new line of revenue beyond Windows. Expect a lot of announcements, a lot of "beta"s, and none of it with any kind of realistic or even well-planned business model behind it--including the Google Car.
Web use on mobile devices decreases in favor of apps. This is going to happen whether the public wants it or not, because companies have figured out that it behooves them to have you "trapped" inside their app (where they can control all the content) rather than on their website. More and more websites are going to try and redirect you to inside their app, rather than allow you to casually browse on their site, because then they think they "own" your eyeballs. The only way this changes is if/when some firm gets crushed in the court of public opinion by doing something really stupid... and that won't happen in 2015. Wait for it in 2016.
Hipster "Uber for X" apps will be all the rage. Every technology event is going to be filled with 20- and 30-somethings who are convinced that the world wants/needs an app that will allow customers to get X (service or good) delivered right to wherever-they're-standing, without doing one iota of business research as to whether that model makes any sense anywhere outside of New York or San Francisco. They will get millions in VC, confidently proclaim themselves the next Facebook, and then implode. (This might actually be what starts the tech bubble implosion.)
Mark Zuckerberg grows up a little. Remember his "You're washed up when you're 30" crack? He turns 30, and I suspect he doesn't think he's "done" quite yet. As he tries to become more of a "technology statesman" (which is really the next step for him), however, he's going to get caught up on that remark over and over again from people who just want to see him fail or be embarrassed, until he either admits, "Yeah, OK, I was wrong" or gets so frustrated he buys a small island in the South Pacific, screams "I hate you all!" and flees the country. (Hopefully he remembers to update his status before he goes.)
Larry Ellison buys a sports team. Yeah, he's got a yacht, but now Ballmer owns the Clippers. Paul Allen has the World Champion Seattle Seahawks. If either of the Google boys puts in for the Buffalo Bills or St Louis Rams, this is a given.... and I think he buys the Golden State Warriors.
Perl makes one final gasp at relevancy, fails, and begins to decompose. C'mon, when's the last time the Perl community made a "This time we mean it! Perl 6 is going to ship!" announcement? They're like the Daikatana or Duke Nukem Forever of programming languages by this point. If you're still using Perl, just accept fate, switch to Python, Ruby, or even Node. Nobody can take your memories from you.

... annnnnnnd I'm spent.

Happy New Year, everyone. May your 2015 be everything you want it to be, and a few more things beyond that. Be kind to yourself, generous to others, humble in the spotlight and lavish in your praise of those around you. Cheers!

Recruiting the Right Way

Wed, 3 Dec 2014 00:00:00 +0000

There's been a fair amount of conversation around how to recruit software developers effectively. I've participated in some of it. But just today, a blog post crossed my desk(top) and finally prompted me enough to get around and blog about it.

In "Resumes Suck. Here's the Data", Aline Lerner talks about a scientific process she used to actually try to gauge the efficacy of resumes in the hiring process. (I probably should say "pseudo-scientifically", only because while it looks pretty legit to me, and frankly it's more legit science than most companies will engage around hiring, it probably has a few statistical/scientific holes in it that could lead to some misleading data; that said, it's better than the traditional answer of "Well, I know how to interview people, so....", which to me is an outright so-you-buy-into-your-own-bullshit answer.)

Her conclusion?

... it may not be a matter of being good or bad at judging resumes but rather a matter of the task itself being flawed — at the end of the day, the resume is a low-signal document. ... Very smart people, who are otherwise fantastic writers, seem to check every ounce of intuition and personality at the door and churn out soulless documents expounding their experience with the software development life cycle or whatever... because they’re scared that sounding like a human being on their resume or not peppering it with enough keywords will eliminate them from the applicant pool before an engineer even has the chance to look at it.

Yeah. I can relate.

If you combine her findings here with my anecdotal evidence around how most companies' interview process is fundamentally broken (which Google seems to have confirmed), what you come to realize is that for most companies, the entire recruiting process is just one giant coin-flip based on equal parts intuition, mutual fakery with the best of intentions ("This company is great!" "I'm an amazing candidate!"), and almost no science--or accountability, or independent verification of the results, or testing--involved.

So here's what we do at iTrellis (the company where I'm currently the CTO):

Our recruiters are instructed to look at much more than the resume. Quite frankly, if you have a Github profile, a BitBucket profile, a blog, or God-help-us-all just your own domain, we take all of that into account. As Aline points out, resumes suck. No resume has ever accurately captured what somebody did on the job: either they exaggerate for effect, or they totally misrepresent (both positively and negatively) what they did. And quite frankly, any employment history, technology, project or customer you helped more than ten years ago is really not going to matter much, unless that's exactly what we're hiring you for. But forking a project to play with it, contributing a pull request, commenting on a blog about something interesting you discovered... folks, that's a more accurate representation of who you are than any resume you could possibly hand me. And if you have your own domain, and you built your own blog... Nice.
Once identified, our recruiter calls you to make sure you pass the Bozo Test. Seriously. The phone call is a pleasant ten- to thirty-minute chat to find out if you have a pulse, you're actually looking for a job, and that you're not some wingnut out of left field who thinks that C++ is a better grade than a B-. More importantly, they are doing no technical evaluation of the candidate. Seriously, think about that: you're going to ask somebody whose specialty is Human Resources to make a call on whether this person has the chops to write code? All they're going to be able to do is look to see if they have the credentials (degree, certifications, etc), and we all know how accurate those are. No way. Don't ask me to put together a pension program, and don't ask the HR guy to evaluate a technologist.
Once you've agreed that you're looking for a job, and you passed the Bozo Gate, we tell you to go write some code. Seriously. No, you're not going to talk to any of our people. No, there's no "preliminary screening". We have code to write. And frankly, the best screen we can put in place to determine if you have technical chops is to have you write some code. Specifically, we ask you to spend no more than 10 hours on one of these projects. Notice how there's both Java and C# versions? If you wanted to do it in Ruby or Node, I'd be OK with that. Hell, if you wanted to take a wild stab in the dark and do it in a language you've never written before, I'd take that, too. (Shows moxie. Gumption! Dare I say, chutzpah!) We don't even care if you finish it. We just want to have a pile of code that you wrote specifically for this purpose, because then we can use it (free of any IP restrictions) for the next stage.
Once you've written the code, you send it to us. Zip it up, or better yet, point us to the pull request, or, even better, your version of the project on your own source control system public server of choice. This is where we evaluate your technical chops: what tools did you use, what results did you get, how closely did you adhere to our requests/"requirements", and so on. Did you ask us questions around the ambiguities? Did you make assumptions about what we wanted without checking with us first? Did you remember to unit test? (We said, right there in the README, "Unit tests are very important to us". If you didn't unit test, you gonna have some 'splainin' to do....) And so on.

By the way, take note: other than ten to thirty minutes of our recruiters' time for the phone call (and whatever time the recruiter spent looking at your profile/resume/whatever, and I've told them to keep that bar pretty low!), no iTrellis time has been consumed as part of this process so far. Your code is what we care about at this point, why should we even talk to you yet?
We evaluate your code sample. Can we build it? Can we run it? Can we run the tests? This step is pretty straightforward.
Assuming your code passes muster, we bring you in. But not to interview--oh, no. The human brain is a deeply flawed instrument, and we don't want to give in to its accidental prejudices and flaws. No, instead, we want you to "pair" with three of our existing employees, over your code. This has a couple of parts to it:
- We want to find out if you fit with our existing staff. Hey, let's face it, you could be the best coder in the world, but if you manage to piss off all three of your pairs, you are Not Going To Work.
- We want to find out if we can put you in front of a client. We are a consulting company. We can't afford to have people that can't stand the light of day.
- We want to see how mentally nimble you are.
And by that last part, what I do is I ask my three pairs to specifically work with you on three different things, from a list of things like:
- "Can you explain your code to me? Why did you choose to do this? Why didn't you do that?" You need to be able to explain your decisions without getting defensive or angry. I may sometimes ask the pair-er to deliberately disagree with the decision, to see what your reaction is. Disagreement happens. How you handle it is what matters.
- "The customer has now changed their requirements; they need the project to do (whatever) different." The ToDo list now needs "categories". The ToDo list now needs to be able to sort by dates. Whatever. How does this break your domain model? More importantly, what changes do you identify that need to be made? Let's write some of that code now. How does this change your tests?
- "We now need to store to a database" or "to a different database" or "to several different databases (sharding)". Ideally, we'll pick something you've never seen before. I want to see how you'll approach using something or doing something you've never done before: where do you start? How do you start?
- "How would you put a DevOps pipeline around this thing?" What's your tools, how would you relate it back to stories, the whole nine yards.
- "Oh, my! We need to secure this thing, stat!" What's insecure? What's already secure? Where do you begin doing the security analysis?

... and so on. The goal here is to push your boundaries a little, and yes, maybe pile on a little stress to see how you react. But at every step, something that is what you'll actually have to deal with, every day, when working for a consulting company.

Assuming you have the chops we're looking for, we hire you. This part is pretty simple, if you ask me. But my CFO, now he takes over. :-)

That's our recruiting process.

By the way, I say all of this here because none of this is Secret Sauce(TM). We want the entire process to be entirely transparent. No "hidden agenda". Yes, I'll admit, right up front, what each stage is designed to do, and why. I'll even tell you, right from the beginning, if you ask. Yes, it's a little longer than your average "one-hour interview with the hiring manager"; but if you notice, the time commitment on the iTrellis side is 30 minutes (recruiter call) + code evaluation time, probably not more than 30 minutes (my or an engineer's time) + pairing time, not more than 30 - 45 minutes per pair (of an engineer's time), which is actually pretty comparable to your average "stand at the whiteboard and code" interview time. Yes, you've put in 10 or so hours into the coding project--but it's coding, and tell me right now, which would you rather do: a full eight-hour "interview loop" in front of people at a whiteboard, or snuggled up with the cat in front of the TV, gloriously hacking using your favorite IDE and your favorite libraries with your favorite tunes blasting through your favorite headphones?

Yeah, I thought so. (So do most of the candidates who come through, whether we accept them or not. Of course, they could be trying to just brown-nose me, hard to say. )

How do I know it works? A couple of reasons. One, we've gotten some great hires out of it. (OK, yes, that could be just luck--random luck could give you that, too.) Two, we've had no "bozo" hires. (OK, we've only been around a year, so again, we could just be lucky.) Three, we've had more than a few candidates that looked great on paper either walk away from the coding challenge entirely (buh-bye!) or turn in some pretty weak results. Sure, they might've been great, but frankly, if your audition tape is bad, your band is not going to get gigs, either.

But mostly, I have faith in this process because it lends itself well to unit-testing; at some point (haven't done it yet, but this is in my plans for 2015), I'm going to slip a few "known quantities" into the system and see how they do: I'm going to slip a complete newbie with great confidence and a few buzzwords into the process to see if they can get all the way through, and I'm going to slip an amazing coder with a crappy cover resume in as well, just to see how we do. And, by the way, note that I'm not part of the recruiting process: as soon as I could get myself out of the way of it, I did, because I didn't want to create a company of Ted-clones. (Which is also something that we tend to do intuitively: prefer the familiar, and nothing is more familiar than the self. This is part of the reason why a team of white men tends to stay a team of white men.)

If you've made it this far, thanks for reading. Two more important points, then I'm out:

Did I mention we're hiring? We're hiring. At this exact moment we're looking for Java folks who love writing automated tests and working in the DevOps pipeline; and we're hiring for people who have good C# skills and want to work with a client who's building a modern (DurandalJS, Azure, etc) web app. Contact our recruiters or contact me if you're interested.
This process is licensed "Creative Commons" (if a process can be so licensed, that's just to make the lawyers happy), so if you want to take it and use it for your own company's hiring process, go for it. Nothing would make me happier than for the industry to reject the coding-at-the-whiteboard nonsense for something like this or derived from this. (Because, in truth, this is in turn derived from lots of different hiring processes I went through over my 25 years.)

Peace, out, y'all.

(UPDATE 2021: Hey, I no longer work for iTrellis, but I think the process is still interesting. I've updated the post accordingly.)

2014 Tech Predictions

Fri, 3 Jan 2014 00:00:00 +0000

Here we go again: the annual review of last year's predictions, and a set of new ones for the new year.

2013 Retrospective

Without further ado, first we examine last year's Gregorian prognostications:

    <ul>
      <li>
        <p>
          <b>THEN:</b>
          <i>"Big data" and "data analytics" will dominate the enterprise landscape.</i>
        </p>
        <p>
          <b>NOW:</b> Yeah, it was a bit of a slam dunk breakaway kind of call, but it clearly

counts. Vendors and consulting companies were climbing all over themselves to talk
about "big data", and startups basing their existence on gathering, analyzing, displaying
and (theoretically) offering insight from "big data" were all the rage in the startup
community, such as local startup Predixion (CTO'ed
by a buddy of mine). If you live anywhere in the Pacific Northwest, chances are there's
a similar kind of startup within spitting distance of you right now. 1-0.

THEN: NoSQL buzz will start to diversify.

NOW: It didn't happen quite as much as I'd expected, but the various vendors are, in fact, starting to use terms other than "NoSQL" to define themselves. In particular, we're seeing database vendors (MongoDB, Neo4J, Cassandra being my principal examples) talking about being a "document database" or a "graph database" instead of being a "NoSQL" database, though they're fairly quick to claim the NoSQL tag when it comes to differentiating against the traditional relational database. Since I said "start" to diversify, I'm going to take the win. 2-0.

THEN: Desktops increasingly become niche products.

NOW: Well, this one is hard to call. Yes, desktop sales have plummeted, but it's hard to see what those remaining sales are being used for. I will point out that the Mac Pro, with it's radically-different internal construction, definitely puts a new spin on the desktop, but I'm not sure that this counts. Since I took the benefit of the doubt on the last one, I'll forgot it on this one. 2-1.

THEN: Home servers will start to grow in interest.

NOW: I wish I had sales numbers to go with some of this stuff, as hard evidence, but the fact that many people are using their console devices (XBox, XBoxOne, PS3, PS4, etc) as media servers means I missed the boat on this one. I think we may still see home servers continue to rise, but the clear trend has been to make the console gaming device into a server, and not purchase servers on their own to serve as media servers. 2-2.

THEN: Private cloud is going to start getting hot.

NOW: Meh. I see certain cloud vendors talking about private cloud, but for the most part the emphasis is still on public cloud. 2-3. Not looking good for the home team.

THEN: Oracle will release Java8, and while several Java pundits will decry "it's not the Java I love!", most will actually come to like it.

NOW: Well, let's start with the fact that Java8 actually didn't ship this year. And even that, what I would have guessed would be a hugely-debated and hotly-contested choice, really went by without much fanfare or complaint, except from some of the usual hard-liner complaint sources. Which means one of two things: either (a) it's finally come to pass that most of the people developing on top of the JVM really don't care about the Java language's growth anymore, or (b) the community felt as Oracle's top engineering brass did, that getting this release "right" was far better than getting it out on the promised deadline. And while I agree with the (b) group on that, it still means that the prediction was way off. 2-4.

THEN: Microsoft will start courting the .NET developers again.

NOW: Quite frankly, this one got left in dust almost the moment that Ballmer's retirement was announced. Whatever emphasis the company as a whole might have put into courting .NET developers back into the fold was immediately shelved, at least until a successor comes in to take Ballmer's place and decide what kind of strategy the company as a whole will pursue. Granted, the individual divisions within Microsoft, most notably DevDiv, continue to try and woo the developer community, but that was always going to be the case. However, the lack of any central "push" from the company effectively meant that the perceived "push" against .NET in favor of WinRT was almost immediately left behind, and the subsequent declaration of the Surface's failure (and Surface was by far the most public and prominent of the WinRT-based devices) from most corners meant that most .NET developers who cared about this breathed a sigh of relief and no longer felt those Microsoft cyclical Darwinian crosshairs (the same ones that claimed first C programmers, then C++ programmers, then COM programmers) on their back. Still, no points. 2-5.

THEN: Samsung will start pushing themselves further and further into the consumer market.

NOW: And boy, howdy, did they. Samsung not only released several new versions of their various devices into the market, but they've also really pushed their consumer electronics in other form factors, too, such as their TVs and such. If there is a rival to Apple in the consumer electronics department, it is clearly Samsung, and the various court cases and patent violation filings are obvious verification of that. 3-5.

THEN: Apple's next release cycle will, again, be "more of the same".

NOW: Can you say "iPhone 5c", and "iPad Air", boys and girls? Even iOS7 is basically the same OS, with a skinnier font and--oh, wow, innovation!--nested folders. 4-5.

THEN: Visual Studio 2014 features will start being discussed at the end of the year.

NOW: Microsoft tossed me a major curve ball with their announcement of quarterly releases, and the subsequent release of Visual Studio 2013, and as a result, we haven't really seen the traditional product hype cycle out of the Microsoft DevDiv that we're used to. Again, how much of that is due to internal confusion over how to project their next-gen products out into the world without a clear Ballmer successor, and how much of that was planned from the beginning isn't clear, but either way, we ain't heard a peep outta nobody about C# 6 at all in 2013, so... 4-6.

THEN: Scala interest wanes.

NOW: If anything, the opposite took place--Typesafe, Scala's owner/pimp/corporate backer, made some pretty splashy headlines within the JVM community, and lots of people talked a lot about it in places where Scala wasn't being discussed before. We can argue about whether that indicates just a strong marketing effort (where before Typesafe's formation there really was none) or actual growth in acceptance, but either way, I can't claim that it "waned", so the score becomes 4-7.

THEN: Interest in native languages will rise.

NOW: Again, this one is hard to judge. There's been some uptick in interest in those native languages (Rust, Go, etc), and certainly there's been some interesting buzz around some kind of Phoenix-like rise of C++, but none of it has really made waves within the mainstream that I can see. (Then again, I don't really spend a lot of time in those areas where native languages would have made a larger mark, so this could be observer's contextual bias at work here.) That said, more native-based languages are emerging, and certainly Apple's interest and support in LLVM (which, contrary to it's name, is not really a "virtual machine", per se) can be seen as such, but not enough to make me feel comfortable saying I got this one right. 4-8.

THEN: Hardware is the new platform.

NOW: Surface was a bust. Chromebooks hardly registered on anybody's radar. Dell threw out an arguable Surface-killer tablet, but for most consumer-minded folks it never even crossed their minds, it seems. Hardware may be the new platform, and certainly we're seeing a lot of non-x86-based machines continuing their race into consumers' hands, but most consumers don't think twice about the hardware as much as they do the visible projection of that hardware choice, in the operating system. (Think about it this way: when you go buy a device, do you care about the CPU, or the OS--iOS, Android, Windows8--running it?) 4-9.

THEN: APIs for lots of things are going to come out.

NOW: Oh, my, yes. More on this later, but for now... 5-9.

Well, with a final tally of 5 "rights" to 9 "wrongs", clearly my 2013 record was about as win-filled as the Baltimore Ravens' 2013 record. *sigh* Oh, well, can't win 'em all every year, right?

2014 Predictions

Now, though, let's do the fun part: What does Ted think 2014 has in store for us geeky types?

iOS, Android and Windows8 start to move into your car. Audi has already announced this. Ford announced this last year with their SDK release. Frankly, with all the emphasis on "wearable tech" and "alternative tech", this seems a natural progression, considering how much time Americans, at least, spend time in their car. What, exactly, people will want software developers to do with this capability remains entirely unclear to me (and, it seems, to everybody else, given the lack of apps for the Ford SDK so far), but auto manufacturers will put it into their 2015 models just because their competitors are starting to, and the auto industry is one place were you cannot be seen as not keeping up with the neighbors.
Wearable tech hypes up (with little to no actual adoption or innovation). The Samsung Smart Watch is out, one of nearly a dozen models introduced in 2013. Then there was Google Glass. And given that the tech industry is a frequent "hype it before we even barely know it's going to work" kind of place, this one seems like another fast breakway layup kind of claim. Note that I fully expect that what we see offered will, in time, be as hip and as cool as the original Newton, meaning that these first iterations will be stumblin', fumblin', bumblin' attempts to try and see what anybody can do with these things to make them even remotely useful, and that unless you like living on the very edge of techno-geekery, there'll be absolutely zero reason for anyone to get one for at least calendar year 2014.
Apple's gadgets will be more of the same. Same one as last year: iPhone, iPad, iPod, MacBook, they're all going to be incremental refinements on what we see already. There will be no AppleTV, there will be no iWatch, there will be no radical new laptop-ish thing. Apple is clearly the market leader, and they are clearly in the grips of the Innovator's Dilemma, and they have no clear challenger (yet) that threatens to dethrone them, leaving them with no reason to shake up the status quo.
Android market consolidates further around Samsung and Motorola. The Android consumer market has slowly been collapsing around those two manufacturers, and I don't see any reason for that trend to change. Yes, other carriers will continue to offer Android on their devices, and yes, other device manufacturers will continue to put Android on their devices, and yes, Android will continue to appear on things other than tablets and phones, but as far as the consumer electronics world goes, the Android market will be classified as Samsung, Motorola, and everybody else.
We'll see one iOS release, two minor Android releases, and maybe two Windows8 minor releases. The players are basically set, the game plans are already in play, and nobody appears to have any kind of major game-changing feature set in the wings. 2014 will be a year of minor releases, tweaks to the existing systems and UIs, minor software improvements, and so on. I can't see the mobile market getting any kind of major shock or surprise this year.
Windows 8/8.1/9/whatever gains a little respect, but not market share gains. Windows8 as a tablet OS has been quietly gathering some converts, particularly among those who didn't find themselves on the WindowsStore-only SurfaceRTs, and as such, I think the "Windows line" will begin to gather more "critics' choice" kinds of respect, but that's not going to translate into much in the way of consumer sales. Unfortunately for the Microsoftians, Windows as of yet doesn't demonstrate any kind of compelling reason to choose it over the other two market leaders (iOS and Android), and until that happens, Windows8, as a device OS, remains a distant third and always will.
UI/UX emphasis is going to start moving to "alternate" input streams. Microsoft's Kinect has demonstrated that gesture is a viable input technology. Google Glass demonstrated that eyeballs can drive a UI. Voice commands are making their way into console gaming/media devices (XBox, etc). This year, enterprise and business developers, looking for ways to make a splash and justify greater research budgets, are going to start experimenting with how those "alternative" kinds of input can be utilized in non-gaming scenarios. Particularly when combined with the rise of automobiles offering programmable SDKs/platforms (see above), this represents a huge, rich area for exploration.
Java-the-language starts to see a resurgence of "mojo". Java8 will ship this year--not even God Himself could stop that at this point. And once it does, Java-the-language will see a revitalization as developers who've been flirting with Groovy, Scala, Clojure, and other lambda-supporting languages but can't use them on the job start to bring those ideas into Java APIs. Google's already been doing this with Guava, but now many of those ideas--already percolating in some libraries--will explode into common usage.
Meanwhile, this will be a quiet year for C#. The big news coming out of Microsoft, "Roslyn", the "compiler-as-a-service" rewrite of the C# and Visual Basic compilers, won't be of much use to most developers on a practical level, and as a result, this will likely be a pretty quiet year for C# and VB.
Functional languages will remain "hipster" tools that most people can't use. Haskell remains far out of reach for most developers, and that's the most approachable of the various functional languages discussed. (Don't even get me started on Julia, Pure, Clean, or any of the others.) As much as I wish to the contrary, this is also likely to remain true for several of the "hybrid" languages, like Scala, F#, and Clojure, though I do think they will see some modest growth as some of the upper-echelon development community begins to grok them. Those who understand them will continue to do some amazing things with them, but this is not the year I would suggest starting a business with anything "functional" as part of its business model, because not only will it be difficult to find developers who can use those tools, but trying to sell developer-facing things with those tools at the core will find a pretty dry and dusty market.
Dynamic languages will see continued growth and success. Ruby doesn't look to be slowing down, Node/JavaScript only looks to get more hyped, and Objective-C remains the dominant language for doing iOS development, which itself doesn't look to be slowing down. More importantly, I think we're going to start to see a rise in hybrid "static/dynamic" languages, wherein developers can choose (based on the way they write their code) compiler enforcement as they wish. Between the introduction of "invokedynamic" in Java7 (and its deeper use in Java8), and "dynamic" in C# getting some serious exercise in the Oak framework, I'm becoming more and more convinced that having a language that supports both static and dynamic typing capabilities represents the best compromise between those two poles of software development languages. That said, neither Java nor C# "gets it all the way right" on this topic, and I suspect that somewhere out there, there's a language hacker who's got a few ideas that he or she will trot out and make us all go "Doh!"
HTML 5 "fragmentation" will start to echo in the industry. Unfortunately, HTML 5 is not the same thing to all browsers, and those who are looking to HTML 5 as a way to save them from platform differences are going to start to feel some pain. That, in turn, is going to lead to some backlash as they are forced to deal with something they thought they were going to be saved from.
"Mobile browsers" become just "browsers". With the explosive growth of devices (tablets and phones) and the explosive growth of the capabilities of those devices (processor(s), memory, and so on), the need for a "crippled" or "low-end-optimized" browser has effectively gone the way of the Dodo bird. As a result...
"Mobile web" starts a slow, steady slide into irrelevancy. ... sites optimized for "mobile" browsing experiences--which represents a non-trivial development effort in most cases--will start to drop away, mostly due to neglect. Instead...
"Responsive web" becomes the new black. ... we'll see web sites using CSS frameworks (among other tools) to build user interfaces that adjust themselves to the physical viewsizes and input capabilities of the target browser. Bootstrap is an obvious frontrunner here for building said kinds of user interfaces, but don't be surprised if a number of other CSS and JavaScript frameworks to achieve the same ends start to spring up.
Microsoft fails to name a Ballmer successor. Yeah, this one's a stretch. It's absolutely inconceivable that they wouldn't. And yet, in all honesty, I can't see the Microsoft board finding somebody that meets Bill's approval from outside of the company, and I can't imagine anyone inside of the company who isn't somehow "tainted" by the various internecine wars that have been fought since Bill's departure. It is, quite frankly, a mess, and I don't know that it'll be cleaned up before this time next year. It would be a horrible result were that to be the case, by the way, but... *shrug* I dunno. Pretty clearly, whomever it is, is going to have a monumental task in front of them.
"Programmable Web" becomes an even bigger thing, leading companies to develop APIs that make no sense to anybody. Right now, as we spin up 2014, it's become the fashionable thing to build your website not as an HTML-based presentation layer website, but as a series of APIs. For some companies, that makes sense; for others, though, that is going to be a seductive Siren song that leads them to a huge development effort for little payoff. Note, I think almost all companies have interesting data and/or resources that can be exposed as APIs that would lead to some powerful mashups--I'm not arguing otherwise. But what I think we're about to stumble into is the cargo-culting blind obedience to the letter of the idea that so many companies undertake when a new concept hits the industry hard, as "Web APIs" are doing now.
Five new single-page JavaScript MVC application frameworks will ship and gather interest. For those of you who know me from the Java world, remember the 2000s and the huge glut of open-source Web frameworks that led us all into analysis paralysis for a half-decade or more? I see absolutely no reason why the exact same thing isn't already under way in the JavaScript Web framework world, with the added delicious twist that in the JavaScript world, we can do it on BOTH the client AND the server. Let the forking begin.
Apple's MacPro machine inspires dozens of knock-off clones. When the MacBook came out, silver-metal cases with chiclet keyboards suddenly appeared all over the PC clone market. When the MacBook Air came out, suddenly thin was in. What on Earth makes us think that the trashcan-sized MacPro desktop/server isn't gong to have exactly the same effect?
Desktop machine sales creep slightly higher. Work this through with me before you shoot it down out of hand: Tablet sales are continuing to skyrocket, and nothing seems to change that. But people still need to produce stuff (reports, articles, etc), and that really requires a keyboard. But if tablets are easier to consume data on the road, you're more likely to carry your tablet instead of your laptop (and most people--myself wildly excluded--don't like carrying more than one or at most two devices with them). Assuming that your mobile workload is light enough that you can "get by" using your tablet, and you don't want to carry a laptop *and* a tablet, you're more likely to leave your laptop at home or at work. Once your laptop is a glorified workstation, why pay that added premium for a laptop at all? In other words, I think people are going to start doing this particular math, and while tablets will continue to eat away at the "I need a mobile computing solution" sales, desktops are going to start to eat away at the "I need a computing solution for my desk" sales. Don't believe me? Look around the office at all the workstations powered by laptops already, and then start looking at whether those laptops are actually being used as laptops, and whether that mobility need could, in fact, be replaced by a far lighter tablet. It's a stretch, and it may not hit in 2014, but I really think that the world is going to slowly stratify into an 80/20 split of tablets and desktops.
Dozens of new "cloud" platforms will be introduced, and most of them will remain entirely irrelevant behind the "Big Three". Lots of the big players are going to start tossing out their version of a cloud platform if they haven't already (HP, Oracle, IBM, I'm looking at you), and smaller players are going to start offering "cloud" platforms of their own (a la Rackspace), but fundamentally, the cloud will remain a "Big Three" place: Amazon's AWS, Microsoft's Azure, and Google's Cloud Platform.
We will never see any kind of official announcement, much less actual working prototypes, around Amazon's "Drone Delivery" program ever again. Sure, Jeff made a splash when he announced it. Sure, it resonates with the geek crowd. Sure, it seems like a spiffy idea on paper. Do you have any idea of how much infrastructure and overhead (and potential for failure that has nothing to do with geeks deploying "anti-drone defenses") would be involved? No way. What's more, Amazon is not really in the shipping business (as the all-but-failed Amazon "deliver groceries to your front door" program highlights), but in the "We'll sell it to you and ship it through somebody else" business. It's a cool idea, but it'll never, ever, EVER, see the light of day.

As always, thanks for reading, and keep this channel open--I've got some news percolating
about my next new adventure that I'm planning to "splash" in mid-January. It won't
be too surprising, but it's exciting (at least to me), and hopefully represents an
adventure that I can still be... uh... adventuring... for years to come.

On (Free) Speaking

Thu, 12 Dec 2013 00:00:00 +0000

Remember when I posted about speaking for free at conferences? And everybody got so upset? Because, you know... community!

Somebody tell me how this is any different.

When a conference chooses not to offer its speakers even a modest stipend beyond expenses (and let's not even begin to discuss the conferences who don't bother covering expenses), they are essentially asking the speaker to do all that work for free. Just like asking a musician to use his music for free. And some musicians will do it for "the exposure", because they want the gig credit to list on their website more than they want the money. But at some point, the band needs to be paid, or else they break up and great music goes unplayed.

Just sayin'.

Speakers, stipends, and expenses

Mon, 26 Aug 2013 00:00:00 +0000

In the past, I've been asked about my thoughts on conferences and the potential "death" of conferences, and the question came up again more recently in a social setting. It's been a while since I commented on it, and if anything, my thoughts have only gotten sharper and clearer.

On speaking professionally

When you go to the dentist's office, who do you want holding the drill--the "enthused, excited amateur", or the "practiced professional"?

The use of the term "professional" here, by the way, is not in its technical use of the term, meaning "one who gets paid to perform a particular task", but more in a follow-on to that, meaning, "one who takes their commitment very seriously, and holds themselves to the same morals and ethics as one who would be acting in a professional capacity, particularly with an eye towards actually being paid to perform said task at some point". There is an implicit separation between someone who plays football because they love it, for example, going out on Sunday afternoons and body-slamming other like-minded individuals just because of the adrenaline rush and the male bonding, and those who go out on Sunday afternoons and command a rather decently-sized salary ($300k at a minimum, I think?) to do so. Being a professional means that not only is there a paycheck associated with the activity, but a number of responsibilities--this means not engaging in stupid activity that prevents you from being able to perform your paid activity. In the aforementioned professional athlete's case, this means not going out and doing backflips on a dance floor (ahem, Gronkowski) or playing some other sport at a dangerous level of activity. (In the professional speaker's case, it means arranging travel plans to arrive at the conference at least a day before your session--never the day of--and so on.)

For a lot of people, speaking at an event is an opportunity for them to share their passion and excitement about a given topic--and I never want to take that opportunity away from them. By all means, go out and speak--and maybe in so doing, you will find that you enjoy it, and will be willing to put the kind of time and energy required into doing it well.

Because, really, at the end of the day, the speakers you see in the industry that are very, very good at what they do, they weren't just "born" that way. They got that way the same way professional athletes got that way, by doing a lot of preparation and work behind the scenes. They got that way because they got a lot of "first team reps", speaking at a variety of events. And they continue to get better because they continue to speak, which means continuously putting effort and energy into new talks, into revising old talks, and so on.

But all of that time can't be for free, or else people won't do it.

Go back to the amateur athlete scenario: the more time said athlete has to work at a different job to pay the bills, the less time they have to prep and master their athletic skills. This is no different for speakers--if someone is already spending 8 hours a day working, and another 6 to 8 hours a day sleeping, then that's 8 to 10 hours in the day for everything else, including time spent with the family, eating, personal hygiene, and so on, including whatever relaxation time they can carve out. (And yes, we all need some degree of relaxation time.) When, exactly, is this individual, excited, passionate, enthused (or not), supposed to get those "first team reps" in? By sacrificing something else: time with the family, sleep, a hobby, whatever.

Don't you think that they deserve some kind of compensation for that time?

I know, I know, the usual response is, "But they're giving back to the community!" Yes, I know, you never really figured anything out on your own, you just ran off to StackOverflow or Google and found all the code you needed in order to learn the new technology--it was never any more effort on your own part than that. You OWE the community this engagement. And, by the way, you should also owe them all the code you ever write, for the same reason, because it's not like your employer ever gave you anything for that code, and it's not like you did all that research and study for the code you work on for them.

See, the tangled threads of "why" we do something are often way too hard to unravel. So let's instead focus on the "what" you did. You submitted an abstract, you created an outline, you concocted some slides, you built some demos, you practiced your talk, you delivered it to the audience, and you submitted yourself to "life's slings and arrows" in the form of evaluations. And for all that, the conference organizers owe you nothing? In fact, you're required to pay for the privilege of doing all that?

On "professional" conferences

One dangerous trend I see in conferences, and it's not the same one I saw in 2009, is that the main focus of a conference is shifting; no longer is it a gathering of like-minded professionals who want to improve their technical skills by learning from others. Instead, it's turning into a gathering of people who want to party, play board games, gorge themselves on bacon, drink themselves to a stupor, play in a waterpark or go catch a Vegas show with naked women in it. Somehow, "professional developer conference" has taken on all the overtones of a Bacchanalian orgy, all in the name of "community".

Don't get me wrong--I think it can be useful to blow off some steam during a show, particularly because for most people, absorbing all this new information is mentally exhausting, and you need time to process it, both socially (in the form of hallway conversations) and physically (meaning, go give your body something to do while your mind is churning away). But when the focus of the conference shifts from "speakers" to "bacon bar", that's a dangerous, dangerous sign.

And you know what the first sign is that the conference doesn't think it's principal offering is the technical content? When they won't even cover the speakers' costs to be at that event.

Seriously, think about it for a moment: if the principal focus of this event is the exchange of intellectual and industrial information, through the medium of a lecture given by an individual, then where should your money go? The bacon bar? Or towards making sure that you have the best damn lecturers your budget can afford?

When a conference doesn't offer to pick up airfare and hotel, then in my mind that conference is automatically telling the world, "We're willing to bring in the best speakers that are willing to do this all for free!" And how many of you would be willing to eat at a restaurant that said, "We're willing to bring in the best chefs that are willing to cook for free!"? Or go to a hospital that brings in "the best doctors that are willing to operate for free!"?

And how many of you are willing to part of your own money to go to it?

For community events like CodeCamps, it's an understood proposition that this is more about the networking and community-building than it is about the quality of the information you're going to get, and frankly, given that the CodeCamp is a free event, there's also an implicit "everybody here is a volunteer" that goes with it that explains--and, to my mind, encourages--people who've never spoken before to get up and speak.

But when you're a CodeMash, a devLink, or some of these other shows that are charging you, the attendee, a non-trivial amount of money to attend, and they're not covering speakers' expenses at a minimum, then they're telling you that your money is going towards bacon bars and waterparks, not the quality of the information you're receiving.

Yes, there are some great speakers who will continue to do those events, and Gods' honest truth, if I had somebody to cover my mortgage and/or paid me to be there, I'd love to do that, too. But many of those people who are paid by a company to be speaking at events are called "evangelists" and "salespeople", and developers have already voted with their feet often enough to make it easy to say that we don't want a conference filled with "evangelists" and "salespeople". You want an unbiased technical view of something? You want people to talk about a technology that don't have an implicit desire to sell it to you, so that they can tell you both what it's good for and where it sucks? Then you want speakers who aren't being paid by a company to be there; instead, you want speakers who can give you the "harsh truth" about a technology without fear of reprisal from their management. (And yes, there are a lot of evangelists who are very straight-shooting speakers, and I love 'em, every one. But there's a lot more of them out there who aren't.)

In many cases, for the conference to deliver both the bacon bar and the speakers' T&E, it would require your attendance fee to go up some. By rough back-of-the-napkin calculations, probably about $50 for each of you, depending on the venue, the length of the conference, the number of speakers (and the number of talks they each do), and the total number of attendees. Is it worth it?

When you go to the dentist's office, do you want the "excited, enthused amateur", or the "practiced professional"?

On 'Exclusive Content'

Mon, 19 Aug 2013 00:00:00 +0000

Although it seems to have dipped somewhat in recent years, periodically I get requests from conferences or webinars or other presentation-oriented organizations/events that demand that the material I present be "exclusive", usually meaning that I've never delivered said content at any other organized event (conference or what-have-you). And, almost without exception, I refuse to speak at those events, or else refuse to abide by the "exclusive" tag (and let them decide whether they still want me to speak for them).

People (by which I mean "organizers"--most speakers seem to get it intuitively if they've spoken at more than five or so conferences in their life) have expressed some surprise and shock at my attitude. So, I decided to answer some of the more frequently-asked questions that I get in response to this, partly so that I don't have to keep repeating myself (yeah, right, as if said organizers are going to read my blog) and partly because putting something into a blog is a curious form of sanity-check, in that if I'm way off, commenters will let me know posthaste.

Thus...:

"Nobody will come to our conference/listen to our webinar if the content is the same as elsewhere." This is, by far, the first and most-used reaction I get, and let me be honest: if people came to your conference or fired up your webinar solely because of the information contained, they would never come to your conference or listen to your webinar. The Internet is huge. Mind-staggeringly huge. Anything you could possibly ever want about any topic you could ever possibly imagine, it's captured it somewhere. (There's a corollary to that, too; I call it "Whittington's Law", which states, "Anything you can possibly imagine, the Internet not only has it, but a porn site version of it, as well".) You will never have exclusive content, because unless I invented the damn thing, and I've never shown it to anybody or ever used it before, somebody will likely have used it, written a blog post or a video tutorial or what-have-you, and posted it to the Internet. Therefore, by definition, it can't be exclusive.
But even on top of that first point, no presentation given by the same guy using the same slides is ever exactly the same. Anybody who's ever seen me give a talk twice knows that a lot of how I give my presentations is extremely ad-hoc; I like to write code on the fly, incorporate audience feedback and participation, and sometimes I even get caught up in a tangent that we explore along the way. None of my presentations are ever scripted, such that if you filmed two of them and played them side-by-side, you'll see marked and stark differences between them. And frankly, if you're a conference organizer, you should be quite happy about this, because one of the first rules of presenting is to "Know thy audience", but if you can't know your audience ahead of time, what course is left to you but to poll the audience when you first get started, and adjust your presentation based on that?
"Sure, the experience won't be as great as if they were in the room at the time, but if they can get the content elsewhere, why should they come to our conference?" Well.... Honestly, that question really needs to be rephrased: "Given all the vast amounts of information out there on the Internet, why should someone come to your conference, period?" If you and your fellow organizers can't answer that question, then my content isn't going to help you in the slightest. TechEd and other big conferences that stream all of their content to the Web seem to be coming to the realization that there is something about the in-person experience that still creates value for attendees, so maybe you should be thinking about that, instead. Yes, you will likely lose a few ticket sales from people watching the content online, but if those numbers are staggeringly large, it means that your conference offered nothing but content in the first place, and you were going to see those numbers drop off significantly anyway once the majority of your audience figured out that the content is available elsewhere. And for free, no less.
"But why is this so important to you?" Because, my friends, everything gets better with practice, and that includes presentations. When I taught for DevelopMentor lo those many years ago, one of the fundamental rules was that "You don't really know a deck until you've delivered it five times". (I call it "Sumida's Law", after the guy who trained me there.) What's more, the more often you've presented on a subject, the more easily you see the "right" order to the topics, and better ways of explaining and analogizing those topics occur to you over time. ("Halloway's Corollary to Sumida's Law": "Once you've delivered a deck five times, you immediately want to rewrite it all".) To be quite honest with you all, the first time I give a talk is much like the beta release of any software product: it takes user interaction and feedback before you start to see the non-obvious bugs.

I still respect the conference or webinar host that insists on exclusive content, and I wish you well finding your next speaker, but it won't be me. Or anyone I mentor or shepherd.

Programming Interviews

Mon, 19 Aug 2013 00:00:00 +0000

Apparently I have become something of a resource on programming interviews: I've had three people tell me they read the last two blog posts, one because his company is hiring and he wants his people to be doing interviews right, and two more expressing shock that I still get interviewed--which I don't really think is all that fair, more on that in a moment--and relief that it's not just them getting grilled on areas that they don't believe to be relevant to the job--and more on that in a moment, too.

A couple of things have emerged in the last few weeks since the saga described earlier, so I thought I'd wrap the thing up with a final post. Besides, I like things that come in threes.

First, go see this video. Jonathan pinged me about it shortly after the second blog post came out, and damn if he and Mitch don't nail a bunch of things directly on the head. Specifically, I want to call out two lists they put into their slides (which I can't find online, or I'd include a link, sorry).

One, what are the things you're trying to answer in an interview? They call it out as three questions an interviewer or interview team is seeking to answer:

Can they do the job?
Will they be motivated?
Would they get along with the team?

Personally, #2 to me is a red herring--frankly, I expect that if you, the candidate, take a job with my company, then either you have determined that you will be motivated to work here, or else you can force yourself to be. I don't really expect you to be the company cheerleader (unless, of course, I'm hiring you for that role), but I do expect professionalism: that you will be at work when you are scheduled or expected to be, that you will do quality work while you are there, and that you will look to make the best decisions possible given the information you have at the time. Motivation is not something I should be interviewing for; it's something you should be bringing.

But the other two? Spot-on.

And this brings me to my interim point: I'm not opposed to a programming test. I think I gave the impression to a number of readers that I think that I'm too good or too famous or whatever to be tested on my skills; that's the furthest thing from the truth. I think you most certainly should be verifying that I have the technical chops to do the job you want me to do; what I do want to suggest, however, is that for a number of candidates (myself included), there are ways to determine my technical chops without forcing me to stand at a whiteboard and code with a pen. For some candidates, you can examine their GitHub profile and see how many repos they have that're public (and have a look through some of the code they wrote). In fact, what I think would be a great interview question would be to look at a repo they haven't touched in a year, find some element of the code inside there, and ask them to explain what they were thinking when they wrote it. If it's well-documented, or if it's simple code, they'll be able to do that fairly quickly (once they context-swap to the codebase--got to give them time to remember, after all). If it's a complex or tricky bit, and they can't explain it...

... well, you just learned something about the code they write, now didn't you?

In my case, I have no public GitHub profile to choose from, but I'm an edge case, in that you can also watch my videos, and/or read my books and articles. Granted, there's a chance that I have amazing editors who save me from incredible stupidity and make me look good... but what are the chances that somebody is doing that for over a decade, across several technology platforms, and all without any credit? Probably pretty close to nil, IMHO. I'm not unique in this case--there's others whose work more or less speaks for itself, and I think you're disrespecting the candidate if you don't do your homework on the interview first.

Which, by the way, brings up another point: As an interviewer, you have a responsibility to do your homework on the candidate before they walk in the door, particularly if you're expecting them to have done their homework on your firm. Don't waste my time (and yours, particularly since yours is probably a LOT more expensive than mine, considering that a lot of companies are doing "interview loops" these days with a team of people, and all of their time adds up). If you're not going to take my candidacy seriously, why should I take your job or job offer or interview seriously?

The second list Jon and Mitch call out is their "interviewing antipatterns" list:

The Riddler
The Disorienter
The Stone Tablet
The Knuth Fanatic
The Cram Session
Groundhog Day
The Gladiator
Hear No Evil

I want you to watch the video, so I'm not going to summarize each here; go watch it. If you're in a position of doing hiring, ask yourself how many of those you yourself are perpetrating.

Second, go read this article. I don't like that he has "Dig into algorithms, data structures, code organization, simplicity" as one of his takeaways, because I think most interviewers are going to see "algorithms" and "data structures" and stop there, but the rest seems pretty spot-on.

Third, ask yourself the critical question: What, exactly, are we doing wrong? You think you're an agile organization? Then ask yourself how much feedback you get on your interviewing process, and how you would know if you screwed it up. Yes, you will know if hire a bad candidate, but how will you know if you're letting good candidates go? Maybe you're the hot company that everybody wants to work at, and you can afford to throw some wheat out with the chaff a few times, but you're not going to be in that position for long if you do, and more importantly, you're not going to be in that position for long, period. If you don't start trying to improve your hiring process now, by the time you need to, it'll be too late.

Fourth, practice! When unit-testing came out, many programmers said, "I don't need to test my code, my code is great!", and then everybody had a good laugh at their expense. Yet I see a lot of companies say essentially the same thing about their hiring and interview practices. How do you test an interview process? Easy--interview yourselves. Work with known-good conditions (people you know, people who work with you already, and so on), and run them through the process, but with the critical stipulation that you must treat them exactly as you would a candidate. If you look at your tech lead and say, "Yeah, this is where I'd ask you a technical question, but I already know...", then unless you're prepared to do that for your candidates, you're cheating yourself on the feedback. It's exactly like saying, "Yeah, this is where I'd write a test checking to see how we handle a null in that second parameter, but I already know...". If you're not prepared to do the latter, don't do the former. (And if you are prepared to do the latter, then I probably don't want to work with you anyway.)

Fifth, remember: Interviewing is not easy! It's not easy on the candidates, and it shouldn't be on you. It would be great if you could just test somebody on one dimension of themselves and call it good, but as much as people want to pretend that a programmer is just a code-spewing cog in a machine, they're not. If you want well-rounded candidates, then you must interview all aspects of that well-roundedness to determine if they are or not.

Whatever you interview for, that's what you will get.

More on the Programming Tests saga

Thu, 25 Jul 2013 00:00:00 +0000

A couple of people had asked how the story with the company that triggered the "I Hate Programming Tests" post ended, so I figured I'd follow up with the rest of that story, and some thoughts.

After handing in the disjoint-set solution I'd come up with, the VP pondered things for a bit, then decided to bring me in for an in-person interview loop with a half-dozen of the others that work there. I said I'd be happy to, and came in, did a brief meet-and-greet with the group of folks I'd be interviewing with (plus, I think, a few others), and then we got to the first interview mono-a-mono, and after a brief "Are you familiar with MVC?", we get into...

... another algorithm challenge. A walk-up-to-the-whiteboard-and-code-this challenge.

OK, whatever. I already said I'm not great with algorithmic challenges like this, but maybe this guy didn't get the memo or he's just trying to see how I reason things through. So, sure, let's attack this, even though I haven't done this kind of problem in like twenty years. (One of the challenges was "How do you sort a file of integer numbers when you can't store the entire collection of numbers in memory?", which wasn't an unfair challenge, just not something that I generally have to mess with. Honestly, in the working world, I'll start by going through the file number by number--or do chunks of the file in parallel using actors, if the file is large enough--and shove them into a database that's indexed on that number. But, of course, as with all of these kinds of challenges, the interviewer continues to throw constraints at the problem until we either get to the solution he wants or Ted runs out of imagination; in this case, I think it was the latter.) End result: not a positive win.

Next interviewer walks in, he wasn't there for the meet-and-greet, which means he has even less context about me than the guy before me, and he immediately asks... another algorithmic challenge. "If you have a tree of nodes, and you want to get a list of the nodes in rank order" (meaning, a breadth-first search, where each node now gets a "sibling" pointer pointing to the sibling on its right in the tree, or null if it's the rightmost node at that depth level) "how would you do it?" Again, a fail, and now I'm getting annoyed. I admitted, from the outset, that this is not the kind of stuff I'm good at. We've already made that point. I accept the "F" on that part of my report card. What's even more annoying, the interviewer keeps sighing and drumming his fingers in an obvious state of "Why is this bozo wasting my time like this, I could be doing something vastly more important" and so on, which, gotta say, was kind of distracting. End result: total fail.

By this point, I'm really annoyed. The VP comes to meet me, asks how it's going, and I tell him, flatly, "Sucks." He nods, says, yeah, we're going to kill the interview loop early, but I want to talk to you over lunch (with another employee along for company) and then have you meet with one more person before we end the exercise.

Lunch goes quite well, actually, and the last interview of the day is with their Product Manager, who then presents me with a challenge: "Suppose I want to build an online system for ordering pizzas. Customers can order pizzas, in other words. Build for me either the UI or the data model for this system." OK, this is different. I choose the data model, and build a ridiculously simple one-to-many relationship of customers to orders, and a similar one-to-many for orders to pizzas. She then proceeds to complicate the model step by step, sometimes in response to my questions, sometimes out of the blue, until we have a fairly complex roughly-sketched data model on the whiteboard. Result: win.

The VP at this point is on the horns of a dilemma: two of the engineers in the interview loop are convinced I'm an idiot. They're clearly voting no on this. But he's read my articles, he's seen some of my presentations, he knows I'm not the idiot the others assume me to be, and he's now trying to figure out what his next steps are. He takes a week to think about it, then emails me yesterday to say that it's not going to work.

Here's my thoughts, and folks, if you interview people or are part of an interview process, I'm trying to generalize this beyond this one experience to take it into a larger context:

Know what you want to prove with your interview. I get the feeling that this interview loop was essentially a repeat of every interview loop they've ever done before, with no consideration to the candidate himself. An interview is a chance for the company to get to know the candidate better, in order to make a well-informed decision. In this particular case, trying to suss out my skills around algorithms was a wasted effort--I'd already conceded that point. Therefore, find new questions! Find new areas in which to challenge the candidate to see what their skills are. (If you can't think of something else to ask, then you're not really thinking about the interview all that hard, and you're just going through the motions.)
Look for the proof you seek in other areas. With the growth of things like Github and open source projects in general, it's becoming easier and easier to prove to yourself as a company that a candidate does or does not have the coding skills you're looking for. Did this guy submit some pull requests to a project? Did this guy post some blogs about interesting technical tidbits? (Or, Lord help us, write articles for major publications?) Did this guy author an open-source project, or work on a project that other people know about? Look at it this way: If Anders Heljsberg, Bjarne Stroustrup or James Gosling walk through the door, are you going to put them through the same interview questions you put the random recruiter-found candidate goes through? Or are you willing to consider their established body of work and call it covered? As an interviewer, it behooves you to look for that established body of work, so that you can spend the interview loop looking at other things.
Be clear in what you want. One of the things the VP said to me was that he was looking for somebody who had a similar skillset to what he had; that is, had a architectural view of things and an interest in managing the people involved. By then submitting my candidacy to a series of tests that didn't really test for those things, he essentially torpedoed whatever chances it might have had.
Be willing to assert your authority. If you're the VP of the company, and the people who work for you disagree with your decisions, sometimes the Right Thing To Do is to simply overrule them. Yes, I know, it's not all politically correct to do that, and if you do it too often you'll ruin whatever sense of empowerment that you want your employees to have within the company, but there are times when you just need to assert that authority and say, "You know what? I appreciate y'all's input, but this is one of those cases where I think I have a different enough perspective that I am going to just overrule and do it anyway." Sometimes you'll be right, yay, and sometimes you'll be wrong, boo, but there is a reason you're the VP or the Director or the Team Lead, and not the others. Leadership means making hard decisions sometimes.
Be willing to change up the process. So your candidate comes in, and they're a junior programmer who's just graduated college, with zero experience. Do you then start asking them questions about their experience? That would be a waste of their time and yours. So you'll have to come up with new questions and a new approach. Not all interviews have to be carbon copies of each other, because certainly your candidates aren't carbon copies of each other. (At least, you'd better hope not, or else you're going to end up with a pretty single-dimensional staff.) If they've proven their strength in some category, or admitted a lack in another, then drop your standard set of questions, and go to something different. There is no honor in asking the exact same questions of every candidate.
Be willing to hire somebody that offers complementary skills. If your company already has a couple of engineers who know algorithms really well, then hire somebody for a different skillset. Likewise, if your company already has a couple of people who are really good with customers, you don't need another one. Look for people that have skills that fall outside the realm of what you currently have, and trust that when that individual is presented with a problem that attacks their weakness, they'll turn to somebody else in the firm to help them with it. When presented with an algorithmic challenge, you're damn well sure that I'm going to turn to somebody next to me and say, "Hey, dude? Help me walk through this for a bit, would you?" And, in turn, if that engineer has to give a presentation to a customer, and they turn to me and say, "Hey, dude? Help me work on this presentation, would you?", I'm absolutely ready to chip in. That's how teams are built. That's why we have teams in the first place.

In the end, this is probably the best of all possible scenarios, not working for them, particularly since I have some other things brewing that will likely consume all of my attention in the coming months, but there's that part of me that hates the fact that I failed at this. That same part of me is now going back through a few of the "interview challenges" books that I picked up, ironically, for my eldest son when he goes out and does his programming interviews, just to work through a few of the problems because I HATE feeling inadequate to a challenge.

And that, in turn, raises my next challenge: I want to create a website, just a static thing, that has a series of questions that, I think, are far better coding challenges than the ones I was given. I don't know when or if I'm going to get to this, but I gotta believe that any of the problems out of the book "Programming Challenges" (by Skiena and Revilla, Springer-Verlag, 2003) or the website from which those challenges were drawn, would be a much better test of the candidate's ability, particularly if you look at the ancillary parts of the challenge: do they write tests, how do they write their tests, do they pair well with somebody, and so on. THOSE are the things you really care about, not how well they remember their college lessons, which are easily accessible over Google or StackOverflow.

Bottom line: Your time is precious, people. Interview well, or just don't bother.

Programming Tests

Tue, 9 Jul 2013 00:00:00 +0000

It's official: I hate them.

Don't get me wrong, I understand their use and the reasons why potential employers give them out. There's enough programmers in the world who aren't really skilled enough for the job (whatever that job may be) that it becomes necessary to offer some kind of litmus test that a potential job-seeker must pass. I get that.

And it's not like all the programming tests in the world are created equal: some are pretty useful ways to demonstrate basic programming facilities, a la the FizzBuzz problem. Or some of the projects I've seen done, a la the "Robot on Mars" problem that ThoughtWorks handed out to candidates (a robot lands on Mars, which happens to be a cartesian grid; assuming that we hand the robot these instructions, such as LFFFRFFFRRFFF, where "L" is a "turn 90 degrees left", "R" is a "turn 90 degrees right", and "F" is "go forward one space, please write control code for the robot such that it ends up at the appropriate-and-correct destination, and include unit tests), are good indicators of how a candidate could/would handle a small project entirely on his/her own.

But the ones where the challenge is to implement some algorithmic doodad or other? shudder.

For example, one I just took recently asks candidates to calculate the "disjoint sets" of a collection of sets; in other words, given sets of { 1, 2, 3 }, { 1, 2, 4 } and { 1, 2, 5 }, the result should be sets of {1,2},{3},{4}, and {5}. Do this and calculate the big-O notation for your solution in terms of time and of space/memory.

I hate to say this, but in twenty years of programming, I've never had to do this. Granted, I see the usefulness of it, and granted, it's something that, given large enough sets and large enough numbers of sets, will make a significant difference that it bears examination, but honestly, in times past when I've been confronted with this problem, I'm usually the first to ask somebody next to me how best to think about this, and start sounding out some ideas with them before writing any bit of code. Unit tests to test input and its expected responses are next. Then I start looking for the easy cases to verify before I start attacking the algorithm in its entirety, usually with liberal help from Google and StackOverflow.

But in a programming test, you're doing this alone (which already takes away a significant part of my approach, because being an "external processor", I think by talking out loud), and if it's timed (such as this one was), you're tempted to take a shortcut and forgo some of the setup (which I did) in order to maximize the time spent hacking, and when you end up down a wrong path (such as I did), you have nothing to fall back on.

Granted, I screwed up, in that I should've stuck to my process and simply said, "Here's how far I got in the hour". But when you've been writing code for twenty years, across three major platforms, for dozens of Fortune 500 companies and architected platforms that others will use to build software and services for thousands of users and customers, you feel like you should be able to hack something like this out fairly quickly.

And when you can't, you feel like a failure.

I hate programming tests.

Update: By the way, as always, I would love some suggestions on how to accomplish the disjoint-set problem. I kept thinking I was close, but was missing one key element. I particularly would LOVE a nudge in doing it in a object-functional language, like F# or Scala (I've only attempted it in C# so far). Just a nudge, though--I want to work through it myself, so I learn.

Postscript An analogy hit me shortly after posting this: it's almost as if, in order to test a master carpenter's skill at carpentry, you ask him to build a hammer. After all, if he's all that good, he should be able to do something as simple as affix a metal head to a wooden shaft and have the result be a superior device to anything he could buy off the shelf, right?

Further update: After writing this, I took a break, had some dinner, played a game of Magic: The Gathering with my wife and kids (I won, but I can't be certain they didn't let me win, since they knew I was grumpy about not getting this test done in time), and then came back to it. I built up a series of little steps, backed by unit tests to make sure I was stepping through my attempts at reasoning out the algorithm correctly, backed up once or twice with a new approach, and finally solved it in about three hours, emailing it to the company at 6am (0600 for those of you reading this across the Atlantic or from a keyboard marked "Property of US Armed Forces"), just for grins. I wasn't expecting to get a response, since I was grossly beyond the time allotted, but apparently it was good enough to merit a follow-up interview, so yay for me. :-) Upshot is, though, I have an implementation that works, though now I find myself wondering if there's a way to do it in a functional/no-side-effect/persistent-data-structure kind of way....

I still hate them, though, at least the algorithm-based ones, and in a fleeting moment of transparent honesty, I will admit it's probably because I'm not very good at them, but if you repeat that to anyone I'll deny it as outrageous slander and demand satisfaction, Nerf guns at ten paces.

More on Types

Wed, 1 May 2013 00:00:00 +0000

With my most recent blog post, some of you were a little less than impressed with the idea of using types.

One reader, in particular, suggested that:

Your encapsulating type aliases don't... encapsulate :|

Actually, it kinda does. But not in the way you described.

using X = qualified.type; merely introduces an alias, and will consequently (a) not prevent assignment of a FirstName to a LastName (b) not even be detectible as such from CLI metadata (i.e. using reflection).

This is true—-the using statement only introduces an alias, in much the same way that C++’s “typedef” does. It’s not perfect, by any real means. Which is why the much better approach would be to introduce an actual type for Name, rather than just an alias--the alias is just a starting point.

Also, the alias is lexically scoped, and doesn't actually declare a public name (so, it would need to be redeclared in all 'client' compilation units. (This won't be done, of course, because the clients would have no clue about this and happily be passing System.String as ever). The same goes for C++ typedefs, or, indeed C++11 template aliases (using FirstName = std::string; and using LastName = std::string;). You'd be better off using BOOST_STRONG_TYPEDEF (or a roll-your-own version of this thing that is basically a CRTP pattern with some inherited constructors. When your compiler has the latter feature, you could probably do without an evil MACRO).

All of which is also true. Frankly, the “using” statement is a temporary stopgap, simply a placeholder designed to say, “In time, this will be replaced with a full-fledged type.”

And even more to the point, he fails to point out that my “Age” class from my example doesn’t really encapsulate the fact that Age is, fundamentally, an “int” under the covers—-because Age possesses type conversion operators to convert it into an int on demand (hence the “implicit” in that operator declaration), it’s pretty easy to get it back to straight “int”-land. Were I not so concerned with brevity, I’d have created a type that allowed for addition on it, though frankly I probably would forbid subtraction, and most certainly multiplication and division. (What does multiplying an Age mean, really?)

See, in truth, I cheated, because I know that the first reaction most O-O developers will have is, “Are you crazy? That’s tons more work—-just use the int!” Which, is fair, wrong, and an old argument-—the C guys said the same thing about these “object” things, and how much work it was compared to just declaring a data structure and writing a few procedures to manipulate them. Creating a full-fledged type for each domain—-or each fraction of a domain—-seems... heavy.

Truthfully, this is much easier to do in F#. And in Scala. And in a number of different languages. Unfortunately, in C#, Java, and even C++ (and frankly, I don’t think the use of an “evil MACRO” is unwarranted, if it doesn’t promote bad things). The fact that “doing it right” in those languages means “doing a ton of work to get it right” is exactly why nobody does it—and suffers the commensurate loss of encapsulation and integrity in their domain model.

Another poster pointed out that there is a much better series on this at http://www.fsharpforfunandprofit.com. In particular, check out the series on "Designing with Types"—-it expresses everything I wanted to say, albeit in F# (where I was trying, somewhat unsuccessfully, to example-code it in C#). By the way, I suspect that almost every linguistic feature he uses would translate pretty easily/smoothly over to Scala (or possibly Clojure) as well.

Another poster pointed out that doing this type-driven design (TDD, anyone?) would create some serious havoc with your persistence. Cry me a river, and then go use a persistence model that fits an object-oriented and type-oriented paradigm. Like, I dunno, an object database. Particularly considering that you shouldn’t want to expose your database schema to anyone outside the project anyway, if you’re concerned about code being tightly coupled. (As in, any other code outside this project—like a reporting engine or an ETL process—that accesses your database directly now is tied to that schema, and is therefore a tight-coupling restriction on evolving your schema.)

Achieving good encapsulation isn’t a matter of trying to hide the methods being used—it’s (partly) a matter of allowing the type system to carry a significant percentage of the cognitive load, so that you don’t have to. Which, when you think on it, is kinda what objects and strongly-typed type systems are supposed to do, isn’t it?

On Types

Fri, 26 Apr 2013 00:00:00 +0000

Recently, having been teaching C# for a bit at Bellevue College, I’ve been thinking more and more about the way in which we approach building object-oriented programs, and particularly the debates around types and type systems. I think, not surprisingly, that the way in which the vast majority of the O-O developers in the world approach types and when/how they use them is flat wrong—both in terms of the times when they create classes when they shouldn’t (or shouldn’t have to, anyway, though obviously this is partly a measure of their language), and the times when they should create classes and don’t.

The latter point is the one I feel like exploring here; the former one is certainly interesting on its own, but I’ll save that for a later date. For now, I want to think about (and write about) how we often don’t create types in an O-O program, and should, because doing so can often create clearer, more expressive programs.

A Person

Common object-oriented parlance suggests that when we have a taxonomical entity that we want to represent in code (i.e., a concept of some form), we use a class to do so; for example, if we want to model a "person" in the world by capturing some of their critical attributes, we do so using a class (in this case, C#):

class Person
{
	public string FirstName { get; set; }
	public string LastName { get; set; }
	public int Age { get; set; }
	public bool Gender { get; set; }
}

Granted, this is a pretty simplified case; O-O enthusiasts will find lots of things wrong with this code, most of which have to do with dealing with the complexities that can arise.

From here, there’s a lot of ways in which this conversation can get a lot more complicated—how, where and when should inheritance factor into the discussion, for example, and how exactly do we represent the relationship between parents and children (after all, some children will be adopted, some will be natural birth, some will be disowned) and the relationship between various members who wish to engage in some form of marital status (putting aside the political hot-button of same-sex marriage, we find that some states respect "civil unions" even where no formal ceremony has taken place, many cultures still recognize polygamy—one man, many wives—as Utah did up until the mid-1800s, and a growing movement around polyamory—one or more men, one or more women—looks like it may be the next political hot-button around marriage) definitely depends on the business issues in question...

... but that’s the whole point of encapsulation, right? That if the business needs change, we can adapt as necessary to the changed requirements without having to go back and rewrite everything.

Genders

Consider, for example, the rather horrible decision to represent "gender" as a boolean: while, yes, at birth, there are essentially two genders at the biological level, there are some interesting birth defects/disorders/conditions in which a person’s gender is, for lack of a better term, screwed up—men born with female plumbing and vice versa. The system might need to track that. Or, there are those who consider themselves to have been born into the wrong gender, and choose to live a lifestyle that is markedly different from what societal norms suggest (the transgender crowd). Or, in some cases, the gender may not have even been determined yet: fetuses don’t develop gender until about halfway through the pregnancy.

Which suggests, offhand, that the use of a boolean here is clearly a Bad Idea. But what suggests as its replacement? Certainly we could maintain an internal state string or something similar, using the get/set properties to verify that the strings being set are correct and valid, but the .NET type system has a better answer: Given that there is a finite number of choices to gender—whether that’s two or four or a dozen—it seems that an enumeration is a good replacement:

enum Gender
{ Male, Female, Indeterminate, Transgender }

class Person
{
	public string FirstName { get; set; }
	public string LastName { get; set; }
	public int Age { get; set; }
	public Gender Gender { get; set; }
}

Don’t let the fact that the property and the type have the same name be too confusing—-not only does it compile cleanly, but it actually provides some clear description of what’s being stored. (Although, I’ll admit, it’s confusing the first time you look at it.) More importantly, there’s no additional code that needs to be written to enforce only the four acceptable values—or, extend it as necessary when that becomes necessary.

Ages

Similarly, the age of a person is not an integer value—people cannot be negative age, nor do they usually age beyond a hundred or so. Again, we could put code around the get/set blocks of the Age property to ensure the proper values, but it would again be easier to let the type system do all the work:

struct Age
{
	int data;
	public Age(int d) { Validate(d); data = d;
}

public static void Validate(int d)
{ 
	if (d < 0) throw new ArgumentException("Age cannot be negative");
	if (d > 120) throw new ArgumentException("Age cannot be over 120");
}

// explicit int to Age conversion operator
public static implicit operator Age(int a) { return new Age(a); }

// explicit Age to int conversion operator 
public static implicit operator int(Age a){ return a.data; }

class Person
{
	public string FirstName { get; set; }
	public string LastName { get; set; }
	public Age Age { get; set; }
	public Gender Gender { get; set; }
}

Notice that we’re still having to write the same code, but now the code is embodied in a type, which is itself intrinsically reusable—we can reuse the Age type in other classes, which is more than we can say if that code lives in the Person.Age property getter/setter. Again, too, now the Person class really has nothing to do in terms of ensuring that age is maintained properly (and by that, I mean greater than zero and less than 120). (The "implicit" in the conversion operators means that the code doesn’t need to explicitly cast the int to an Age or vice versa.)

Technically, what I’ve done with Age is create a restriction around the integer (System.Int32 in .NET terms) type; were this XSD Schema types, I could do a derivation-by-restriction to restrict an xsd:int to the values I care about (0 – 120, inclusive). Unfortunately, no O-O language I know of permits derivation-by-restriction, so it requires work to create a type that "wraps" another, in this case, an Int32.

#### Names
Names are another point of problem, in that there’s all kinds of crazy cases that (as much as we’d like to pretend otherwise) turn out to be far more common than we’d like—not only do most people have middle names, but sometimes women will take their husband’s last name and hyphenate it with their own, making it sort of a middle name but not really, or sometimes people will give their children to multiple middle names, Japanese names put family names first, sometimes people choose to take a single name, and so on. This is again a case where we can either choose to bake that logic into property getters/setters, or bake it into a single type (a "Name" type) that has the necessary code and properties to provide all the functionality that a person’s name represents.

So, without getting into the actual implementation, then, if we want to represent names in the system, then we should have a full-fledged "Name" class that captures the various permutations that arise:

```csharp
class Name
{
	public Title Honorific { get { ... } }
	public string Individual { get { ... } }
	public string Nickname { get { ... } }
	public string Family { get { ... } }
	public string Full { get { ... } }
	public static Name Parse(string incoming) { ... }
}

See, ultimately, everything will have to boil back to the core primitives within the language, but we need to build stronger primitives for the system—Name, Title, Age, and don’t even get me started on relationships.

Relationships

Parent-child relationships are also a case where things are vastly more complicated than just the one-to-many or one-to-one (or two-to-one) that direct object references encourage; in the case of families, given how complex the modern American family can get (and frankly, it’s not any easier if we go back and look at medieval families, either—go have a look at any royal European genealogical line and think about how you’d model that, particularly Henry VIII), it becomes pretty quickly apparent that modeling the relationships themselves often presents itself as the only reasonable solution.

I won’t even begin to get into that example, by the way, simply because this blog post is too long as it is. I might try it for a later blog post to explore the idea further, but I think the point is made at this point.

Summary

The object-oriented paradigm often finds itself wading in tens of thousands of types, so it seems counterintuitive to suggest that we need more of them to make programs more clear. I agree, many O-O programs are too type-heavy, but part of the problem there is that we’re spending too much time creating classes that we shouldn’t need to create (DTOs and the like) and not enough time thinking about the actual entities in the system.

I’ll be the first to admit, too, that not all systems will need to treat names the way that I’ve done—sometimes an age is just an integer, and we’re OK with that. Truthfully, though, it seems more often than not that we’re later adding the necessary code to ensure that ages can never be negative, have to fall within a certain range, and so on.

As a suggestion, then, I throw out this idea: Ensure that all of your domain classes never expose primitive types to the user of the system. In other words, Person never exposes an int for the idea of age, only an Age type. C# makes this easy via using declarations, like so:

using FirstName = System.String;
using LastName = System.String;

which can then, if you’re thorough and disciplined about using the FirstName and LastName types instead of string, evolve into fully-formed types later in their own right if they need to. C++ provides typedef for this purpose—unfortunately, Java lacks any such facility, making this a much harder prospect. (This is something I’d stick at the top of my TODO list were I nominated to take Brian Goetz’s place at the head of Java9 development.)

In essence, encapsulate the primitive types away so that when they don’t need to be primitives, or when they need to be more complex than just simple holders of data, they don’t have to be, and clients will never know the difference. That, folks, is what encapsulation is trying to be about.

Say that part about HTML standards, again?

Sat, 13 Apr 2013 00:00:00 +0000

In incarnations past, I have had debates, public and otherwise, with friends and colleagues who have asserted that HTML5 (by which we really mean HTML5/JavaScript/CSS3) will essentially become the platform of choice for all applications going forward—that essentially, this time, standards will win out, and companies that try to subvert the open nature of the web by creating their own implementations with their own extensions and proprietary features that aren’t part of the standards, lose.

Then, I read the Wired news post about Google’s departure from WebKit, and I’m a little surprised that the Internet (and by “the Internet”, I mean “the very people who get up in arms about standards and subverting them and blah blah blah”) hasn’t taken more issues with some of the things cited therein:

Google’s decision is in tune with its overall efforts to improve the infrastructure of the internet. When it comes to browser software and other web technologies that directly effect the how quickly and effectively your machine grabs and displays webpages, the company likes to use open source technologies. That way, it can feed their adoption outside the company — and ultimately improve the delivery of its many online services (including all important advertisements). But if it believes the rest of the web is moving too slowly, it has no problem starting up its own project.

Just to be clear, Google is happy to use open-source technologies, so it can feed adoption of those technologies, but if it’s something that Google thinks is being adopted too slowly—like, say, Google’s extensions to the various standards that aren’t being picked up by its competitors—then Google feels the need to kick off its own thing. Interesting.

… [T]he trouble with WebKit is that is used different “multi-process architecture” than its Chrome browser, which basically means it didn’t handle concurrent tasks in the same way. When Chrome was first released in 2008 WebKit didn’t have a multi-process architecture, so Google had to build its own. WebKit2, released in 2010, adds multi-process features, but is quite different from what Google had already built. Apple and Google don’t see eye to eye on the project, and it became too difficult and too time-consuming for the company juggle the two architectures. “Supporting multiple architectures over the years has led to increasing complexity for both [projects],” the post says. “This has slowed down the collective pace of innovation.”

So… Google tried to use some open-source software, but discovered that the project didn’t work the way they built the rest of their application to work. (I’m certain that’s the first time that has happened, ever.) When the custodians of the project did add the feature Google wanted, the feature was implemented in a manner that still wasn’t in lockstep with the way Google wanted things to work in their application. This meant that “innovation” is “slowed down”.

(As an aside, I find it fascinating that whenever a company adopts open-source, it’s to “foster interoperability and open standards”, but when they abandon open-source, it’s to “foster innovation and faster evolution”. And I’m sure it’s entirely accidental that most of the time, adopting “open standards” is usually when the company is way behind on the technology curve for a given thing, and adopting “faster innovation” is usually when that same company thinks they’ve caught up the distance or surged ahead of their competitors in that space.)

Of course, a new implementation has its risks of bugs and incompatibilities, but Google has a plan for that:

“Throughout this transition, we’ll collaborate closely with other browser vendors to move the web forward and preserve the compatibility that made it a successful ecosystem,” the announcement reads.

Ah, there. See? By collaborating closely with their competitors, they will preserve compatibility. Because when Microsoft did that, everybody was totally OK with that…. uh, and… yeah… it worked pretty well, too, and….

Look, it seems pretty reasonable to assume that even if the tags and the DOM and the APIs are all 100% unchanged from Chrome v.Past to v.Next, there’s still going to be places where they optimize differently than WebKit does, which means now that developers will need to learn (and implement) optimizations in their Web-based applications differently. And frankly, the assumption that Chrome’s Blink and WebKit will somehow be bug-for-bug compatible/identical with each other is a pretty steep bar to accept blindly, considering the history.

Once again, we see the cycle coming around: in the beginning, when a technology is fleshing out, companies yearn for standards in order to create adoption. After a certain tipping point of adoption, however, the major players start to seek ways to avoid becoming a commodity, and start introducing “extensions” and “innovations” that for some odd reason their competitors in the standards meetings don’t seem all that inclined to adopt. That’s when they start forking and shying away from staying true to the standard, and eventually, the standard becomes either a least-common-denominator… or a joke.

Anybody want to bet on which outcome emerges for HTML5?

(Before you reach for the “Comment” link to flame me all to Hell, yes, even an HTML 5 standard that is 80% consistent across all the browsers is still pretty damn useful—just as a SQL standard that is 80% consistent across all the databases is useful. But this is a far cry from the utopia of interconnectedness and interoperability that was promised to us by the HTMLophiles, and it simply demonstrates that the Circle of TechnoLife continues, unabated, as it has ever since PC manufacturers—and the rest of us watching them--discovered what happens to them when they become a commodity.)

Programming Language 'laws'

Tue, 19 Mar 2013 00:00:00 +0000

As is pretty typical for that site, Lambda the Ultimate has a great discussion on some insights that the creators of Mozart and Oz have come to, regarding the design of programming languages; I repeat the post here for convenience:

Now that we are close to releasing Mozart 2 (a complete redesign of the Mozart system), I have been thinking about how best to summarize the lessons we learned about programming paradigms in CTM. Here are five "laws" that summarize these lessons:

A well-designed program uses the right concepts, and the paradigm follows from the concepts that are used. [Paradigms are epiphenomena]

A paradigm with more concepts than another is not better or worse, just different. [Paradigm paradox]

Each problem has a best paradigm in which to program it; a paradigm with less concepts makes the program more complicated and a paradigm with more concepts makes reasoning more complicated. [Best paradigm principle]

If a program is complicated for reasons unrelated to the problem being solved, then a new concept should be added to the paradigm. [Creative extension principle]

A program's interface should depend only on its externally visible functionality, not on the paradigm used to implement it. [Model independence principle]

Here a "paradigm" is defined as a formal system that defines how computations are done and that leads to a set of techniques for programming and reasoning about programs. Some commonly used paradigms are called functional programming, object-oriented programming, and logic programming. The term "best paradigm" can have different meanings depending on the ultimate goal of the programming project; it usually refers to a paradigm that maximizes some combination of good properties such as clarity, provability, maintainability, efficiency, and extensibility. I am curious to see what the LtU community thinks of these laws and their formulation.

This just so neatly calls out to me, based on my own very brief and very informal investigation into multi-paradigm programming (based on James Coplien's work from C++ from a decade-plus ago). I think they really have something interesting here.

"We Accept Pull Requests"

Tue, 26 Feb 2013 00:00:00 +0000

There are times when the industry in which I find myself does things that I just don't understand.

Consider, for a moment, this blog by Jeff Handley, in which he essentially says that the phrase "We accept pull requests" is "cringe-inducing":

Why do the words “we accept pull requests” have such a stigma? Why were they cringe-inducing when I spoke them? Because too many OSS projects use these words as an easy way to shut people up. We (the collective of OSS project owners) can too easily jump to this phrase when we don’t want to do something ourselves. If we don’t see the value in a feature, but the requester persists, we can simply utter, “We accept pull requests,” and drop it until the end of days or when a pull request is submitted, whichever comes first. The phrase now basically means, “Buzz off!”

OK, I admit that I'm somewhat removed from the OSS community--I don't have any particular dogs in that race, as the old saying goes--and the idea that "We accept pull requests" is a "Buzz off!" phrase is news to me. But I understand what Jeff is saying: a phrase has taken on a meaning of its own, and as is often the case, it's a meaning that's contrary to its stated one:

At Microsoft, having open source projects that actually accept pull requests is a fairly new concept. I work on NuGet, which is an Outercurve project that accepts contributions from Microsoft and many others. I was the dev lead for Razor and Web Pages at the time it went open source through Microsoft Open Tech. I collaborate with teams that work on EntityFramework, SignalR, MVC, and several other open source projects. I spend virtually all my time thinking about projects that are open source. Just a few years ago, this was unimaginable at Microsoft. Sometimes I feel like it still hasn’t sunk in how awesome it is that we have gotten to where we are, and I think I’ve been trigger happy and I’ve said “We accept pull requests” too often I typically use the phrase in jest, but I admit that I have said it when I was really thinking “Buzz off!”

Honestly, I've heard the same kind of thing from the mouths of Microsoft developers during Software Development Reviews (SDRs), in the form of the phrase "Thank you for your feedback"--it's usually at the end of a fervent discussion when one of the reviewers is commenting on a feature being done (or not being done) and the team is in some kind of disagreement about the feature's relative importance or the implementation used. It's usually uttered in a manner that gives the crowd a very clear intent: "You can stop talking now, because I've stopped listening."

The weekend after the MVP summit, I was still regretting having said what I said. I wished all week I could take the words back. And then I saw someone else fall victim. On a highly controversial NuGet issue, the infamous Phil Haack used a similar phrase as part of a response stating that the core team probably wouldn’t be taking action on the proposed changes, but that there was nothing stopping those affected from issuing a pull request. With my mistake still fresh in my mind, I read Phil’s words just as I’m sure everyone in the room at the MVP summit heard my own. It sounded flippant and it had the opposite effect from what Phil intended or what I would want people thinking of the NuGet core team. From there, the thread started turning nasty. We were stuck arguing opinions and we were no longer discussing the actual issue and how it could be solved.

As Jeff goes on to mention, I got involved in that Twitter conversation, along with a number of others, and as he says, the conversation moved on to JabbR, but without me--I bailed on it for a couple of reasons. Phil proposed a resolution to the problem, though, that seemed to satisfy at least a few folks:

With that many mentions on the tweets, we ran out of characters and eventually moved into JabbR. By the end of the conversation, we all agreed that the words “we accept pull requests” should never be used again. Phil proposed a great phrase to use instead: “Want to take a crack at it? We’ll help.”

But frankly, I don't care for this phraseology. Yes, I understand the intent--the owners of open-source projects shouldn't brush off people's suggestions about things to do with the project in the future and shouldn't reach for a handy phrase that will essentially serve the purpose of saying "Buzz off". And keeping an open ear to your community is a good thing, yes.

What I don't like about the new phrase is twofold. First, if people use the phrase casually enough, eventually it too will be overused and interpreted to mean "Buzz off!", just as "Thank you for your feedback" became. But secondly, where in the world did it somehow become a law that open source projects MUST implement every feature that their users suggest? This is part of the strange economics of open source--in a commercial product, if the developers stray too far away from what customers need or want, declining sales will serve as a corrective force to bring them back around (or, if they don't, bankruptcy of either the product or the company will eventually follow). But in an open-source project, there's no real visible marker to serve as that accountability and feedback--and so the project owners, those who want to try and stay in tune with their users anyway, feel a deeper responsibility to respond to user requests. And on its own, that's a good thing.

The part that bothers me, though, is that this new phraseology essentially implies that any open-source project has a responsibility to implement the features that its users ask for, and frankly, that's not sustainable. Open-source projects are, for the most part, maintained by volunteers, but even those that are backed by commercial firms (like Microsoft or GitHub) have finite resources--they simply cannot commit resources, even just "help", to every feature request that any user makes of them. This is why the "We accept pull requests" was always, to my mind, an acceptable response: loosely translated, to me at least, it meant, "Look, that's an interesting idea, but it either isn't on our immediate roadmap, or it takes the project in a different direction than we'd intended, or we're not even entirely sure that it's feasible or doable or easily managed or what-have-you. Why don't you take a stab at implementing it in your own fork of the code, and if you can get it to some point of implementation that you can show us, send us a copy of the code in the form of a pull request so we can take a look and see if it fits with how we see the project going." This is not an unreasonable response: if you care passionately about this feature, either because you think it should be there or because your company needs that feature to get its work done, then you have the time, energy and motivation to at least take a first pass at it and prove the concept (or, sometimes, prove to yourself that it's not such an easy request as you thought). Cultivating a sense of entitlement in your users is not a good practice--it's a step towards a completely unsustainable model that could, if not curbed, eventually lead to the death of the project as the maintainers essentially give up when faced with feature request after feature request.

I applaud the efforts on the part of project maintainers, particularly those at large commercial corporations involved in open source, to avoid "Buzz off" phrases. But it's not OK for project maintainers to feel like they are under a responsibility to implement any particular feature or idea suggested by a user. Some ideas are going to be good ones, some are going to be just "off the radar" of the project's core committers, and some are going to be just plain bad. You think your idea is one of those? Take a stab at it. Write the code. And if you've got it to a point where it seems to be working, then submit a pull request.

But please, let's not blow this out of proportion. Users need to cut the people who give them software for free some slack.

(EDIT: I accidentally referred to Jeff as "Anthony" in one place and "Andrew" in another. Not really sure how or why, but... Edited.)

More on 'Craftsmanship'

Fri, 25 Jan 2013 00:00:00 +0000

tl;df: To all those who dissented, you're right, but you're wrong. Craftsmanship is a noble meme, when it's something that somebody holds as a personal goal, but it's often coming across as a way to beat up and denigrate on others who don't choose to invest significant time and energy into programming. The Zen Masters didn't walk around the countryside, proclaiming "I am a Zen Master!"

Wow. Apparently I touched a nerve.

It's been 48 hours since I posted On the Dark Side of 'Craftsmanship', and it's gotten a ton of interest, as well as a few syndicated re-posts (DZone and a few others). Comments to the blog included a response from Dave Thomas, other blog posts have been brought to my attention, and Twitter was on FIRE with people pinging me with their thoughts, which turn out to be across the spectrum, approving and dissenting. Not at all what I really expected to happen, to be honest--I kinda thought it would get lost in the noise of others commenting around the whole thing.

But for whatever reason, it's gotten a lot of attention, so I feel a certain responsibility to respond and explain to some of the dissenters who've responded. Not to defend, per se, but to at least demonstrate some recognition and attempt to clarify my position where I think it's gotten mis-heard. (To those who approved of the message, thank you for your support, and I'm happy to have vocalized something you felt unable, unwilling, unheard, or too busy to vocalize yourself. I hope my explanations here continue to represent your opinions, but if not, please feel free to let me know.)

A lot of the opinions centered around a few core ideas, it seems, so let me try and respond to those first.

You're confusing "craftsmanship" with a few people behaving badly. That may well be, but those who behaved badly included at least one who holds himself up as a leader of the craftsman movement and has held his actions up as indications of how "craftsmen" should behave. When you do this, you invite this kind of criticism and association. So if the movement is being given a black eye because of the actions of a single individual, well, now you know how a bunch of moderate Republicans feel about Paul Ryan.

Corey is a nice guy, he apologized, don't crucify him. Of course he is. Corey is a nice guy--and, speaking well to his character, he apologized almost immediately when it all broke. I learned a long time ago that "true sorry" means you (a) apologize for your actions, (b) seek to remedy the damage your actions have caused ("make it right", in other words), and (c) avoid making the same mistake in the future. From a distance, it seems like he feels contrition, and has publicly apologized for his actions. I would hope he's reached out to Heather directly to try and make things right with her, but that's between the two of them. Whether he avoids this kind of activity in the future remains to be seen. I think he will, but that's because I think he's learned a harsh lesson about being in the spotlight--it tends to be a harsh place to be. The rest of this really isn't about Corey and Heather anymore, so as far as I'm concerned, that thread complete.

You misunderstand the nature of "craftsmanship". Actually, no, I don't. At its heart, the original intent of "craftsmanship" was a constant striving to be better about what you do, and taking pride in the things that you do. It's related to the Japanese code of the samurai (kaizen) that says, in essence, that we are constantly striving to get better. The samurai sought to become better swordsmen, constantly challenging each other to prove the mettle against one another, improving their skills and, conditioning, but also their honor, by how they treated each other, their lord, their servants, and those they sought to protect. Kanban is a wonderful code, and one I have tried to live my entire life, even before I'd discovered it. Please don't assume that I misunderstand the teachings of your movement just because I don't go to the meetings.

Why you pick on "craftsmanship", anyway? If I want to take pride in what I do, what difference does it make? This is me paraphrasing on much of the dissent, and my response boils down to two basic thoughts:

If you think your movement is "just about yourself", why invent a label to differentiate yourself from the rest?
If you invent a label, it becomes almost automatic to draw a line between "us" and "them", and that in of itself almost automatically leads to "us vs them" behavior and mentality.

Look, I view this whole thing as kind of like religion: whatever you want to do behind closed doors, that's your business. But when you start waving it in other peoples' faces, then I have a problem with it. You want to spend time on the weekends improving your skills, go for it. You want to spend time at night learning a bunch of programming languages so you can improve your code and your ability to design systems, go for it. You want to study psychology and philosophy so you can understand other people better when it comes time to interact with them, go for it. And hey, you want to put some code up somewhere so people can point to it and help you get it better, go for it. But when you start waving all that time and dedication in my face, you're either doing it because you want recognition, or you want to suggest that I'm somehow not as good as you. Live the virtuous life, don't brag about it.

There were some specific blogs and comments that I think deserve discusson, too:

Dave Thomas was kind enough to comment on my blog:

I remember the farmer comment :) I think I said 30%, but I stand by what I said. And it isn't really an elitist stance. Instead, I feel that programming is hard work. At the end of a day of coding, I'm tired. And so I believe that if you are asking someone to do programming, then it is in both your and their interest that they are doing something they enjoy. Because if they don't enjoy it, then they are truly just a laborer, working hard at something that has no meaning to them. And as you spend 8 hours a day, 5 days a week doing it, that seems like an awful waste of an intelligent person's life.

Sure, programming is hard. So is house painting. They're different kinds of exhaustion, but it's exhaustion all the same. But, frankly, if somebody has chosen to take up a job that they do just because it's a job, that's their choice, and not ours to criticize, in my opinion. (And I remember it as 50%, because I very clearly remember saying the "way to insult half the room" crack after it, but maybe I misheard you. I do know others also heard it at 50%, because an attendee or two came up to talk about it after the panel. At least, that's how I remember it at the time. But the number itself is kinda meaningless, now that I think about it.)

The farming quote was a deliberate attempt at being shocking to make a point. But I still think it is valid. I'd guess that 30% of the developers I meet are not happy in their work. And I think those folks would be happier and more fulfilled doing something else that gave them more satisfaction.

Again, you and I are both in agreement, that people should be doing what they love, but that's a personal judgment that each person is permitted to make for themselves. There are aspects of our lives that we don't love, but we do because they make other people happy (Juliet and Charlotte driving the boys around to their various activities comes to mind, for example), and it is not our position to judge how others choose for themselves, IMHO.

No one should have to be a laborer.

And here, you and I will disagree quite fundamentally: as I believe it was Martin Luther King, Jr, who said, "If you are going to be a janitor, be the best janitor you know how to be." It seems by that statement that you are saying that people who labor with their bodies rather than your minds (and trust me, you may not be a laborer anymore, big publishing magnate that you are, but I know I sure still am) are somehow less well-off than those who have other people working for them. Some people don't want the responsibility of being the boss, or the owner. See the story of the mexican fisherman at the end of this blog.

Nate commented:

You have a logical fallacy by lumping together the people that derided Heather's code and people that are involved in software craftmanship. It's actually a huge leap of logic to make that connection, and it really retracts from the article.

As I point out later, the people who derided Heather's code were some of the same folks who hold up software craftsmanship. That wasn't me making that up.

Now you realise that you are planting your flag firmly in the 'craftmanship' camp while propelling your position upwards by drawing a line in the sand to define another group of people as 'labourers'. Or in other words attempt to elevate yourself by patronising others with the position you think you are paying them a compliment.
Maybe you do not realise this?

No, I realize it, and it's a fair critique, which is why I don't label myself as a "craftsman". I have more to say on this below.

However, have you considered that the craft is not how awesome and perfect you and your code are, but what is applicable for the task at hand. I think most people who you would put into either camp share the same mix of attributes whether good or bad. The important thing is if the solution created does what it is designed to do, is delivered on time for when it is needed and if the environment that the solution has been created for warrants it, that the code is easily understandable by yourself and others (that matter) so it can be developed further over time and maintained.

And the very people who call themselves "craftsmen" criticized a piece of code that, as near as I can tell, met all of those criteria. Hence my reaction that started this whole thing.

I don't wish to judge you, and maybe you are a great, smart guy who does good in the world, but like you I have not researched anything about you, I have simply read your assessment above and come to a conclusion, that's being human I guess.

Oh, people judge each other all the time, and it's high time we stopped beating them up for it. It's human to judge. And while it would be politically correct to say, "You shouldn't judge me before you know me", fact is, of course you're going to do exactly that, because you don't have time to get to know me. And the fact that you don't know me except but through the blog is totally acceptable--you shouldn't have to research me in order to have an opinion. So we're all square on that point. (As to whether I'm a great smart guy who does good in the world, well, that's for others to judge in my opinion, not mine.)

The above just sounds like more of the same 'elitism' that has been ripe in this world from playground to the workplace since the beginning.

It does, doesn't it? And hopefully I clarify the position more clearly later.

In It's OK to love your job, Chad McCallum says that

The basic premise (or at least the one the author start out with) is that because there’s a self-declared group of “software craftspeople”, there is going to be an egotistical divide between those who “get it” and those who don’t.

Like it or not, Chad, that egotistical divide is there. You can "call bullshit" all day long, but look at the reactions that have popped up over this--people feel that divide, and frankly, it's one that's been there for a long, long time. This isn't just me making this up.

Chad also says,

It’s true the feedback that Heather got was unnecessarily negative. And that it came from people who are probably considered “software craftspeople”. That said, correlation doesn’t equal causation. I’m guessing the negative feedback was more because those original offenders had a bad day and needed to vent. And maybe the comments after that one just jumped on the bandwagon because someone with lots of followers and/or respect said it.

These are both things that can and have happened to anyone, regardless of the industry they work in. It’s extremely unfair to associate “someone who’s passionate about software development” to “person who’s waiting to jump on you for your mistakes”.

Unfortunately, Chad, the excuse that "others do it, too" is not an acceptable excuse. If everybody jumped off a cliff, would you do it, too? I understand the rationale--it's extremely hard being the one to go against the herd (I've got the psychological studies I can cite at you that prove it), but that doesn't make it OK or excuse it. Saying "it happens in other industries" is just an extension of that. In other industries, women are still explicitly discriminated against--does that make it OK for us to do that, too?

Chad closes his blog with "Stop calling us egotistical jerks just because we love what we do." To which I respond, "I am happy to do so, as soon as those 'craftsmen' who are acting like one, stop acting like one." If you're not acting like one, then there should be no argument here. If you're trying to tell me that your label is somehow immune to criticism, then I think we just have to agree to disagree.

Paul Pagel (on a site devoted to software craftsmanship, no less) responded as well with his Humble Pursuit of Mastery. He opens with:

I have been reading on blogs and tweets the sentiment that "software craftsmanship is elitism". This perception is formed around comments of code, process, or techniques. I understand a craftsman's earned sense of pride in their work can sometimes be inappropriately communicated.

I don't think I commented on code, process or technique, so I can't be sure if this is directly refuting what I'm saying, but I note that Paul has already touched on the meme he wants to communicate in his last phrase: the craftsman's "earned sense of pride". I have no problem with the work being something that you take pride in; I note, however, that "pride goeth before a fall", and note that, again, Ozymandias was justifiably proud of his accomplishments, too.

Paul then goes through a summation of his career, making sure to smallcaps certain terms with which I have no argument: "sacrifice", "listen", "practicing", "critique" and "teaching". And, in all honesty, these are things that I embrace, as well. But I start getting a little dubious about the sanctity of your terminology, Paul, when it's being used pretty blatantly as an advertising slogan and theme all over the site--if you want the term to remain a Zen-like pursuit, then you need to keep the commercialism out of it, in my opinion, or you invite the kind of criticism that's coming here (explicit or implicit).

Paul's conclusion wraps up with:

Do sacrificing, listening, practice, critiquing, and teaching sound like elitist qualities to you? Software craftsmanship starts out as a humble endeavor moving towards mastery. I won't let 140 or 1000 characters redefine the hours and years spent working hard to become a craftsman. It gave me humility and the confidence to be a professional software developer. Sometimes I let confidence get the better of me, but I know when that happens I am not honoring the spirit of craftsmanship which I was trained.

Humility enough to trademark your phrase "Software is our craft"? Humility enough to call yourself a "driving force" behind software craftsmanship? Don't get me wrong, Paul, there is a certain amount of commercialism that any consultant must adopt in order to survive--but either please don't mix your life-guiding principles with your commercialism, or else don't be surprised when others take aim at your "humility" when you do. It's the same when ministers stand in a multi-million dollar building on a Sunday morning and talk about the parable of the widow giving away her last two coppers--that smacks of hypocrisy.

Finally, Matt van Horn wrote in Crafsmanship, a rebuttal that:

there is an allusion to software craftsmen as being an exclusive group who agre on the “right” tools and techniques. This could not be further from the truth. Anyone who is serious about their craft knows that for every job there are some tools that are better and some that are worse.

... but then he goes right into making that exact mistake:

Now, I may not have a good definition of elegant code, but I definitely know it when I see it – regardless of who wrote it. If you can’t see that
(1..10).each{|i| puts i}
is more elegant than
x = 0
while true do
  x = x + 1
  if x > 10
    break
  end
  puts x
end
then you must near the beginning of your journey towards mastery. Practicing your craft develops your ability to recognize these differences, just as a skilled tailor can more easily spot the difference between a bespoke suit and something from Men’s Wearhouse.

Matt, you kind of make my point for me. What makes it elegant? You take it as self-evident. I don't. As a matter of fact, I've been asking this question for some years now, "What makes code 'elegant', as opposed to 'ugly'? Ironically, Elliott Rusty Harold just blogged about how this style of coding is dangerous in Java, and got crucified for it, but he has the point that functional style (your first example) doesn't JIT as well as the more imperative style right now on the JVM (or on the CLR, from what I can tell). Are you assuming that this will be running on a native Ruby implementation, on JRuby, IronRuby, ...? You have judged the code in the second example based on an intrinsic value system that you may have never questioned. To judge, you have to be able to explain your judgments in terms of the value system. And the fact that you judge without any context, kind of speaks directly to the point I was trying to make: "craftsmen", it seems, have this tendency to judge in absence of context, because they are clearly "further down their journey towards mastery", to use your own metaphor.

Or, to put it much more succinctly, "Beauty is in the eye of the beholder".

Matt then tells me I missed the point of the samurai and tea master story:

inally, he closes with a famous zen story, but he entirely misses the point of it. The story concerns a tea master, and a samurai, who get into a duel. The tea master prevails by bringing the same concentration to the duel that he brings to his tea ceremony. The point that Ted seems to miss here is that the tea master is a craftsman of the highest order. A master of cha-do (the way of tea) is able to transform the simple act of making and pouring a cup of tea into something transcendant by bringing to this simple act a clear mind, a good attitude, and years of patient, humble practice. Arguably he prevails because he has perfected his craft to a higher degree than the samurai has perfected his own. That is why he has earned the right to wear the garb of a samurai, and why he is able to face down his opponent.

Which, again, I find funny, because most Zen masters will tell you that the story--any Zen story, in fact--has no "definitive" meaning, but has meaning based on how you interpret it. (There are a few Zen parables that reinforce this point, but it gets a little meta to justify my understanding of a Zen story by quoting another Zen story.) How Matt chooses to interpret that parable is, of course, up to him. I choose to interpret the story thusly: the insulted samurai felt that his "earned sense of pride" at his sword mastery was insulted by the tea master--clearly no swordsman, as it says in the story--wore robes of a rank and honor that he had not earned. And clearly, the tea master was no swordsman. But what the tea master learned from his peer was not how to use his concentration and discipline to improve his own swordsmanship, but how to demonstrate that he had, in fact, earned a note of mastery through an entirely different discipline than the insulted samurai's. The tea master still has no mastery of the sword, but in his own domain, he is an expert. This was all the insulted samurai needed to see, that the badge of honor had been earned, and not just imposed by a capricious (and disrespectful) lord. Put a paintbrush and canvas into the hands of a house painter, and you get pretty much a mess--but put a spray painter in the hands of Leonardo, and you still get a mess. In fact, to really do the parable justice, we should see how much "craft" Matt can bring when asked to paint a house, because that's about how much relevance swordsmanship and house painting have in relationship to one another. (All analogies fail eventually, by the way, and we're probably reaching the boundaries of this one.)

Billy Hollis is a master with VB, far more than I ever will be; I know C++ far better than he ever will. I respect his abilities, and he, mine. There is no argument here. But more importantly, there are friends I've worked with in the past who are masters with neither VB nor C++, nor any other programming language, but chose instead to sink their time and energy into skiing, pottery, or being a fan of a television show. They chose to put their energies--energies the "craftsmen" seem to say should be put towards their programming--towards things that bring them joy, which happen to not be programming.

Which brings me to another refrain that came up over and over again: You criticize the craftsman, but then you draw a distinction between "craftsman" and "laborer". You're confusing (or confused). First of all, I think it important to disambiguate along two axes: those who are choosing to invest their time into learning to write better software, and those who are choosing to look at writing code as "just" a job as one axis, and along a second axis, the degree to which they have mastered programming. By your own definitions, "craftsmen", can one be early in your mastery of programming and still be a "craftsman"? Can one be a master bowler who's just picked up programming and be considered a "craftsman"? Is the nature of "craftsmanship" a measure of your skill, or is it your dedication to programming, or is it your dedication to something in your life, period? (Remember, the tea master parable says that a master C++ developer will see the master bowler and respect his mastery of bowling, even though he can't code worth a crap. Would you call him a "craftsman"?)

Frankly, I will say, for the record, that I think there are people programming who don't want to put a ton of time and energy into learning how to be better programmers. (I suspect that most of them won't ever read this blog, either.) They see the job as "just a job", and are willing to be taught how to do things, but aren't willing to go off and learn how to do them on their own. They want to do the best job they can, because they, like any human being, want to bring value to the world, but don't have that passion for programming. They want to come in at 9, do their job, and go home at 5. These are those whom I call "laborers". They are the "fisherman" in the following story:

The businessman was at the pier of a small coastal Mexican village when a small boat with just one fisherman docked. Inside the small boat were several large yellowfin tuna. The businessman complimented the Mexican on the quality of his fish and asked how long it took to catch them. The Mexican replied only a little while.

The businessman then asked why he didn't stay out longer and catch more fish? The Mexican said he had enough to support his family's immediate needs. The businessman then asked, but what do you do with the rest of your time? The Mexican fisherman said, "I sleep late, fish a little, play with my children, take a siesta with my wife, Maria, stroll into the village each evening where I sip wine and play guitar with my amigos; I have a full and busy life, señor."

The businessman scoffed, "I am a Harvard MBA and I could help you. You should spend more time fishing and with the proceeds buy a bigger boat. With the proceeds from the bigger boat you could buy several boats; eventually you would have a fleet of fishing boats. Instead of selling your catch to a middleman, you would sell directly to the processor and eventually open your own cannery. You would control the product, processing and distribution. You would need to leave this small coastal fishing village and move to Mexico City, then LA and eventually New York City where you would run your expanding enterprise."

The Mexican fisherman asked, "But señor, how long will this all take?" To which the businessman replied, "15-20 years." "But what then, señor?" The businessman laughed and said, "That's the best part! When the time is right you would announce an IPO and sell your company stock to the public and become very rich. You would make millions." "Millions, señor? Then what?" The businessman said, "Then you would retire. Move to a small coastal fishing village where you would sleep late, fish a little, play with your kids, take a siesta with your wife, stroll to the village in the evenings where you could sip wine and play your guitar with your amigos."

What makes all of this (this particular subject, craftsmanship) particularly hard for me is that I like the message that craftsmanship brings, in terms of how you conduct yourself. I love the book Apprenticeship Patterns, for example, and think that anyone, novice or master, should read this book. I have taken on speaking apprentices in the past, and will continue to do so well into the future. The message that underlies the meme of craftsmanship--the constant striving to improve--is a good one, and I don't want to throw the baby out with the bathwater. If you have adopted "craftsmanship" as a core value of yours, then please, by all means, continue to practice it! Myself, I choose to do so, as well. I have mentored programmers, I have taken speaking apprentices, and I strive to learn more about my craft by branching my studies out well beyond software--I am reading books on management, psychology, building architecture, and business, because I think there is more to software than just the choice of programming language or style.

But be aware that if you start telling people how you're living your life, there is an implicit criticism or expectation that they should be doing that, as well. And when you start criticizing other peoples' code as being "unelegant" or "unbeautiful" or "unclean", you'd better be able to explain your value system and why you judged it as so. Humility is a hard, hard path to tread, and one that I have only recently started to see the outlines of; I am guilty of just about every sin imaginable when it comes to this subject. I have created "elegant" systems that failed their original intent. I have criticized "ugly" code that, in fact, served the purpose well. I have bragged of my own accomplishments to those who accomplished a lot more than I did, or ever will. And I consider it amazing to me that my friends who've been with me since long before I started to eat my justly-deserved humble pie are still with me. (And that those friends are some amazing people in their own right.; if a man is judged by the company he keeps, then by looking around at my friends, I am judged to be a king.) I will continue to strive to be better than I am now, though, even within this discussion right now: those of you who took criticism with my post, you have good points, all of you, and I certainly don't want to stop you from continuing on your journeys of self-discovery, either.

And if we ever cross paths in person, I will buy you a beer so that we can sit down, and we can continue this discussion in person.

On the Dark Side of 'Craftsmanship'

Wed, 23 Jan 2013 00:00:00 +0000

I don't know Heather Arthur from Eve. Never met her, never read an article by her, seen a video she's in or shot, or seen her code. Matter of fact, I don't even know that she is a "she"--I'm just guessing from the name. But I'm coming out, right now, to say that the way she's being treated is absolutely beyond the pale, and she deserves better.

Apparently she got quite an ugly reaction from a few folks when she open-sourced some code:

So I went to see what people were saying about this project. I searched Twitter and several tweets came up. One of them, I guess the original one, was basically like “hey, this is cool”, but then the rest went like this:
"I cannot even make this stuff up." --@steveklabnik
"Ever wanted to make sed or grep worse?" --@zeeg
"@steveklabnik or just point to the actual code file. eyes bleeding!" --@coreyhaines
At this point, all I know is that by creating this project I’ve done something very wrong. It seemed liked I’d done something fundamentally wrong, so stupid that it flabbergasts someone. So wrong that it doesn’t even need to be explained. And my code is so bad it makes people’s eyes bleed. So of course I start sobbing.

Now, to be fair, Corey later apologized. But I'm still going to criticize the response. Not because Heather's a "she" and we should be more supportive of women in IT. Not because somebody took something they found interesting and put it up on github for anyone to take a look at and use if they found it useful. Not even because it's good code when they said it was bad code or vice versa. (To be honest, I haven't even looked at the code--that's how immaterial it is to my point.)

I'm criticizing because this is what "software craftsmanship" gets us: an imposed segregation of those who "get it" from those who "don't" based on somebody's arbitrary criteria of what we should or shouldn't be doing. And if somebody doesn't use the "right" tools or code it in the "right" way, then bam! You clearly aren't a "craftsman" (or "craftswoman"?) and you clearly don't care about your craft and you clearly aren't worth the time or energy necessary to support and nourish and grow and....

Frankly, I've not been a fan of this movement since its inception. Dave Thomas (Ruby Dave) was on a software panel with me at a No Fluff Just Stuff show about five years ago when we got on to this subject, and Dave said, point blank, "About half of the programmers in the world should just go take up farming." He paused, and in the moment that followed, I said, "Wow, Dave, way to insult half the room." He immediately pointed out that the people in the room were part of the first half, since they were at a conference, but it just sort of underscored to me how high-handed and high-minded that kind of talk and position can be.

Not all of us writing code have to be artists. Frankly, in the world of painting, there are those who will spend hours and days and months, tiny brushes in hand, jars of pigment just one lumens different from one another, laboring over the finest details, creating just one piece... and then there are those who paint houses with paint-sprayers, out of cans of mass-produced "Cream Beige" found at your local Lowes. And you know what? We need both of them.

I will now coin a term that I consider to be the opposite of "software craftsman": the "software laborer". In my younger days, believing myself to be one of those "craftsmen", a developer who knew C++ in and out, who understood memory management and pointers, who could create elegant and useful solutions in templates and classes and inheritance, I turned up my nose at those "laborers" who cranked out one crappy app after another in (what else?) Visual Basic. My app was tight, lean, and well-tuned; their apps were sloppy, bloated, and ugly. My app was a paragon of reused code; their apps were cut-and-paste cobbled-together duct-tape wonders. My app was a shining beacon on a hill for all the world to admire; their apps were mindless drones, slogging through the mud.... Yeah, OK, so you get the idea.

But the funny thing was, those "laborers" were going home at 5 every day. Me, I was staying sometimes until 9pm, wallowing in the wonderment of my code. And, I have to wonder, how much of that was actually not the wonderment of my code, but the wonderment of "me" over the wonderment of "code".

Speaking of, by the way, there appear to be the makings of another such false segregation, in the areas of "functional programming". In defense of Elliott Rusty Harold's blog the other day (which I criticized, and still stand behind, for the reasons I cited there), there are a lot of programmers that are falling into the trap of thinking that "all the cool kids are using functional programming, so if I want to be a cool kid, I have to use functional programming too, even though I'm not sure what I'm doing....". Not all the cool kids are using FP. Some aren't even using OOP. Some are just happily humming along using good ol' fashioned C. And producing some really quality stuff doing so.

See, I have to wonder just how much of the software "craftsmanship" being touted isn't really a narcissistic "Look at me, world! Look at how much better I am because I care about what I do! Look upon my works, ye mighty, and despair!" kind of mentality. Too much of software "craftsmanship" seems to be about the "me" part of "my code". And when I think about why that is, I come to an interesting assertion: That if we take the name away from the code, and just look at the code, we can't really tell what's "elegant" code, what's "hack" code, and what was "elegant hack because there were all these other surrounding constraints outside the code". Without the context, we can't tell.

A few years after my high point as a C++ "craftsman", I was asked to do a short, one-week programming gig/assignment, and the more I looked at it, the more it screamed "VB" at me. And I discovered that what would've taken me probably a month to do in C++ was easily accomplished in a few days in VB. I remember looking at the code, and feeling this sickening, sinking sense of despair at how stupid I must've looked, crowing. VB isn't a bad language--and neither is C++. Or Java. Or C#. Or Groovy, or Scala, or Python, or, heck, just about any language you choose to name. (Except Perl. I refuse to cave on that point. Mostly for comedic effect.)

But more importantly, somebody who comes in at 9, does what they're told, leaves at 5, and never gives a rat's ass about programming except for what they need to know to get their job done, I have respect for them. Yes, some people will want to hold themselves up as "painters", and others will just show up at your house at 8 in the morning with drop cloths. Both have their place in the world. Neither should be denigrated for their choices about how they live their lives or manage their careers. (Yes, there's a question of professional ethics--I want the house painters to make sure they do a good job, too, but quality can come just as easily from the nozzle of a spray painter as it does from the tip of a paintbrush.)

I end this with one of my favorite parables from Japanese lore:

Several centuries ago, a tea master worked in the service of Lord Yamanouchi. No-one else performed the way of the tea to such perfection. The timing and the grace of his every move, from the unfurling of mat, to the setting out of the cups, and the sifting of the green leaves, was beauty itself. His master was so pleased with his servant, that he bestowed upon him the rank and robes of a Samurai warrior.
When Lord Yamanouchi travelled, he always took his tea master with him, so that others could appreciate the perfection of his art. On one occasion, he went on business to the great city of Edo, which we now know as Tokyo.

When evening fell, the tea master and his friends set out to explore the pleasure district, known as the floating world. As they turned the corner of a wooden pavement, they found themselves face to face with two Samurai warriors.

The tea master bowed, and politely step into the gutter to let the fearsome ones pass. But although one warrior went by, the other remained rooted to the spot. He stroked a long black whisker that decorated his face, gnarled by the sun, and scarred by the sword. His eyes pierced through the tea maker’s heart like an arrow.

He did not quite know what to make of the fellow who dressed like a fellow Samurai, yet who would willingly step aside into a gutter. What kind of warrior was this? He looked him up and down. Where were broad shoulders and the thick neck of a man of force and muscle? Instinct told him that this was no soldier. He was an impostor who by ignorance or impudence had donned the uniform of a Samurai. He snarled: “Tell me, oh strange one, where are you from and what is your rank?”

The tea master bowed once more. “It is my honour to serve Lord Yamanouchi and I am his master of the way of the tea.”

“A tea-sprout who dares to wear the robes of Samurai?” exclaimed the rough warrior.

The tea master’s lip trembled. He pressed his hands together and said: “My lord has honoured me with the rank of a Samurai and he requires me to wear these robes. “

The warrior stamped the ground like a raging a bull and exclaimed: “He who wears the robes of a Samurai must fight like a Samurai. I challenge you to a duel. If you die with dignity, you will bring honour to your ancestors. And if you die like a dog, at least you will be no longer insult the rank of the Samurai!”

By now, the hairs on the tea master’s neck were standing on end like the feet of a helpless centipede that has been turned upside down. He imagined he could feel that edge of the Samurai blade against his skin. He thought that his last second on earth had come.

But the corner of the street was no place for a duel with honour. Death is a serious matter, and everything has to be arranged just so. The Samurai’s friend spoke to the tea master’s friends, and gave them the time and the place for the mortal contest.

When the fierce warriors had departed, the tea master’s friends fanned his face and treated his faint nerves with smelling salts. They steadied him as they took him into a nearby place of rest and refreshment. There they assured him that there was no need to fear for his life. Each one of them would give freely of money from his own purse, and they would collect a handsome enough sum to buy the warrior off and make him forget his desire to fight a duel. And if by chance the warrior was not satisfied with the bribe, then surely Lord Yamanouchi would give generously to save his much prized master of the way of the tea.

But these generous words brought no cheer to the tea master. He thought of his family, and his ancestors, and of Lord Yamanouchi himself, and he knew that he must not bring them any reason to be ashamed of him.

“No,” he said with a firmness that surprised his friends. “I have one day and one night to learn how to die with honour, and I will do so.”

And so speaking, he got up and returned alone to the court of Lord Yamanouchi. There he found his equal in rank, the master of fencing, he was skilled as no other in the art of fighting with a sword.

“Master,” he said, when he had explained his tale, “Teach me to die like a Samurai.”

But the master of fencing was a wise man, and he had a great respect for the master of the Tea ceremony. And so he said: “I will teach you all you require, but first, I ask that you perform the way of the Tea for me one last time.”

The tea master could not refuse this request. As he performed the ceremony, all trace of fear seemed to leave his face. He was serenely concentrated on the simple but beautiful cups and pots, and the delicate aroma of the leaves. There was no room in his mind for anxiety. His thoughts were focused on the ritual.

When the ceremony was complete, the fencing master slapped his thigh and exclaimed with pleasure : “There you have it. No need to learn anything of the way of death. Your state of mind when you perform the tea ceremony is all that is required. When you see your challenger tomorrow, imagine that you are about to serve tea for him. Salute him courteously, express regret that you could not meet him sooner, take off your coat and fold it as you did just now. Wrap your head in a silken scarf and and do it with the same serenity as you dress for the tea ritual. Draw your sword, and hold it high above your head. Then close your eyes and ready yourself for combat. “

And that is exactly what the tea master did when, the following morning, at the crack of dawn he met his opponent. The Samurai warrior had been expecting a quivering wreck and he was amazed by the tea master’s presence of mind as he prepared himself for combat. The Samurai’s eyes were opened and he saw a different man altogether. He thought he must have fallen victim to some kind of trick or deception ,and now it was he who feared for his life. The warrior bowed, asked to be excused for his rude behaviour, and left the place of combat with as much speed and dignity as he could muster.

(excerpted from http://storynory.com/2011/03/27/the-samurai-and-the-tea-master/)

My name is Ted Neward. And I bow with respect to the "software laborers" of the world, who churn out quality code without concern for "craftsmanship", because their lives are more than just their code.

On Functional Programming in Java

Mon, 21 Jan 2013 00:00:00 +0000

Elliott Rusty Harold is blogging that functional programming in Java is dangerous. He's wrong, and he's way late to the party on this one--it's coming to Java whether he likes it or not.

Go read his post first, while I try to sum up the reasons he cites for saying it's dangerous:

Java is not a lazy-evaluated language. Programmers in Java will screw up and create heap and stack errors as a result.
See? Here's a naive implementation of Clojure code taken directly over to Java and look how it blows up.
Programmers can do bad things with this idea, so therefore we should avoid it.
Oh, and by the way, it's "dangerously inefficient" in Java/JVM, even though I offer no perf benchmarks or comparisons to back this statement, and I'm somehow ignoring that Clojure and Scala run on the JVM as well, apparently without problem.

That kind of about sums it up, I think.

Look, as Elliott points out, Java is not Haskell. Neither is it Lisp. It's its own language, rooted in imperative and object-oriented history, but no less able to incorporate functional features into its development than Lisp could incorporate object-oriented features. However, if you do stupid things, like trying to re-create an infinite (implicitly lazily-evaluated) list in Clojure by creating an actualized list that stretches to infinity... you're going to blow the JVM up. Duh. Not even the supercomputer on the USS Enterprise five hundred years from now will be able to construct that list.

Porting code from one language to another is not a trivial exercise. If you attempt to port line-for-line and expression-for-expression, you can expect that your ported code will not be idiomatically correct. (I know this already, having done the exercise myself.) The root of the problem in his ported code is twofold. One, he (rather foolishly and in elegant strawman fashion) badly simulates what an infinite list would look like in Java--a commenter does the better job by showing how an Iterator can be made to perform the same thing that Haskell actually does under the hood by producing the next value on demand, rather than trying to create a list of Integers stretching to infinity. For someone who professes to have some Haskell experience and love, it surprises me that Elliott makes this kind of mistake, which leads me to conclude that he's trying to create the strawman. Two, he assumes that anyone who programs in Java functionally will have to create all of their functional tools by hand, and frankly, using Guava or FJ here would make this code sample a LOT easier to swallow. The fact that he ignores both of these in his stawman again sort of reinforces the idea that he's deliberately crippling his strawman to make his point.

His underlying point, though, seems to be simple: "I work with bad programmers, who don't seem to understand how to write code functionally in Java without screwing it up." Dude. Sucks to be you. "Bad programmers will move heaven and earth to do the wrong thing." --Glenn Vanderburg.

What really sucks for him is that these features are coming in Java 8, including lambda expressions and library support including a Stream interface that will allow for exactly this kind of code to be written without pain. Those programmers Elliott is working with are going to be even more on fire to use their functional approaches (and all the associated goodness of doing so, including composability and what-not) in their Java code. What might make Elliott more happy is that at least they won't have written it; it'll all be written by guys much smarter than any of them.

Book Review: Metaprogramming in .NET

Sat, 5 Jan 2013 00:00:00 +0000

Metaprogramming in .NET (Kevin Hazzard, Jason Bock; Manning)

tl;dr This is a great book (not perfect), but not an easy read for everyone, not because the writing is bad, but because the subject is a whole new level of abstraction above what most developers deal with.

Full disclosure: Manning Publications is a publisher I've published with before, and Kevin and Jason are both friends of mine in the .NET community. I write a column for MSDN Magazine, and metaprogramming was one of the topics in one of the series I've written ("Metaparadigmatic Programming") for the column, so this subject is not unfamiliar to me.

Kevin and Jason have done a great job covering a pretty diverse subject, in my opinion. Because metaprogramming is "programming about programming", it's sometimes a hard concept for people who've never really investigated it to wrap their heads around, but Kevin and Jason do a great job opening with some concepts first, then exploring .NET Reflection, which is most developers' first introduction to metaprogramming. If you can understand how Reflection is programming against code and code metadata, then you're in a good place to start exploring metaprogramming in further depth.

And explore it they do. From code generation with T4, CodeDOM and Reflection.Emit to code-level Expressions to low-level IL munging, they take you through a lot of the metaprogramming tools. They've also tried to include some practical places where these techniques are useful, though I do wish the examples had been a bit "larger", meaning they were integrated into the larger picture of a "real-world" system, but that's hard to do sometimes, and most readers sufficiently senior enough to read this book should be able to see how to apply them to their own problems. I also wish they'd approached generics a bit more thoroughly, since that's another metaprogrammatic technique that often doesn't get much love from developers (most of whom seem to view generics as a necessary evil, not a huge opportunity for design power), but maybe that would've been too much head-exploding for one book. Writing a LINQ provider would've been a good enhancement to the book, but again, that may have been a little too much for one book. I also wish they had put an IL overview into its own chapter, since it comes up in several chapters at once and would've been good as a reference, but there's books out there on IL, which hasn't changed much since .NET 2.0 days, so readers finding IL challenging should pick up one of those if they're finding their heads spinning a little on the IL syntax.

Having taken you through those techniques, though, they then take a different tack and take you through scripting languages and the Microsoft Dynamic Language Runtime (DLR), as well as into a few "alternative" languages for the CLR, which is an entirely different way of approaching metaprogrammatic techniques. Nemerle, for example, is a language that supports macros defined within the language, a technique that generally is limited to Lisps. (I admit it, Nemerle is one of my favorite CLR languages, and should be something every .NET developer plays with for at least a weekend.) They also include the first published coverage that I'm aware of on Roslyn, the Compiler-as-a-Service project under way at Microsoft, so readers intrigued by how they might use the compiler as part of their development efforts in v.Next of Visual Studio should definitely have a look.

Overall, the writing style is crisp, clean, not too academic but not too folksy, and entirely representative of two men I've been privileged to meet, have interesting technical conversations with, and have over to my house for dinner. Both are extremely approachable, and their text reflects this. Every .NET developer that wants to claim "senior" or "guru" level status should read this book and experiment with one or more of these techniques; these are the things that the "cool kids" in the .NET world know how to do, and if you want to hang with the best, this is the book you'll read cover to cover.

(This review was posted to Amazon at the above link on 5 Jan 2013, then copy-and-pasted here because I like posting reviews to my blog as well as to Amazon.)

2013 Tech Predictions

Tue, 1 Jan 2013 00:00:00 +0000

Here we go again: the annual review of last year's predictions, and a set of new ones for the new year.

Once again, it's time for my annual prognostication and review of last year's efforts. For those of you who've been long-time readers, you know what this means, but for those two or three of you who haven't seen this before, let's set the rules: if I got a prediction right from last year, you take a drink, and if I didn't, you take a drink. (Best. Drinking game. EVAR!)

Let's begin....

Recap: 2012 Predictions

THEN: Lisps will be the languages to watch.

With Clojure leading the way, Lisps (that is, languages that are more or less loosely based on Common Lisp or one of its variants) are slowly clawing their way back into the limelight. Lisps are both functional languages as well as dynamic languages, which gives them a significant reason for interest. Clojure runs on top of the JVM, which makes it highly interoperable with other JVM languages/systems, and Clojure/CLR is the version of Clojure for the CLR platform, though there seems to be less interest in it in the .NET world (which is a mistake, if you ask me).

NOW: Clojure is definitely cementing itself as a "critic's darling" of a language among the digital cognoscenti, but I don't see its uptake increasing--or decreasing. It seems that, like so many critic's darlings, those who like it are using it, and those who aren't have either never heard of it (the far more likely scenario) or don't care for it. Datomic, a NoSQL written by the creator of Clojure (Rich Hickey), is interesting, but I've not heard of many folks taking it up, either. And Clojure/CLR is all but dead, it seems. I score myself a "0" on this one.

THEN: Functional languages will....

I have no idea. As I said above, I'm kind of stymied on the whole functional-language thing and their future. I keep thinking they will either "take off" or "drop off", and they keep tacking to the middle, doing neither, just sort of hanging in there as a concept for programmers to take and run with. Mind you, I like functional languages, and I want to see them become mainstream, or at least more so, but I keep wondering if the mainstream programming public is ready to accept the ideas and concepts hiding therein. So this year, let's try something different: I predict that they will remain exactly where they are, neither "done" nor "accepted", but continue next year to sort of hang out in the middle.

NOW: Functional concepts are slowly making their way into the mainstream of programming topics, but in some cases, programmers seem to be picking-and-choosing which of the functional concepts they believe in. I've heard developers argue vehemently about "lazy values" but go "meh" about lack-of-side-effects, or vice versa. Moreover, it seems that developers are still taking an "object-first, functional-when-I-need-it" kind of approach, which seems a little object-heavy, if you ask me. So, since the concepts seem to be taking some sort of shallow root, I don't know that I get the point for this one, but at the same time, it's not like I was wildly off. So, let's say "0" again.

THEN: F#'s type providers will show up in C# v.Next.

This one is actually a "gimme", if you look across the history of F# and C#: for almost every version of F# v."N", features from that version show up in C# v."N+1". More importantly, F# 3.0's type provider feature is an amazing idea, and one that I think will open up language research in some very interesting ways. (Not sure what F#'s type providers are or what they'll do for you? Check out Don Syme's talk on it at BUILD last year.)

NOW: C# v.Next hasn't been announced yet, so I can't say that this one has come true. We should start hearing some vague rumors out of Redmond soon, though, so maybe 2013 will be the year that C# gets type providers (or some scaled-back version thereof). Again, a "0".

THEN: Windows8 will generate a lot of chatter.

As 2012 progresses, Microsoft will try to force a lot of buzz around it by keeping things under wraps until various points in the year that feel strategic (TechEd, BUILD, etc). In doing so, though, they will annoy a number of people by not talking about them more openly or transparently.

NOW: Oh, my, did they. Windows8 was announced with a bang, but Microsoft (and Sinofsky, who ran the OS division up until recently) decided that they could go it alone and leave critical partners (like Dropbox!) out of the loop entirely. As a result, the Windows8 Store didn't have a lot of apps in it that people (including myself) really expected would be there. And THEN, there was Surface... which took everybody by surprise, as near as I can tell. Totally under wraps. I'm scoring myself "+2" for that one.

THEN: Windows8 ("Metro")-style apps won't impress at first.

The more I think about it, the more I'm becoming convinced that Metro-style apps on a desktop machine are going to collectively underwhelm. The UI simply isn't designed for keyboard-and-mouse kinds of interaction, and that's going to be the hardware setup that most people first experience Windows8 on--contrary to what (I think) Microsoft thinks, people do not just have tablets laying around waiting for Windows 8 to be installed on it, nor are they going to buy a Windows8 tablet just to try it out, at least not until it's gathered some mojo behind it. Microsoft is going to have to finesse the messaging here very, very finely, and that's not something they've shown themselves to be particularly good at over the last half-decade.

NOW: I find myself somewhat at a loss how to score this one--on the one hand, the "used-to-be-called-Metro"-style applications aren't terrible, and I haven't really heard anyone complain about them tremendously, but at the same time, I haven't heard anyone really go wild and ga-ga over them, either. Part of that, I think, is because there just aren't a lot of apps out there for it yet, aside from a rather skimpy selection of games (compared to the iOS App Store and Android Play Store). Again, I think Microsoft really screwed themselves with this one--keeping it all under wraps helped them make a big "Oh, WOW" kind of event buzz within the conference hall when they announced Surface, for example, but that buzz sort of left the room (figuratively) when people started looking for their favorite apps so they could start using that device. (Which, by the way, isn't a bad piece of hardware, I'm finding.) I'll give myself a "+1" for this.

THEN: Scala will get bigger, thanks to Heroku.

With the adoption of Scala and Play for their Java apps, Heroku is going to make Scala look attractive as a development platform, and the adoption of Play by Typesafe (the same people who brought you Akka) means that these four--Heroku, Scala, Play and Akka--will combine into a very compelling and interesting platform. I'm looking forward to seeing what comes of that.

NOW: We're going to get to cloud in a second, but on the whole, Heroku is now starting to make Scala/Play attractive, arguably as attractive as Ruby/Rails is. Play 2.0 unfortunately is not backwards-compatible with Play 1.x modules, which hurts it, but hopefully the Play community brings that back up to speed fairly quickly. "+1"

THEN: Cloud will continue to whip up a lot of air.

For all the hype and money spent on it, it doesn't really seem like cloud is gathering commensurate amounts of traction, across all the various cloud providers with the possible exception of Amazon's cloud system. But, as the different cloud platforms start to diversify their platform technology (Microsoft seems to be leading the way here, ironically, with the introduction of Java, Hadoop and some limited NoSQL bits into their Azure offerings), and as we start to get more experience with the pricing and costs of cloud, 2012 might be the year that we start to see mainstream cloud adoption, beyond "just" the usage patterns we've seen so far (as a backing server for mobile apps and as an easy way to spin up startups).

NOW: It's been whipping up air, all right, but it's starting to look like tornadoes and hurricanes--the talk of 2012 seems to have been more around notable cloud outages instead of notable cloud successes, capped off by a nationwide Netflix outage on Christmas Eve that seemed to dominate my Facebook feed that night. Later analysis suggested that the outage was with Amazon's AWS cloud, on which Netflix resides, and boy, did that make a few heads spin. I suspect we haven't yet (as of this writing) seen the last of that discussion. Overall, it seems like lots of startups and other greenfield apps are being deployed to the cloud, but it seems like corporations are hesitating to pull the trigger on an "all-in" kind of cloud adoption, because of some of the fears surrounding cloud security and now (of all things) robustness. "+1"

THEN: Android tablets will start to gain momentum.

Amazon's Kindle Fire has hit the market strong, definitely better than any other Android-based tablet before it. The Nooq (the Kindle's principal competitor, at least in the e-reader world) is also an Android tablet, which means that right now, consumers can get into the Android tablet world for far, far less than what an iPad costs. Apple rumors suggest that they may have a 7" form factor tablet that will price competitively (in the $200/$300 range), but that's just rumor right now, and Apple has never shown an interest in that form factor, which means the 7" world will remain exclusively Android's (at least for now), and that's a nice form factor for a lot of things. This translates well into more sales of Android tablets in general, I think.

NOW: Google's Nexus 7 came to dominate the discussion of the 7" tablet, until...

THEN: Apple will release an iPad 3, and it will be "more of the same".

Trying to predict Apple is generally a lost cause, particularly when it comes to their vaunted iOS lines, but somewhere around the middle of the year would be ripe for a new iPad, at the very least. (With the iPhone 4S out a few months ago, it's hard to imagine they'd cannibalize those sales by releasing a new iPhone, until the end of the year at the earliest.) Frankly, though, I don't expect the iPad 3 to be all that big of a boost, just a faster processor, more storage, and probably about the same size. Probably the only thing I'd want added to the iPad would be a USB port, but that conflicts with the Apple desire to present the iPad as a "device", rather than as a "computer". (USB ports smack of "computers", not self-contained "devices".)

NOW: ... the iPad Mini. Which, I'd like to point out, is just an iPad in a 7" form factor. (Actually, I think it's a little bit bigger than most 7" tablets--it looks to be a smidge wider than the other 7" tablets I have.) And the "new iPad" (not the iPad 3, which I call a massive FAIL on the part of Apple marketing) is exactly that: same iPad, just faster. And still no USB port on either the iPad or iPad Mini. So between this one and the previous one, I score myself at "+3" across both.

THEN: Apple will get hauled in front of the US government for... something.

Apple's recent foray in the legal world, effectively informing Samsung that they can't make square phones and offering advice as to what will avoid future litigation, smacks of such hubris and arrogance, it makes Microsoft look like a Pollyanna Pushover by comparison. It is pretty much a given, it seems to me, that a confrontation in the legal halls is not far removed, either with the US or with the EU, over anti-cometitive behavior. (And if this kind of behavior continues, and there is no legal action, it'll be pretty apparent that Apple has a pretty good set of US Congressmen and Senators in their pocket, something they probably learned from watching Microsoft and IBM slug it out rather than just buy them off.)

NOW: Congress has started to take a serious look at the patent system and how it's being used by patent trolls (of which, folks, I include Apple these days) to stifle innovation and create this Byzantine system of cross-patent licensing that only benefits the big players, which was exactly what the patent system was designed to avoid. (Patents were supposed to be a way to allow inventors, who are often independents, to avoid getting crushed by bigger, established, well-monetized firms.) Apple hasn't been put squarely in the crosshairs, but the Economist's article on Apple, Google, Microsoft and Amazon in the Dec 11th issue definitely points out that all four are squarely in the sights of governments on both sides of the Atlantic. Still, no points for me.

THEN: IBM will be entirely irrelevant again.

Look, IBM's main contribution to the Java world is/was Eclipse, and to a much lesser degree, Harmony. With Eclipse more or less "done" (aside from all the work on plugins being done by third parties), and with IBM abandoning Harmony in favor of OpenJDK, IBM more or less removes themselves from the game, as far as developers are concerned. Which shouldn't really be surprising--they've been more or less irrelevant pretty much ever since the mid-2000s or so.

NOW: IBM who? Wait, didn't they used to make a really kick-ass laptop, back when we liked using laptops? "+1"

THEN: Oracle will "screw it up" at least once.

Right now, the Java community is poised, like a starving vulture, waiting for Oracle to do something else that demonstrates and befits their Evil Emperor status. The community has already been quick (far too quick, if you ask me) to highlight Oracle's supposed missteps, such as the JVM-crashing bug (which has already been fixed in the _u1 release of Java7, which garnered no attention from the various Java news sites) and the debacle around Hudson/Jenkins/whatever-the-heck-we-need-to-call-it-this-week. I'll grant you, the Hudson/Jenkins debacle was deserving of ire, but Oracle is hardly the Evil Emperor the community makes them out to be--at least, so far. (I'll admit it, though, I'm a touch biased, both because Brian Goetz is a friend of mine and because Oracle TechNet has asked me to write a column for them next year. Still, in the spirit of "innocent until proven guilty"....)

NOW: It is with great pleasure that I score myself a "0" here. Oracle's been pretty good about things, sticking with the OpenJDK approach to developing software and talking very openly about what they're trying to do with Java8. They're not entirely innocent, mind you--the fact that a Java install tries to monkey with my browser bar by installing some plugin or other and so on is not something I really appreciate--but they're not acting like Ming the Merciless, either. Matter of fact, they even seem to be going out of their way to be community-inclusive, in some circles. I give myself a "-1" here, and I'm happy to claim it. Good job, guys.

THEN: VMWare/SpringSource will start pushing their cloud solution in a major way.

Companies like Microsoft and Google are pushing cloud solutions because Software-as-a-Service is a reoccurring revenue model, generating revenue even in years when the product hasn't incremented. VMWare, being a product company, is in the same boat--the only time they make money is when they sell a new copy of their product, unless they can start pushing their virtualization story onto hardware on behalf of clients--a.k.a. "the cloud". With SpringSource as the software stack, VMWare has a more-or-less complete cloud play, so it's surprising that they didn't push it harder in 2011; I suspect they'll start cramming it down everybody's throats in 2012. Expect to see Rod Johnson talking a lot about the cloud as a result.

NOW: Again, I give myself a "-1" here, and frankly, I'm shocked to be doing it. I really thought this one was a no-brainer. CloudFoundry seemed like a pretty straightforward play, and VMWare already owned a significant share of the virtualization story, so.... And yet, I really haven't seen much by way of significant marketing, advertising, or developer outreach around their cloud story. It's much the same as what it was in 2011; it almost feels like the parent corporation (EMC) either doesn't "get" why they should push a cloud play, doesn't see it as worth the cost, or else doesn't care. Count me confused. "0"

THEN: JavaScript hype will continue to grow, and by years' end will be at near-backlash levels.

JavaScript (more properly known as ECMAScript, not that anyone seems to care but me) is gaining all kinds of steam as a mainstream development language (as opposed to just-a-browser language), particularly with the release of NodeJS. That hype will continue to escalate, and by the end of the year we may start to see a backlash against it. (Speaking personally, NodeJS is an interesting solution, but suggesting that it will replace your Tomcat or IIS server is a bit far-fetched; event-driven I/O is something both of those servers have been doing for years, and the rest of it is "just" a language discussion. We could pretty easily use JavaScript as the development language inside both servers, as Sun demonstrated years ago with their "Phobos" project--not that anybody really cared back then.)

NOW: JavaScript frameworks are exploding everywhere like fireworks at a Disney theme park. Douglas Crockford is getting more invites to conference keynote opportunities than James Gosling ever did. You can get a job if you know how to spell "NodeJS". And yet, I'm starting to hear the same kinds of rumblings about "how in the hell do we manage a 200K LOC codebase written in JavaScript" that I heard people gripe about Ruby/Rails a few years ago. If the backlash hasn't started, then it's right on the cusp. "+1"

THEN: NoSQL buzz will continue to grow, and by years' end will start to generate a backlash.

More and more companies are jumping into NoSQL-based solutions, and this trend will continue to accelerate, until some extremely public failure will start to generate a backlash against it. (This seems to be a pattern that shows up with a lot of technologies, so it seems entirely realistic that it'll happen here, too.) Mind you, I don't mean to suggest that the backlash will be factual or correct--usually these sorts of things come from misuing the tool, not from any intrinsic failure in it--but it'll generate some bad press.

NOW: Recently, I heard that NBC was thinking about starting up a new comedy series called "Everybody Hates Mongo", with Chris Rock narrating. And I think that's just the beginning--lots of companies, particularly startups, decided to run with a NoSQL solution before seriously contemplating how they were going to make up for the things that a NoSQL doesn't provide (like a schema, for a lot of these), and suddenly find themselves wishing they had spent a little more time thinking about that back in the early days. Again, if the backlash isn't already started, it's about to. "+1"

THEN: Ted will thoroughly rock the house during his CodeMash keynote.

Yeah, OK, that's more of a fervent wish than a prediction, but hey, keep a positive attitude and all that, right?

NOW: Welllll..... Looking back at it with almost a years' worth of distance, I can freely admit I dropped a few too many "F"-bombs (a buddy of mine counted 18), but aside from a (very) vocal minority, my takeaway is that a lot of people enjoyed it. Still, I do wish I'd throttled it back some--InfoQ recorded it, and the fact that it hasn't yet seen public posting on the website implies (to me) that they found it too much work to "bleep" out all the naughty words. Which I call "my bad" on, because I think they were really hoping to use that as part of their promotional activities (not that they needed it, selling out again in minutes). To all those who found it distasteful, I apologize, and to those who chafe at the fact that I'm apologizing, I apologize. I take a "-1" here.

2013 Predictions:

Having thus scored myself at a "9" (out of 17) for last year, let's take a stab at a few for next year:

"Big data" and "data analytics" will dominate the enterprise landscape. I'm actually pretty late to the ballgame to talk about this one, in fact--it was starting its rapid climb up the hype wave already this year. And, part and parcel with going up this end of the hype wave this quickly, it also stands to reason that companies will start marketing the hell out of the term "big data" without being entirely too precise about what they mean when they say "big data".... By the end of the year, people will start building services and/or products on top of Hadoop, which appears primed to be the "Big Data" platform of choice, thus far.
NoSQL buzz will start to diversify. The various "NoSQL" vendors are going to start wanting to differentiate themselves from each other, and will start using "NoSQL" in their marketing and advertising talking points less and less. Some of this will be because Pandora's Box on data storage has already been opened--nobody's just assuming a relational database all the time, every time, anymore--but some of this will be because the different NoSQL vendors, who are at different stages in the adoption curve, will want to differentiate themselves from the vendors that are taking on the backlash. I predict Mongo, who seems to be leading the way of the NoSQL vendors, will be the sacrificial scapegoat for a lot of the NoSQL backlash that's coming down the pike.
Desktops increasingly become niche products. Look, does anyone buy a desktop machine anymore? I have three sitting next to me in my office, and none of the three has been turned on in probably two years--I'm exclusively laptop-bound these days. Between tablets as consumption devices (slowly obsoleting the laptop), and cloud offerings becoming more and more varied (slowly obsoleting the server), there's just no room for companies that sell desktops--or the various Mom-and-Pop shops that put them together for you. In fact, I'm starting to wonder if all those parts I used to buy at Fry's Electronics and swap meets will start to disappear, too. Gamers keep desktops alive, and I don't know if there's enough money in that world to keep lots of those vendors alive. (I hope so, but I don't know for sure.)
Home servers will start to grow in interest. This may seem paradoxical to the previous point, but I think techno-geek leader-types are going to start looking into "servers-in-a-box" that they can set up at home and have all their devices sync to and store to. Sure, all the media will come through there, and the key here will be "turnkey", since most folks are getting used to machines that "just work". Lots of friends, for example, seem to be using Mac Minis for exactly this purpose, and there's a vendor here in Redmond that sells a ridiculously-powered server in a box for a couple thousand. (This is on my birthday list, right after I get my maxed-out 13" MacBook Air and iPad 3.) This is also going to be fueled by...
Private cloud is going to start getting hot. The great advantage of cloud is that you don't have to have an IT department; the great disadvantage of cloud is that when things go bad, you don't have an IT department. Too many well-publicized cloud failures are going to drive corporations to try and find a solution that is the best-of-both-worlds: the flexibility and resiliency of cloud provisioning, but staffed by IT resources they can whip and threaten and cajole when things fail. (And, by the way, I fully understand that most cloud providers have better uptimes than most private IT organizations--this is about perception and control and the feelings of powerlessness and helplessness when things go south, not reality.)
Oracle will release Java8, and while several Java pundits will decry "it's not the Java I love!", most will actually come to like it. Let's be blunt, Java has long since moved past being the flower of fancy and a critic's darling, and it's moved squarely into the battleship-gray of slogging out code and getting line-of-business apps done. Java8 adopting function literals (aka "closures") and retrofitting the Collection library to use them will be a subtle, but powerful, extension to the lifetime of the Java language, but it's never going to be sexy again. Fortunately, it doesn't need to be.
Microsoft will start courting the .NET developers again. Windows8 left a bad impression in the minds of many .NET developers, with the emphasis on HTML/JavaScript apps and C++ apps, leaving many .NET developers to wonder if they were somehow rendered obsolete by the new platform. Despite numerous attempts in numerous ways to tell them no, developers still seem to have that opinion--and Microsoft needs to go on the offensive to show them that .NET and Windows8 (and WinRT) do, in fact, go very well together. Microsoft can't afford for their loyal developer community to feel left out or abandoned. They know that, and they'll start working on it.
Samsung will start pushing themselves further and further into the consumer market. They already have started gathering more and more of a consumer name for themselves, they just need to solidify their tablet offerings and get closer in line with either Google (for Android tablets) or even Microsoft (for Windows8 tablets and/or Surface competitors) to compete with Apple. They may even start looking into writing their own tablet OS, which would be something of a mistake, but an understandable one.
Apple's next release cycle will, again, be "more of the same". iPhone 6, iPad 4, iPad Mini 2, MacBooks, MacBook Airs, none of them are going to get much in the way of innovation or new features. Apple is going to run squarely into the Innovator's Dilemma soon, and their products are going to be "more of the same" for a while. Incremental improvements along a couple of lines, perhaps, but nothing Earth-shattering. (Hey, Apple, how about opening up Siri to us to program against, for example, so we can hook into her command structure and hook our own apps up? I can do that with Android today, why not her?)
Visual Studio 2014 features will start being discussed at the end of the year. If Microsoft is going to hit their every-two-year-cycle with Visual Studio, then they'll start talking/whispering/rumoring some of the v.Next features towards the middle to end of 2013. I fully expect C# 6 will get some form of type providers, Visual Basic will be a close carbon copy of C# again, and F# 4 will have something completely revolutionary that anyone who sees it will be like, "Oh, cool! Now, when can I get that in C#?"
Scala interest wanes. As much as I don't want it to happen, I think interest in Scala is going to slow down, and possibly regress. This will be the year that Typesafe needs to make a major splash if they want to show the world that they're serious, and I don't know that the JVM world is really all that interested in seeing a new player. Instead, I think Scala will be seen as what "the 1%" of the Java community uses, and the rest will take some ideas from there and apply them (poorly, perhaps) to Java.
Interest in native languages will rise. Just for kicks, developers will start experimenting with some of the new compile-to-native-code languages (Go, Rust, Slate, Haskell, whatever) and start finding some of the joys (and heartaches) that come with running "on the metal". More importantly, they'll start looking at ways to use these languages with platforms where running "on the metal" is more important, like mobile devices and tablets.

As always, folks, thanks for reading. See you next year.

UPDATE: Two things happened this week (7 Jan 2013) that made me want to add to this list:

Hardware is the new platform. A buddy of mine (Scott Davis) pointed out on a mailing list we share that "hardware is the new platform", and with Microsoft's Surface out now, there's three major players (Apple, Google, Microsoft) in this game. It's becoming apparent that more and more companies are starting to see opportunities in going the Apple route of owning not just the OS and the store, but the hardware underneath it. More and more companies are going to start playing this game, too, I think, and we're going to see Amazon take some shots here, and probably a few others. Of course, already announced is the Ubuntu Phone, and a new Android-like player, Tizen, but I'm not thinking about new players--there's always new players--but about some of the big standouts. And look for companies like Dell and HP to start looking for ways to play in this game, too, either through partnerships or acquisitions. (Hello, Oracle, I'm looking at you.... And Adobe, too.)
APIs for lots of things are going to come out. Ford just did this. This is not going away--this is going to proliferate. And the startup community is going to lap it up like kittens attacking a bowl of cream. If you're looking for a play in the startup world, pursue this.

Envoy (in Scala, JavaScript, and more)

Thu, 20 Dec 2012 00:00:00 +0000

A little over a decade ago, Eugene Wallingford wrote a paper for the PloP '99 conference, describing the Envoy pattern language, "a pattern language for managing state in a functional program". It's a good read, but the implementation language for the paper is Scheme--given that it's a Lisp dialect, often isn't particularly obvious or easy to understand at first, I thought it might be interesting (both for me and any readers that wanted to follow along) to translate the implementation examples into a variety of different languages. In this case, I thought it would be relatively easy to do it in Scala and F#, given their hybrid object-functional nature, but it's also an interesting exercise to demonstrate it in [Javascript](I'll use NodeJS v0.8.15, running on my Mac, and Rhino, with the JVM), Yeti (an ML dialect that runs on the JVM), Jaskell (a Haskell dialect that also runs on the JVM), and, hey, what the heck, let's do it in C# while we're at it, just so the .NET guys don't feel too badly outnumbered.

(I'm posting this now with the intent of filling in the Yeti, Jaskell, F# and C# implementations later.)

Note that with lambdas coming in Java 8, it'll be possible to adapt this pattern language to work with Java, too--I'll leave that as exercise to do for myself (and update this blog entry) once I get a Java8 build on the machine on which I'm writing this.

One reason for doing this in Yeti and Jaskell is to demonstrate the original purpose of the Envoy pattern language--that we can achieve object-like semantics even in languages that don't directly support object semantics (like Scheme). But for the other languages, it's fair to ask why anyone would bother doing this in languages that do directly support objects (a la Scala, F#, etc), since it would seem a lot easier to just use the object features directly. And, truth be told, it's true--when looking to model objects in a language that has first-class support for objects, just use that support and those features, and call it a day. The point of this exercise is, for me, to exercise the functional features of those languages, and see exactly how functional languages can provide some of the same benefits that an O-O language enjoys, without having to use the O-O features directly. (There's been a lot of people writing functional-isms in O-O languages, yours truly included, so it seems a good exercise to flip that on its head.) This will also help me figure out where/when/how to use these features, when the need arises.

If you've not yet read the Envoy pattern language, take a moment and do that now; I don't want to annoy Mr. Wallingford in any way by repeating his prose here (not to mention that I'm going to have enough to do as it is just translating the code into several different languages). But I will toss in a brief summary of each of the elements in the pattern language, just so we're all on the same page about what's happening in each of these code samples.

Implementation notes

These are a few notes for each of the implementation langauges.

JavaScript

Because I want to be able to run the JavaScript code on either the Node platform directly or on the Rhino engine (via the Java JDK "jrunscript" command that installs on Java implementations starting with Java 6), and because those two environments provide different mechanisms for printing to the console ("console.log" in Node, and "println" in Rhino), I create a top-level function "out" that aliases to one or the other of those, depending on what's defined in the environment:

var out = (function() {
  if (typeof(console) !== "undefined" &amp;&amp;
      typeof(console.log) !== "undefined")
    return console.log
  else if (typeof(println) !== "undefined")
    return println
  else
    throw new Error("No idea what to use for output")
})();

(This actually gives away one of the punchlines in the first element of the pattern language, Function as Object, below, because here we're pretty clearly using "out" as a function-as-object.)
I used Rhino that ships with Java6, and node v0.8.15 for these.

Scala

I used Scala v2.9.2 running on Java6 for this.

Yeti

Yeti is an ML-based language that compiles to Java bytecode. Unlike Scala, it's a functional-only language (well, sort of), with Hindley-Milner type inference. As the Yeti home page describes, it supports polymorphic structure and variant types, property fields, lazy lists, pattern-matching on values, and a decent interop facility against Java code (meaning it can call Java classes, as well as compile to classes to be called from Java if desired.)

Yeti was at v0.9.7 at the time I wrote this, and again, running on the Java6 VM.

F#

I'm using the Visual Studio 2012 release to write the F# bits, which corresponds to F# 3.0. As far as I can tell, there's nothing really all that "3.0-specific" that I'm using, so it should work with F# 2.0, which shipped with Visual Studio 2010, and there's nothing Windows-specific here either, which means it should run fine on F# 3.0-on-Mono. Note that, like what I'm doing with the JavaScript version, I'm binding each of the pattern elements into a function for execution, thus creating a scope block that is dissociated from the larger global scope:

let example = fun () ->
    Console.WriteLine "Howdy world"

If (like I tried once) we were to use the more naive approach:

let example =
    Console.WriteLine "Howdy world"

... then each of the functions is executed and the results bound to the name described ("example", in this case) at the time the compiler sees it; in other words, each is eagerly-evaluated, instead of waiting to be invoked in the main entry point of the program later. By binding an anonymous function literal, it essentially lazy-fies each of them, and won't execute them until they are deliberately invoked in Main, as in:

[<EntryPoint>]
let main argv = 
    example()
    // ... the others go here
    0 // return an integer exit code

With the platform (and prelude) details out of the way, let's begin.

C#

Wow, the C# version is going to be ugly. Let me explain what I mean.

Let's start with the syntax for an anonymous function literal (a lambda, in C# parlance):

() => { return 5; };

This is a function that takes no arguments and yields an int. (Actually, to be exact, this is a delegate, since the lambda wouldn't need an explicit return or the curly braces, since it's a single-expression block and the lambda assumes that the result of the single expression should be implicitly returned.)

Ideally, we'd be able to capture this in an implicitly-typed local variable, like so:

var giveMeFive = () => { return 5; };

But unfortunately, C# doesn't allow this, saying that it "Cannot assign lambda to implicitly-typed local variable". (Doesn't get much more straightforward than that when it comes to an error message.) So, we have to explicitly-type the local variable, which is a Func<> of some type:

Func<int> giveMeFive = () => { return 5; };

Hold on to this thought, because things are going to get even uglier when we want to invoke an anonymous block like this later (when we get into the Closure parts of the pattern language).

Function as Object

In pure functional languages, it's actually difficult to keep state and data tied together--in fact, part of the whole point of a functional language is to write functions that operate on data, ideally on lots of different kinds of data. "Therefore, create a function that acts like an object. Such a function carries the data it needs along with the expression that operates on the data. More importantly, an object encapsulates its data, ensuring that only the allowed operations are applied to them." In other words, by writing a function and keeping the data buried inside of it, we achieve the same kind of encapsulation that object-orientation has traditionally reserved for itself as its principal advantage. This is done via a closure, which is the next element in the language.

Scheme:

The original Scheme implementation looked like this:

(define balance 0)
(define withdraw
  (lambda (balance amount)
    (if (<= amount balance)
      (- balance amount)
      (error "Insufficient funds" balance))
  ))
(define deposit
  (lambda (balance amount)
    (+ balance amount)
  ))
(define accrue-interest
  (lambda (balance interest-rate)
    (+ balance (* balance interest-rate))
  ))

There's a few things wrong with this approach, as Wallingford points out, but to be faithful, recreating this in our target languages is pretty straightforward: three functions, each of which operate on parameters passed in. "You could create new accounts simply by binding values to names. Operating on accounts involves passing the account to the appropriate procedure and binding the new value as appropriate."

JavaScript:

In JavaScript, we can bind function values to names just as we can in Scheme, so it's not actually all that different, once you get past the lack of parentheses and added curly braces. Thus, it looks like:

(function() {
  out("function-as-object =========")
  
  var balance = 0
  var withdraw = function(amount) {
    if (amount <= balance)
      balance = balance - amount
    else
      throw new Error("Insufficient funds")
  }
  var deposit = function(amount) {
    balance += amount
  }
  var accrueInterest = function(interestRate) {
    balance += (balance * interestRate)
  }
})()

Note that I wrap all of it into its own function so as to give the whole thing some scope--makes it easier to define in a single .js file and execute.

Scala:

Similarly, Scala allows us to bind functions to names, too:

def functionAsObject() = {
  def withdraw(balance : Int, amount : Int) = {
    if (amount <= balance) balance - amount else throw new RuntimeException("Insufficient funds")
  }
  def deposit(balance: Int, amount : Int) = {
    balance + amount
  }
  def accrueInterest(balance : Int, rate : Float) = {
    balance + (balance * rate)
  }
}

Again, all of it is wrapped into a function for easier (on me, while I was experimenting with all of this) scoping.

F#:

F#, like most functional/object hybrid languages, also offers the ability to bind functions to values, so this is also pretty straightforward. I choose to just operate against the "global" balance value, rather than do the more functional "pass the balance in" that the previous two use:

let functionAsObject = fun () ->
    let balance = ref 0
    let withdraw = 
        fun amt ->
            if amt <= !balance then
                balance := (!balance) - amt
                !balance
            else
                raise (Exception("Insufficient funds"))
    let deposit = 
        fun amt -> 
            balance := (!balance) + amt
            !balance
    let accrueInterest = 
        fun (intRate : float) -> 
            balance := (!balance) + (int (float !balance * intRate))
            !balance
    
    Console.WriteLine "=========> Function as Object"
    printfn "%d" (deposit 200)
    printfn "%d" (withdraw 50)

Yeti (ML):

Although Yeti supports a slightly more succinct syntax for defining a function, I choose to use the syntax that more closely matches what we're doing in the other examples--bind a function literal (do ... done;) to a name (withdraw, deposit and accrueInterest). Again, since this is running on top of the JVM, we have full access to the underlying Java library, which means we can make use of RuntimeException again as a cheap way of signaling a bad withdrawal.

withdraw = 
  do bal amt:
    if amt <= bal then
      bal - amt
    else
      throw new RuntimeException("Insufficient funds")
    fi
  done;

deposit =
  do bal amt: bal + amt done;
  
accrueInterest =
  do bal intRate:
    bal + (bal * intRate)
  done;

balance = 100;
println (withdraw balance 10)

Jaskell (Haskell):

C#:

This is a little more verbose than some of the other versions we've seen thus far, because C# lacks the type-inference that F# or Yeti or Scala has, yet requires explicit typing (in some places) because it is a statically-typed language. Again, because the language explicitly forbids the assignment of a lambda/delegate to an implicitly-typed local variable, the local names "withdraw", "deposit", and "accrueInterest" have to be explicitly typed.

static void FunctionAsObject()
{
    var balance = 0;
    Func<int, int> withdraw = (amount) =>
    {
        if (amount <= balance)
        {
            balance = balance - amount;
            return balance;
        }
        else
            throw new Exception("Insufficient funds");
    };
    Func<int, int> deposit = (amount) => 
    {
        balance += amount; return balance;
    };
    Func<float, int> accrueInterest = (intRate) => 
    { 
        balance += (int)(intRate * balance); return balance; 
    };

    Console.WriteLine("=============> FunctionAsObject");
    Console.WriteLine("{0}", deposit(100));
    Console.WriteLine("{0}", withdraw(10));
}

Notice that again, I choose to operate on the "global" variable "balance", rather than pass it in. (It's fairly easy to imagine how it would look if "balance" were passed in.)

Closure

"You are writing a function with a free variable. How do you bundle a function with a data value defined outside the procedure's body?" If the data value is defined inside the procedure, remember, it gets reset to the same value each time, and obviously this isn't going to track state at all well. "So you might try defining the balance outside the function." But that doesn't work, because now the value isn't encapsulated anymore. "Therefore, create the function in an environment where its free variables are bound to local variables."

This is something that O-O folks won't see right away, but it's a powerful mechanism for reuse. Traditional O-O says to tuck the encapsulated value (balance) away as a private field, but in environments like JavaScript, which lack any sort of formal access control, or in environments like the JVM or CLR, both of which offer a means by which to bypass access control directives (via the Reflection libraries in both), what's marked as "private" often isn't as private as we might want. By creating a local variable that's outside the scope of the returned function object but inside of the scope of the function returning the function (see where "balance" is declared in the JavaScript version, for example), the language or platform has to "close over" that variable (hence the name "closure"), thus making it accessible to the returned function for use, but effectively hidden away from any prying eyes that might want to screw with it outside of permitted access channels.

Scheme:

The only key thing to note here is that "withdraw" references a lambda, a function literal in Scheme. We'll try to keep this flavor in the other language implementations, just to be faithful:

(define withdraw
  (let ((balance 100)) ;; balance is defined here,
    (lambda (amount)
      (if (>= balance amount) ;; so this reference is bound
        (begin
          (set! balance (- balance amount))
          balance)
        (error "Insufficient funds" balance)))
    ))

JavaScript:

JavaScript is, surprisingly to some "old-school" JavaScript programmers, a full-fledged member of the family of languages that support closures, so all that's necessary here is to define a function that returns a function that "closes over" the local variable "balance". But, in order to make sure that balance isn't reset to its original value of 100 each time we call the function, we have to actually invoke the outer function to return the inner function, which is then bound to the name "withdraw"; that way, the variable "balance" is initialized once, yet still referenced:

(function() {
  out("closure ====================")

  var withdraw = function() {
    var balance = 100
    return function(amount) {
      if (balance >= amount) {
        balance -= amount
        return balance
      }
      else
        throw new Error("Insufficient funds")
    }
  }()
  out("withdraw 20 " + withdraw(20))
  out("withdraw 30 " + withdraw(30))
})()

Scala:

We can do the same thing in Scala, and the syntax looks somewhat similar to the JavaScript version--create a function literal, invoke it, and bind the result to the name "withdraw", where the return is another anonymous function literal:

  def closure() = {
    val withdraw = (() => {
      var balance = 100
      (amount: Int) => {
        if (amount <= balance) {
          balance -= amount
          balance
        }
        else
          throw new RuntimeException("Insufficient funds")
      }
    })()
    println(withdraw(20))
    println(withdraw(20))
  }

F#:

The F# version gets interesting because when we try to do the same thing that the JavaScript (or other languages) do--that is, "close over" a local variable defined in the outer scope--the compiler immediately rejects that, saying point-blank that the language does not support that, and to use "reference variables" (the `ref` keyword) instead:

let closure = fun () ->
    let withdraw =
        let balance = ref 100
        fun amt ->
            if amt <= !balance then
                balance := (!balance) - amt
                !balance
            else
                raise (Exception("Insufficient funds"))

    Console.WriteLine "=========> Closure"
    printfn "%d" (withdraw 20)
    printfn "%d" (withdraw 30)

What essentially we're doing, then, is capturing a pointer/reference to balance, and carrying that into the returned function literal, rather than letting the language capture that for us. The ref is dereferenced using the "!" operator, and assigned through using the ":=" operator, as can be seen above. Other than that, this is pretty much identical to the other languages' versions.

By the way, it should be noted that F#'s "printfn" function is actually type-safe, so attempts to write "printfn "%d" x" where "x" is a non-integer value will yield a compile-time error. That's an incredibly spiffy feature, and I wish it were something we could apply to our own F# APIs, but from what I understand from Don Syme (the F# language creator), it's something that's baked into the compiler somehow. :-/

Yeti (ML):

Yeti works just as any of the others have, since we can define a function literal that returns a function literal, so just like the JavaScript and Scala versions, we can bind a variable (as opposed to a value, which is immutable) just outside the inner function literal, and Yeti will "close over" that variable and use it for modifiable state:

withdraw = 
  (do:
    var balance = 100;
    do amt:
      if amt <= balance then
        balance := balance - amt;
        balance
      else
        throw new RuntimeException("Insufficient funds")
      fi
    done;
  done;) ();

println (withdraw 10);  // prints 90
println (withdraw 10);  // prints 80
println (withdraw 10);  // prints 70

Jaskell (Haskell):

C#:

Brace yourself--things are about to get really ugly here. The other versions suggest that we obtain encapsulation by capturing the "balance" value inside an outer function scope which is then referenced from an inner function scope, that inner function scope being the returned function literal. But... C# doesn't let us invoke function literals directly, except if they're cast to Func<> instances:

static void Closure()
{
    Func<int, int> withdraw = ((Func<Func<int, int>)(() => {
        var balance = 100;
        Func<int, int> result = delegate(int amount)
        {
            if (balance >= amount)
            {
                balance -= amount;
                return balance;
            }
            else
                throw new Exception("Insufficient funds");
        };
        return result;
    }))();
    Console.WriteLine("=============> Closure");
    Console.WriteLine("{0}", withdraw(20));
}

Did all that make sense? It might be clearer if I go back to the version I wrote that I had to use in order to figure all this out on my own:

static void Closure()
{
    Func<Func<int, int> withdrawMaker = (delegate {
        var balance = 100;
        Func<int, int> result = delegate(int amount)
        {
            if (balance >= amount)
            {
                balance -= amount;
                return balance;
            }
            else
                throw new Exception("Insufficient funds");
        };
        return result;
    });
    Func<int, int> withdraw = withdrawMaker();

    Console.WriteLine("=============> Closure");
    Console.WriteLine("{0}", withdraw(20));
}

Why bother will all of this--why not just write it as a generalized method like O-O folks have done since the beginning of time? Because we want that "balance" tucked away somewhere where Reflection can't find it. So the double-level of function indirection is necessary; to cap things off, we don't want to have to write a one-use "maker" function every time.

Constructor Function

"You are creating a Function as Object using a Closure. How do you create instances of the object? [M]ake a function that returns your Function as Object. Give the function an Intention Revealing Name (Beck) such as make-object."

Scheme:

Now things get more interesting, because the Scheme code is defining "make-withdraw" to be a lambda that in turn nests a lambda inside of it. This makes the syntax a little weird--since the returned value from "make-withdraw" is a lambda, the bound lambda must be executed in order to do the actual withdrawal.

(define make-withdraw
  (lambda (balance)
    (lambda (amount)
      (if (>= balance amount)  ;; balance is still bound,
        (begin                 ;; but to a new object on each call
          (set! balance (- balance amount))
          balance)
        (error "Insufficient funds" balance)))
    ))
(define account-for-eugene (make-withdraw 100))
(account-for-eugene 20)    => 80
(define account-for-tom (make-withdraw 1000))
(account-for-tom 20)       => 980

JavaScript:

It's pretty common in JavaScript to create a function that returns a function, and that's the heart of what Constructor Function is doing: returning a function:

(function() {
  out("constructorFunction ========")

  var makeWithdraw = function(balance) {
    return function(amount) {
      if (balance >= amount) {
        balance -= amount
        return balance
      }
      else
        throw new Error("Insufficient funds")
    }
  }
  var acctForEugene = makeWithdraw(100)
  out(acctForEugene(20))
  var acctForTed = makeWithdraw(1000)
  out(acctForTed(20))
})()

Scala:

Ditto for Scala, though the idiom/pattern of function-literal-returning- function-literal isn't always quite this obvious in Scala:

  def constructorFunction() = {
    def makeWithdraw(bal : Int) = {
      var balance = bal
      (amt : Int) => {
        if (balance >= amt) {
          balance = (balance - amt) 
          balance
        }
        else 
          throw new RuntimeException("Insufficient funds")
      }
    }
    val acctForEugene = makeWithdraw(100)
    println(acctForEugene(20))
    val acctForTed = makeWithdraw(1000)
    println(acctForTed(20))
  }

F#:

Really, not any different from the other languages: a function binding that returns a function, with the passed-in "balance" captured as a reference (see the earlier pattern element discussion for why it's a ref) inside the outer function scope, and used from the inner function scope.

let constructorFunction = fun () ->
    let makeAccount =
        fun bal ->
            let balance = ref bal
            fun amt ->
                if amt <= !balance then
                    balance := (!balance) - amt
                    !balance
                else
                    raise (Exception("Insufficient funds"))                
            
    Console.WriteLine "=========> Constructor Function"
    let acctForEugene = makeAccount 100
    printfn "%d" (acctForEugene 20)

Yeti (ML):

Same exercise--a function binding that returns a function, with the passed-in "balance" stored as a variable (var) inside the outer function scope, such that it is closed over by the inner function scope.

makeWithdraw =
  (do bal:
    var balance = bal;
    do amt:
      if amt <= balance then
        balance := balance - amt;
        balance
      else
        throw new RuntimeException("Insufficient funds")
      fi
    done;
  done;);

acctForEugene = makeWithdraw 100;
println (acctForEugene 10);   // 90
println (acctForEugene 10);   // 80

Jaskell (Haskell):

C#:

The constructor function must be explicitly typed, again, but we gain a tiny bit of brevity by changing the "delegate" literals into (slightly) shorter C# lambdas:

static void ConstructorFunction()
{
    Func<int,Func<int, int> makeAccount = 
        ((Func<int,Func<int,int>)( (bal) => {
            var balance = bal;
            return (int amount) =>
            {
                if (balance >= amount)
                {
                    balance -= amount;
                    return balance;
                }
                else
                    throw new Exception("Insufficient funds");
            };
        }));

    Console.WriteLine("=============> Closure");
    var acctForEugene = makeAccount(100);
    Console.WriteLine("{0}", acctForEugene(20));
}

Were it not for the implicitly-typed local variable declaration syntax around "acctForEugene", it would be acutely obvious that "makeAccount" isn't creating any kind of object at all, but a function to be executed. Even so, the explicit typing requirement for the lambdas is kind of annoying, and will only get worse as we move through the pattern language.

Method Selector

"You are creating a Function as Object using a Closure. A Constructor Function creates new instances of the object. How do you provide shared access to the closure's state?" After all, an account can do more than just withdraw, but all of the operations on the account have to share the same state--the account balance--without violating encapsulation.

Scheme:

Again we see the nested lambdas, but now there's a third level of nesting; the first invocation (make-account) returns a second invocation that will take a single string, switch on the string, and return a third lambda that will do the actual work of manipulating the balance.

(define make-account
  (lambda (balance)
    (lambda (transaction)
      (case transaction
        ('withdraw
          (lambda (amount)
            (if (>= balance amount)
              (begin
                (set! balance (- balance amount)
                balance)
              (error "Insufficient funds" balance)))))
        ('deposit
          (lambda (amount)
            (set! balance (+ balance amount))
            balance))
        ('balance
          (lambda ()
            balance))
        (else
          (error "Unknown request -- ACCOUNT"
            transaction))))
  ))
(define account-for-eugene (make-account 100))
((account-for-eugene 'withdraw) 10)  => 90
((account-for-eugene 'withdraw) 10)  => 80
((account-for-eugene 'deposit) 100)  => 180

JavaScript:

Doing this in JavaScript is, again, straightforward, though it does seem a little too subtle for idiomatic JavaScript:

(function() {
  out("methodSelector ========")

  var makeAccount = function(bal) {
    var balance = bal
    return function(transaction) {
      if (transaction === "withdraw") {
        return function(amount) {
          if (balance >= amount)
            return (balance = (balance - amount))
          else
            throw new Error("Insufficient funds")
        }
      }
      else if (transaction === "deposit") {
        return function(amount) {
          return (balance = (balance + amount))
        }
      }
      else if (transaction === "balance") {
        return function() {
          return balance
        }
      }
      else {
        throw new Error("Insufficient funds")
      }
    }
  }
  var acctForEugene = makeAccount(100)
  out(acctForEugene("withdraw")(20))
  out(acctForEugene("balance")())
})();

Scala:

This style of interface--passing in a string and a variable list of arguments--really isn't quite Scala's style, since (being a strongly- typed language) it prefers to be able to compile-time-check as much as it can, but that doesn't mean we can't build it when the need and opportunity mesh:

  def methodSelector() = {
    def makeAccount(bal : Int) = {
      var balance = bal
      (transaction : String) => {
        transaction match {
          case "withdraw" =>
            (amt : Int) => {
              if (balance >= amt) {
                balance = (balance - amt) 
                balance
              }
              else 
                throw new RuntimeException("Insufficient funds")
            }
          case "deposit" => {
            (amt : Int) => {
              balance += amt
              balance
            }
          }
          case _ => 
            throw new RuntimeException("Unknown request")
        }
      }
    }
    val acctForEugene = makeAccount(100)
    println(acctForEugene("deposit")(50))
    val acctForTed = makeAccount(100)
    println(acctForTed("withdraw")(50))
  }

F#:

This is, again, like the Yeti version and the Scala version, going to require some sacrifice in terms of flexibility in order to stay true to the original Scheme version--in F#, like in Scala and other statically-typed languages, we have to make sure that all branches of a pattern-match yield the same type of result, so the "balance" branch has to yield a function that takes a parameter (and it must be of the same type of parameter as the other two branches), even though "balance" never makes use of it. This also means that when calling the return value from "makeAccount", even for balance, we have to pass along some parameter that will be ignored.

let methodSelector = fun () ->
    let makeAccount =
        fun (bal : int) ->
            let balance = ref bal
            fun transaction ->
                match transaction with
                | "balance" ->
                    fun _ -> !balance
                | "deposit" ->
                    fun (amt : int) ->
                        balance := (!balance) + amt
                        !balance
                | "withdraw" ->
                    fun (amt : int) ->
                        if amt <= !balance then
                            balance := (!balance) - amt
                            !balance
                        else
                            raise (Exception("Insufficient funds"))
                | _ ->
                    raise (Exception("Unrecognized operation" + transaction))
                            
    Console.WriteLine "=========> Method Selector"
    let acctForEugene = makeAccount 100
    printfn "%d" ((acctForEugene "withdraw") 20)
    printfn "%d" ((acctForEugene "balance") 0)

We can address this required-uniformity-of-access a little bit more consistently with the next pattern element, but whether it's an improvement is debatable.

Yeti (ML):

Nothing new here: the makeAccount function now nests three function literals, just like the JavaScript and Scala ones do. Like the other languages, we use a pattern-match/switch-case construct to decide between the different action strings ("deposit", "withdraw", "balance") and then return the appropriate function literal for further execution. Note that Yeti, like JavaScript, actually has a way of returning an "object" here (a structure, which is a data type the contains one or more named fields, a la objects in JavaScript or case classes in Scala), but since the goal is to remain as faithful as possible to the original Scheme implementation, I stick with the more "functional-only" approach.

makeAccount =
  (do bal:
    var balance = bal;
    do action:
      case action of
        "withdraw": 
          do amt:
            if amt <= balance then
              balance := balance - amt;
              balance
            else
              throw new RuntimeException("Insufficient funds")
            fi
          done;
        "deposit": 
          do amt: 
            balance := balance + amt;
            balance;
          done;
        "balance": 
          do: 
            balance; 
          done;
        _ : throw new RuntimeException("Unknown operation")
      esac
    done;
  done;);

acctForEugene = makeAccount 100;
println ((acctForEugene "withdraw") 20);
println ((acctForEugene "deposit") 20);

Jaskell (Haskell):

C#:

If you stopped reading right here, I wouldn't blame you; this is some ugly C#, without question, particularly considering that there are other ways to accomplishing this same effect without requiring quite so much nesting.

static void MethodSelector()
{
    Func<int, Func<string, Func<int, int> makeAccount =
        ((Func<int, Func<string, Func<int, int>)((bal) =>
        {
            var balance = bal;
            return (string transaction) =>
                {
                    switch (transaction)
                    {
                        case "deposit":
                            return (int amount) =>
                                {
                                    if (balance >= amount)
                                    {
                                        balance -= amount;
                                        return balance;
                                    }
                                    else
                                        throw new Exception("Insufficient funds");
                                };
                        case "withdraw":
                            return (int amount) =>
                                {
                                    balance += amount;
                                    return balance;
                                };
                        case "balance":
                            return (int unused) =>
                                {
                                    return balance;
                                };
                        default:
                            throw new Exception("Illegal operation");
                    }
                };
        }));
    Console.WriteLine("=============> MethodSelector");
    var acctForEugene = makeAccount(100);
    Console.WriteLine("{0}", acctForEugene("deposit")(20));
    Console.WriteLine("{0}", acctForEugene("withdraw")(20));
    Console.WriteLine("{0}", acctForEugene("balance")(0));
}

Bear in mind, too, that there are some other ways to accomplish what the C# code here tries to do, one using dynamic types (from 4.0):

static void MethodSelector2()
{
    Func<int, dynamic> makeAccount = (int bal) =>
    {
        var balance = bal;
        dynamic result = new System.Dynamic.ExpandoObject();
        result.withdraw = (Func<int, int>)((amount) => {
            if (balance >= amount)
            {
                balance -= amount;
                return balance;
            }
            else
                throw new Exception("Insufficient funds");
        });
        result.deposit = (Func<int, int>)((amount) =>
        {
            balance += amount;
            return balance;
        });
        result.balance = (Func<int>)(() => balance);
        return result;
    };

    Console.WriteLine("=============> MethodSelector2");
    var acctForEugene = makeAccount(100);
    Console.WriteLine("{0}", acctForEugene.deposit(20));
    Console.WriteLine("{0}", acctForEugene.balance());
    var acctForTed = makeAccount(100);
    Console.WriteLine("{0}", acctForTed.withdraw(10));
    Console.WriteLine("{0}", acctForTed.balance());
}

... or even using ye old plain ol' Dictionary type, taking a string as a key and yielding Func<> as values for execution:

static void MethodSelector3()
{
    Func<int, Dictionary<string,Func<int,int> makeAccount = 
    (int bal) =>
    {
        var balance = bal;
        var result = new Dictionary<string, Func<int,int>();
        result["withdraw"] = (Func<int, int>)((amount) =>
        {
            if (balance >= amount)
            {
                balance -= amount;
                return balance;
            }
            else
                throw new Exception("Insufficient funds");
        });
        result["deposit"] = (Func<int, int>)((amount) =>
        {
            balance += amount;
            return balance;
        });
        result["balance"] = (Func<int, int>)((unused) => balance);
        return result;
    };

    Console.WriteLine("=============> MethodSelector3");
    var acctForEugene = makeAccount(100);
    Console.WriteLine("{0}", acctForEugene["deposit"](20));
    Console.WriteLine("{0}", acctForEugene["balance"](0));
    var acctForTed = makeAccount(100);
    Console.WriteLine("{0}", acctForTed["withdraw"](10));
    Console.WriteLine("{0}", acctForTed["balance"](0));
}

The second of these two is closer to strict intent of Method Selector from the Scheme example, but the first allows for flexible arity (numbers of parameters) in the functions handed back when dereferenced (so that "balance" doesn't have to take a bogus unused parameter). Frankly, had I to choose, I'd probably go with the dynamic version, just because of that flexibility.

Message-Passing Interface

"You have created a Method Selector for a Function as Object. You prefer to use your object in code that has an object-oriented feel. How do you invoke the methods of an object? [P]rovide a simple message-passing interface for using the closure."

Scheme:

Everything in a Lisp is a list, and the Scheme implementation uses that to full effect by taking the argument list passed in to "send" and splits it up into the object (the account), message (withdraw/deposit/etc), and the arguments (if any) that are left.

(define send
  (lambda argument-list
    (let ((object  (car argument-list))
          (message (car (cdr argument-list)))
          (args    (cdr (cdr argument-list))))
      (apply (get-method object message) args))
  ))
(define get-method
  (lambda (object selector)
    (object selector)
  ))
(define account-for-eugene (make-account 100))
(send account-for-eugene 'withdraw 50)  => 50
(send account-for-eugene 'deposit 100)  => 150
(send account-for-eugene 'balance)      => 150

JavaScript:

In JavaScript, peeling off the head and tail of the arguments reference is trickier here, because unlike Scheme, JavaScript sees "arguments" as an array, not a list. While I could've created "car" and "cdr" functions in JavaScript to perform the relevant operations on an array, it felt more idiomatic to provide a function "slice" to do the "slicing" (which is actually a copy) of elements off the end of the array instead. More importantly, "slice" is a primitive method on Array objects in ECMAScript 5, though neither Node nor Rhino in Java 6 recognize it (I suspect because neither is a compliant ECMAScript 5 environment yet), and if this code ever gets run in a ECMAScript 5 world, then it would/should use that version, instead, since it'll likely be faster than mine.

The other interesting tidbit in here is that when I wrote it the first time, when doing a deposit, the "balance" became "8020", instead of the mathematically-correct "100". JavaScript's "promiscuous typing" thought that the "+" operator wanted to do a string concatenation, instead of a mathematical add of two numbers, so I had to convince it that the value coming out of arguments[1] was, in fact, a number, and the easiest way (it seemed to me at the time) was to just do a quick redundant math operation on it (multiply by 10, then divide by 10 again). There's likely a more idiomatic way to do that, I suspect.

I also note that getMethod() in JavaScript is a bit unnecessary; we could inline its functionality directly inside of send().

(function() {
  out("messagePassingInterface ========")

  var slice = function(src, start, end) {
    var returnVal = []
    var j = 0
    if (end === undefined)
      end = src.length
    for (var i = start; i < end; i++) {
      if (src.length > i)
        returnVal[j++] = src[i]
    }
    return returnVal;
  }
  
  var makeAccount = function(bal) {
    var balance = bal
    return function(transaction) {
      if (transaction === "withdraw") {
        return function(amount) {
          if (balance >= amount)
            return (balance = (balance - amount))
          else
            throw new Error("Insufficient funds")
        }
      }
      else if (transaction === "deposit") {
        return function(amount) {
          return (balance = (balance + (amount * 10.0 / 10.0)))
        }
      }
      else if (transaction === "balance") {
        return function() {
          return balance
        }
      }
      else {
        throw new Error("Insufficient funds")
      }
    }
  }
  var getMethod = function(object, selector) {
    return object(selector)
  }
  var send = function(object, message) {
    return (getMethod(object, message))(slice(arguments, 2))
  }
  var acctForEugene = makeAccount(100)
  out(send(acctForEugene, "withdraw", 20)) // 80
  out(send(acctForEugene, "balance"))      // 80
  out(send(acctForEugene, "deposit", 20))  // 100
  out(send(acctForEugene, "balance"))      // 100
})();

Scala:

The Scala version of this follows the JavaScript version in that it works off of a variable-argument list, but since Scala doesn't give us the built-in "arguments" reference, we have to specify it at the method declaration:

  def messagePassingInterface() = {
    def makeAccount(bal : Int) = {
      var balance = bal
      def send(key:String, args:Any*) = {
        key match {
          case "withdraw" => {
            val amt = args.head.asInstanceOf[Int]
            if (balance >= amt) {
              balance = (balance - amt) 
              balance
            }
            else 
              throw new RuntimeException("Insufficient funds")
          }
          case "deposit" => {
            val amt = args.head.asInstanceOf[Int]
            balance += amt
            balance
          }
        }
      }
      send _
    }
    val acctForEugene = makeAccount(100)
    println(acctForEugene("withdraw", 10))
  }

F#:

By now taking an "obj list" for the parameters, we unify all of the calls to the account to take a consistent parameter list that still allows for a flexible number of parameters, but.... It still requires that callers that don't want to pass any arguments have to pass an empty list. And, on top of that, it doesn't really feel "F#-ish".

let messagePassingInterface = fun () ->
    let makeAccount =
        fun (bal : int) ->
            let balance = ref bal
            fun transaction ->
                match transaction with
                | "balance" ->
                    fun _ -> !balance
                | "deposit" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        balance := (!balance) + amt
                        !balance
                | "withdraw" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        if amt <= !balance then
                            balance := (!balance) - amt
                            !balance
                        else
                            raise (Exception("Insufficient funds"))
                | _ ->
                    raise (Exception("Unrecognized operation" + transaction))
    let getMethod = fun (acct : string -> obj list -> int) selector -> acct selector
    let send = 
        fun (acct : string -> obj list -> int) (message : string) (arglist : obj list) ->
            (getMethod acct message)(arglist)

    Console.WriteLine "=========> Message Passing Interface"
    let acctForEugene = makeAccount 100
    printfn "%d" (send acctForEugene "withdraw" [20])
    printfn "%d" (send acctForEugene "balance" [])

Note that F# does have language facilities for allowing a variable-argument list to be passed, but it only works on method members:

// From the MSDN documentation
open System

type X() =
    member this.F([<ParamArray>] args: Object[]) =
        for arg in args do
            printfn "%A" arg

[<EntryPoint>]
let main _ =
    // call a .NET method that takes a parameter array, passing values of various types
    Console.WriteLine("a {0} {1} {2} {3} {4}", 1, 10.0, "Hello world", 1u, true)

    let xobj = new X()
    // call an F# method that takes a parameter array, passing values of various types
    xobj.F("a", 1, 10.0, "Hello world", 1u, true)
    0

We could go back and rewrite all of the F# samples to be class member methods (that is, return actual objects), but that sort of gets away from the spirit of what the blog exercise is trying to do, so I'll leave that as an exercise to the reader. (Which, by the way, is author-speak for "I'm feeling lazy and I don't want to bother".)

Yeti (ML):

Unfortunately, while Yeti (like most functional languages) has a built-in list type, it doesn't recognize arguments to a function as a list, so we either have to explicitly put the arguments in, or we have to explicitly state that the arguments to the returned function literal are a list. I choose the latter tactic, even though it's not the world's most impressive syntax:

makeAccount =
  (do bal:
    var balance = bal;
    do action:
      case action of
        "withdraw":
          do argList:
            amt = head argList;
            if amt <= balance then
              balance := balance - amt;
              balance;
            else
              throw new RuntimeException("Insufficient funds")
            fi
          done;
        "deposit": 
          do argList:
            amt = head argList; 
            balance := balance + amt;
            balance;
          done;
        "balance": 
          do: 
            balance; 
          done;
        _ : throw new RuntimeException("Unknown operation")
      esac
    done;
  done;);

acctForEugene = makeAccount 100;
println  ((acctForEugene "withdraw")[20]);  // 80

If there's a way to get a Yeti function to accept a variable number of arguments, I've not seen it in the language overview. I don't know if any ML-derivative has this, to be honest. Of course, the other thing to do, since this is a statically-typed environment, is to just return function literals that expect the proper number of arguments, which will get us the compile-time safety that these languages are supposed to provide; the below does exactly that--the last line will fail to compile if you uncomment it:

makeAccount =
  (do bal:
    var balance = bal;
    do action:
      case action of
        "withdraw":
          do amt:
            if amt <= balance then
              balance := balance - amt;
              balance;
            else
              throw new RuntimeException("Insufficient funds")
            fi
          done;
        "deposit": 
          do amt:
            balance := balance + amt;
            balance;
          done;
        "balance": 
          do: 
            balance; 
          done;
        _ : throw new RuntimeException("Unknown operation")
      esac
    done;
  done;);

acctForEugene = makeAccount 100;
println ((acctForEugene "withdraw") 20);      // 80
println ((acctForEugene "balance") 0);        // 80
//println ((acctForEugene "withdraw") "fred");  // won't compile

(Truthfully, we should do this for the Scala version, too.) This choice is going to cause us a little bit of heartache, though, because in order to use "balance", we have to pass in a number--if we leave off the "_" in the function literal returned from the "balance" arm of the selector, we don't need to pass "0" when we invoke it, but what's returned isn't a number, but a function. I can't figure out how to make Yeti take that function and just invoke it--the syntax guide doesn't seem to say out loud exactly how I can invoke that function without having to pass in a number argument. If I'd left it as taking a list, then I could pass an empty list and all would look consistent, if a little weird.

(Note that this is deliberately opposite what I chose to do for the F# version.)

Jaskell (Haskell):

C#:

Generic Function

"You have created a Method Selector for a Function as Object. You want to take full advantage of the tools available in your functional language. How do you invoke the methods of an object? ... [P]rovide a simple interface to the Method Selector that more closely follows the functional style."

Scheme:

In the Scheme implementation, it's interesting that having written the send function in the last element of the pattern language, we don't really use it here, but instead just inline its functionality in each of the named functions (which, in turn, take the argument list, peel off the head of the argument list as the account object, and pass the remainder of the arguments on to the selected function):

(define withdraw
  (lambda argument-list
    (let ((object (car argument-list))
          (withdraw-arguments (cdr argument-list)))
      (apply (object 'withdraw) withdraw-arguments)
    )))
(define deposit
  (lambda argument-list
    (let ((object (car arguments))
          (deposit-arguments (cdr arguments)))
      (apply (object 'deposit) deposit-arguments)
    )))
(define balance
  (lambda (object)
    (object 'balance)
  ))
  
(define account-for-eugene (make-account 100))
(withdraw account-for-eugene 10)
(map  (lambda (account) (deposit account 10)) account-for-eugene)

Interestingly enough, I sort of expected the Scheme version to use "deposit" directly, rather than write a trampoline that calls "deposit", since we could've avoided the Generic Function part of the language just by using "send" directly, as well:

(map  (lambda (account) (send account 'deposit 10)) account-for-eugene)

And, to be honest, calling "map" on a single object doesn't really seem to be a profoundly functional experience, so in my examples I'm going to create a collection of accounts (called a "bank", naturally enough), and map across that collection.

JavaScript:

The JavaScript version of this is, again, pretty similar to the Scheme version. Again, ECMAScript 5 environments are supposed to have a "map" function natively built in, but previous environments don't, so I have to write one to verify that we can, in fact, use the named functions as the mapped operation. I also write a "map2", another version of map that takes the function to apply to the collection but also takes any additional arguments after that and passes them to the function being applied across the collection; it allows me to use "deposit" directly, instead of having to write a trampoline for it, and besides, it's trivial to write in JavaScript:

(function() {
  out("genericFunction ==============")

  var slice = function(src, start, end) {
    var returnVal = []
    var j = 0
    if (end === undefined)
      end = src.length
    for (var i = start; i < end; i++) {
      if (src.length > i)
        returnVal[j++] = src[i]
    }
    return returnVal
  }
  
  var map = function(fn, src) {
    var retVal = []
    for (i in src)
      retVal[i] = fn(src[i])
    return retVal
  }
  var map2 = function(src, fn) {
    var retVal = []
    for (i in src)
      retVal[i] = fn(src[i], slice(arguments, 2))
    return retVal
  }
  
  var makeAccount = function(bal) {
    var balance = bal
    return function(transaction) {
      if (transaction === "withdraw") {
        return function(amount) {
          if (balance >= amount)
            return (balance = (balance - amount))
          else
            throw new Error("Insufficient funds")
        }
      }
      else if (transaction === "deposit") {
        return function(amount) {
          return (balance = (balance + (amount * 10.0 / 10.0)))
        }
      }
      else if (transaction === "balance") {
        return function() {
          return balance
        }
      }
      else {
        throw new Error("Insufficient funds")
      }
    }
  }
  var withdraw = function() {
    var object = arguments[0]
    var argumentList = slice(arguments, 1)
    return object("withdraw")(argumentList)
  }
  var deposit = function() {
    var object = arguments[0]
    var argumentList = slice(arguments, 1)
    return object("deposit")(argumentList)
  }
  var balance = function(object) {
    return object("balance")()
  }

  var acctForEugene = makeAccount(100)
  out(withdraw(acctForEugene, 20))
  out(deposit(acctForEugene, 20))
  
  var bank = [
    makeAccount(100),  // acctForEugene
    makeAccount(1000)  // acctForTed
  ]
  map(function(it) { deposit(it, 20) }, bank)
  out(balance(bank[0]))
  out(balance(bank[1]))
  
  map2(bank, deposit, 20)
  out(balance(bank[0]))
  out(balance(bank[1]))
})();

Scala:

Scala, of course, has functional methods built onto its List type (which we can use instead of an array, since Scala has much better support for lists than arrays):

  def genericFunction() = {
    def makeAccount(bal : Int) = {
      var balance = bal
      def send(key:String, args:Any*) = {
        key match {
          case "withdraw" => {
            val amt = args.head.asInstanceOf[Int]
            if (balance >= amt) {
              balance = (balance - amt) 
              balance
            }
            else 
              throw new RuntimeException("Insufficient funds")
          }
          case "deposit" => {
            val amt = args.head.asInstanceOf[Int]
            balance += amt
            balance
          }
          case "balance" => {
            balance
          }
          case _ =>
            throw new RuntimeException("Unknown request")
        }
      }
      send _
    }
    def withdraw(account : (String, Any*) => Int, amount : Int) = {
      account("withdraw", amount)
    }
    def deposit(account : (String, Any*) => Int, amount : Int) = {
      account("deposit", amount)
    }
    def balance(account : (String, Any*) => Int) = {
      account("balance")
    }
    val accounts = List(makeAccount(100), makeAccount(200), makeAccount(300))
    accounts.foreach(withdraw(_, 20))
    accounts.foreach((in) => { println(balance(in)) })
  }

F#:

The F# version helps clean up some of the syntax a little, sort of:

let genericFunction = fun () ->
    let makeAccount =
        fun (bal : int) ->
            let balance = ref bal
            fun transaction ->
                match transaction with
                | "balance" ->
                    fun _ -> !balance
                | "deposit" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        balance := (!balance) + amt
                        !balance
                | "withdraw" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        if amt <= !balance then
                            balance := (!balance) - amt
                            !balance
                        else
                            raise (Exception("Insufficient funds"))
                | _ ->
                    raise (Exception("Unrecognized operation" + transaction))
    let deposit = 
        fun amt acct->
            acct "deposit" [amt :> obj]
    let withdraw =
        fun amt acct ->
            acct "withdraw" [amt :> obj]
    let balance =
        fun acct ->
            acct "balance" []

    Console.WriteLine "=========> Generic Function"
    let bank = [ makeAccount 100; makeAccount 200; makeAccount 300 ]
    let balances = List.map (fun it -> deposit 20 it) bank
    List.iter (fun it -> printfn "%d" it) balances
    let balances = List.map (deposit 20) bank
    List.iter (printfn "%d") balances

Notice that by putting the account counter-intuitively as the last parameter to the generic "deposit" and "withdraw" functions, we can avoid having to write the "trampoline" function that we would've had to write when using "map"; the account gets curried from the List directly (as shown in the second example). We could do the same thing in the Scala version, too, and then wouldn't have to use the explicit "_" syntax that Scala provides. Of course, if the desire is instead to pass the amount in a curried fashion, instead of the account, then the original ordering of the parameters is better.

Yeti (ML):

Writing this in Yeti/ML is definitely trickier than it was in JavaScript, despite the built-in "map" and other functions, because getting the arguments to "trampoline" right is a little harder. Fortunately, the generic method hides the "balance 0" weirdness from the last pattern element, making it a tad easier to use:

makeAccount =
  (do bal:
    var balance = bal;
    do action:
      case action of
        "withdraw":
          do amt:
            if amt <= balance then
              balance := balance - amt;
              balance
            else
              throw new RuntimeException("Insufficient funds")
            fi
          done;
        "deposit": 
          do amt:
            balance := balance + amt;
            balance
          done;
        "balance": 
          do: 
            balance
          done;
        _ : throw new RuntimeException("Unknown operation")
      esac
    done;
  done;);

withdraw =
  (do acct amt:
    (acct "withdraw") amt;
  done;);
deposit =
  (do acct amt:
    (acct "deposit") amt;
  done;);
balance =
  (do acct:
    acct "balance" 0;
  done;);

acctForEugene = makeAccount 100;
println (withdraw acctForEugene 20);      // 80
println (deposit acctForEugene 20);       // 100
println (balance acctForEugene);          // 100

accounts = [(makeAccount 100), (makeAccount 200), (makeAccount 300)];
balances = map (do acct: (deposit acct 20) done) accounts;
for accounts do acct: println(deposit acct 20) done;

Yeti complained if I didn't bind the result of the "map" call to a value, hence the "balances" value there, even though the balances are actually also stored in the relevant closures for each account. Note that the "for" line that follows it actually does the same thing, and prints the results out, to boot. In fact, it's high time people started to realize that the "for" loop in most imperative languages is just a non-functional way of doing a "map" without yielding a value. Languages like Scala and Yeti/ML essentially blur that line significantly enough to the point where we should just eschew "for" altogether and use "map", if you ask me.

Jaskell (Haskell):

C#:

Delegation

"You are creating a Function as Object. How do you create a new object that extends the behavior of an existing object? ... [U]se delegation. Make a Function as Object that has an instance variable an instance of the object you want to extend. Implement behaviors specific to the new object as methods in a Method Selector. Pass all other messages onto the instance variable."

Again, in a traditional O-O language, we'd just inherit, and in an object- functional hybrid, we could do the same. There's no real point not to, to be honest. But the interesting thing about this implementation is that it demonstrates the runtime relationship between a JavaScript object and its prototype: calling a function passing in the "derived" object causes the "derived" to try its "base" (its prototype) in the event that the method in question isn't defined on the "derived".

Note also that this particular trick is really only feasible because the "object" presents a uniform interface: all interaction with the "object" (whether it is a standard account or an interest-bearing one) is done through the Method Selector mechanism, which allows for this extension without having to modify any sort of base interface. This isn't so much a knock on O-O as a whole as it is on statically-typed traditional O-O.

Scheme:

This is pretty straightforward, if you understood the Message-Passing Interface implementation of earlier.

(define make-interest-bearing-account
  (lambda (balance interest-rate)
    (let ((my-account (make-account balance)))
      (lambda (transaction)
        (case transaction
          ('accrue-interest
            (lambda ()
              ((my-account 'deposit)
                (* ((my-account 'balance))
                   interest-rate)) ))
        (else
          (my-account transaction))
        )))
  ))
(define account-for-eugene (make-interest-bearing-account 100 0.05))
((account-for-eugene 'balance))         => 100
((account-for-eugene 'deposit) 100)     => 200
((account-for-eugene 'balance))         => 200
((account-for-eugene 'accrue-interest)) => 210
((account-for-eugene 'balance))         => 210

JavaScript:

Despite the fact that the JavaScript implementation just keeps getting longer and longer, it's actually not that much harder to add in this delegation functionality--again, as has been the case for a lot of the JavaScript code, it's almost a direct one-to-one port from the Scheme:

(function() {
  out("delegation =======")
  
  var slice = function(src, start, end) {
    var returnVal = []
    var j = 0
    if (end === undefined)
      end = src.length
    for (var i = start; i < end; i++) {
      if (src.length > i)
        returnVal[j++] = src[i]
    }
    return returnVal
  }
  
  var makeAccount = function(bal) {
    var balance = bal
    return function(transaction) {
      if (transaction === "withdraw") {
        return function(amount) {
          if (balance >= amount)
            return (balance = (balance - amount))
          else
            throw new Error("Insufficient funds")
        }
      }
      else if (transaction === "deposit") {
        return function(amount) {
          return (balance = (balance + (amount * 10.0 / 10.0)))
        }
      }
      else if (transaction === "balance") {
        return function() {
          return balance
        }
      }
      else {
        throw new Error("Insufficient funds")
      }
    }
  }
  var makeInterestBearingAccount = function(bal, intRate) {
    var myAccount = makeAccount(bal)
    return function(transaction) {
      if (transaction === "accrueInterest") {
        return function() {
          var balance = myAccount("balance")()
          var interest = (balance * intRate)
          return myAccount("deposit")(interest)
        }
      }
      else
        return myAccount(transaction)
    }
  }
  
  var acctForEugene = makeInterestBearingAccount(100, 0.05)
  out(acctForEugene("balance")())
  out(acctForEugene("deposit")(20))
  out(acctForEugene("accrueInterest")())
  out(acctForEugene("balance")())
})();

Scala:

The Scala version of this is tricky, because it relies on a very subtle bit of Scala syntax; specifically, when we try to pass the "args" sequence (which, in actual implementation, is a WrappedArray) from the "makeInterestBearingAccount" function to the "makeAccount" function (by which I mean, the functions returned from those two functions), if we don't use the peculiar ": _*" syntax, Scala interprets "args" to be a single parameter (a single parameter whose type is a collection), instead of the intended "pass the arguments through" behavior. (If you're a Java or C# developer, it's like having a varargs method calling another varargs method, and passing the array of arguments from the first as an array instead of each element on its own to form the array of arguments in the second. Yeah, I know--it's a little brain-twisty.)

  def delegation() = {
    def makeAccount(bal : Int) = {
      var balance = bal
      def send(key:String, args:Any*) = {
        key match {
          case "withdraw" => {
            val amt = args.head.asInstanceOf[Int]
            if (balance >= amt) {
              balance = (balance - amt) 
              balance
            }
            else 
              throw new RuntimeException("Insufficient funds")
          }
          case "deposit" => {
            val amt = args.head.asInstanceOf[Int]
            balance += amt
            balance
          }
          case "balance" => {
            balance
          }
          case _ =>
            throw new RuntimeException("Unknown request")
        }
      }
      send _
    }
    def makeInterestBearingAccount(bal : Int, intRate : Double) = {
      val account = makeAccount(bal)
      def send(key: String, args:Any*) = {
        key match {
          case "accrueInterest" => {
            val amt = (int2float(account("balance")) * intRate).toInt
            account("deposit", amt)
          }
          case _ =>
            account(key, args : _*)
        }
      }
      send _
    }
    val acctForEugene = makeInterestBearingAccount(100, 0.05)
    println(acctForEugene("deposit", 20))
    println(acctForEugene("accrueInterest"))
    println(acctForEugene("balance"))
  }

F#:

Aside from the aforementioned weirdness about the obj list as a generic parameter mechanism, this is really straightforward:

let delegation = fun () ->
    let makeAccount =
        fun (bal : int) ->
            let balance = ref bal
            fun transaction ->
                match transaction with
                | "balance" ->
                    fun _ -> !balance
                | "deposit" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        balance := (!balance) + amt
                        !balance
                | "withdraw" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        if amt <= !balance then
                            balance := (!balance) - amt
                            !balance
                        else
                            raise (Exception("Insufficient funds"))
                | _ ->
                    raise (Exception("Unrecognized operation" + transaction))
    let makeInterestBearingAccount =
        fun (bal : int) (intRate : float) ->
            let account = makeAccount bal
            fun transaction ->
                match transaction with
                | "accrueInterest" ->
                    fun _ ->
                        let balance = (account "balance" [])
                        let interest : float = float balance * intRate 
                        account "deposit" [int interest]
                | _ -> account transaction
    Console.WriteLine "=========> Delegation"
    let acctForEugene = makeInterestBearingAccount 100 0.05
    printfn "%d" (acctForEugene "deposit" [20])
    printfn "%d" (acctForEugene "accrueInterest" [])

Note the explicit casts in the "accrueInterest" code: this is because F#, like a lot of functional languages, won't do automatic type-promotion for you. So the "int"s have to be explicitly converted to "float"s, and back again.

Yeti (ML):

Since we didn't go down the path of trying to do the variable-argument list in Yeti, we don't have the same problems the Scala version presented, and the generic methods (the top-level "withdraw", "deposit" and "balance" functions) actually help hide the syntactic weirdness that we ran into in the last pattern element:

makeAccount =
  (do bal:
    var balance = bal;
    do action:
      case action of
        "withdraw":
          do amt:
            if amt <= balance then
              balance := balance - amt;
              balance
            else
              throw new RuntimeException("Insufficient funds")
            fi
          done;
        "deposit": 
          do amt:
            balance := balance + amt;
            balance
          done;
        "balance": 
          do: 
            balance
          done;
        _ : throw new RuntimeException("Unknown operation")
      esac
    done;
  done;);

withdraw =
  (do acct amt:
    (acct "withdraw") amt;
  done;);
deposit =
  (do acct amt:
    (acct "deposit") amt;
  done;);
balance =
  (do acct:
    acct "balance" 0;
  done;);

acctForEugene = makeAccount 100;
println (withdraw acctForEugene 20);      // 80
println (deposit acctForEugene 20);       // 100
println (balance acctForEugene);          // 100

accounts = [(makeAccount 100), (makeAccount 200), (makeAccount 300)];
balances = map (do acct: (deposit acct 20) done) accounts;
for accounts do acct: println(deposit acct 20) done;

Note the last two lines--the "for" construct in most imperative languages is actually akin to the "map" construct in most functional languages, except that in the imperative "for" there's no return value from the expression, and in a functional "map" there (usually) is. This is why we have to bind the result from the "map" to a name, and we don't have any results from the "for". (The "map" also insists on having a returned value--a list of Unit isn't acceptable, which is what would be returned if we used the "println" expression in the "map".)

Jaskell (Haskell):

C#:

Private Method

"You have created a Method Selector. How do you factor common behavior out of the methods in the Method Selector? ... [D]efine the common code in a Local Procedure (Wallingford). Invoke this procedure in place of the duplicated code within the Method Selector."

Scheme:

(define make-account
  (lambda (balance)
    (let ((transaction-log '())
      (log-transaction
        (lambda type amount)
          (set! transaction-log
                (cons (list type amount)
                      transaction-log)))) )
      (lamba (transaction)
        (case transaction
          ('withdraw
            (lambda (amount)
              (if (>= balance amount)
                (begin
                  (set! balance (- balance amount))
                  (log-transaction 'withdraw amount)
                  balance)
                (error "Insufficient funds" balance))))
          ('deposit
            (lambda (amount)
              (set! balance (+ balance amount))
              (log-transaction 'deposit amount)
              balance))
        ...))
    ))

JavaScript:

Again, in JavaScript, we rely on the fact that anything declared inside the "makeAccount" function but outside the function returned by "makeAccount" is encapsulated, and create both the "transactionLog" (an array, since JavaScript likes those better than lists) and the function to append to it ("logTransaction") within that "neutral zone". Just to prove that the transaction log is being written, I add another method to the method selector table, "viewLog", to return the contents of the transaction log.

(function() {
  out("privateMethod ===========")
  
  var makeAccount = function(bal) {
    var transactionLog = []
    var logTransaction = function(type, amount) {
      transactionLog.push("Action: " + type + " for " + amount)
    }
    
    var balance = bal
    return function(transaction) {
      if (transaction === "withdraw") {
        return function(amount) {
          if (balance >= amount) {
            logTransaction("withdraw", amount)
            return (balance = (balance - amount))
          }
          else
            throw new Error("Insufficient funds")
        }
      }
      else if (transaction === "deposit") {
        return function(amount) {
          logTransaction("deposit", amount)
          return (balance = (balance + (amount * 10.0 / 10.0)))
        }
      }
      else if (transaction === "balance") {
        return function() {
          logTransaction("balance", balance)
          return balance
        }
      }
      else if (transaction === "viewLog") {
        return function() {
          return (transactionLog)
        }
      }
      else {
        throw new Error("Insufficient funds")
      }
    }
  }
  var acctForEugene = makeAccount(100)
  out(acctForEugene("withdraw")(20))
  out(acctForEugene("balance")())
  out(acctForEugene("deposit")(20))
  out(acctForEugene("balance")())
  out(acctForEugene("viewLog")())
})();

Scala:

The Scala version is also pretty straightforward--we've already seen that Scala supports nested functions, so it is simply a matter of defining the logTransaction() function and an empty List[String] in the same "neutral zone" in which the "balance" variable lives. Instead of adding a new selector to the list, I chose this time to just print the transaction log as part of the "balance" operation.

  def privateMethod() = {
    def makeAccount(bal : Int) = {
      var balance = bal
      var transactionLog = List[String]()
      def logTransaction(action:String, amount:Int) = {
        val msg = ("Action: " + action + " for " + amount)
        transactionLog = transactionLog :+ msg
      }
      def send(key:String, args:Any*) = {
        key match {
          case "withdraw" => {
            val amt = args.head.asInstanceOf[Int]
            if (balance >= amt) {
              logTransaction("withdraw", amt)
              balance = (balance - amt) 
              balance
            }
            else 
              throw new RuntimeException("Insufficient funds")
          }
          case "deposit" => {
            val amt = args.head.asInstanceOf[Int]
            logTransaction("deposit", amt)
            balance += amt
            balance
          }
          case "balance" => {
            println(transactionLog)
            balance
          }
          case _ =>
            throw new RuntimeException("Unknown request")
        }
      }
      send _
    }
    val acctForEugene = makeAccount(100)
    println(acctForEugene("deposit", 20))
    println(acctForEugene("balance"))
  }

F#:

Binding a local function is, by this point, somewhat trivial and uninspiring, but it's just as easily done in F# as it is in any of the other languages:

let privateMethod = fun () ->
    let makeAccount =
        fun (bal : int) ->
            let transactionLog = ref []
            let logTransaction act (amt : int) =
                let message = "Action: " + act + " for " + amt.ToString()
                transactionLog := List.append !transactionLog [message]
            let balance = ref bal
            fun transaction ->
                match transaction with
                | "balance" ->
                    fun _ -> 
                        List.iter (printfn "%s") !transactionLog
                        !balance
                | "deposit" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        balance := (!balance) + amt
                        logTransaction "deposit" amt
                        !balance
                | "withdraw" ->
                    fun (arglist : obj list) ->
                        let amt = arglist.Head :?> int
                        if amt <= !balance then
                            balance := (!balance) - amt
                            logTransaction "withdraw" amt
                            !balance
                        else
                            raise (Exception("Insufficient funds"))
                | _ ->
                    raise (Exception("Unrecognized operation" + transaction))
    Console.WriteLine "=========> Private Method"
    let acctForEugene = makeAccount 100
    printfn "%d" (acctForEugene "deposit" [20])
    printfn "%d" (acctForEugene "withdraw" [50])
    printfn "%d" (acctForEugene "balance" [])

Yeti (ML):

The private method in Yeti is, again, just a nested function hiding out in the closure that is returned by "makeAccount"; the fact that Yeti supports expressions embedded inside of strings makes it easy to create the transaction log string:

makeAccount =
  (do bal:
    var balance = bal;
    var transactionLog is list<string> = [];
    logTransaction action amount = 
      transactionLog := "Action: \(action) for \(amount)" :: transactionLog;
    do action:
      case action of
        "withdraw":
          do amt:
            if amt <= balance then
              logTransaction "withdraw" amt;
              balance := balance - amt;
              balance
            else
              throw new RuntimeException("Insufficient funds")
            fi
          done;
        "deposit": 
          do amt:
            logTransaction "deposit" amt;
            balance := balance + amt;
            balance
          done;
        "balance": 
          do: 
            println transactionLog;
            balance
          done;
        _ : throw new RuntimeException("Unknown operation")
      esac
    done;
  done;);

Jaskell (Haskell):

C#:

Summary

JavaScript is, of course, the de-facto golden child right now.

And Scala is, undoubtedly, one of my favorites. It's syntax is a little quirky in places, but no more so than any other language I've used.

I like the Yeti code style and syntax, and could definitely see doing some small projects in it, particularly some service-y kinds of things with it, a la Web or REST services; the Yeti source code has some examples of how to create (for example) servlets and WARs, and it's a nice syntax. I don't know that I'd want to create a full-fledged MVC framework on top of Yeti, but as something that's basically taking input, doing processing and sending back JSON or XML results, it's not a bad approach. Considering you can also create classes in Yeti, which puts it into the same grounds as F#, it's worth looking into if you've got some ML in your background and want to go back to it while staying on top of the JVM.

The F# version is a nice mix of ML and objects, though the casting operators are definitely a syntax that only a mother could love, and the distinction between what is allowed on functions vs. methods (such as the parameter arrays) feels a little arbitrary at times. (I'm sure there's good reasons for it, but it still feels a little arbitrary, at least to me.) The "cannot close over local variables, use refs instead" rule is also a little annoying, although it does make it explicitly clear that now you're closing over a reference, not the actual value, so now the "what happens if I modify the closed-over value" question becomes self-explanatory. (This sometimes trips people up in other languages that don't make the by-value or by-reference closing-over semantics explicit.)

Honestly, I don't really expect that anyone reading this piece is going to immediately turn around, abandon all their domain objects, and take up this approach as a replacement--in some cases, taking this "all functional" style creates more angst than it really provides benefits--but we can use parts of it to generate some really interesting new patterns.

On Knowledge

Thu, 29 Nov 2012 00:00:00 +0000

Back during the Bush-Jr Administration, Donald Rumsfeld drew quite a bit of fire for his discussion of knowledge, in which he said (loosely paraphrasing) "There are three kinds of knowledge: what you know you know, what you know you don't know, and what you don't know you don't know". Lots of Americans, particularly those who were not kindly disposed towards "Rummy" in the first place, took this to be canonical Washington doublespeak, and berated him for it.

I actually think that was one of the few things Rumsfeld said that was worth listening to, and I have a slight amendment to the statement; but first, let's level-set and make sure we're all on the same page about what those first three categories mean, in real life, with a few assumptions along the way to simplify the discussion (as best we can, anyway):

What you know you know. This is the category of information that the individual in question has studied to some level of depth: for a student of International Relations (as I was), this would be the various classes that they took and received (presumably) a passing grade in. For you, the reader of my blog, that would probably be some programming language and/or platform. This is knowledge that you have, in some depth, at a degree that most people would consider "factually accurate".
What you know you don't know. This is the category of information that the individual in question has heard about, but has never studied to any level or degree: for the student of International Relations, this might be the subject of biochemistry or electrical engineering. For you, the reader of my blog, it might be certain languages that you've heard of, perhaps through this blog (Erlang, F#, Scala, Clojure, Haskell, etc) or data-storage systems (Cassandra, CouchDB, Riak, Redis, etc) that you've never investigated or even sat through a lecture about. This is knowledge that you realize you don't have.
What you don't know you don't know. This is the category of information that the individual in question has never even heard about, and so therefore, by definition, has not only the lack of knowledge of the subject, but lacks the realization that they lack the knowledge of the subject. For the student of International Relations, this might be phrenology or Schrodinger's Cat. For you, the reader of my blog, it might be languages like Dylan, Crack, Brainf*ck, Ook, or Shakespeare (which I'm guessing is going to trigger a few Google searches) or platforms like BeOS (if you're in your early 20's now), AmigaOS (if you're in your early 30's now) or database tools/platforms/environments like Pick or Paradox. This is knowledge that you didn't realize you don't have (but, paradoxically, now that you know you don't have it, it moves into the "know you don't know" category).

Typically, this discussion comes up in my "Pragmatic Architecture" talk, because an architect needs to have a very clear realization of what technologies and/or platforms are in which of those three categories, and (IMHO) push as many of them from category #3 (don't know that you don't know) into category #2 (know you don't know) or, ideally, category #1 (know you know). Note that category #1 doesn't mean that you are the world's foremost expert on the thing, but you have some working knowledge of the thing in question--I don't consider myself to be an expert on Cassandra, for example, but I know enough that I can talk reasonably intelligently to it, and I know where I can get more in the way of details if that becomes important, so therefore I peg it in category #1.

But what if I'm wrong?

See, here's where I think there's a new level of knowledge, and it's one I think every software developer needs to admit exists, at least for various things in their own mind:

What you think you know. (Or, perhaps, another variant of "What you don't know that you don't know.") This is knowledge that you believe, in your heart of hearts, you have about a given subject, and it is flat-out incorrect and wrong.

Be honest with yourself: we've all met somebody in this industry who claims to have knowledge/expertise on a subject, and damn if they can't talk a good game. They genuinely believe, in fact, that they know the subject in question, and speak with the confidence and assurance that comes with that belief. (I'm assuming that the speaker in question isn't trying to deliberately deceive anyone, which may, in some cases, be a naive and/or false assumption, but I'm leaving that aside for now.) But, after a while, it becomes apparent, either to themselves or to the others around them, that the knowledge they have is either incorrect, out of date, out of context, or some combination of all three.

As much as "what you don't know you don't know" information is dangerous, "what you think you know" information is far, far more so, particularly because until you demonstrate to yourself that your information is actually correct, you're a danger and a liability to anyone who listens to you. Without regularly challenging yourself to some form of external review/challenge, you'll never exactly know whether what you know is real, or just made up from your head.

This is why, at every turn, your assumption should be that any information you have is some or all incorrect until proven otherwise. Find out why you know something--what combination of facts/data lead you to believe that this is the case?--and you will quickly begin to discover whether that knowledge is real, or just some kind of elaborate self-deception.

On Equality

Fri, 12 Oct 2012 00:00:00 +0000

Recently (over the last half-decade, so far as I know) there's been a concern about the numbers of women in the IT industry, and in particular the noticeable absence of women leaders and/or industry icons in the space. All of the popular languages (C, C++, Java, C#, Scala, Groovy, Ruby, you name it) have been invented by or are represented publicly by men. The industry speakers at conferences are nearly all men. The rank-and-file that populate the industry are men. And this strikes many as a bad thing.

(UPDATE 2021: As part of porting this post over to the new system, I took a hard look at this post, and find that I'm feeling a little differently than what I expressed a decade prior. I leave it here for posterity, but some of my statements below are definitely not what I'd say today, and will post a followup at some point down the road to get deeper into the nuance of my current belief.)

Honestly, I used to be a lot more concerned than I am today. While I'm sure that many will see my statements and position that follows as misogynistic and/or discriminatory, let me be the first to suggest quite plainly that I have nothing against any woman who wants to be a programmer, who wants to be an industry speaker, or who wants to create a startup and/or language and/or library and/or framework and/or tool and/or any other role of leadership and authority within the industry. I have always felt that this industry is more merit-based than any other I have ever had direct or indirect contact with. There is no need for physical strength, there is no need for dexterity or mobility, there is no need for any sort of physical stress tolerances (such as the G forces fighter pilots incur during aerial combat which, by the way, women are actually scientifically better at handling than men), there really even is no reason that somebody who is physically challenged couldn't excel here. So long as you can type (or, quite frankly, have some other mechanism by which you can put characters into an IDE), you can program.

And no, I have no illusions that somehow men are biologically wired better to be leaders. In fact, I think that as time progresses, we will find that the stereotypical characteristics that we ascribe to each of the genders (male competitiveness and female nuturing) each serve incredibly useful purposes in the IT world. Cathi Gero, for example, was once referred to by a client in my presence as "the Mom of the IT department"--by which they meant, Cathi would simply not rest until everything was exactly as it should be, a characteristic that they found incredibly comforting and supportive. Exactly the kind of characteristic you would want from a highly-paid consultant: that they will stick with you through all the mess until the problem is solved.

And no, I also have no illusions that somehow I understand what it's been like to be a woman in IT. I've never experienced the kind of "automatic discrimination" that women describe, being mistaken for recruiters at a technical conference, rather than as a programmer. I won't even begin to try and pretend that I know what that's like.

Unless, of course, I can understand it by analogy, such as when a woman sees me walking down the street, and crosses the street ahead of me so that she won't have to share the sidewalk, for even a second, with a long-haired, goateed six-foot-plus stranger. She has no reason to assume I represent any threat to her other than my physical appearance, but still, her brain makes the association, and she chooses to avoid the potential possibility of threat. Still, that's probably not the same.

What I do think, quite bluntly, is that one of the reasons we don't have more women in IT is because women simply choose not to be here.

Yes, I know, there are dozens of stories of misogynistic behavior at conferences, and dozens more stories of discriminatory behavior. Dozens of stories of "good ol' boys behavior" making women feel isolated, and dozens of stories of women feeling like they had to over-compensate for their gender in order to be heard and respected. But for each conference story where a woman felt offended by a speakers' use of a sexual epithet or joke, there are dozens of conferences where no such story ever emerges.

I'm reminded of a story, perhaps an urban myth, of a speaker at a leadership conference that stood in front of a crowd, took a black marker, made a small circle in the middle of a flip board, and asked a person in the first row what they saw. "A black spot", they replied. A second person said the same thing, and a third. Finally, after about a half-dozen responses of "a block spot", the speaker said, "All of you said you saw the same thing: a black spot. I'm curious as to why none of you saw the white background behind it".

It's easy for us to focus on the outlier and give that attention. It's even easier when we see several of them, and if they come in a cluster, we call it a "dangerous trend" and "something that must be addressed". But how easy it is, then, to miss the rest of the field, in the name of focusing on the outlier.

My ex-apprentice wants us to proactively hire women instead of men in order to address this lack:

Bring women to the forefront of the field. If you're selecting a leader and the best woman you can find is not as qualified as the best man you can find, (1) check your numbers to make sure unintentional bias isn't working against her, and (2) hire her anyway. She is smart and she will rise to the occasion. She is not as experienced because women haven't been given these opportunities in the past. So give it to her. Next round, she will be the most qualified.
Am I advocating affirmative action in hiring? No, I'm advocating blind hiring as much as is feasible. This has worked for conferences that do blind session selection and seek out submissions from women. However, I am advocating deliberate bias in favor of a woman in promotions, committee selection, writing and speaking solicitation, all technical leadership positions. The small biases have multiplied until there are almost no women in the highest technical levels of the field.

But you can't claim that you're advocating "blind hiring" while you're saying "hire her anyway" if she "is not as qualified as the best man you can find". This is, by definition, affirmative action, and while it does put women into those positions, it doesn't address the underlying problem--that she isn't as qualified. There is no reason that she shouldn't be as qualified as the man, so why are we giving her a pass? Why is it this company's responsibility to fix the industry at a cost to themselves? (I'm assuming, of course, that there is a lost productivity or lost innovation or some other cost to not hiring the best candidate they can find; if such a loss doesn't exist, then there's no basis for assuming that she isn't equally qualified as the man.)

Did women routinely get "railroaded" out of technical directions (math and science) and into more "soft areas" (English and fine arts) in schools back when I was a kid? Yep. Studies prove that. My wife herself tells me that she was "strongly encouraged" to take more English classes than math or science back in Junior high and high school, even when her grades in math and science were better than those in English. That bias happened. But does it happen with girls today? Studies I'm reading about third-hand suggest not appreciably. And even if you were discriminated against back then, what stops you now? If you're reading this, you have a computer, so what stops you now from pursuing that career path? Programming today is not about math and science--it's about picking up a book, downloading a free SDK and/or IDE, and diving in. My background was in International Relations--I was never formally trained, either. Has it held me back? You betcha--there are a few places that refused to hire me because I didn't have the formal CS background to be able to select the right algorithm or do big-O analysis. Didn't seem to stop me--I just went and interviewed someplace else.

Equality means equality. If a woman wants to be given the same respect as a man, then she has to earn it the same way he does, by being equally qualified and equally professional. It is this "we should strengthen the weak" mentality that leads to soccer games with no score kept, because "we're all winners". That in turn leads to children that then can't handle it when they actually do lose at something, which they must, eventually, because life is not fair. It never will be. Pretending otherwise just does a disservice to the women who have put in the blood, sweat, and tears to achieve the positions of prominence and respect that they earned.

Am I saying this because I worry that preferential treatment to women speakers at conferences and in writing will somehow mean there are fewer opportunities for me, a man? Some will accuse me of such, but those who do probably don't realize that I turn down more conferences than I accept these days, and more writing opportunities as well. In fact, regardless of your gender, there are dozens, if not hundreds, of online portals and magazines that are desperate for authors to write quality work--if you're at all stumped trying to write for somebody, then you're not trying very hard. And every week user groups across the country are canceled for a lack of a speaker--if you're trying to speak and you're not, then you're either setting your bar too high ("If I don't get into TechEd, having never spoken before in my life, it must be because I'm a woman, not that I'm not a qualified speaker!") or you're really not trying ("Why aren't the conferences calling me about speaking there?").

If you're a woman, and you're thinking about a career in IT, more power to you. This industry offers more opportunity and room for growth than any other I've yet come across. There are dozens of meetings and meetups and conferences that are springing into place to encourage you and help you earn that distinction. Yes, as you go you will want and/or need help. So did I. You need people that will help you sharpen your skills and improve your abilities, yes. But a specific and concrete bias in your favor? No. You don't need somebody's charity.

Because if you do, then it means that you're admitting that you can't do it on your own, and you aren't really equal. And that, I think, would be the biggest tragedy of the whole issue.

Flame away.

Just Say No to SSNs

Fri, 16 Mar 2012 00:00:00 +0000

Two things conspire to bring you this blog post.

Of Contracts and Contracts

First, a few months ago, I was asked to participate in an architectural review for a project being done for one of the states here in the US. It was a project dealing with some sensitive information (Child Welfare Services), and I was required to sign a document basically promising not to do anything bad with the data. Not a problem to sign, since I was going to be more focused on the architecture and code anyway, and would stay away from the production servers and data as much as I possibly could. But then the state agency asked for my social security number, and when I pushed back asking why, they told me it was “mandatory” in order to work on the project. I suspect it was for a background check—but when I asked how long they were going to hold on to the number and what their privacy policy was regarding my data, they refused to answer, and I never heard from them again. Which, quite frankly, was something of a relief.

Second, just tonight there was a thread on the Seattle Tech Startup mailing list about SSNs again. This time, a contractor who participates on the list was being asked by the contracting agency for his SSN, not for any tax document form, but... just because. This sounded fishy. It turned out that the contract was going to be with AT&T, and that they commonly use a contractor’s SSN as a way of identifying the contractor in their vendor database. It was also noted that many companies do this, and that it was likely that many more would do so in the future. One poster pointed out that when the state’s attorney general’s office was contacted about this practice, it isn’t illegal.

Folks, this practice has to stop. For both your sake, and the company’s.

Of Data and Integrity

Using SSNs in your database is just a bad idea from top to bottom. For starters, it makes your otherwise-unassuming enterprise application a ripe target for hackers, who seek to gather legitimate SSNs as part of the digital fingerprinting of potential victims for identity theft. What’s worse, any time I’ve ever seen any company store the SSNs, they’re almost always stored in plaintext form (“These aren’t credit cards!”), and they’re often used as a primary key to uniquely identify individuals.

There’s so many things wrong with this idea from a data management perspective, it’s shameful.

SSNs were never intended for identification purposes. Yeah, this is a weak argument now, given all the de facto uses to which they are put already, but when FDR passed the Social Security program back in the 30s, he promised the country that they would never be used for identification purposes. This is, in fact, why the card reads “This number not to be used for identification purposes” across the bottom. Granted, every financial institution with whom I’ve ever done business has ignored that promise for as long as I’ve been alive, but that doesn’t strike me as a reason to continue doing so.
SSNs are not unique. There’s rumors of two different people being issued the same SSN, and while I can’t confirm or deny this based on personal experience, it doesn’t take a rocket scientist to figure out that if there are 300 million people living in the US, and the SSN is a nine-digit number, that means that there are 999,999,999 potential numbers in the best case (which isn’t possible, because the first three digits are a stratification mechanism—-for example, California-issued numbers are generally in the 5xx range, while East Coast-issued numbers are in the 0xx range). What I can say for certain is that SSNs are, in fact, recycled—-so your new baby may (and very likely will) end up with some recently-deceased individual’s SSN. As we start to see databases extending to a second and possibly even third generation of individuals, these kinds of conflicts are going to become even more common. As US population continues to rise, and immigration brings even more people into the country to work, how soon before we start seeing the US government sweat the problems associated with trying to go to a 10- or 11-digit SSN? It’s going to make the IPv4 and IPv6 problems look trivial by comparison. (Look for that to be the moment when the US government formally adopts a hexadecimal system for SSNs.)
SSNs are sensitive data. You knew this already. But what you may not realize is that data not only has a tendency to escape the organization that gathered it (databases are often sold, acquired, or stolen), but that said data frequently lives far, far longer than it needs to. Look around in your own company—how many databases are still online, in use, even though the data isn’t really relevant anymore, just because “there’s no cost to keeping it”? More importantly, companies are increasingly being held accountable for sensitive information breaches, and it’s just a matter of time before a creative lawyer seeking to tap into the public’s sensitivities to things they don’t understand leads him/her takes a company to court, suing them for damages for such a breach. And there’s very likely more than a few sympathetic judges in the country to the idea. Do you really want to be hauled up on the witness stand to defend your use of the SSN in your database?

Given that SSNs aren’t unique, and therefore fail as their primary purpose in a data management scheme, and that they represent a huge liability because of their sensitive nature, why on earth would you want them in your database?

A Call

But more importantly, companies aren’t going to stop using them for these kinds of purposes until we make them stop. Any time a company asks you for your SSN, challenge them. Ask them why they need it, if the transaction can be completed without it, and if they insist on having it, a formal declaration of their sensitive information policy and what kind of notification and compensation you can expect when they suffer a sensitive data breach. It may take a while to find somebody within the company who can answer your questions at the places that legitimately need the information, but you’ll get there eventually. And for the rest of the companies that gather it “just in case”, well, if it starts turning into a huge PITA to get them, they’ll find other ways to figure out who you are.

This is a call to arms, folks: Just say NO to handing over your SSN.

Leveling Up 'DDD'

Fri, 2 Mar 2012 00:00:00 +0000

Eric Evans, a number of years ago, wrote a book on “Domain Driven Design”. Around the same time, Martin Fowler coined the “Rich Domain Model” pattern.

Ever since then, people have been going bat-shit nutso over building these large domain object models, then twisting and contorting them in all these various ways to make them work across different contexts—-across tiers, for example, and into databases, and so on. It created a cottage industry of infrastructure tools, toolkits, libraries and frameworks, all designed somehow to make your objects less twisted and more usable and less tightly-coupled to infrastructure (I’ll pause for a moment to let you think about the absurdity of that—infrastructure designed to reduce coupling to other infrastructure—before we go on), and so on.

All the time, though, we were shying away from really taking the plunge, and thinking about domain entities in domain terms.

Jessica Kerr nails it, on the head. Her post is in the context of Java (with, ironically, some F# thrown in for clarity), but the fact is, the Java parts could’ve been written in C# or C++ and the discussion would be the exact same.

To think about building domain objects, if you are really looking to build a domain model, means to think beyond the implementation language you’re building them in. That means you have to stop thinking in terms of “Strings” and “ints”, but in terms of “FirstName” and “Age” types. Ironically, Java is ill-suited as a language to support this. C# is not great about this, but it is easier than Java. C++, ironically, may be best suited for this, given the ease with which we can set up “aliased” types, via either the typedef or even the lowly preprocessor macro (though it hurts me to say that).

I disagree with her when she says that it’s a problem that FirstName can’t inherit from String—-frankly, I hold the position that having FirstName inherit from String would be putting too much implementation detail into FirstName then, and would hurt FirstName’s chances for evolution and enhancement—-but the rest of the post is so spot-on, it’s scary.

And the really ironic thing? I remember having this conversation nearly twenty years ago, in the context of C++ at the time.

Want another mind-warping discussion around DDD and how to think about domain objects correctly? Read Allen Holub’s “Getters and Setters Considered Harmful” article of nine (!) years ago.

Read those two entries, think on them for a bit, then give it a whirl in your own projects. Or as a research spike. I think you’ll start to find a lot of that infrastructure code starting to drop away and become unnecessary. And that will let you get back to the essence of objects, and level up your DDD.

(Unfortunately, I don’t know what leveled-up DDD is called. DDD++, maybe?)

Is Programming Less Exciting Today?

Wed, 25 Jan 2012 00:00:00 +0000

As discriminatory as this is going to sound, this one is for the old-timers. If you started programming after the turn of the milennium, I don’t know if you’re going to be able to follow the trend of this post—not out of any serious deficiency on your part, hardly that. But I think this is something only the old-timers are going to identify with. (And thus, do I alienate probably 80% of my readership, but so be it.)

Is it me, or is programming just less interesting today than it was two decades ago?

By all means, shake your smartphones and other mobile devices at me and say, “Dude, how can you say that?”, but in many ways programming for Android and iOS reminds me of programming for Windows and Mac OS two decades ago. HTML 5 and JavaScript remind me of ten years ago, the first time HTML and JavaScript came around. The discussions around programming languages remind me of the discussions around C++. The discussions around NoSQL remind me of the arguments both for and against relational databases. It all feels like we’ve been here before, with only the names having changed.

Don’t get me wrong—-if any of you comment on the differences between HTML 5 now and HTML 3.2 then, or the degree of the various browser companies agreeing to the standard today against the “browser wars” of a decade ago, I’ll agree with you. This isn’t so much of a rational and logical discussion as it is an emotive and intuitive one. It just feels similar.

To be honest, I get this sense that across the entire industry right now, there’s a sort of malaise, a general sort of “Bah, nothing really all that new is going on anymore”. NoSQL is re-introducing storage ideas that had been around before but were discarded (perhaps injudiciously and too quickly) in favor of the relational model. Functional languages have obviously been in place since the 50’s (in Lisp). And so on.

More importantly, look at the Java community: what truly innovative ideas have emerged here in the last five years? Every new open-source project or commercial endeavor either seems to be a refinement of an idea before it (how many different times are we going to create a new Web framework, guys?) or an attempt to leverage an idea coming from somewhere else (be it from .NET or from Ruby or from JavaScript or….). With the upcoming .NET 4.5 release and Windows 8, Microsoft is holding out very little “new and exciting” bits for the community to invest emotionally in: we hear about “async” in C# 5 (something that F# has had already, thank you), and of course there is WinRT (another platform or virtual machine… sort of), and… well, honestly, didn’t we just do this a decade ago? Where is the WCFs, the WPFs, the Silverlights, the things that would get us fired up? Hell, even a new approach to data access might stir some excitement. Node.js feels like an attempt to reinvent the app server, but if you look back far enough you see that the app server itself was reinvented once (in the Java world) in Spring and other lightweight frameworks, and before that by people who actually thought to write their own web servers in straight Java. (And, for the record, the whole event-driven I/O thing is something that’s been done in both Java and .NET a long time before now.)

And as much as this is going to probably just throw fat on the fire, all the excitement around JavaScript as a language reminds me of the excitement about Ruby as a language. Does nobody remember that Sun did this once already, with Phobos? Or that Netscape did this with LiveScript? JavaScript on the server end is not new, folks. It’s just new to the people who’d never seen it before.

In years past, there has always seemed to be something deeper, something more exciting and more innovative that drives the industry in strange ways. Artificial Intelligence was one such thing: the search to try and bring computers to a state of human-like sentience drove a lot of interesting ideas and concepts forward, but over the last decade or two, AI seems to have lost almost all of its luster and momentum. User interfaces—specifically, GUIs—were another force for a while, until GUIs got to the point where they were so common and so deeply rooted in their chosen pasts (the single-button of the Mac, the menubar-per-window of Windows, etc) that they left themselves so little room for maneuver. At least this is one area where Microsoft is (maybe) putting the fatted sacred cow to the butcher’s knife, with their Metro UI moves in Windows 8... but only up to a point.

Maybe I’m just old and tired and should hang up my keyboard and go take up farming, then go retire to my front porch’s rocking chair and practice my Hey you kids! Getoffamylawn! or something. But before you dismiss me entirely, do me a favor and tell me: what gets you excited these days? If you’ve been programming for twenty years, what about the industry today gets your blood moving and your mind sharpened?

2012 Tech Predictions

Sun, 1 Jan 2012 00:00:00 +0000

Here we go again: the annual review of last year's predictions, and a set of new ones for the new year.

Well, friends, another year has come and gone, and it's time for me to put my crystal ball into place and see what the upcoming year has for us. But, of course, in the long-standing tradition of these predictions, I also need to put my spectacles on (I did turn 40 last year, after all) and have a look at how well I did in this same activity twelve months ago.

Let's see what unbelievable gobs of hooey I slung last year came even remotely to pass. For 2011, I said....

THEN: Android’s penetration into the mobile space is going to rise, then plateau around the middle of the year. Android phones, collectively, have outpaced iPhone sales. That’s a pretty significant statistic—and it means that there’s fewer customers buying smartphones in the coming year. More importantly, the first generation of Android slates (including the Galaxy Tab, which I own), are less-than-sublime, and not really an “iPad Killer” device by any stretch of the imagination. And I think that will slow down people buying Android slates and phones, particularly since Google has all but promised that Android releases will start slowing down.
- NOW: Well, I think I get a point for saying that Android's penetration will rise... but then I lose it for suggesting that it would slow down. Wow, was I wrong on that. Once Amazon put the Kindle Fire out, suddenly for the first time Android tablets began to appear in peoples' hands in record numbers. The drawback here is that most people using the Fire don't realize it's an Android tablet, which certainly hurts Google's brand-awareness (not that Amazon really seems to mind), but the upshot is simple: people are still buying devices, even though they may already own one. Which amazes me.
THEN: Windows Phone 7 penetration into the mobile space will appear huge, then slow down towards the middle of the year. Microsoft is getting some pretty decent numbers now, from what I can piece together, and I think that’s largely the “I love Microsoft” crowd buying in. But it’s a pretty crowded place right now with Android and iPhone, and I’m not sure if the much-easier Office and/or Exchange integration is enough to woo consumers (who care about Office) or business types (who care about Exchange) away from their Androids and iPhones.
- NOW: Despite the catastrophic implosion of RIM (thus creating a huge market of people looking to trade their Blackberrys in for other mobile phones, ones which won't all go down when a RIM server implodes), WP7 has definitely not emerged as the "third player" in the mobile space; or, perhaps more precisely, they feel like a distant third, rather than a creditable alternative to the other two. In fact, more and more it just feels like this is a two-horse race and Microsoft is in it still because they're willing to throw loss after loss to stay in it. (For what reason, I'm not sure--it's not clear to me that they can ever reach a point of profitability here, even once Nokia makes the transition to WP7, which is supposedly going to take years. On the order of a half-decade or so.) Even living here in Redmon, where I would expect the WP7 concentration to be much, much higher than anywhere else in the world, it's still more common to see iPhones and 'droids in peoples' hands than it is to see WP7 phones.
THEN: Android, iOS and/or Windows Phone 7 becomes a developer requirement. Developers, if you haven’t taken the time to learn how to program one of these three platforms, you are electing to remove yourself from a growing market that desperately wants people with these skills. I see the “mobile native app development” space as every bit as hot as the “Internet/Web development” space was back in 2000. If you don’t have a device, buy one. If you have a device, get the tools—in all three cases they’re free downloads—and start writing stupid little apps that nobody cares about, so you can have some skills on the platform when somebody cares about it.
- NOW: Wow, yes. Right now, if you are a developer and you haven't spent at least a little time learning mobile development, you are excluding yourself from a development "boom" that rivals the one around Web sites in the mid-90's. Seriously: remember when everybody had to have a website? That's the mentality right now with a ton of different companies--"we have to have a mobile app!" "But we sell condom lubricant!" "Doesn't matter! We need a mobile app! Build us something! Go go go go go!"
THEN: The Windows 7 slates will suck. This isn’t a prediction, this is established fact. I played with an “ExoPC” 10” form factor slate running Windows 7 (Dell I think was the manufacturer), and it was a horrible experience. Windows 7, like most OSes, really expects a keyboard to be present, and a slate doesn’t have one—so the OS was hacked to put a “keyboard” button at the top of the screen that would slide out to let you touch-type on the slate. I tried to fire up Notepad and type out a haiku, and it was an unbelievably awkward process. Android and iOS clearly own the slate market for the forseeable future, and if Dell has any brains in its corporate head, it will phone up Google tomorrow and start talking about putting Android on that hardware.
- NOW: Yeah, that was something of a "gimme" point (but I'll take it). Windows7 on a slate was a Bad Idea, and I'm pretty sure the sales reflect that. Conduct your own anecdotal poll: see if you can find a store somewhere in your town or city that will actually sell you a Windows7 slate. Can't find one? I can--it's the Microsoft store in town, and I'm not entirely sure they still stock them. Certainly our local Best Buy doesn't.
THEN: DSLs mostly disappear from the buzz. I still see no strawman (no “pet store” equivalent), and none of the traditional builders-of-strawmen (Microsoft, Oracle, etc) appear interested in DSLs much anymore, so I think 2010 will mark the last year that we spent any time talking about the concept.
- NOW: I'm going to claim a point here, too. DSLs have pretty much left us hanging. Without a strawman for developers to "get", the DSL movement has more or less largely died out. I still sometimes hear people refer to something that isn't a programming language but does something technical as a "DSL" ("That shipping label? That's a DSL!"), and that just tells me that the concept never really took root.
THEN: Facebook becomes more of a developer requirement than before. I don’t like Mark Zuckerburg. I don’t like Facebook’s privacy policies. I don’t particularly like the way Facebook approaches the Facebook Connect experience. But Facebook owns enough people to be the fourth-largest nation on the planet, and probably commands an economy of roughly that size to boot. If your app is aimed at the Facebook demographic (that is, everybody who’s not on Twitter), you have to know how to reach these people, and that means developing at least some part of your system to integrate with it.
- NOW: Facebook, if anything, has become more important through 2011, particularly for startups looking to get some exposure and recognition. Facebook continues to screw with their user experience, though, and they keep screwing with their security policies, and as "big" a presence as they have, it's not invulnerable, and if they're not careful, they're going to find themselves on the other side of the relevance curve.
THEN: Twitter becomes more of a developer requirement, too. Anybody who’s not on Facebook is on Twitter. Or dead. So to reach the other half of the online community, you have to know how to connect out with Twitter.
- NOW: Twitter's impact has become deeper, but more muted in some ways--people don't think of Twitter as a "new" channel, but one that they've come to expect and get used to. At the same time, how Twitter is supposed to factor into different applications isn't always clear, which hinders Twitter's acceptance and "must-have"-ness. Of course, Twitter could care less, it seems, though it still confuses me how they actually make money.
THEN: XMPP becomes more of a developer requirement. XMPP hasn’t crossed a lot of people’s radar screen before, but Facebook decided to adopt it as their chat system communication protocol, and Google’s already been using it, and suddenly there’s a whole lotta traffic going over XMPP. More importantly, it offers a two-way communication experience that is in some scenarios vastly better than what HTTP offers, yet running in a very “Internet-friendly” way just as HTTP does. I suspect that XMPP is going to start cropping up in a number of places as a useful alternative and/or complement to using HTTP.
- NOW: Well, unfortunately, XMPP still hides underneath other names and still doesn't come to mind when people are thinking about communication, leaving this one way unfulfilled. *sigh* Maybe someday we will learn that not everything has to go over HTTP, but it didn't happen in 2011.
THEN: “Gamification” starts making serious inroads into non-gaming systems. Maybe it’s just because I’ve been talking more about gaming, game design, and game implementation last year, but all of a sudden “gamification”—the process of putting game-like concepts into non-game applications—is cresting in a big way. FourSquare, Yelp, Gowalla, suddenly all these systems are offering achievement badges and scoring systems for people who want to play in their worlds. How long is it before a developer is pulled into a meeting and told that “we need to put achievement badges into the call-center support application”? Or the online e-commerce portal? It’ll start either this year or next.
- NOW: Gamification is emerging, but slowly and under the radar. It's certainly not as strong as I thought it would be, but gamification concepts are sneaking their way into a variety of different scenarios (beyond games themselves). Probably can't claim a point here, no.
THEN: Functional languages will hit a make-or-break point. I know, I said it last year. But the buzz keeps growing, and when that happens, it usually means that it’s either going to reach a critical mass and explode, or it’s going to implode—and the longer the buzz grows, the faster it explodes or implodes, accordingly. My personal guess is that the “F/O hybrids”—F#, Scala, etc—will continue to grow until they explode, particularly since the suggested v.Next changes to both Java and C# have to be done as language changes, whereas futures for F# frequently are either built as libraries masquerading as syntax (such as asynchronous workflows, introduced in 2.0) or as back-end library hooks that anybody can plug in (such as type providers, introduced at PDC a few months ago), neither of which require any language revs—and no concerns about backwards compatibility with existing code. This makes the F/O hybrids vastly more flexible and stable. In fact, I suspect that within five years or so, we’ll start seeing a gradual shift away from pure O-O systems, into systems that use a lot more functional concepts—and that will propel the F/O languages into the center of the developer mindshare.
- NOW: More than any of my other predictions (or subjects of interest), functional languages stump me the most. On the one hand, there doesn't seem to be a drop-off of interest in the subject, based on a variety of anecdotal evidence (books, articles, etc), but on the other hand, they don't seem to be crossing over into the "mainstream" programming worlds, either. At best, we can say that they are entering the mindset of senior programmers and/or project leads and/or architects, but certainly they don't seem to be turning in to the "go-to" language for projects being done in 2011.
THEN: The Microsoft Kinect will lose its shine. I hate to say it, but I just don’t see where the excitement is coming from. Remember when the Wii nunchucks were the most amazing thing anybody had ever seen? Frankly, after a slew of initial releases for the Wii that made use of them in interesting ways, the buzz has dropped off, and more importantly, the nunchucks turned out to be just another way to move an arrow around on the screen—in other words, we haven’t found particularly novel and interesting/game-changing ways to use the things. That’s what I think will happen with the Kinect. Sure, it’s really freakin’ cool that you can use your body as the controller—but how precise is it, how quickly can it react to my body movements, and most of all, what new user interface metaphors are people going to have to come up with in order to avoid the “me-too” dancing-game clones that are charging down the pipeline right now?
- NOW: Kinect still makes for a great Christmas or birthday present, but nobody seems to be all that amazed by the idea anymore. Certainly we aren't seeing a huge surge in using Kinect as a general user interface device, at least not yet. Maybe it needed more time for people to develop those new metaphors, but at the same time, I would've expected at least a few more games to make use of it, and I haven't seen any this past year.
THEN: There will be no clear victor in the Silverlight-vs-HTML5 war. And make no mistake about it, a war is brewing. Microsoft, I think, finds itself in the inenviable position of having two very clearly useful technologies, each one’s “sphere of utility” (meaning, the range of answers to the “where would I use it?” question) very clearly overlapping. It’s sort of like being a football team with both Brett Favre and Tom Brady on your roster—both of them are superstars, but you know, deep down, that you have to cut one, because you can’t devote the same degree of time and energy to both. Microsoft is going to take most of 2011 and probably part of 2012 trying to support both, making a mess of it, offering up conflicting rationale and reasoning, in the end achieving nothing but confusing developers and harming their relationship with the Microsoft developer community in the process. Personally, I think Microsoft has no choice but to get behind HTML 5, but I like a lot of the features of Silverlight and think that it has a lot of mojo that HTML 5 lacks, and would actually be in favor of Microsoft keeping both—so long as they make it very clear to the developer community when and where each should be used. In other words, the executives in charge of each should be locked into a room and not allowed out until they’ve hammered out a business strategy that is then printed and handed out to every developer within a 3-continent radius of Redmond. (Chances of this happening: .01%)
- NOW: Well, this was accurate all the way up until the last couple of months, when Microsoft made it fairly clear that Silverlight was being effectively "put behind" HTML 5, despite shipping another version of Silverlight. In the meantime, though, they've tried to support both (and some Silverlighters tell me that the Silverlight team is still looking forward to continuing supporting it, though I'm not sure at this point what is rumor and what is fact anymore), and yes, they confused the hell out of everybody. I'm surprised they pulled the trigger on it in 2011, though--I expected it to go a version or two more before they finally pulled the rug out.
THEN: Apple starts feeling the pressure to deliver a developer experience that isn’t mired in mid-90’s metaphor. Don’t look now, Apple, but a lot of software developers are coming to your platform from Java and .NET, and they’re bringing their expectations for what and how a developer IDE should look like, perform, and do, with them. Xcode is not a modern IDE, all the Apple fan-boy love for it notwithstanding, and this means that a few things will happen:
- Eclipse gets an iOS plugin. Yes, I know, it wouldn’t work (for the most part) on a Windows-based Eclipse installation, but if Eclipse can have a native C/C++ developer experience, then there’s no reason why a Mac Eclipse install couldn’t have an Objective-C plugin, and that opens up the idea of using Eclipse to write iOS and/or native Mac apps (which will be critical when the Mac App Store debuts somewhere in 2011 or 2012).
- Rumors will abound about Microsoft bringing Visual Studio to the Mac. Silverlight already runs on the Mac; why not bring the native development experience there? I’m not saying they’ll actually do it, and certainly not in 2011, but the rumors, they will be flyin….
- Other third-party alternatives to Xcode will emerge and/or grow. MonoTouch is just one example. There’s opportunity here, just as the fledgling Java IDE market looked back in ‘96, and people will come to fill it.
- NOW: Xcode 4 is "better", but it's still not what I would call comparable to the Microsoft Visual Studio or JetBrains IDEA experience. LLVM is definitely a better platform for the company's development efforts, long-term, and it's encouraging that they're investing so heavily into it, but I still wish the overall development experience was stronger. Meanwhile, though, no Eclipse plugin has emerged (that I'm aware of), which surprised me, and neither did we see Microsoft trying to step into that world, which doesn't surprise me, but disappoints me just a little. I realize that Microsoft's developer tools are generally designed to support the Windows operating system first, but Microsoft has to cut loose from that perspective if they're going to survive as a company. More on that later.
THEN: NoSQL buzz grows. The NoSQL movement, which sort of got started last year, will reach significant states of buzz this year. NoSQL databases have a lot to offer, particularly in areas that relational databases are weak, such as hierarchical kinds of storage requirements, for example. That buzz will reach a fever pitch this year, and the relational database moguls (Microsoft, Oracle, IBM) will start to fight back.
- NOW: Well, the buzz certainly grew, and it surprised me that the big storage guys (Microsoft, IBM, Oracle) didn't do more to address it; I was expecting features to emerge in their database products to address some of the features present in MongoDB or CouchDB or some of the others, such as "schemaless" or map/reduce-style queries. Even just incorporating JavaScript into the engine somewhere would've generated a reaction.

Overall, it appears I'm running at about my usual 50/50 levels of prognostication. So be it. Let's see what the ol' crystal ball has in mind for 2012:

Lisps will be the languages to watch. With Clojure leading the way, Lisps (that is, languages that are more or less loosely based on Common Lisp or one of its variants) are slowly clawing their way back into the limelight. Lisps are both functional languages as well as dynamic languages, which gives them a significant reason for interest. Clojure runs on top of the JVM, which makes it highly interoperable with other JVM languages/systems, and Clojure/CLR is the version of Clojure for the CLR platform, though there seems to be less interest in it in the .NET world (which is a mistake, if you ask me).
Functional languages will.... I have no idea. As I said above, I'm kind of stymied on the whole functional-language thing and their future. I keep thinking they will either "take off" or "drop off", and they keep tacking to the middle, doing neither, just sort of hanging in there as a concept for programmers to take and run with. Mind you, I like functional languages, and I want to see them become mainstream, or at least more so, but I keep wondering if the mainstream programming public is ready to accept the ideas and concepts hiding therein. So this year, let's try something different: I predict that they will remain exactly where they are, neither "done" nor "accepted", but continue next year to sort of hang out in the middle.
F#'s type providers will show up in C# v.Next. This one is actually a "gimme", if you look across the history of F# and C#: for almost every version of F# v."N", features from that version show up in C# v."N+1". More importantly, F# 3.0's type provider feature is an amazing idea, and one that I think will open up language research in some very interesting ways. (Not sure what F#'s type providers are or what they'll do for you? Check out Don Syme's talk on it at BUILD last year.)
Windows8 will generate a lot of chatter. As 2012 progresses, Microsoft will try to force a lot of buzz around it by keeping things under wraps until various points in the year that feel strategic (TechEd, BUILD, etc). In doing so, though, they will annoy a number of people by not talking about them more openly or transparently. What's more....
Windows8 ("Metro")-style apps won't impress at first. The more I think about it, the more I'm becoming convinced that Metro-style apps on a desktop machine are going to collectively underwhelm. The UI simply isn't designed for keyboard-and-mouse kinds of interaction, and that's going to be the hardware setup that most people first experience Windows8 on--contrary to what (I think) Microsoft thinks, people do not just have tablets laying around waiting for Windows 8 to be installed on it, nor are they going to buy a Windows8 tablet just to try it out, at least not until it's gathered some mojo behind it. Microsoft is going to have to finesse the messaging here very, very finely, and that's not something they've shown themselves to be particularly good at over the last half-decade.
Scala will get bigger, thanks to Heroku. With the adoption of Scala and Play for their Java apps, Heroku is going to make Scala look attractive as a development platform, and the adoption of Play by Typesafe (the same people who brought you Akka) means that these four--Heroku, Scala, Play and Akka--will combine into a very compelling and interesting platform. I'm looking forward to seeing what comes of that.
Cloud will continue to whip up a lot of air. For all the hype and money spent on it, it doesn't really seem like cloud is gathering commensurate amounts of traction, across all the various cloud providers with the possible exception of Amazon's cloud system. But, as the different cloud platforms start to diversify their platform technology (Microsoft seems to be leading the way here, ironically, with the introduction of Java, Hadoop and some limited NoSQL bits into their Azure offerings), and as we start to get more experience with the pricing and costs of cloud, 2012 might be the year that we start to see mainstream cloud adoption, beyond "just" the usage patterns we've seen so far (as a backing server for mobile apps and as an easy way to spin up startups).
Android tablets will start to gain momentum. Amazon's Kindle Fire has hit the market strong, definitely better than any other Android-based tablet before it. The Nooq (the Kindle's principal competitor, at least in the e-reader world) is also an Android tablet, which means that right now, consumers can get into the Android tablet world for far, far less than what an iPad costs. Apple rumors suggest that they may have a 7" form factor tablet that will price competitively (in the $200/$300 range), but that's just rumor right now, and Apple has never shown an interest in that form factor, which means the 7" world will remain exclusively Android's (at least for now), and that's a nice form factor for a lot of things. This translates well into more sales of Android tablets in general, I think.
Apple will release an iPad 3, and it will be "more of the same". Trying to predict Apple is generally a lost cause, particularly when it comes to their vaunted iOS lines, but somewhere around the middle of the year would be ripe for a new iPad, at the very least. (With the iPhone 4S out a few months ago, it's hard to imagine they'd cannibalize those sales by releasing a new iPhone, until the end of the year at the earliest.) Frankly, though, I don't expect the iPad 3 to be all that big of a boost, just a faster processor, more storage, and probably about the same size. Probably the only thing I'd want added to the iPad would be a USB port, but that conflicts with the Apple desire to present the iPad as a "device", rather than as a "computer". (USB ports smack of "computers", not self-contained "devices".)
Apple will get hauled in front of the US government for... something. Apple's recent foray in the legal world, effectively informing Samsung that they can't make square phones and offering advice as to what will avoid future litigation, smacks of such hubris and arrogance, it makes Microsoft look like a Pollyanna Pushover by comparison. It is pretty much a given, it seems to me, that a confrontation in the legal halls is not far removed, either with the US or with the EU, over anti-cometitive behavior. (And if this kind of behavior continues, and there is no legal action, it'll be pretty apparent that Apple has a pretty good set of US Congressmen and Senators in their pocket, something they probably learned from watching Microsoft and IBM slug it out rather than just buy them off.)
IBM will be entirely irrelevant again. Look, IBM's main contribution to the Java world is/was Eclipse, and to a much lesser degree, Harmony. With Eclipse more or less "done" (aside from all the work on plugins being done by third parties), and with IBM abandoning Harmony in favor of OpenJDK, IBM more or less removes themselves from the game, as far as developers are concerned. Which shouldn't really be surprising--they've been more or less irrelevant pretty much ever since the mid-2000s or so.
Oracle will "screw it up" at least once. Right now, the Java community is poised, like a starving vulture, waiting for Oracle to do something else that demonstrates and befits their Evil Emperor status. The community has already been quick (far too quick, if you ask me) to highlight Oracle's supposed missteps, such as the JVM-crashing bug (which has already been fixed in the _u1 release of Java7, which garnered no attention from the various Java news sites) and the debacle around Hudson/Jenkins/whatever-the-heck-we-need-to-call-it-this-week. I'll grant you, the Hudson/Jenkins debacle was deserving of ire, but Oracle is hardly the Evil Emperor the community makes them out to be--at least, so far. (I'll admit it, though, I'm a touch biased, both because Brian Goetz is a friend of mine and because Oracle TechNet has asked me to write a column for them next year. Still, in the spirit of "innocent until proven guilty"....)
VMWare/SpringSource will start pushing their cloud solution in a major way. Companies like Microsoft and Google are pushing cloud solutions because Software-as-a-Service is a reoccurring revenue model, generating revenue even in years when the product hasn't incremented. VMWare, being a product company, is in the same boat--the only time they make money is when they sell a new copy of their product, unless they can start pushing their virtualization story onto hardware on behalf of clients--a.k.a. "the cloud". With SpringSource as the software stack, VMWare has a more-or-less complete cloud play, so it's surprising that they didn't push it harder in 2011; I suspect they'll start cramming it down everybody's throats in 2012. Expect to see Rod Johnson talking a lot about the cloud as a result.
JavaScript hype will continue to grow, and by years' end will be at near-backlash levels. JavaScript (more properly known as ECMAScript, not that anyone seems to care but me) is gaining all kinds of steam as a mainstream development language (as opposed to just-a-browser language), particularly with the release of NodeJS. That hype will continue to escalate, and by the end of the year we may start to see a backlash against it. (Speaking personally, NodeJS is an interesting solution, but suggesting that it will replace your Tomcat or IIS server is a bit far-fetched; event-driven I/O is something both of those servers have been doing for years, and the rest of it is "just" a language discussion. We could pretty easily use JavaScript as the development language inside both servers, as Sun demonstrated years ago with their "Phobos" project--not that anybody really cared back then.)
NoSQL buzz will continue to grow, and by years' end will start to generate a backlash. More and more companies are jumping into NoSQL-based solutions, and this trend will continue to accelerate, until some extremely public failure will start to generate a backlash against it. (This seems to be a pattern that shows up with a lot of technologies, so it seems entirely realistic that it'll happen here, too.) Mind you, I don't mean to suggest that the backlash will be factual or correct--usually these sorts of things come from misuing the tool, not from any intrinsic failure in it--but it'll generate some bad press.
Ted will thoroughly rock the house during his CodeMash keynote. Yeah, OK, that's more of a fervent wish than a prediction, but hey, keep a positive attitude and all that, right?
Ted will continue to enjoy his time working for Neudesic. So far, it's been great working for these guys, and I'm looking forward to a great 2012 with them. (Hopefully this will be a prediction I get to tack on for many years to come, too.)

I hope that all of you have enjoyed reading these, and I wish you and yours a very merry, happy, profitable and fulfilling 2012. Thanks for reading.

2011 Tech Predictions

Sat, 1 Jan 2011 00:00:00 +0000

Here we go again: the annual review of last year's predictions, and a set of new ones for the new year.

Long-time readers of this blog know what’s coming next: it’s time for Ted to prognosticate on what the coming year of tech will bring us. But I believe strongly in accountability, even in my offered-up-for-free predictions, so one of the traditions of this space is to go back and revisit my predictions from this time last year. So, without further ado, let’s look back at Ted’s 2010 predictions, and see how things played out; 2010 predictions are prefixed with “THEN”, and my thoughts on my predictions are prefixed with “NOW”:

For 2010, I predicted....

THEN: ... I will offer 3- and 4-day training classes on F# and Scala, among other things. OK, that's not fair—yes, I have the materials, I just need to work out locations and times. Contact me if you're interested in a private class, by the way.

NOW: Well, I offered them… I just didn’t do much to advertise them or sell them. I got plenty busy just with the other things I had going on. Besides, this and the next prediction were pretty much all advertisement anyway, so I don’t know if anybody really counts these two.

THEN: ... I will publish two books, one on F# and one on Scala. OK, OK, another plug. Or, rather, more of a resolution. One will be the "Professional F#" I'm doing for Wiley/Wrox, the other isn't yet finalized. But it'll either be published through a publisher, or self-published, by JavaOne 2010.

NOW: “Professional F# 2.0” shipped in Q3 of 2010; the other Scala book I decided not to pursue—too much stuff going on to really put the necessary time into it. (Cue sad trombone.)

THEN: ... DSLs will either "succeed" this year, or begin the short slide into the dustbin of obscure programming ideas. Domain-specific language advocates have to put up some kind of strawman for developers to learn from and poke at, or the whole concept will just fade away. Martin's book will help, if it ships this year, but even that might not be enough to generate interest if it doesn't have some kind of large-scale applicability in it. Patterns and refactoring and enterprise containers all had a huge advantage in that developers could see pretty easily what the problem was they solved; DSLs haven't made that clear yet.

NOW: To be honest, this one is hard to call. Martin Fowler published his DSL book, which many people consider to be a good sign of what’s happening in the world, but really, the DSL buzz seems to have dropped off significantly. The strawman hasn’t appeared in any meaningful public way (I still don’t see an example being offered up from anybody), and that leads me to believe that the fading-away has started.

THEN: ... functional languages will start to see a backlash. I hate to say it, but "getting" the functional mindset is hard, and there's precious few resources that are making it easy for mainstream (read: O-O) developers make that adjustment, far fewer than there was during the procedural-to-object shift. If the functional community doesn't want to become mainstream, then mainstream developers will find ways to take functional's most compelling gateway use-case (parallel/concurrent programming) and find a way to "git 'er done" in the traditional O-O approach, probably through software transactional memory, and functional languages like Haskell and Erlang will be relegated to the "What Might Have Been" of computer science history. Not sure what I mean? Try this: walk into a functional language forum, and ask what a monad is. Nobody yet has been able to produce an answer that doesn't involve math theory, or that does involve a practical domain-object-based example. In fact, nobody has really said why (or if) monads are even still useful. Or catamorphisms. Or any of the other dime-store words that the functional community likes to toss around.

NOW: I think I have to admit that this hasn’t happened—at least, there’s been no backlash that I’ve seen. In fact, what’s interesting is that there’s been some movement to bring those functional concepts—including monads, which surprised me completely—into other languages like C# or Java for discussion and use. That being said, though, I don’t see Haskell and Erlang taking center stage as application languages—instead, I see them taking supporting-cast kinds of roles building other infrastructure that applications in turn make use of, a la CouchDB (written in Erlang). Monads still remain a mostly-opaque subject for most developers, however, and it’s still unclear if monads are something that people should think about applying in code, or if they are one of those “in theory” kinds of concepts. (You know, one of those ideas that change your brain forever, but you never actually use directly in code.)

THEN: ... Visual Studio 2010 will ship on time, and be one of the buggiest and/or slowest releases in its history. I hate to make this prediction, because I really don't want to be right, but there's just so much happening in the Visual Studio refactoring effort that it makes me incredibly nervous. Widespread adoption of VS2010 will wait until SP1 at the earliest. In fact....

NOW: Wow, did I get a few people here in Redmond annoyed with me about that one. And, as it turned out, I was pretty off-base about its stability. (It shipped pretty close if not exactly on the ship date Microsoft promised, as I recall, though I admit I wasn’t paying too much attention to it.) I’ve been using VS 2010 for a lot of .NET work in the last six months, and I’ve yet (knock on wood) to have it crash on me. /bow Visual Studio team.

THEN: ... Visual Studio 2010 SP 1 will ship within three months of the final product. Microsoft knows that people wait until SP 1 to think about upgrading, so they'll just plan for an eager SP 1 release, and hope that managers will be too hung over from the New Year (still) to notice that the necessary shakeout time hasn't happened.

NOW: Uh…. nope. In fact, SP 1 has just reached a beta/CTP state. As for managers being too hung over, well…

THEN: ... Apple will ship a tablet with multi-touch on it, and it will flop horribly. Not sure why I think this, but I just don't think the multi-touch paradigm that Apple has cooked up for the iPhone will carry over to a tablet/laptop device. That won't stop them from shipping it, and it won't stop Apple fan-boiz from buying it, but that's about where the interest will end.

NOW: Oh, WOW did I come so close and yet missed the mark by a mile. Of course, the “tablet” that Apple shipped was the iPad, and it did pretty much everything except flop horribly. Apple fan-boys bought it… and then about 24 hours later, so did everybody else. My mom got one, for crying out loud. And folks, the iPad—along with the whole “slate” concept—is pretty clearly here to stay.

THEN: ... JDK 7 closures will be debated for a few weeks, then become a fait accompli as the Java community shrugs its collective shoulders. Frankly, I think the Java community has exhausted its interest in debating new language features for Java. Recent college grads and open-source groups with an axe to grind will continue to try and make an issue out of this, but I think the overall Java community just... doesn't... care. They just want to see JDK 7 ship someday.

NOW: Pretty close—except that closures won’t ship as part of JDK 7, largely due to the Oracle acquisition in the middle of the year here. And I was spot-on vis-à-vis the “they want to see JDK 7 ship someday”; when given the chance to wait for a year or so for a Java-with-closures to ship, the community overwhelmingly voted to get something sooner rather than later.

THEN: ... Scala either "pops" in 2010, or begins to fall apart. By "pops", I mean reaches a critical mass of developers interested in using it, enough to convince somebody to create a company around it, a la G2One.

NOW: … and by “somebody”, it turns out I meant Martin Odersky. Scala is pretty clearly a hot topic in the Java space, its buzz being disturbed only by Clojure. Scala and/or Clojure, plus Groovy, makes a really compelling JVM-based stack.

THEN: ... Oracle is going to make a serious "cloud" play, probably by offering an Oracle-hosted version of Azure or AppEngine. Oracle loves the enterprise space too much, and derives too much money from it, to not at least appear to have some kind of offering here. Now that they own Java, they'll marry it up against OpenSolaris, the Oracle database, and throw the whole thing into a series of server centers all over the continent, and call it "Oracle 12c" (c for Cloud, of course) or something.

NOW: Oracle made a play, but it was to continue to enhance Java, not build a cloud space. It surprises me that they haven’t made a more forceful move in this space, but I suspect that a huge amount of time and energy went into folding Sun into their corporate environment.

THEN: ... Spring development will slow to a crawl and start to take a left turn toward cloud ideas. VMWare bought SpringSource for a reason, and I believe it's entirely centered around VMWare's movement into the cloud space—they want to be more than "just" a virtualization tool. Spring + Groovy makes a compelling development stack, particularly if VMWare does some interesting hooks-n-hacks to make Spring a virtualization environment in its own right somehow. But from a practical perspective, any community-driven development against Spring is all but basically dead. The source may be downloadable later, like the VMWare Player code is, but making contributions back? Fuhgeddabowdit.

NOW: The Spring One show definitely played up Cloud stuff, and springsource.com seems to be emphasizing cloud more in a couple of subtle ways. Not sure if I call this one a win or not for me, though.

THEN: ... the explosion of e-book readers brings the Kindle 2009 edition way down to size. The era of the e-book reader is here, and honestly, while I'm glad I have a Kindle, I'm expecting that I'll be dusting it off a shelf in a few years. Kinda like I do with my iPods from a few years ago.

NOW: Honestly, can’t say that I’m using my Kindle a lot, but I am reading using the Kindle app on non-Kindle hardware more than I thought I would be. That said, I am eyeing the new Kindle hardware generation with an acquisitive eye…

THEN: ... "social networking" becomes the "Web 2.0" of 2010. In other words, using the term will basically identify you as a tech wannabe and clearly out of touch with the bleeding edge.

NOW: Um…. yeah.

THEN: ... Facebook becomes a developer platform requirement. I don't pretend to know anything about Facebook—I'm not even on it, which amazes my family to no end—but clearly Facebook is one of those mechanisms by which people reach each other, and before long, it'll start showing up as a developer requirement for companies looking to hire. If you're looking to build out your resume to make yourself attractive to companies in 2010, mad Facebook skillz might not be a bad investment.

NOW: I’m on Facebook, I’ve written some code for it, and given how much the startup scene loves the “Like” button, I think developers who knew Facebook in 2010 did pretty well for themselves.

THEN: ... Nintendo releases an open SDK for building games for its next-gen DS-based device. With the spectacular success of games on the iPhone, Nintendo clearly must see that they're missing a huge opportunity every day developers can't write games for the Nintendo DS that are easily downloadable to the device for playing. Nintendo is not stupid—if they don't open up the SDK and promote "casual" games like those on the iPhone and those that can now be downloaded to the Zune or the XBox, they risk being marginalized out of existence.

NOW: Um… yeah. Maybe this was me just being hopeful.

In general, it looks like I was more right than wrong, which is not a bad record to have. Of course, a couple of those “wrong”s were “giving up the big play” kind of wrongs, so while I may have a winning record, I still may have a defense that’s given up too many points to be taken seriously. *shrug* Oh, well.

What portends for 2011?

Android’s penetration into the mobile space is going to rise, then plateau around the middle of the year. Android phones, collectively, have outpaced iPhone sales. That’s a pretty significant statistic—and it means that there’s fewer customers buying smartphones in the coming year. More importantly, the first generation of Android slates (including the Galaxy Tab, which I own), are less-than-sublime, and not really an “iPad Killer” device by any stretch of the imagination. And I think that will slow down people buying Android slates and phones, particularly since Google has all but promised that Android releases will start slowing down.
Windows Phone 7 penetration into the mobile space will appear huge, then slow down towards the middle of the year. Microsoft is getting some pretty decent numbers now, from what I can piece together, and I think that’s largely the “I love Microsoft” crowd buying in. But it’s a pretty crowded place right now with Android and iPhone, and I’m not sure if the much-easier Office and/or Exchange integration is enough to woo consumers (who care about Office) or business types (who care about Exchange) away from their Androids and iPhones.
Android, iOS and/or Windows Phone 7 becomes a developer requirement. Developers, if you haven’t taken the time to learn how to program one of these three platforms, you are electing to remove yourself from a growing market that desperately wants people with these skills. I see the “mobile native app development” space as every bit as hot as the “Internet/Web development” space was back in 2000. If you don’t have a device, buy one. If you have a device, get the tools—in all three cases they’re free downloads—and start writing stupid little apps that nobody cares about, so you can have some skills on the platform when somebody cares about it.
The Windows 7 slates will suck. This isn’t a prediction, this is established fact. I played with an “ExoPC” 10” form factor slate running Windows 7 (Dell I think was the manufacturer), and it was a horrible experience. Windows 7, like most OSes, really expects a keyboard to be present, and a slate doesn’t have one—so the OS was hacked to put a “keyboard” button at the top of the screen that would slide out to let you touch-type on the slate. I tried to fire up Notepad and type out a haiku, and it was an unbelievably awkward process. Android and iOS clearly own the slate market for the forseeable future, and if Dell has any brains in its corporate head, it will phone up Google tomorrow and start talking about putting Android on that hardware.
DSLs mostly disappear from the buzz. I still see no strawman (no “pet store” equivalent), and none of the traditional builders-of-strawmen (Microsoft, Oracle, etc) appear interested in DSLs much anymore, so I think 2010 will mark the last year that we spent any time talking about the concept.
Facebook becomes more of a developer requirement than before. I don’t like Mark Zuckerburg. I don’t like Facebook’s privacy policies. I don’t particularly like the way Facebook approaches the Facebook Connect experience. But Facebook owns enough people to be the fourth-largest nation on the planet, and probably commands an economy of roughly that size to boot. If your app is aimed at the Facebook demographic (that is, everybody who’s not on Twitter), you have to know how to reach these people, and that means developing at least some part of your system to integrate with it.
Twitter becomes more of a developer requirement, too. Anybody who’s not on Facebook is on Twitter. Or dead. So to reach the other half of the online community, you have to know how to connect out with Twitter.
XMPP becomes more of a developer requirement. XMPP hasn’t crossed a lot of people’s radar screen before, but Facebook decided to adopt it as their chat system communication protocol, and Google’s already been using it, and suddenly there’s a whole lotta traffic going over XMPP. More importantly, it offers a two-way communication experience that is in some scenarios vastly better than what HTTP offers, yet running in a very “Internet-friendly” way just as HTTP does. I suspect that XMPP is going to start cropping up in a number of places as a useful alternative and/or complement to using HTTP.
“Gamification” starts making serious inroads into non-gaming systems. Maybe it’s just because I’ve been talking more about gaming, game design, and game implementation last year, but all of a sudden “gamification”—the process of putting game-like concepts into non-game applications—is cresting in a big way. FourSquare, Yelp, Gowalla, suddenly all these systems are offering achievement badges and scoring systems for people who want to play in their worlds. How long is it before a developer is pulled into a meeting and told that “we need to put achievement badges into the call-center support application”? Or the online e-commerce portal? It’ll start either this year or next.
Functional languages will hit a make-or-break point. I know, I said it last year. But the buzz keeps growing, and when that happens, it usually means that it’s either going to reach a critical mass and explode, or it’s going to implode—and the longer the buzz grows, the faster it explodes or implodes, accordingly. My personal guess is that the “F/O hybrids”—F#, Scala, etc—will continue to grow until they explode, particularly since the suggested v.Next changes to both Java and C# have to be done as language changes, whereas futures for F# frequently are either built as libraries masquerading as syntax (such as asynchronous workflows, introduced in 2.0) or as back-end library hooks that anybody can plug in (such as type providers, introduced at PDC a few months ago), neither of which require any language revs—and no concerns about backwards compatibility with existing code. This makes the F/O hybrids vastly more flexible and stable. In fact, I suspect that within five years or so, we’ll start seeing a gradual shift away from pure O-O systems, into systems that use a lot more functional concepts—and that will propel the F/O languages into the center of the developer mindshare.
The Microsoft Kinect will lose its shine. I hate to say it, but I just don’t see where the excitement is coming from. Remember when the Wii nunchucks were the most amazing thing anybody had ever seen? Frankly, after a slew of initial releases for the Wii that made use of them in interesting ways, the buzz has dropped off, and more importantly, the nunchucks turned out to be just another way to move an arrow around on the screen—in other words, we haven’t found particularly novel and interesting/game-changing ways to use the things. That’s what I think will happen with the Kinect. Sure, it’s really freakin’ cool that you can use your body as the controller—but how precise is it, how quickly can it react to my body movements, and most of all, what new user interface metaphors are people going to have to come up with in order to avoid the “me-too” dancing-game clones that are charging down the pipeline right now?
There will be no clear victor in the Silverlight-vs-HTML5 war. And make no mistake about it, a war is brewing. Microsoft, I think, finds itself in the inenviable position of having two very clearly useful technologies, each one’s “sphere of utility” (meaning, the range of answers to the “where would I use it?” question) very clearly overlapping. It’s sort of like being a football team with both Brett Favre and Tom Brady on your roster—both of them are superstars, but you know, deep down, that you have to cut one, because you can’t devote the same degree of time and energy to both. Microsoft is going to take most of 2011 and probably part of 2012 trying to support both, making a mess of it, offering up conflicting rationale and reasoning, in the end achieving nothing but confusing developers and harming their relationship with the Microsoft developer community in the process. Personally, I think Microsoft has no choice but to get behind HTML 5, but I like a lot of the features of Silverlight and think that it has a lot of mojo that HTML 5 lacks, and would actually be in favor of Microsoft keeping both—so long as they make it very clear to the developer community when and where each should be used. In other words, the executives in charge of each should be locked into a room and not allowed out until they’ve hammered out a business strategy that is then printed and handed out to every developer within a 3-continent radius of Redmond. (Chances of this happening: .01%)
Apple starts feeling the pressure to deliver a developer experience that isn’t mired in mid-90’s metaphor. Don’t look now, Apple, but a lot of software developers are coming to your platform from Java and .NET, and they’re bringing their expectations for what and how a developer IDE should look like, perform, and do, with them. Xcode is not a modern IDE, all the Apple fan-boy love for it notwithstanding, and this means that a few things will happen:

Eclipse gets an iOS plugin. Yes, I know, it wouldn’t work (for the most part) on a Windows-based Eclipse installation, but if Eclipse can have a native C/C++ developer experience, then there’s no reason why a Mac Eclipse install couldn’t have an Objective-C plugin, and that opens up the idea of using Eclipse to write iOS and/or native Mac apps (which will be critical when the Mac App Store debuts somewhere in 2011 or 2012).
Rumors will abound about Microsoft bringing Visual Studio to the Mac. Silverlight already runs on the Mac; why not bring the native development experience there? I’m not saying they’ll actually do it, and certainly not in 2011, but the rumors, they will be flyin….
Other third-party alternatives to Xcode will emerge and/or grow. MonoTouch is just one example. There’s opportunity here, just as the fledgling Java IDE market looked back in ‘96, and people will come to fill it.

NoSQL buzz grows. The NoSQL movement, which sort of got started last year, will reach significant states of buzz this year. NoSQL databases have a lot to offer, particularly in areas that relational databases are weak, such as hierarchical kinds of storage requirements, for example. That buzz will reach a fever pitch this year, and the relational database moguls (Microsoft, Oracle, IBM) will start to fight back.

I could probably go on making a few more, but I think these are enough to get me into trouble for the year.

To all of you who’ve been readers of this blog for the past year, I thank you—blog-gathered statistics tell me that I get, on average, about 7,000 hits a day, which just stuns me—and it is a New Years’ Resolution that I blog more and give you even more reason to stick around. Happy New Year, and may your 2011 be just as peaceful, prosperous, and eventful as you want it to be.

Death to Best Practices

Tue, 10 Aug 2010 00:00:00 +0000

Can we please put the whole term “Best Practices” to rest now? Apparently, according to this link (forwarded to me by John Dietz, thanks!), the very place where it originated (or was best popularized, depending on your interpretation of history) has now seen the whole concept basically debunked:

For example, Jim Collins’ blockbuster business book Good to Great, published in 2001, featured 11 supposedly great companies. All of them did extraordinarily well on the stock market for 10-20 years. But by 2008, when Steven Levitt posted Good to Great to Below Average on Freakonomics, two of them had died.
(Read more: http://timberry.bplans.com/2010/07/the-sad-truth-about-best-practices.html#ixzz0wBOxDrkh)

The point is, best practices just don’t exist. They are an attempt to take a solution to a problem out of the context and apply them across the entire spectrum, and that essentially invalidates the entire thing. A solution is only useful when considered in context.

Don’t believe me? Consider this challenge: when is it not a best practice to breathe? When you’re under water, of course.

(Unless you’re a fish. Or a frog. Or maybe a prince turned into a frog.)

Point is… context matters, folks.

Blind application of best practices don’t work, as Tim Berry’s article quotes from Jim Collins’ book (my emphasis):

Nine of the eleven companies remain more or less intact. Of these, Nucor is the only one that has dramatically outperformed the stock market since the book came out. Abbott Labs and Wells Fargo have done okay. Overall, a portfolio of the “good to great” companies looks like it would have underperformed the S&P 500.

Still think that the “best practices” idea might work? Prove it to yourself: do a detailed study of CEO performance across any given CEO’s career and across a variety of different companies who changed CEOs. In short windows, yes, you can find scenarios where a CEO had a stellar performance with a particular company for a particular period of time. But when you pan back and look across the CEO’s entire career (during which he/she practiced the same “best practices” at different firms), almost none of them have repeatable successes at a variety of different firms in any consistent manner (despite the millions handed to them).

In other words, for a good many of them, their success was nothing but blind luck. A monkey could have done just as well. The “superstar CEO” is generally a product of the five or so years in which his one firm was wildly successful. Attempts to repeat the success at other firms (that is, in a different context) typically have failed or generated mediocre results. (Anybody know what Lee Iacocca is up to these days, he of “I will rescue Chrysler” fame? Come to think of it, can anybody name any of the 70’s or 80’s “superstar CEOs” that is still wildly successful today? Just one?)

How did this “best practices” thing get to be such a common meme? Because “best practices” mean, essentially, that the questioner doesn’t want to have to think. And it’s a seductive premise—if I just push the right buttons, type the right keywords, call the right methods and/or use the right classes, I can get something that “just works” without having to think about all those nasty little details that seem to trip people up: performance, scalability, security, blah blah blah.

Here’s the dirty little secret of our industry: Software development is hard.

Computer science is about tradeoffs and hard choices. Optimizing the code or design or architecture one way means taking hits another way. Trading static typing for dynamic typing means losing a set of already-written unit tests in exchange for a degree of flexibility in certain parts of the design. Using inheritance instead of parametric polymorphism offers some benefits but also adds some restrictions, and vice versa. Choosing an agile approach gains you greater feedback and closer connection to your customer (which typically means you’re closer to budget and critical features being completed on time), but requires more work and expertise to pull it off over other, more waterfall-ish, processes. And so on, and so on, and so on.

Tim says this well:

Don’t ever just blindly follow. You always think about it, consider the options, how it might be different in your case, and then, if it still sounds good, try it. Carefully.

If I ever give you any advice, I want you to please never take it without thinking first, analyzing, and deciding for yourself whether or not, and how, and to what extent what I say fits your situation.

(Read more: http://timberry.bplans.com/2010/07/the-sad-truth-about-best-practices.html#ixzz0wBQU3yCB)

But it’s not just the questioners’ fault: speakers, in their zeal to prove that they were smarter than everybody else, bought into the idea, too, because if I’m the one holding the “best practices”, then clearly I’m the one you want to come to with the development or consulting work. After all, who better to hire than the guy/gal with all the answers? In essence, we fed their addiction by tossing off “best practices” in pithy one-line answers (like “every class a service”) that turned out to be utter B/S pronounced by people who had, in some cases, never actually put that practice into practice for anything other than a demo or an example.

It’s time to say “no” to “best practices”.

Speakers: The next time somebody asks you for the “best practices” on a technology, respond with “The best practice you could possibly employ is to hire me.” That, or else with “There are no such thing. You cannot answer a question about a problem outside of its context.”

Attendees: The next time a speaker starts talking about “best practices”, walk out, because clearly the speaker is trying to feed you easy answers when in fact there are only hard choices.

It’s time for our industry to break the habit of taking hits off the ~~crack~~ best practices pipe, and start facing the fact that software development is hard.

Architectural Katas

Thu, 17 Jun 2010 00:00:00 +0000

By now, the Twitter messages have spread, and the word is out: at Uberconf this year, I did a session ("Pragmatic Architecture"), which I've done at other venues before, but this time we made it into a 180-minute workshop instead of a 90-minute session, and the workshop included breaking the room up into small (10-ish, which was still a teensy bit too big) groups and giving each one an "architectural kata" to work on.

The architectural kata is a take on PragDave's coding kata, except taken to a higher level: the architectural kata is an exercise in which the group seeks to create an architecture to solve the problem presented. The inspiration for this came from Frederick Brooks' latest book, The Design of Design, in which he points out that the only way to get great designers is to get them to design. The corollary, of course, is that in order to create great architects, we have to get them to architect. But few architects get a chance to architect a system more than a half-dozen times or so over the lifetime of a career, and that's only for those who are fortunate to be given the opportunity to architect in the first place. Of course, the problem here is, you have to be an architect in order to get hired as an architect, but if you're not an architect, then how can you architect in order to become an architect?

Um... hang on, let me make sure I wrote that right.

Anyway, the "rules" around the kata (which makes it more difficult to consume the kata but makes the scenario more realistic, IMHO):

you may ask the instructor questions about the project
you must be prepared to present a rough architectural vision of the project and defend questions about it
you must be prepared to ask questions of other participants' presentations
you may safely make assumptions about technologies you don't know well as long as those assumptions are clearly defined and spelled out
you may not assume you have hiring/firing authority over the development team
any technology is fair game (but you must justify its use)
any other rules, you may ask about

The groups were given 30 minutes in which to formulate some ideas, and then three of them were given a few minutes to present their ideas and defend it against some questions from the crowd.

(UPDATE 2021: The full list of katas is available at the architectural katas website, and available for forking as a repository from my Github. The website code is available here for those who'd like to play with it, but it's pretty simple and probably much better re-thought from the ground up.)

An example kata is below:

**Architectural Kata #5: I'll have the BLT**
a national sandwich shop wants to enable "fax in your order" but over the Internet instead

users: millions+

requirements: users will place their order, then be given a time to pick up their sandwich and directions to the shop (which must integrate with Google Maps); if the shop offers a delivery service, dispatch the driver with the sandwich to the user; mobile-device accessibility; offer national daily promotionals/specials; offer local daily promotionals/specials; accept payment online or in person/on delivery

As you can tell, it's vague in some ways, and this is somewhat deliberate--as one group discovered, part of the architect's job is to ask questions of the project champion (me), and they didn't, and felt like they failed pretty miserably. (In their defense, the kata they drew--randomly--was pretty much universally thought to be the hardest of the lot.) But overall, the exercise was well-received, lots of people found it a great opportunity to try being an architect, and even the team that failed felt that it was a valuable exercise.

I'm definitely going to do more of these, and refine the whole thing a little. (Thanks to everyone who participated and gave me great feedback on how to make it better.) If you're interested in having it done as a practice exercise for your development team before the start of a big project, ping me. I think this would be a great exercise to do during a user group meeting, too.

Code Kata: RoboStack

Mon, 10 May 2010 00:00:00 +0000

Code Katas are small, relatively simple exercises designed to give you a problem to try and solve. I like to use them as a way to get my feet wet and help write something more interesting than "Hello World" but less complicated than "The Internet's Next Killer App".

This one is from the UVa online programming contest judge system, which I discovered after picking up the book Programming Challenges, which is highly recommended as a source of code katas, by the way. Much of the advice parts of the book can be skimmed or ignored by the long-time professional developer, but it's still worth a read, since it can be an interesting source of ideas and approaches when solving real-world scenarios.

Problem: You work for a manufacturing company, and they have just received their newest piece of super-modern hardware, a highly efficient assembly-line mechanized pneumatic item manipulator, also known in some circles as a "robotic arm". It is driven by a series of commands, and your job is to write the software to drive the arm. The initial test will be to have the arm move a series of blocks around.

Context: The test begins with n number of blocks, laid out sequentially next to each other, each block with a number on it. (You may safely assume that n never exceeds 25.) So, if n is 4, then the blocks are laid out (starting from 0) as:

0: 0

1: 1

2: 2

3: 3

The display output here is the block-numbered "slot", then a colon, then the block(s) that are stacked in that slot, lowest to highest in left to right order. Thus, in the following display:

0:

1:

2: 0 1 2 3

3:

The 3 block is stacked on top of the 2 block is stacked on top of the 1 block is stacked on top of the 0 block, all in slot 2. This can be shortened to the representation [0:, 1:, 2: 0 1 2 3, 3:] for conciseness.

The arm understands a number of different commands, as well as an optic sensor. (Yeah, the guys who created the arm were good enough to write code that knows how to read the number off a block, but not to actually drive the arm. Go figure.) The commands are as follows, where a and b are valid block numbers (meaning they are between 0 and n-1):

"move a onto b" This command orders the arm to find block a, and return any blocks stacked on top of it to their original position. Do the same for block b, then stack block a on top of b.
"move a over b" This command orders the arm to find block a, and return any blocks stacked on top of it to their original position. Then stack block a on top of the stack of blocks containing b.
"pile a onto b" This command orders the arm to find the stack of blocks containing block b, and return any blocks stacked on top of it to their original position. Then the arm must find the stack of blocks containing block a, and take the stack of blocks starting from a on upwards (in other words, don't do anything with any blocks on top of a) and put that stack on top of block b.
"pile a over b" This command orders the arm to find the stack of blocks containing block a and take the stack of blocks starting from a on upwards (in other words, don't do anything with any blocks on top of a) and put that stack on top of the stack of blocks containing block b (in other words, don't do anything with the stack of blocks containing b, either).
"quit" This command tells the arm to shut down (and thus terminates the simulation).

Note that if the input command sequence accidentally offers a command where a and b are the same value, that command is illegal and should be ignored.

As an example, then, if we have 4 blocks in the state [0: 0, 1: 1, 2: 2, 3: 3], and run a "move 2 onto 3", we get [0: 0, 1: 1, 2:, 3: 3 2]. If we then run a "pile 3 over 1", we should end up with [0: 0, 1: 1 3 2, 2:, 3:]. And so on.

Input: n = 10. Run these commands:

move 9 onto 1
move 8 over 1
move 7 over 1
move 6 over 1
pile 8 over 6
pile 8 over 5
move 2 over 1
move 4 over 9
quit

The result should be [0: 0, 1: 1 9 2 4, 2:, 3: 3, 4:, 5: 5 8 7 6, 6:, 7:, 8:, 9:]

Challenges:

Implement the Towers of Hanoi (or as close to it as you can get) using this system.
Add an optimizer to the arm, in essence reading in the entire program (up to "quit"), finding shorter paths and/or different commands to achieve the same result.
Add a visual component to the simulation, displaying the arm as it moves over each block and moves blocks around.
Add another robotic arm, and allow commands to be given simultaneously. This will require some thought—does each arm execute a complete command before allowing the other arm to execute (which reduces the performance having two arms might offer), or can each arm act entirely independently? The two (or more) arms will probably need separate command streams, but you might try running them with one command stream just for grins. Note that deciding how to synchronized the arms so they don't conflict with one another will probably require adding some kind of synchronization instructions into the stream as well.

Code Kata: Compressing Lists

Thu, 6 May 2010 00:00:00 +0000

Rick Minerich mentioned this one on his blog already, but here is the original "problem"/challenge as it was presented to me and which I in turn shot to him over a Twitter DM:

I have a list, say something like [4, 4, 4, 4, 2, 2, 2, 3, 3, 2, 2, 2, 2, 1, 1, 1, 5, 5], which consists of varying repetitions of integers. (We can assume that it's always numbers, and the use of the term "list" here is generic—it could be a list, array, or some other collection class, your choice.) The goal is to take this list of numbers, and "compress" it down into a (theoretically smaller) list of numbers in pairs, where the first of the pair is the occurrence number of the value, which is the second number. So, since the list above has four 4's, followed by three 2's, two 3's, four 2's, three 1's and two 5's, it should compress into [4, 4, 3, 2, 2, 3, 4, 2, 3, 1, 2, 5].

This is a pretty easy challenge, but I wanted to try and solve it in a functional mindset, which the challenger had never seen before. I also thought it made for an interesting challenge for people who've never programming in functional languages before, because it requires a very different approach than the imperative solution.

Extensions to the kata (a.k.a. "extra credit"):

How does the implementation change (if any) to generalize it to a list of any particular type? (Assume the list is of homogenous type—always strings, always ints, always whatever.)
How does the implementation change (if any) to generalize it to a list of any type? (In other words, a list of strings, ints, Dates, whatever, mixed together within the list: [1, 1, "one", "one", "one", ...] .)
How does the implementation change (if any) to generate a list of two-item tuples (the first being the occurence, the second being the value) as the result instead? Are there significant advantages to this?
How does the implementation change (if any) to parallelize/multi-thread it? For your particular language how many elements have to be in the list before doing so yields a significant payoff?

By the way, some of the extension questions make the Kata somewhat interesting even for the imperative/O-O developer; have at, and let me know what you think.

Don't Fear the dynamic/VARIANT/Reaper...

Sun, 14 Feb 2010 00:00:00 +0000

A couple of days ago, a buddy of mine, Scott Hanselman, wrote a nice little intro to the "dynamic" type in C# 4.0. In particular, I like (though don't necessarily 100% agree with) his one-sentence summation of dynamic as "There's no way for you or I to know the type of this now, compiler, so let's hope that the runtime figures it out." It's an interesting characterization, but my disagreement with his characterization is not the point here, at least not of this particular blog entry.

I've been waiting for it for a while, ever since C# 4 was announced, and sure enough, here we go: Scott's blog is the victim of the Static-Typing Fundamentalist, the bearded and grizzled veteran of the Static/Dynamic Code Wars, come out to proclaim the sins of dynamic programming, the evils of those who use(d) it, and why C#/C++/Java was so much better than Visual Basic/Ruby/Python/whatever. Be careful of these creatures. They rival Al-Qaeda in their ferocity and zeal, Fox News in their attention to detail and evidence, and George Bush in their pronouncements of gloom and doom for the future if we don't act now and eliminate this evil.

Allow me to quote (liberally) from Rob's comment on Scott's blog, and comment in turn as we go:

It's such a shame that you promote this stuff. You should've seen the horrific devastation that "Variant" caused in the old VB days. Variant single-handedly create job security for so many people since the late 90's, because of the horrible, horrible, horrible things that developers did with that ridiculous, 12-byte data type!

I just love it when people make comments like "horrific devastation". Nothing like a little hyperbole to liven things up! I mean, it didn't cause exceptions, it didn't make code hard to read, it didn't make it tricky for developers to modify and refactor safely, it leveled cities! burned forests! slaughtered kittens! and even worse, it was 12 bytes in size!

Never mind the fact that Visual Basic developers frequently churned out apps twice, three, five times faster than their C++ cousins did. (I know this—I was one of those C++ developers, and routinely mocked the VB guys across the hall for their crappy language and code.... until they built an app in a few days that I tried to build at home in C++ and gave up after two weeks. And all the damn thing did was basic dialogs-and-data kinds of stuff, too.)

This weak-typing with late-binding is just such a bad idea. I know you'll say "But wait, these are powerful tools that skilled developers can leverage!" - and maybe so, but 98% of the people that truly use these sorts of techniques out in the real world, are unskilled developers making a mess of software all across this great land, because the compiler is so forgiving.

Ah, the "All Developers (Except Me) Are Idiots" argument. I love this one—-the hubris involved here is just too precious for words. I have no doubt that the author of this post, being (of course) the classically-trained object-oriented developer and therefore too smart/disciplined/experienced/whatever to fall into such a ridiculous temptation as to use dynamic typing, would never use this feature except in the Most Dire of Emergencies, but his fellow programmers, all of them being much less disciplined/smart/trained/whatever than he is, will fall for the temptation and write code that levels cities! burns forests! kills kittens! and worse, uses 12 bytes! (Oh, wait, it's only 3 bytes, because dynamic is just a placeholder for an object reference, and all object references are 3 bytes in the CLR. Or at least they used to be—-I admit, I haven't checked in CLR 4.) Those poor souls, they won't have any hope! There they'll be, staring at Visual Studio, wanting desperately to do the Right Thing, and that evil little programmer devil on their shoulder (probably wearing a T-shirt that says, "P3rl is l33t" or something equally blasphemous) will whisper, "You know, if you just make it a dynamic, you can get the compiler to shut up and you can go home early...."

Oh, right—-sorry, I forgot. That devil will whisper, "You know, if you write this code in Visual Basic .NET, you can make the entire codebase Option Strict Off and Option Explicit Off, make the compiler shut up and you can go home early...." Hell, they've been whispering that bit of subversion since 2001. And ye Gods! The leveled cities! burned forests! cute little kitten bodies! all over the place! It's fortunate that we C# developers have kept all those Visual Basic developers on the straight-and-narrow path of ~~true salvation~~ static typing.

This is a huge step backwards for C#, in my opinion - and creates the same scenario VB always did - where it is so forgiving, that it allows developers to write horrible code and you won't so much as see a compiler warning!! I've always tauted that C# was better, simply because it gave the developer "tough love", and forced him/her to be better coder and to "make good choices"! :-)

Ah, yes, the C# compiler and its "tough love". The "prefer compile errors over runtime errors" argument, vis-a-vis Scott Meyers' "Effective C++" circa 1994 or so. It's vastly preferable to see errors early, before the big demo in front of the VP/President/potential customer. (Anybody who disagrees with this obviously hasn't had a demo fail in front of a VP/President/potential customer.) How fortunate that the C# compiler catches all these ugly errors at compile-time, like

static void DoSomething()
{
    List<object> intList = new List<object>();
    intList.Add(5);
    string s = (string) intList[0];
    Console.WriteLine(s);
}

... because boy, that would be embarrassing if it didn't. I mean, can you imagine the horror other disciplined/smart/experienced developers would feel if a lenient compiler actually allowed code like this:

class Point
{
    internal int x;
    internal int y;
    public Point(int x, int y) { x = x; y = y; }
}

or this:

class Point
{
    internal int x;
    internal int y;
    public Point(int x, int y)
    {
        this.x = x;
        this.y = y;
    }
    public override string ToString()
    { return String.Format("({0},{1})", x, y); }
}

static void DoSomething()
{
    Point pt = new Point(12, 12);
    pt.GetType()
        .GetField("x", BindingFlags.Instance | BindingFlags.NonPublic)
        .SetValue(pt, 24);
    Console.WriteLine(pt);
}

to compile? Cities! Forests! Kittens! Thank God C# isn't that kind of lustfully promiscuous... I mean, "lenient"... compiler!

(Now if only we could tout blog comment engines with spellcheck....)

Specific to this blog post, if you are doing somewhere where you can't even quantify what the data type that is coming back? Guess what, you've got yourself a bad design.

Wow. There's just no arguing with that one. I mean, knowing the actual type on which the method is being dispatched is such a huge part of the C# development experience:

static void DoSomething()
{
    List<Point> ptList = new List<Point>();
    ptList.Add(new Point(12, 12));
    object o = ptList[0];
    Console.WriteLine(o.ToString());
}

Gah. Just the thought of not knowing the concrete type on which the method is being dispatched gives me the heebie-jeebies.

Just because the framework allows you use weak-typing and late-binding, doesn't mean you should - nor should you endorse it's use, in my opinion.

Somebody better tell all those users of NHibernate, NUnit, Spring.NET, MEF and all those other Reflection-based tools... including WinForms, ASP.NET, WPF, Workflow and WCF, come to think about it... that they're using frameworks that clearly were designed by idiots. (The gall of those people.)

I'm just saying, it's a shame that popular "nerd celebrities" like you (and I mean zero offense by that!) - endorse all this loosey-goosey typing. I say that becuase I've never seen a single case where weak typing or late binding: A) made a design better or B) where it didn't make the component or application worse, because it was a looser design.

I'm so glad you were here to set Scott and me straight, Rob. Because otherwise, we might actually get something done. God forbid.

Little tidbits of thought for those who are still thinking about this one.

Ola Bini describes the application of the right language at the right level of the stack as a three-layer pyramid.
Any C# or Java developer who's not writing unit tests to test their code "because the compiler will catch all those errors" and provide "tough love" needs to be fired. Immediately. I cannot conceive of a situation where unit tests can be passed over in favor of static typing in a professionally-responsible development project. (Oh, don't mis-read that, I can see lots of situations where unit tests aren't necessary. But not on code that's going to reach Production.)
The argument for the degree of static typing in C# or Java is completely indefensible compared to what statically-typed type-inferenced languages like Haskell, F# or Scala provide. And their syntax frequently looks like let x = [ 1; 2; 3; 4; ], which isn't all that far off from what a dynamically-typed language looks like, despite very very different things happening under the compiler's hood. Until you, the Statically-Typed Fundamentalist, have written code in a Haskell/ML-derived language, you have no right arguing the merits of static typing. (In fact, that's probably also true if you've never written code in Ruby, Python, or PowerShell, either.)
There's lots more arguments the Static-Typing Fundamentalist can throw, by the way. I'm disappointed Rob never mentioned performance, for one—that's a classic line of attack, too. Never mind the fact that most of those guys are still looping down and doing other silly micro-optimizations because that's way C++ taught them to do it....
Oh, and never ever show the Static Typing Fundamentalist an XML document and using something like XPath to extract data from it. They inevitably fall into XML Schema and the "if we just write the schema flexibly enough" and.... The last time I did that.... I still visit his gravesite, all these years later, and it still hurts, losing him that way.
Java guys argued against dynamic typing for years, too... until they tried Groovy and JRuby and Clojure. Now.... not so much.

Peace out.

2010 Tech Predictions

Tue, 5 Jan 2010 00:00:00 +0000

Here we go again: the annual review of last year's predictions, and a set of new ones for the new year.

Here we go again—another year, another set of predictions revisited and offered up for the next 12 months. And maybe, if I'm feeling really ambitious, I'll take that shot I thought about last year and try predicting for the decade. Without further ado, I'll go back and revisit, unedited, my predictions for 2009 ("THEN"), and pontificate on those subjects for 2010 before adding any new material/topics. Just for convenience, here's a link back to last years' predictions.

Last year's predictions went something like this (complete with basketball-scoring):

THEN: "Cloud" will become the next "ESB" or "SOA", in that it will be something that everybody will talk about, but few will understand and even fewer will do anything with. (Considering the widespread disparity in the definition of the term, this seems like a no-brainer.) NOW: Oh, yeah. Straight up. I get two points for this one. Does anyone have a working definition of "cloud" that applies to all of the major vendors' implementations? Ted, 2; Wrongness, 0.
THEN: Interest in Scala will continue to rise, as will the number of detractors who point out that Scala is too hard to learn. NOW: Two points for this one, too. Not a hard one, mind you, but one of those "pass-and-shoot" jumpers from twelve feet out. James Strachan even tweeted about this earlier today, pointing out this comparison. As more Java developers who think of themselves as smart people try to pick up Scala and fail, the numbers of sour grapes responses like "Scala's too complex, and who needs that functional stuff anyway?" will continue to rise in 2010. Ted, 4; Wrongness, 0.
THEN: Interest in F# will continue to rise, as will the number of detractors who point out that F# is too hard to learn. (Hey, the two really are cousins, and the fortunes of one will serve as a pretty good indication of the fortunes of the other, and both really seem to be on the same arc right now.) NOW: Interestingly enough, I haven't heard as many F# detractors as Scala detractors, possibly because I think F# hasn't really reached the masses of .NET developers the way that Scala has managed to find its way in front of Java developers. I think that'll change mighty quickly in 2010, though, once VS 2010 hits the streets. Ted, 4; Wrongness 2.
THEN: Interest in all kinds of functional languages will continue to rise, and more than one person will take a hint from Bob "crazybob" Lee and liken functional programming to AOP, for good and for ill. People who took classes on Haskell in college will find themselves reaching for their old college textbooks again. NOW: Yep, I'm claiming two points on this one, if only because a bunch of Haskell books shipped this year, and they'll be the last to do so for about five years after this. (By the way, does anybody still remember aspects?) But I'm going the opposite way with this one now; yes, there's Haskell, and yes, there's Erlang, and yes, there's a lot of other functional languages out there, but who cares? They're hard to learn, they don't always translate well to other languages, and developers want languages that work on the platform they use on a daily basis, and that means F# and Scala or Clojure, or its simply not an option. Ted 6; Wrongness 2.
THEN: The iPhone is going to be hailed as "the enterprise development platform of the future", and companies will be rolling out apps to it. Look for Quicken iPhone edition, PowerPoint and/or Keynote iPhone edition, along with connectors to hook the iPhone up to a presentation device, and (I'll bet) a World of Warcraft iPhone client (legit or otherwise). iPhone is the new hotness in the mobile space, and people will flock to it madly. NOW: Two more points, but let's be honest—this was a fast-break layup, no work required on my part. Ted 8; Wrongness 2.
THEN: Another Oslo CTP will come out, and it will bear only a superficial resemblance to the one that came out in October at PDC. Betting on Oslo right now is a fools' bet, not because of any inherent weakness in the technology, but just because it's way too early in the cycle to be thinking about for anything vaguely resembling production code. NOW: If you've worked at all with Oslo, you might argue with me, but I'm still taking my two points. The two CTPs were pretty different in a number of ways. Ted 10; Wrongness 2.
THEN: The IronPython and IronRuby teams will find some serious versioning issues as they try to manage the DLR versioning story between themselves and the CLR as a whole. An initial hack will result, which will be codified into a standard practice when .NET 4.0 ships. Then the next release of IPy or IRb will have to try and slip around its restrictions in 2010/2011. By 2012, IPy and IRb will have to be shipping as part of Visual Studio just to put the releases back into lockstep with one another (and the rest of the .NET universe). NOW: Pressure is still building. Let's see what happens by the time VS 2010 ships, and then see what the IPy/IRb teams start to do to adjust to the versioning issues that arise. Ted 8; Wrongness 2.
THEN: The death of JSR-277 will spark an uprising among the two leading groups hoping to foist it off on the Java community--OSGi and Maven--while the rest of the Java world will breathe a huge sigh of relief and look to see what "modularity" means in Java 7. Some of the alpha geeks in Java will start using--if not building--JDK 7 builds just to get a heads-up on its impact, and be quietly surprised and, I dare say, perhaps even pleased. NOW: Ah, Ted, you really should never underestimate the community's willingness to take a bad idea, strip all the goodness out of it, and then cycle it back into the mix as something completely different yet somehow just as dangerous and crazy. I give you Project Jigsaw. Ted 10; Wrongness 2;
THEN: The invokedynamic JSR will leapfrog in importance to the top of the list. NOW: The invokedynamic JSR begat interest in other languages on the JVM. The interest in other languages on the JVM begat the need to start thinking about how to support them in the Java libraries. The need to start thinking about supporting those languages begat a "Holy sh*t moment" somewhere inside Sun and led them to (re-)propose closures for JDK 7. And in local sports news, Ted notched up two more points on the scoreboard. Ted 12; Wrongness 2.
THEN: Another Windows 7 CTP will come out, and it will spawn huge media interest that will eventually be remembered as Microsoft promises, that will eventually be remembered as Microsoft guarantees, that will eventually be remembered as Microsoft FUD and "promising much, delivering little". Microsoft ain't always at fault for the inflated expectations people have--sometimes, yes, perhaps even a lot of times, but not always. NOW: And then, just when the game started to turn into a runaway, airballs started to fly. The Windows7 release shipped, and contrary to what I expected, the general response to it was pretty warm. Yes, there were a few issues that emerged, but overall the media liked it, the masses liked it, and Microsoft seemed to have dodged a bullet. Ted 12; Wrongness 5.
THEN: Apple will begin to legally threaten the clone market again, except this time somebody's going to get the DOJ involved. (Yes, this is the iPhone/iTunes prediction from last year, carrying over. I still expect this to happen.) NOW: What clones? The only people trying to clone Macs are those who are building Hackintosh machines, and Apple can't sue them so long as they're using licensed copies of Mac OS X (as far as I know). Which has never stopped them from trying, mind you, and I still think Steve has some part of his brain whispering to him at night, calculating all the hardware sales lost to Hackintosh netbooks out there. But in any event, that's another shot missed. Ted 12; Wrongness 7.
THEN: Alpha-geek developers will start creating their own languages (even if they're obscure or bizarre ones like Shakespeare or Ook#) just to have that listed on their resume as the DSL/custom language buzz continues to build. NOW: I give you Ioke. If I'd extended this to include outdated CPU interpreters, I'd have made that three-pointer from half-court instead of just the top of the key. Ted 14; Wrongness 7.
THEN: Roy Fielding will officially disown most of the "REST"ful authors and software packages available. Nobody will care--or worse, somebody looking to make a name for themselves will proclaim that Roy "doesn't really understand REST". And they'll be right--Roy doesn't understand what they consider to be REST, and the fact that he created the term will be of no importance anymore. Being "REST"ful will equate to "I did it myself!", complete with expectations of a gold star and a lollipop. NOW: Does anybody in the REST community care what Roy Fielding wrote way back when? I keep seeing "REST"ful systems that seem to have designers who've never heard of Roy, or his thesis. Roy hasn't officially disowned them, but damn if he doesn't seem close to it. Still.... No points. Ted 14; Wrongness 9.
THEN: The Parrot guys will make at least one more minor point release. Nobody will notice or care, except for a few doggedly stubborn Perl hackers. They will find themselves having nightmares of previous lives carrying around OS/2 books and Amiga paraphernalia. Perl 6 will celebrate it's seventh... or is it eighth?... anniversary of being announced, and nobody will notice. NOW: Does anybody still follow Perl 6 development? Has the spec even been written yet? Google on "Perl 6 release", and you get varying reports: "It'll ship 'when it's ready'", "There are no such dates because this isn't a commericially-backed effort", and "Spring 2010". Swish—nothin' but net. Ted 16; Wrongness 9.
THEN: The debate around "Scrum Certification" will rise to a fever pitch as short-sighted money-tight companies start looking for reasons to cut costs and either buy into agile at a superficial level and watch it fail, or start looking to cut the agilists from their company in order to replace them with cheaper labor. NOW: Agile has become another adjective meaning "best practices", and as such, has essentially lost its meaning. Just ask Scott Bellware. Ted 18; Wrongness 9.
THEN: Adobe will continue to make Flex and AIR look more like C# and the CLR even as Microsoft tries to make Silverlight look more like Flash and AIR. Web designers will now get to experience the same fun that back-end web developers have enjoyed for near-on a decade, as shops begin to artificially partition themselves up as either "Flash" shops or "Silverlight" shops. NOW: Not sure how to score this one—I haven't seen the explicit partitioning happen yet, but the two environments definitely still seem to be looking to start tromping on each others' turf, particularly when we look at the rapid releases coming from the Silverlight team. Ted 16; Wrongness 11.
THEN: Gartner will still come knocking, looking to hire me for outrageous sums of money to do nothing but blog and wax prophetic. NOW: Still no job offers. Damn. Ah, well. Ted 16; Wrongness 13.

A close game. Could've gone either way. *shrug* Ah, well. It was silly to try and score it in basketball metaphor, anyway—that's the last time I watch ESPN before writing this.

For 2010, I predict....

... I will offer 3- and 4-day training classes on F# and Scala, among other things. OK, that's not fair—yes, I have the materials, I just need to work out locations and times. Contact me if you're interested in a private class, by the way.
... I will publish two books, one on F# and one on Scala. OK, OK, another plug. Or, rather, more of a resolution. One will be the "Professional F#" I'm doing for Wiley/Wrox, the other isn't yet finalized. But it'll either be published through a publisher, or self-published, by JavaOne 2010.
... DSLs will either "succeed" this year, or begin the short slide into the dustbin of obscure programming ideas. Domain-specific language advocates have to put up some kind of strawman for developers to learn from and poke at, or the whole concept will just fade away. Martin's book will help, if it ships this year, but even that might not be enough to generate interest if it doesn't have some kind of large-scale applicability in it. Patterns and refactoring and enterprise containers all had a huge advantage in that developers could see pretty easily what the problem was they solved; DSLs haven't made that clear yet.
... functional languages will start to see a backlash. I hate to say it, but "getting" the functional mindset is hard, and there's precious few resources that are making it easy for mainstream (read: O-O) developers make that adjustment, far fewer than there was during the procedural-to-object shift. If the functional community doesn't want to become mainstream, then mainstream developers will find ways to take functional's most compelling gateway use-case (parallel/concurrent programming) and find a way to "git 'er done" in the traditional O-O approach, probably through software transactional memory, and functional languages like Haskell and Erlang will be relegated to the "What Might Have Been" of computer science history. Not sure what I mean? Try this: walk into a functional language forum, and ask what a monad is. Nobody yet has been able to produce an answer that doesn't involve math theory, or that does involve a practical domain-object-based example. In fact, nobody has really said why (or if) monads are even still useful. Or catamorphisms. Or any of the other dime-store words that the functional community likes to toss around.
... Visual Studio 2010 will ship on time, and be one of the buggiest and/or slowest releases in its history. I hate to make this prediction, because I really don't want to be right, but there's just so much happening in the Visual Studio refactoring effort that it makes me incredibly nervous. Widespread adoption of VS2010 will wait until SP1 at the earliest. In fact....
... Visual Studio 2010 SP 1 will ship within three months of the final product. Microsoft knows that people wait until SP 1 to think about upgrading, so they'll just plan for an eager SP 1 release, and hope that managers will be too hung over from the New Year (still) to notice that the necessary shakeout time hasn't happened.
... Apple will ship a tablet with multi-touch on it, and it will flop horribly. Not sure why I think this, but I just don't think the multi-touch paradigm that Apple has cooked up for the iPhone will carry over to a tablet/laptop device. That won't stop them from shipping it, and it won't stop Apple fan-boiz from buying it, but that's about where the interest will end.
... JDK 7 closures will be debated for a few weeks, then become a fait accompli as the Java community shrugs its collective shoulders. Frankly, I think the Java community has exhausted its interest in debating new language features for Java. Recent college grads and open-source groups with an axe to grind will continue to try and make an issue out of this, but I think the overall Java community just... doesn't... care. They just want to see JDK 7 ship someday.
... Scala either "pops" in 2010, or begins to fall apart. By "pops", I mean reaches a critical mass of developers interested in using it, enough to convince somebody to create a company around it, a la G2One.
... Oracle is going to make a serious "cloud" play, probably by offering an Oracle-hosted version of Azure or AppEngine. Oracle loves the enterprise space too much, and derives too much money from it, to not at least appear to have some kind of offering here. Now that they own Java, they'll marry it up against OpenSolaris, the Oracle database, and throw the whole thing into a series of server centers all over the continent, and call it "Oracle 12c" (c for Cloud, of course) or something.
... Spring development will slow to a crawl and start to take a left turn toward cloud ideas. VMWare bought SpringSource for a reason, and I believe it's entirely centered around VMWare's movement into the cloud space—they want to be more than "just" a virtualization tool. Spring + Groovy makes a compelling development stack, particularly if VMWare does some interesting hooks-n-hacks to make Spring a virtualization environment in its own right somehow. But from a practical perspective, any community-driven development against Spring is all but basically dead. The source may be downloadable later, like the VMWare Player code is, but making contributions back? Fuhgeddabowdit.
... the explosion of e-book readers brings the Kindle 2009 edition way down to size. The era of the e-book reader is here, and honestly, while I'm glad I have a Kindle, I'm expecting that I'll be dusting it off a shelf in a few years. Kinda like I do with my iPods from a few years ago.
... "social networking" becomes the "Web 2.0" of 2010. In other words, using the term will basically identify you as a tech wannabe and clearly out of touch with the bleeding edge.
... Facebook becomes a developer platform requirement. I don't pretend to know anything about Facebook—I'm not even on it, which amazes my family to no end—but clearly Facebook is one of those mechanisms by which people reach each other, and before long, it'll start showing up as a developer requirement for companies looking to hire. If you're looking to build out your resume to make yourself attractive to companies in 2010, mad Facebook skillz might not be a bad investment.
... Nintendo releases an open SDK for building games for its next-gen DS-based device. With the spectacular success of games on the iPhone, Nintendo clearly must see that they're missing a huge opportunity every day developers can't write games for the Nintendo DS that are easily downloadable to the device for playing. Nintendo is not stupid—if they don't open up the SDK and promote "casual" games like those on the iPhone and those that can now be downloaded to the Zune or the XBox, they risk being marginalized out of existence.

And for the next decade, I predict....

... colleges and unversities will begin issuing e-book reader devices to students. It's a helluvalot cheaper than issuing laptops or netbooks, and besides....
... netbooks and e-book readers will merge before the decade is out. Let's be honest—if the e-book reader could do email and browse the web, you have almost the perfect paperback-sized mobile device. As for the credit-card sized mobile device....
... mobile phones will all but disappear as they turn into what PDAs tried to be. "The iPhone makes calls? Really? You mean Voice-over-IP, right? No, wait, over cell signal? It can do that? Wow, there's really an app for everything, isn't there?"
... wireless formats will skyrocket in importance all around the office and home. Combine the iPhone's Bluetooth (or something similar yet lower-power-consuming) with an equally-capable (Bluetooth or otherwise) projector, and suddenly many executives can leave their netbook or laptop at home for a business presentation. Throw in the Whispersync-aware e-book reader/netbook-thing, and now most executives have absolutely zero reason to carry anything but their e-book/netbook and their phone/PDA. The day somebody figures out an easy way to combine Bluetooth with PayPal on the iPhone or Android phone, we will have more or less made pocket change irrelevant. And believe me, that day will happen before the end of the decade.
... either Android or Windows Mobile will gain some serious market share against the iPhone the day they figure out how to support an open and unrestricted AppStore-like app acquisition model. Let's be honest, the attraction of iTunes and AppStore is that I can see an "Oh, cool!" app on a buddy's iPhone, and have it on mine less than 30 seconds later. If Android or WinMo can figure out how to offer that same kind of experience without the draconian AppStore policies to go with it, they'll start making up lost ground on iPhone in a hurry.
... Apple becomes the DOJ target of the decade. Microsoft was it in the 2000's, and Apple's stunning rising success is going to put it squarely in the sights of monopolist accusations before long. Coupled with the unfortunate health distractions that Steve Jobs has to deal with, Apple's going to get hammered pretty hard by the end of the decade, but it will have mastered enough market share and mindshare to weather it as Microsoft has.
... Google becomes the next Microsoft. It won't be anything the founders do, but Google will do "something evil", and it will be loudly and screechingly pointed out by all of Google's corporate opponents, and the star will have fallen.
... Microsoft finds its way again. Microsoft, as a company, has lost its way. This is a company that's not used to losing, and like Bill Belichick's Patriots, they will find ways to adapt and adjust to the changed circumstances of their position to find a way to win again. What that'll be, I have no idea, but historically, the last decade notwithstanding, betting against Microsoft has historically been a bad idea. My gut tells me they'll figure something new to get that mojo back.
... a politician will make himself or herself famous by standing up to the TSA. The scene will play out like this: during a Congressional hearing on airline security, after some nut/terrorist tries to blow up another plane through nitroglycerine-soaked underwear, the TSA director will suggest all passengers should fly naked in order to preserve safety, the congressman/woman will stare open-mouthed at this suggestion, proclaim, "Have you no sense of decency, sir?" and immediately get a standing ovation and never have to worry about re-election again. Folks, if we want to prevent any chance of loss of life from a terrorist act on an airplane, we have to prevent passengers from getting on them. Otherwise, just accept that it might happen, do a reasonable job of preventing it from happening, and let private insurance start offering flight insurance against the possibility to reassure the paranoid.

See you all next year.

Book Review: Debug It!

Sun, 22 Nov 2009 00:00:00 +0000

Debug It! (Paul Butcher, Pragmatic Bookshelf)

Paul asked me to review this, his first book, and my comment to him was that he had a pretty high bar to match; being of the same "series" as Release It!, Mike Nygard's take on building software ready for production (and, in my repeatedly stated opinion, the most important-to-read book of the decade), Debug It! had some pretty impressive shoes to fill. Paul's comment was pretty predictable: "Thanks for keeping the pressure to a minimum."

My copy arrived in the mail while I was at the NFJS show in Denver this past weekend, and with a certain amount of dread and excitement, I opened the envelope and sat down to read for a few minutes. I managed to get halfway through it before deciding I had to post a review before I get too caught up in my next trip and forget.

Short version

Debug It! is a great resource for anyone looking to learn the science of good debugging. It is entirely language- and platform-agnostic, preferring to focus entirely on the process and mindset of debugging, rather than on edge cases or command-line switches in a tool or language. Overall, the writing is clear and straightforward without being preachy or judgmental, and is liberally annotated with real-life case stories from both the authors' and the Pragmatic Programmers' own history, which keeps the tone lighter and yet still proving the point of the text. Highly recommended for the junior developers on the team; senior developers will likely find some good tidbits in here as well.

Long version

Debug It! is an excellently-written and to-the-point description of the process of not only identifying and fixing defects in software, but also of the attitudes required to keep software from failing. Rather than simply tossing off old maxims or warming them over with new terminology ("You should always verify the parameters to your procedure calls" replaced with "You should always verify the parameters entering a method and ensure the fields follow the invariants established in the specification"), Paul ensures that when making a point, his prose is clear, the rationale carefully explained, and the consequences of not following this advice are clearly spelled out. His advice is pragmatic, and takes into account that developers can't always follow the absolute rules we'd like to—he talks about some of his experiences with "bug priorities" and how users pretty quickly figured out to always set the bug's priority at the highest level in order to get developer attention, for example, and some ways to try and address that all-too-human failing of bug-tracking systems.

It needs to be said, right from the beginning, that Debug It! will not teach you how to use the debugging features of your favorite IDE, however. This is because Paul (deliberately, it seems) takes a platform- and language-agnostic approach to the book—there are no examples of how to set breakpoints in gdb, or how to attach the Visual Studio IDE to a running Windows service, for example. This will likely weed out those readers who are looking for "Google-able" answers to their common debugging problems, and that's a shame, because those are probably the very readers that need to read this book. Having said that, however, I like this agnostic approach, because these ideas and thought processes, the ones that are entirely independent of the language or platform, are exactly the kinds of things that senior developers carry over with them from one platform to the next. Still, the junior developer who picks this book up is going to still need a reference manual or the user manual for their IDE or toolchain, and will need to practice some with both books in hand if they want to maximize the effectiveness of what's in here.

One of the things I like most about this book is that it is liberally adorned with real-life discussions of various scenarios the author team has experienced; the reason I say "author team" here is because although the stories (for the most part) remain unattributed, there are obvious references to "Dave" and "Andy", which I assume pretty obviously refer to Dave Thomas and Andy Hunt, the Pragmatic Programmers and the owners of Pragmatic Bookshelf. Some of the stories are humorous, and some of them probably would be humorous if they didn't strike so close to my own bitterly-remembered experiences. All of them do a good job of reinforcing the point, however, thus rendering the prose more effective in communicating the idea without getting to be too preachy or bombastic.

The book obviously intends to target a junior developer audience, because most senior developers have already intuitively (or experientially) figured out many of the processes described in here. But, quite frankly, I think it would be a shame for senior developers to pass on this one; though the temptation will be to simply toss it aside and say, "I already do all this stuff", senior developers should resist that urge and read it through cover to cover. If nothing else, it'll help reinforce certain ideas, bring some of the intuitive process more to light and allow us to analyze what we do right and what we do wrong, and perhaps most importantly, give us a common backdrop against which we can mentor junior developers in the science of debugging.

One of the chapters I like in particular, "Chapter 7: Pragmatic Zero Tolerance", is particularly good reading for those shops that currently suffer from a deficit of management support for writing good software. In it, Paul talks specifically about some of the triage process about bugs ("When to fix bugs"), the mental approach developers should have to fixing bugs ("The debugging mind-set") and how to get started on creating good software out of bad ("How to dig yourself out of a quality hole"). These are techniques that a senior developer can bring to the team and implement at a grass-roots level, in many cases without management even being aware of what's going on. (It's a sad state of affairs that we sometimes have to work behind management's back to write good-quality code, but I know that some developers out there are in exactly that situation, and simply saying, "Quit and find a new job", although pithy and good for a laugh on a panel, doesn't really offer much in the way of help. Paul doesn't take that route here, and that alone makes this book worth reading.)

Another of the chapters that resonates well with me is the first one in Part III ("Debug Fu"), Chapter 8, entitled "Special Cases", in which he tackles a number of "advanced" debugging topics, such as "Patching Existing Releases" and "Hesenbugs" (Concurrency-related bugs). I won't spoil the punchline for you, but suffice it to say that I wish I'd had that chapter on hand to give out to teammates on a few projects I've worked on in the past.

Overall, this book is going to be a huge win, and I think it's a worthy successor to the Release It! reputation. Development managers and team leads should get a copy for the junior developers on their team as a Christmas gift, but only after the senior developers have read through it as well. (Senior devs, don't despair—at 190 pages, you can rip through this in a single night, and I can almost guarantee that you'll learn a few ideas you can put into practice the next morning to boot.)

Agile is treating the symptoms, not the disease

Mon, 12 Oct 2009 00:00:00 +0000

The above quote was tossed off by Billy Hollis at the patterns&practices Summit this week in Redmond. I passed the quote out to the Twitter masses, along with my +1, and predictably, the comments started coming in shortly thereafter. Rather than limit the thoughts to the 120 or so characters that Twitter limits us to, I thought this subject deserved some greater expansion.

But before I do, let me try (badly) to paraphrase the lightning talk that Billy gave here, which sets context for the discussion:

Keeping track of all the stuff Microsoft is releasing is hard work: LINQ, EF, Silverlight, ASP.NET MVC, Enterprise Library, Azure, Prism, Sparkle, MEF, WCF, WF, WPF, InfoCard, CardSpace, the list goes on and on, and frankly, nobody (and I mean nobody) can track it all.
Microsoft released all this stuff because they were chasing the "enterprise" part of the developer/business curve, as opposed to the "long tail" part of the curve that they used to chase down. They did this because they believed that this was good business practice—like banks, "enterprises are where the money is". (If you're not familiar with this curve, imagine a graph with a single curve asymptotically reaching for both axes, where Y is the number of developers on the project, and X is the number of projects. What you get is a curve of a few high-developer-population projects on the left, to a large number of projects with just 1 or 2 developers. This right-hand portion of the curve is known as "the long tail" of the software industry.)
A lot of software written back in the 90's was written by 1 or 2 guys working for just a few months to slam something out and see if it was useful. What chances do those kinds of projects have today? What tools would you use to build them?
The problem is the complexity of the tools we have available to us today preclude that kind of software development.
Agile doesn't solve this problem—the agile movement suggests that we have to create story cards, we have to build unit tests, we have to have a continuous integration server, we have to have standup meetings every day, .... In short, particularly among the agile evangelists (by which we really mean zealots), if you aren't doing a full agile process, you are simply failing. (If this is true, how on earth did all those thousands of applications written in FoxPro or Access ever manage to succeed? –-Me) At one point, an agilist said point-blank, "If you don't do agile, what happens when your project reaches a thousand users?" As Billy put it, "Think about that for a second: This agile guy is threatening us with success."
Agile is for managing complexity. What we need is to recognize that there is a place for outright simplicity instead.

By the way, let me say this out loud: if you have not heard Billy Hollis speak, you should. Even if you're a Java or Ruby developer, you should listen to what he has to say. He's been developing software for a long time, has seen a lot of these technology-industry trends come and go, and even if you disagree with him, you need to listen to him.

Let me rephrase Billy's talk this way:

Where is this decade's Access?

It may seem like a snarky and trolling question, but think about it for a moment: for a decade or so, I was brought into project after project that was designed to essentially rebuild/rearchitect the Access database created by one of the department's more tech-savvy employees into something that could scale beyond just the department.

(Actually, in about half of them, the goal wasn't even to scale it up, it was just to put it on the web. It was only in the subsequent meetings and discussions that the issues of scale came up, and if my memory is accurate, I was the one who raised those issues, not the customer. I wonder now, looking back at it, if that was pure gold-plating on my part.)

Others, including many people I care about (Rod Paddock, Markus Eggers, Ken Levy, Cathi Gero, for starters) made a healthy living off of building "line of business" applications in FoxPro, which Microsoft has now officially shut down. For those who did Office applications, Visual Basic for Applications has now been officially deprecated in favor of VSTO (Visual Studio Tools for Office), a set of libraries that are available for use by any .NET application language, and of course classic Visual Basic itself has been "brought into the fold" by making it a fully-fledged object-oriented language complete with XML literals and LINQ query capabilities.

Which means, if somebody working for a small school district in western Pennsylvania wants to build a simple application for tracking students' attendance (rather than tracking it on paper anymore), what do they do?

Bruce Tate alluded to this in his Beyond Java, based on the realization that the Java space was no better—to bring a college/university student up to speed on all the necessary technologies required of a "productive" Java developer, he calculated at least five or six weeks of training was required. And that's not a bad estimate, and might even be a bit on the shortened side. You can maybe get away with less if they're joining a team which collectively has these skills distributed across the entire team, but if we're talking about a standalone developer who's going to be building software by himself, it's a pretty impressive list. Here's my back-of-the-envelope calculations:

Week one: Java language. (Nobody ever comes out of college knowing all the Java language they need.)
Week two: Java virtual machine: threading/concurrency, ClassLoaders, Serialization, RMI, XML parsing, reference types (weak, soft, phantom).
Week three: Infrastructure: Ant, JUnit, continuous integration, Spring.
Week four: Data access: JDBC, Hibernate. (Yes, I think you need a full week on Hibernate to be able to use it effectively.)
Week five: Web: HTTP, HTML, servlets, filters, servlet context and listeners, JSP, model-view-controller, and probably some Ajax to boot.

I could go on (seriously! no JMS? no REST? no Web services?), but you get the point. And lest the .NET community start feeling complacent, put together a similar list for the standalone .NET developer, and you'll come out to something pretty equivalent. (Just look at the Pluralsight list of courses—name the one course you would give that college kid to bring him up to speed. Stumped? Don't feel bad—I can't, either. And it's not them—pick on any of the training companies.)

Now throw agile into that mix: how does an agile process reduce the complexity load? And the answer, of course, is that it doesn't—it simply tries to muddle through as best it can, by doing all of the things that developers need to be doing: gathering as much feedback from every corner of their world as they can, through tests, customer interaction, and frequent releases. All of which is good. I'm not here to suggest that we should all give up agile and immediately go back to waterfall and Big Design Up Front. Anybody who uses Billy's quote as a sound bite to suggest that is a subversive and a terrorist and should have their arguments refuted with extreme prejudice.

But agile is not going to reduce the technology complexity load, which is the root cause of the problem.

Or, perhaps, let me ask it this way: your 16-year-old wants to build a system to track the cards in his Magic deck. What language do you teach him?

We are in desperate need of simplicity in this industry. Whoever gets that, and gets it right, defines the "Next Big Thing".