Programming with Intention
From TheCommandLineWiki
Part of The Inner Chapters Unbook.
Originally part of episode 70.
Contents |
Notes
Transcripts
39:56
As promised, in this Inner Chapter I wanted to talk about intentional programming or programming with intention. I'll get to, in a minute, why I use those two terms and, maybe perhaps after doing a little research, favor more the descriptive "programming with intention". Anyway, what got me thinking about this is some of the discussion by chromatic and others on readable code on ONLamp. And I found that it was interesting in looking at those articles in a detailed fashion that nobody really set out any real quality definition of what readable or maintainable code is. It just seems to be an assumption that the reader knows what readable code is. Now perhaps readable code is like art: you know it when you see it (or pornography, for the more cynical bent). But I think it really actually boils back to a contention that I had hinted at, that if code were meant for machines alone, we'd still be writing machine code. We wouldn't have built increasing layers of abstraction that approach certain domains of natural human language.
Speaking of natural language, one of the things that I came across when doing a little bit of research for this piece was a mention on Wikipedia of a Microsoft research project, apparently based on some of Charles Simonyi's ideas. I believe Simonyi was also the developer of C# and a contributor to the .NET technologies. And in reading through the article on Wikipedia, which I've got linked in the show notes, the crux of Microsoft's intentional programming seems to be the idea that, rather than expressing an algorithm in the semantics of a programming language, no matter how high-level or abstract that language may be, to express it in more of a human-readable form and then use tools to parse that into the underlying semantics of some existing implementation language like C, C# or Java. Thankfully, I have to say (because I'm very critical of this idea), Microsoft's research efforts around intentional programming seem to have ended around the early 2000s.
I don't think you need any sort of special source code representation in order to maintain or communicate the programmer's intention when practicing programming. I think this is a typical Microsoft solution, instead of trying to find the simplest way of doing something, of bringing some more thoughtfulness to what we do in our practice, they try to develop a tool and wrap it in a GUI, or some other code generator or a lot of... When I read through this Wikipedia article, a lot of what they talked about seemed to talk about code generation, automatic translation between different domains, and I think that's really tricky. And it actually, if you think about it, if you have any familiarity with some of these so-called business languages that others developed in the past — COBOL actually is sometimes held up as an example of this — if you look at the case tools from I believe the 80s when those were kind of at their height before they faded away, there was a similar value proposition with those: that, rather than having to train very hard on a programming language that was like learning a new written or spoken language, these tools proposed to allowed the programmer to just use normal language in order to specify what a program needs to do.
Now, the problem I have with this, in a nutshell, is that natural human language is clumsy at best when it comes to describing algorithms precisely. And if you need any more compelling proof of this, read a legal document sometime and try to parse through the legalese that it has been built up with, and understand that legal documents are an attempt to specify some conditions of a contract, some conditions or clauses of a law — legislative legalese very much falls into this same example category, trying to pin things down without ambiguity, and it's just unreadable. So if you set off to express something very, very precise — something unambiguous, something that there can be no mistake, that there's only one way of interpreting it — and in a language rife with metaphor and other ambiguities, you're really kind of cutting against the grain.
I think it's somewhat naïve of the Microsoft researchers — as smart as Charles Simonyi is, and I have to give him a lot of credit, that he is a very, very smart computer scientist — I think that these two things are just at odds. And personally, when I talked about programming language polyglots last week or the week before, when we were talking about the problem with learning dynamic languages or simpler, better languages like Haskell or Python, versus languages that don't quite have the same underlying architecture and don't support the same idioms, and you know I stressed this a lot more when I talked about idiomatic programming last week. Part of what I was thinking and what informed my view of people being able to pick up multiple programming languages more easily is this idea that if you want to be able to express an algorithm precisely and correctly, you need a language that's appropriate for that and, I think more importantly for the practicing hacker, for the practicing programmer, you need the ability to learn and utilize that language. So it's not enough just to have the language in the first place. And I think that's okay. I don't think there's a problem with that, as many of these natural-language-programming bigots (if you will)... you know, they want to bring the implementation domain, the solution domain, and layer it directly on top of the problem domain and say, "They're the same thing. If I can express a problem directly with a tool..." And I think this is some of the problem inherent in object orientation and some of the causes of where that tends to break down.
46:07
But anyway, I've bogged down a little bit in talking about programming with intention. What I mean by that, to contrast that from intentional programming — this natural language hoo-ha that Microsoft looked into earlier this decade — is really that code that expresses clearly to the reader what is being done. Intention speaks to why. I think the code itself does a good enough job if you understand the programming language at hand, of what exactly it is doing and how it's doing it. I don't think you need to — one of my pet peeves, one of the things that drives me crazy, is seeing a source code comment that says, "This is a for loop that iterates over all of the elements of the collection". Okay, that's nice; I could tell that just from reading the source code myself. I could tell that from the particular for loop that you used, and I even know what collection... Why do you need a source code comment for that?! What does that tell me about why you're iterating it and what's being done inside of the body of the iteration, inside of the body of the loop?
I think that the naming of variables and functions can very much help along with proper code documentation. Again, speaking to why: so your index variable going back to that looping construct for a minute. If that indexed variable is, like, I don't know, a current element... Say you're doing a binary search, and it's the split point in your search. Why not call it SplitPoint or PlaceToSplit or something? I don't mind using lengthier variable names. In fact, I tend to favor them. Now, writing a whole sentence as a variable name gets a little ridiculous, but having something that's more expressive of what that particular variable is being used for, and more importantly, why that variable is being used in a particular operation versus something else, I think is what drives programming with intention. This is the point I am trying to make.
The best comments also clarify the implicit knowledge or the constraints that inform formation of the code itself. So we talked about implicit knowledge; we also talk about implicit knowledge in terms of domain knowledge: that you're working on a financial application, you're working on a security application, you're working on an e-commerce application. There's a lot of domain knowledge that's implicit in that, that's not maybe directly expressed in the code in some way: either through the domain objects that you've coded up as your persistence objects or your transfer objects or whatever, your components. So source code comments sometimes can do a good job of filling in that implicit knowledge that the code can't speak to saying, you know, this... Especially when you're thinking about component-level or type-level source code comments, documenting what a class or a component is for, what it represents in the problem domain when you have to make especially some strong variation from what's natural to talk about in meat space, in the real world when talking about a widget or, you know, a stock item or something like that. If you have some artificial attributes that you've had to add on there, kind of muddied the waters, kind of muddied the mapping. Or you have something that's maybe not directly represented physically in the problem space, something that you introduced as part of your design to make the implementation on a computer easier. Because let's face it, folks: even with Moore's Law, even with the sheer amount of raw computation power that we have, reality and computation — not quite the same thing. Or I should say, computation on our high-end servers, workstations, and PCs — not the same thing as reality.
So, like I said, there's going to be the code itself, which speaks to the algorithm, the function, the work flow, the intelligence of the system. The code comments need to supplement that with why a particular algorithmic choice is made, why a particular sign choice was made, and then also perhaps as I said, some implicit knowledge that a new reader, a fresh reader, maybe somebody returning to it after some span might need some reminders of some of the subtleties of the domain knowledge involved with the solution being built.
I think that, that being said, intention may operate very much in that design phase or in that domain-mapping phase. I think that intention can often rely on that implicit knowledge, that the whys and wherefores of what we are doing are driven by the particular problem space that we're dealing with. Capturing that intention may seem more redundant when there are a lot of assumptions around that shared knowledge, that people already have those assumptions, people already know... There are a lot of common domains that we work in. Networking: okay, most programmers have some exposure to networking stacks. You don't really have to, I think, expend as much effort explicitly capturing your intent when opening a socket and pushing bytes or reading bytes off of a socket. I think that's pretty clear. Some of that we can leave lie. And unfortunately there's no hard line here when it comes down to it. I realize I'm throwing a lot of fuzzy information out there and a lot of fuzzy guidance, but really it does boil down to discretion. Some of it is going to boil down to: who are you writing this documentation for? Are you writing for yourself in six months? So what can you rely on remembering in six months? I think enlightened self interest, I've said this before, is a very, very good thing, and clear, readable code does as much a service to yourself as maintainers or anyone else. Your team: you have a high degree of domain expertise across your team. Then maybe you don't have to spend as much time capturing that, and you can concentrate more on some of the real edge cases.
51:59
I think intention also ranges quite a bit across scale: that when you're dealing with a single function point — whether it's a method, a function, a procedure, or whatever — when you're dealing with a class, or an object, or a structure, when you're dealing with a component, when you're dealing with a subsystem, all the way up to a grand server system, each one of those is going to have intention operating at that scale appropriately. You know, how do you wire a couple of different sub-systems together into a cohesive total system? How do you wire a couple of components together? Why did you choose this component versus that one? Why are you using a component in an non-obvious, non-standard way? Those whys: address those. That's the intention. You have to speak to that so somebody reading it doesn't have to ask you that question.
This is another angle of enlightened self-interest is, if you're sharing knowledge with a team, if you're sharing work with a team, the more your code expresses the whys of you doing something and the implicit knowledge behind your implementation choices, behind even your design choices, the less of your time is wasted having face-to-face conversations on those kinds of questions that can be easily answered by the code and the source-code documentation itself, and the time that's left then can be used far more effectively on much more interesting things. Think about much more forward-looking: how are we going to evolve the system into the future? What are some novel and interesting problems we can solve? You know, how can we better compete? How can we make our open-source project more compelling, more usable? So you can kind of think about it that way if you're struggling for a grip on why programming with intention is a good thing.
I think that domain knowledge, conversely from intention, may not be as useful at all scales. When you get down especially to the smaller scale when you're dealing with single lines of code, when you're within a function point, domain knowledge may not inform what's going on and the design in the small choices that you are making at that point, the implementation pressures that are at play. I think even in the presence of this shared knowledge that I talked about earlier, having a team, having a group that you can rely on and having some a priori shared expertise, intention is still important to understand in code. Even then, you may not have to express as much about your intention because there's a lot more you can assume somebody that already understands, somebody already knows. But still, you want to be mindful, you want to be thoughtful of the whys and wherefores. Why are you making this decision? Again, why are you implementing this particular algorithm? Why are you implementing this particular algorithm in this particular way?
54:45
I think that the last thing I'm going to say about this is that intention may actually be a legitimate part of discovery that's part of our coding practice. I don't know if I've expressed this idea in the past Inner Chapters, so I'll just go over it very briefly: this idea that no design can encapsulate everything about a code base beforehand. If you think about it in terms of a map, if you had a scale one-to-one map, it would be identical to the terrain, and why would you need the map at that point? It would be useless. Design is about a certain amount of reduction. You're breaking things down in coarse granularity, you're dialing it in to finer and finer granularity, up to a certain point where the ambiguities, the constraints, some of the other properties and things that you have to deal with in the implementation are answered well enough that the remaining unknowns, the remaining questions to be answered, can be answered as part of a coding exercise, don't require a lot of whiteboard time and scheming and group discussion. A single developer, a single hacker can sit down and make reasonable choices along the way.
That being said, I think that intention can be part of that discovery. When there are open-ended questions about how to do the best thing when you kind of are whacking around with that and experimenting with that. As you're doing your experimentation and you're making careful observations about which particular approaches work, which ones work better, which ones work worse, that what should normally follow when you strike upon your final solution, when you have that open-ended question that design leaves open and you work through what you normally work through to come up with your solution with your solving your problem in the small, your "design in the small" as I like to call it, you have to necessarily sit back — and I know I've talked about this before — you want to try to reduce and refactor your solution so that it's streamlined, it's clear, it makes sense, it's readable, and — ah, here's the tie-in! — when we talked about readability, when we talked about maintainability, again, we're talking about, what was your intention? So you need to understand why that particular solution out of all the possible little solutions that you tried works best.
I'm going to share an anecdote with you very quickly to illustrate this point. We had — and I may have talked about this before, forgive me if I have. I'm just going to close with this anecdote about intention and about discovery and the rôle intention plays in discovery and then, hence, maintainability. And this was, oh, maybe within the first couple of months at my current job. We had a generation of hardware that was on an older network. It was actually the original network that the Blackberry PDAs utilized for their network communications, Mobitex — a very cost-effective, by the way, a very reliable, very novel architecture for messaging systems. So it seemed to make a lot of sense for this hardware. We still, on the networking side, it was one of the more reliable pieces of our overall offering. Anyway, the network provider was changing their networks over to operate from local time everywhere — so essentially, in any given network node, it just ran in whatever time it ran in; it didn't have any conception of time zone; it didn't need to because the devices were fixed, they weren't traveling around — to UTC for them all.
So the entire network would be synchronized off of Greenwich Mean Time, Universal Coordinated Time: same thing. And as a consequence, all of these devices now needed to have some awareness of their time zones so that they could offset correctly from UTC. Now, unfortunately, as good as the Mobitex network protocol is, as reliable and cost-effective as it is, as suitable a solution as it seemed to be at the time that it was decided on, the implementation on our side was overly complex. And there were legitimate reasons for this. At the outset they envisioned multiple different networks and different gateways, so having to have some intelligent routing from one application on our side out through multiple potential gateways into multiple different networks. So it made some sense in hindsight, but it definitely violates Extreme Programming's tenant of "Don't build today what you don't need today" — you know, don't build things that you think you're going to need tomorrow because that need may never materialize. And this was definitely a case, in retrospect: this need never materialized to have multiple networks.
Anyway, the engineer assigned to find the spot where time stamps were getting sent down for activating devices — opening and closing doors, things of that nature, all the scheduled operations on the devices in the field — he took many, many more weeks to try to come up with a solution. And when it was finally getting ready to go into QA, and we had a discussion around this to say, "Okay, do you think it's ready to go? It's going to go into QA this weekend. We're going to pull people in over the weekend to take a look at this..." And we were doing a code walk-through because Trithemius had some concern about the quality of the solution, and we found a magic number in here. And a magic number, if you've never heard of that term before, is just what seems like... it's just a bare, literal number in the source code with no explanation of what it is. So it's "magic". And in this case that definitely turned out to be true in the truest sense of the term. He had come across an offset, a plus-2 minus-2 offset that just seemed to work. So Trithemius pinned him against the wall — verbally speaking, metaphorically speaking — and said, "Well, why two?" And the engineer in question just look somewhat puzzled and didn't quite understand, "Well, what do you mean, 'Why 2'? Just, it's 2. That's what it is". And Trithemius backed up and tried from a different perspective and said, "Well, let me rephrase that. Does two have some significance? Does it have some meaning? Does it represent something, i.e. is there a semantic to the value 2? Or did you just try different values until you found something that worked?" And he said, "Oh, I just tried different values until I found something that worked". So he had no idea why two, so he couldn't explain the intention, the meaning of two, and couldn't clarify beyond that.
The upshot of this story is that plus-2 minus-2 wasn't the ultimate solution. It fixed an incidental problem along the way in some very, very flawed code. The weekend of QA testing went very, very poorly. There were clearly some sloppy state-handling issues where schedules were getting generated multiple times with increasing frequency every time they went through the system. So it had to be re-written from scratch the following week in a marathon coding session, Trithemius and I in a conference room, me barely knowing the domain I was programming for yet, so him lending a lot of great expertise on that. He missed a trade show in New York that he wanted to go to very much for this.
So the upshot was good, but it really highlights what I was talking about: that reduction, that refactoring. Once you've gone that experimentation, it really behooves you to try to understand, "Why that plus-2 minus-2? What does that plus-2 minus-2 mean? Where does it come from? What offset is it addressing? What is that?" And if you can't answer that, then you can't in any reasonable definition of the term have confidence in your solution, that it's right. How do you know? How do you know that plus-7 minus-7, or plus-13 minus-2, isn't a better solution if you don't know what that offset represents, if you don't know what that solution that you stumbled upon actually represents.
Now don't get me wrong. Experimentation is good. Trying out different things is good. But, you know, part of what I'm a fan of, the forensic or scientific method, is: put a hypothesis out there, formulate a theory, and then prove it or invalidate it. So if you don't have those surrounding notions of why you're changing variables, to what end, what is it that you're trying to stress and get at to improve your understanding, then it's pure "programming by coincidence" as I like to put it. It's programming without intention, if you want to put it that way, to bring things to a close and dovetail that with everything I've been talking about.
So there you go. I hope you enjoyed that. I hope that made some sense to you. If you have some questions or comments about programming with intention, some anecdotes that you'd like to share, both examples and counterexamples, I'd love to hear them. Send them along: feedback@thecommandline.net.
