As programmers, we tend to do what we do for two main reasons: to build something cool, or to draw a paycheck. Those paychecks tend to be pretty decently sized, because we create a lot of value for the companies we work for. But how is that value quantified?
To an employer, value is probably quantified financially: are we generating more income than the cost to keep us on? This is why (I’ve read – I don’t have personal experience here) developers at a company that makes and sells software get treated better than developers making software for internal use only at a non-software company; the first group are revenue producers, while the second are operating expenses.Creative Commons by CheapFullCoverageAutoInsurance.com
The trouble is, it can be difficult to find a direct link between much of what we do and how much money is coming in. If I spend a week writing unit tests and refactoring code to reduce technical debt, I haven’t created anything new that we can sell, nor have I done anything that will let us increase the cost of our product. From a strictly code->dollars perspective, I haven’t done anything productive this week.
So if we’re not directly creating value for our company, are we creating value for our customers? By refactoring the code, I’m making it easier to add new features in the future, features that will help to better meet customers’ needs. When we create software that meets customers’ needs and wants, we are providing value. But how do we quantify this? If we have a good way to estimate the size of a new feature, then one way would be to measure how long it takes the implement the feature and how many bugs are produced; if these numbers decline over time, then either we’re getting better at developing (and thus our personal value to the company has increased) or we’ve improved our processes (or, in this case, the codebase we’re working with). So one measure of value would be the amount of (relatively) bug-free software (in terms of small, medium, large, or very large features) we can provide in a given increment of time.Creative Commons BY-SA 3.0 Nick Youngson
Of course, not all features are created equally. In an ideal world, we would be able to have our customers put a dollar value on each thing we might spend time on (a new feature, a 10% reduction in bugs, etc), sort those items to find the ones with the highest value per increment of time, and do those first; in that way we could be sure of providing the maximum amount of value. In practice, customers naturally don’t care about things like that refactoring that don’t pay off immediately but make all the other stuff easier in the long run.
So how do you quantify value in software development? I don’t think you can put a value on any given hour’s activity; software development is too much of a creative process in which one thing affects too many others. In the long run, it comes down to a binary: are we able to give our customers what they want and need at a price they’re willing to pay? If so, we’re providing them the value they desire and earning those paychecks.ommons
I heard a phrase I liked recently: software testing is ensuring goodness. [Alexander Tarlinder on Software Engineering Radio]
How do we ensure that our software is good, and further, what does good even mean? For a first approximation, let’s say that good software does what it’s supposed to.
How do we define what the software is supposed to do? That could mean it does what the design says it does, that it does what the user wants, or that it does what the user actually needs. This gives us the first few things to test against: we check that the design accurately reflects the users’ wants and needs, and that the software accurately reflects the design.
Of course, there’s a lot involved in writing software that isn’t generally covered in the design. Ideally we’ve specified how the program will react to any given class of input, but in practice users tend to do things that don’t make any sense. I had a bug recently that only appeared if the user opened a section of the page, clicked on a table row, opened a second section, opened a third section, hit the button to revert all the changes, reopened the third section, and then clicked a row in that section. There was certainly nothing in the design stating that if this particular sequence of events was to happen, the activity would not crash!
Ok, so that’s reaching a bit – of course we assume that the software shouldn’t crash (provided we’re not making a kart racing game). The design covers the expected responses to user stimuli, but we assume that it will not crash, freeze up, turn the screen all black, etc. Unfortunately, for a non-trivial piece of software the number of possible things to try quickly becomes exponential. How do we ensure that we’ve tested completely enough to have a reasonable chance of catching any critical bugs?
Finding the Bugs
Photograph of a mantis shrimp. Public domain, National Science Foundation.At some point, we have to determine how much effort to put into finding (or better yet, avoiding) bugs in our software. The more mission-critical the program is, obviously, the more time and money it’s worth investing into finding bugs and the less disruptive a bug has to be to be worth finding and fixing.
I’m a strong believer in the value of having a separate, independent quality assurance team to test the software. Testing is a completely separate skill from coding – rather than trying to build software that works, you’re trying to find all the ways that it could possibly break. So I think it’s valuable to have people skilled in the art of creative destruction, who can approach the software from the perspective of the end user, and who have the authority to stop code changes from moving forward if they believe those changes to be damaging to the quality of the code.
At the same time, there’s no guarantee that a few QAers will be able to try all the weird things that thousands of users might do once your software is out in the wild, which is why we also need code review (or PQA, programmer quality assurance). In code review we have a better chance of catching the one-time-in-a-million bugs that will never show up in testing and yet, somehow, will always pop up in the released code. One of the senior developers on my team was really good at this; I hated having him review my code because I knew he would nitpick every little thing, but I would still choose him to do PQA on my development for the same reason – he was really good at finding anything that might end up being a problem.
How to not code bugs
Speaking for myself, I’m not a fan of doing PQA – it gets boring really fast. Ironically, the better the code is the more boring PQA can be: with the junior developers there tends to be a lot of things you can make suggestions on, but when the code you’re looking at is very well done, you can spend an awfully long time examining it without finding anything wrong, and it takes an effort to get more in depth and concentrate more on finding subtle logic errors and race conditions rather than correcting bad habits and obvious errors in less developed code. Not that you don’t look for the subtle errors in the less developed code too, of course, but you’re not spending an hour looking through the code without finding anything.
On the other side of that, of course, you want to be the person writing the code where your PQAers can’t find anything to complain about. I have not yet figured out how to do this – not even close – but there are a few things that help.
Testing. Ok, this one is obvious, but it can be surprising how often someone makes a very minor change and then doesn’t go back and test afterwards. Even a change that can’t possibly break anything, often does.
Following standards. My team has a set way of doing things, and yours probably does as well. Why? Because we’ve broken stuff in the past, and figured out how to not break them the same way in the future.
Refactoring. See below.
So we’ve been doing unit testing for about the last year and a half, now. Most of the unit tests so far are in code that hasn’t really been updated a lot since the tests were written, so they aren’t yet serving one of the primary purposes of unit tests: letting the next person to work on your code (who might be you) know if he breaks it. What they are doing is making the code better.
When I’m writing unit tests for a function, sometimes I run into problems:
It’s not obvious what the function is doing. This generally means that the function (and possibly some parameters) needs to be renamed so that it’s very clear what will happen when it’s called.
The function is doing too much. In this case, I split it up into multiple functions – even when the subfunctions will only be called from one place. This gives me several functions that only do one thing, which makes it easier to verify that the logic behind each piece of functionality is correct.
The function depends on something being set prior to being called (generally uncovered by a crash when the unit test first calls it with no prep work). This is a good time to verify that under no circumstances can the function ever be called without its prerequisites ever being set (and possibly document the preconditions).
The function contains logic which is actually wrong. In this case, I’ve just uncovered a bug, and can fix it, before my code goes to testing.
Most of the time when I’m refactoring a function there aren’t actually any bugs present, yet… but conditions exist that make it easier for bugs to be inserted. By cleaning up the code, I then improve my chances of being able to make changes without introducing new bugs in the code. Further, because the code is now easier to read, not only are future changes less likely to break it but they’ll be made faster and with less frustration.
So how do we ensure goodness? Very carefully, I suppose. In my experience, the best way to do this is just to make it easy to do the right thing. Make it clear from the names what functions do, keep the code as simple and straightforward as possible, and be sure you understand exactly what the code is doing before you make changes (or, if you’ve written it, before you commit it).
Of course, having a can of RAID on hand never hurts.
One of the disadvantages of doing web development (to me, anyway) is that I don’t often get to use my background in theoretical computer science, especially with the front end stuff. It’s much more likely that I’ll be fighting with javascript than calculating asymptotic runtimes!
A few weeks ago was an exception. We had a number of jobs what could be scheduled and needed the software to determine, when a user made a scheduling request, whether that request could cause either the job being scheduled or any job that depended on it to exceed the time allowed. This was a fun little problem that I solved by using a modified depth-first-search to find the longest dependency chain for each job, then using the depths obtained in this preprocessing step to efficiently [O(n+m)] determine which jobs, if any, would end up running past the allotted time so that we could warn the user before allowing the scheduling.
Tools. Image is in the public domain.
In this case, the problem instance is generally small enough that we could have just brute forced a solution and it would likely still have had a reasonable runtime, but going with an efficient solution straight off means we don’t have to worry about scaling this up in the future, and the code is just nicer to work with.
You often see knowing the standard algorithms being compared to having more tools in your toolbox, and I think that’s a good comparison. I’m not the super-handyman type, but I still keep a good set of power tools around, and when something around the house needs to be fixed I often have the tools I need to just go ahead and take care of it, and I don’t have to force a tool into a situation it’s not meant for because I don’t have the correct tool available. Similarly, some types of development may not require you to pull out the graph algorithms all that often, but when those situations arrive they’re helpful to have around.
Recently I read a post about productivity. The point of it was that you have a fixed amount of effort that you can put in every day, and you’ll get a lot further if you concentrate that effort on one task rather than dividing it between many tasks.
I’d actually like to think of this in terms of physics. Imagine that you have a number of heavy blocks you need to move; these represent all the work you have to get done. If you go up to each block and give it a little tap, you’re unlikely to overcome the block’s inertia; it will stay where it is, and you’ve gone all day without accomplishing anything. On the other hand, if you apply all of your effort to one block, you may be able to move it quite some distance. It may look like you’re doing less – you’ve only worked on one project, rather than many – but you’re actually getting more done.
Dwarf sculpture. Picture in public domain. Credit Max Pixel.
We already know that multitasking doesn’t work, but even when we avoid it, how often do we engage in serial single-tasking – that is, doing one thing at a time, but jumping back and forth between different types of tasks? If each task only takes a few minutes, but comes from a different domain than the previous task, we still have to deal with the context switches that we’re trying to avoid by not multi-tasking.
On Monday, I spent my day focused entirely on doing fixes to a particular section of code. While I was doing a lot of different things, they all had the same basic context, which meant less for my brain to keep track of. I ended up having one of my most productive days ever, designing, developing, and checking in five discrete fixes. Last week I spent more time working on some larger fixes, but I let other work pile up while I focused on getting those big fixes done, so I didn’t have to take attention away from what was most important.
This isn’t a new concept – productivity gurus will speak of the importance of batching, where you save up related tasks (say, responding to emails) and tackle them all at once, also to avoid the context switches. How can you be more productive by doing less?
One thing that I’ve always struggled with is a short attention span; I tend to get bored pretty easily. One thing I like about my current job is that I get to work on a lot of different stuff, which helps keep me interested, but it’s still difficult to maintain focus, and I’ve been looking into different systems to help with that. Earlier this month my son was born, which is awesome but means I have less time and energy, more expenses, and less income (due to my wife staying home to take care of the baby), which adds that much more pressure to try to be really good at my job.
One of the things I tried last year was KanbanFlow, which is a free (and paid, though I haven’t felt the need for the paid features) tool for tracking your work; I liked the ease of use but didn’t really feel that I was getting that much out of it and more or less stopped using it. Recently, however, I’ve been reading the book Kanban in Action, which clarified for me what I should be doing, and I’ve started using my Kanban board again. I’m actually feeling more productive, so I thought I’d document what I’ve been doing.
My Kanban board is set up like this:
My Kanban board as of this morning.
One of the big things in Kanban is the Work in Progress (WIP) limit; the idea being that if you limit how much work you (or your team) can take on at any one time, then each piece of work flows through the process more quickly. This leads to less work overall, because any issues get found more quickly, before you have a chance to forget the details of what you’ve been working on. By limiting how many balls you have in the air at any given time, you also reduce the number of context switches you need to make, which helps you to be more efficient. In this case, you can see that my “In progress” column – the things that I’m actively working on – has a limit of three items; once that’s full, I can’t start working on anything else until I’ve moved something out of that column. Or as the book says – stop starting, start finishing!
My process here is to start the week by making all the decisions about what to work on and put those things on the to-do list in rough order of priority. At the start of the day, I can move a day’s worth of work over to the “Do today” column (under anything that might be left over from the day before); as things move off my “In progress” list, new items get pulled in from the top of “Do today”. Once I’m done with development that someone else needs to look at, it moves to “Waiting on someone else”; when it comes back it goes back into my “Do today” column. Any new tasks that come in can go into any of the first three columns or simply thrown on the backlog, depending on urgency; a customer issue will be worked on immediately but anything else needs to wait until the current task is complete.
Pretty straightforward so far, right? Even if I’m bored with working on something, it still needs to get done so I can move it out of the “In progress” column, and I need to stay on task to be able to clear out the “Do today” column. This brings us to the next part, the pomodoro timer:
The pomodoro technique works with the understanding that it’s extremely difficult to stay focused for long periods of time, and that our brains need to rest every so often. Thus, you work for a timed 25 minutes and then get a 5 minute break to walk around, visit the restroom, check Facebook, whatever. The timer here lets you select the task you’re working on, so you can use it to keep track of how much time you’ve spent on each task. Meanwhile, it’s easier to stay focused because you know there’s a clock ticking down and you just have to stay on task for a little while longer before you can take a break. When the pomodoro ends, you’ll be asked if you stayed on task, if so, you get rewarded with FlowPoints. It’s a simple motivational system, but it seems to work.
Another technique is to break up those tasks that require the most decision making, so we’re not constantly making demands on the same part of the brain. Do the important development first, before spending time answering emails. Put in some mindless tasks, like filling out time cards and walking to meetings, in between heavy programming sessions.
Probably most software developers have at least heard of Donald Knuth’s The Art of Computer Programming. Supposedly a 12-chapter book in seven volumes, the first volume was published in 1968 and we’re now partway through volume four (the fifth portion thereof expected to release in June). I’ve never read a significant amount of the work, but over the years I’ve occasionally dipped into it when researching a particular topic; it’s written as a reference book, and it works well for that.
I bring up these books not because they are particularly readable (they’re universally acknowledged as being quite dense, and very few people even claim to have read through them) or because of the speed with which they were written (although spending half a century on one project is impressive!) but because of Knuth’s dedication to getting the work exactly right. Aside from spending an astonishing amount of time on each book, he has long offered a reward of one hexadecimal dollar ($2.56) to anyone finding an error (including typos).
Most of us, of course, can’t put in that kind of effort (or time) to our writing. We (hopefully) have something to say, but there’s always the pressure to get it out there quickly and move on to the next project. I’ve seen more than a few programming blog posts that looked like they covered interesting material, but the writing was so bad that it wasn’t worth the effort to puzzle out what they were trying to say. (To be fair, many programmers have a first language other than English and deserve some leeway there). Books are even worse, due to the greater investment; I’ve given up on more than a few books due to the number of typos in the first chapter. Technical books, fortunately, seem to be better edited than most, but are by no means exempt from this problem.
Writing is meant to be read; make sure yours can be. If blog posts must be written quickly, consider cutting down on the number posted so as to take more care with each. When writing a more significant work, hire a good editor. My first book took only three weekends to write, but it was only 26 pages (of non-technical material) and still went through a round of editing before being published. I expect the book I’m working on now (which will be more technical) to take over a year to write and require several editors to make sure both the content and the language are correct. It still won’t be perfect; errors inevitably slip through no matter how many times you proofread (I’m still irritated over the one error I’m aware of in a graphic in my dissertation). But a focus on quality reflects well on you, and makes things easier for your readers.
I’m interested in what you have to say. Make it easy for me to hear it.
Last week, I found out that I’ve been promoted to senior developer. From a practical standpoint, this doesn’t actually affect me at all; the titles system was put into place to deal with immigration requirements, and the ‘promotion’ doesn’t signify a change in my responsibilities or compensation. In fact, this actually happened over a month ago; it’s just of so little import that nobody bothered to tell me about it!
Still, this seems as good a time as any to reflect on where my career has been going, where I’d like it to go in the future, and what I need to do in order to get there.
Over the last year I’ve spent less time working on code (although that’s still what I’m usually doing) and more time doing project management-type activities: setting priorities, writing designs, explaining things and providing feedback to other developers. I’ve said in the past that I’m not interested in become a team lead (partially because I want to focus on becoming a better developer and partially because I don’t want to spend all day in meetings) but I find that I actually do enjoy overseeing a project.
I’m now feeling that I’d like to move into more of a software architect role. I’ve seen this defined as the person who takes the blame for the quality of the software; he sets the overall direction for the project. For the project that I’ve been overseeing, I’ve worked to set standards that everyone working on the project follows, and the result is code that’s much easier to work with; I actually feel that my biggest contribution to the company will be how much easier it will be to make updates to my team’s part of the software in the future.
So what do I need to do for this?
On the interpersonal side, pretty much every job I’ve ever had has involved me being in a leadership role, one way or another, so I have no issues with that and I enjoy helping the other developers; my only real challenge in this area is that I’m hard of hearing and many of the other developers have accents, so face to face meetings are challenging for me. Unfortunately that’s not really anything I can change; all I can do is work around it.
On the technical side, I feel like my largest challenge is in testing; when I’m doing PQA (programmer quality assurance) on other people’s code (or even my own) I don’t feel that I’m catching as many issues as I would like. So that’s something for me to focus on this year: improving my testing skills. Interestingly enough, this area is also one of my strengths; since I have a bit of an unusual background (teaching and editing) I do tend to catch different problems than the other developers. So my technical goal for this year is to improve my testing process. What I’m going to try to make time for is to go back and look at development I’ve touched where a bug got through, and see if that’s something I could have caught. If it is, maybe I can add something to my testing checklist so that kind of bug doesn’t get by me the next time.
In other words – I need to discover new steps I can take to improve my work and refine my processes to include those steps, playing architect on a small scale. The better the process – assuming it’s followed – the better the end result. What processes can you improve this year?
Not a big deal – I’ve done it before and it’s not that difficult, just a bit of a pain (and hard on my back). But I’m still counting it as my accomplishment for the day.
When I was a kid, I always planned to get stuff done over the summer – learn Java or whatever – but more often than not it didn’t happen. When your primary focus is on big projects, it’s easy to have days when you’re not seeing any progress, or worse, actually not making any progress. I had that problem with my dissertation – it was so involved that it was difficult to sit down and write out the next lemma. I have that problem still; I’m working on a new book, but it’s such a large project that it’s difficult to just work on a small part of it.
It’s no secret that the way to get a big thing done is to break it into a lot of little things; sometimes it’s just hard to work up the motivation to do the little things. What I’m trying this year is simple: at the end of every day, I want to be able to look back and say that I accomplished at least one thing. If it’s getting towards the end of the day and there’s no sense of accomplishment yet, well, that should be my kick to keep going until I have something.
Thursday it was finishing off a decent-sized piece of functionality I’ve been working on. Today it was replacing a faucet and writing a blog post. I don’t even remember what it was the other days, but I do remember that there was something – even if small – each day of the year thus far that made me feel that, that day, I had accomplished SOMETHING.
And when you keep accomplishing little things? Maybe it helps you find the motivation to get the big things done.
Recently my friend Janie has done a series of blog posts on her disagreement with using computer science questions as a major part of the hiring process. I can agree with her, to a point.
If you’re hiring for someone to do a specific job using a specific language or framework, and the person being interviewed can demonstrate experience with and competence in that language or framework, then that’s probably as far as you need to go. You have a job that needs to be done and you’ve found someone who can do that job. Win!
I think that when the job is less well-defined is when algorithms questions can more usefully come into play. But let me back up and relate a few stories from my personal experience.
As a GTA (graduate teaching assistant) part of my job was to monitor the computer lab and help people as needed. The undergraduate classes at my school used Java; I don’t, but I grabbed a book and picked up enough of it to be able to get people unstuck. I guarantee that pretty much everyone I helped had much more experience with and knowledge of Java than I did – they were using it every week for their classes, while I’d picked up the bare minimum to be able to read it. But I was getting people unstuck because they didn’t understand how to take a problem and write a program that would solve it; they knew how to program, but they didn’t know how to think.
The class that I taught, Foundations of Computer Science, involved no programming whatsoever (and many people put off taking it, for that reason). We covered finite state machines, regular expressions, Turing completeness, that sort of thing. I can recall thinking that if I was hiring someone for just about any job, I would consider an A in that class as a strong recommendation, because it meant that you could think logically and solve problems.
When I was working on my PhD I did almost no programming; I was focused on theory. I think I wrote three programs in four years. Towards the end of that, Microsoft was doing interviews on campus, and I decided to give it a try; the interview turned out to be two questions along the lines of “Here’s a problem. Solve it, then find the fastest possible solution.” I found the problems to be reasonably straightforward (though you had to be a little creative for one of them) but the interviewer told me that most of the people he’d interviewed couldn’t even answer the first problem.
I don’t mean to suggest here that programming skills or experience are not important, only that unless the problem is very well defined, they may be less important than knowing how to figure things out. The advantage that knowing algorithms (and everything that goes with it) gives you is that you have more tools available to you; in the case of the Microsoft questions, one of the problems was very obviously susceptible to dynamic programming, and knowing big-O analysis made it easy to say “this is the best asymptotic runtime of any solution to this problem.”
My current employer has a similar philosophy. The initial test I took allowed you to use any programming language you liked; another one taught you an obscure language and then asked you to use it. Combined with a preference for people with strong academic backgrounds, they’re selecting for employees with a demonstrated ability to both solve problems efficiently and pick up new things quickly. Now I regularly use half a dozen languages at work, none of which I knew when I started, and the turnover at the company is low compared to the larger tech industry. Microsoft and Google have actually pouched half of my team over the last few years, but that’s another story..
I’ve been rewriting a utility that moves certain tables and their indexes from their old tablespaces (many, many old tablespaces) to a new one. This week I ran into a new (to me) error message:
SQL Error: ORA-25191: cannot reference overflow table of an index-organized table
My utility wasn’t providing enough special handling for index-organized tables (IOTs), which were throwing up this error (and a few others). So what’s an index-organized table, and why are they so difficult to work with?
A refresher: a SQL database is relational, and rows are stored unsorted; Oracle calls standard tables heap-organized tables. For a given SQL query, unless an ORDER BY clause is used, results can be returned in any order. Oracle also has IOTs, which are tables that are instead stored in a B*Tree index structure. This structure stores all columns of the table, sorted by primary key; because this is both the table and the index, accessing rows is much faster and less storage space is needed than for a heap-organized table with a separate index.
There are, however, exceptions to the rule that everything is stored in one place. If some columns are rarely used, they can be moved into an overflow area (which could be in a different tablespace) to speed up access to the main segment. Additionally, any columns that cannot fit into a single block must go into the overflow area.
When an IOT is created using an overflow segment, Oracle will automatically create a second table with a name like SYS_IOT_OVER_XXXXX. Trying to use this table directly results in the above error.
If I query dba_tables, I see that SYS_IOT_OVER_XXXXX has an IOT_TYPE of IOT_OVERFLOW (confirming that it’s an overflow table, as expected) and the IOT_NAME column gives me the name of the original table that this is holding overflow from. I can then resolve the issue by moving the IOT, specifying both the regular and overflow tablespaces:
alter table SCHEMA.”TABLE_NAME” move tablespace TABLESPACE_NAME overflow tablespace OVERFLOW_TABLESPACE_NAME;
Now Oracle moves the overflow table along with the heap-organized table, and everyone is happy.
![On Writing [and Editing]](http://www.whatwilliamsaid.com/wp-content/uploads/2017/01/Spellchek.png)