There's a lot of frustration about the government's ability to build things in the US. Subways. Bridges. High-speed rail. Electricity transmission. But there's another crucial area where the public sector often struggles, and that is software. We saw it with the infamous rollout of Obamacare. We see it in the UX of the Treasury Direct website. And we saw it in the way state unemployment insurance systems broke during the pandemic. So why is it so hard for the public sector to build and maintain software? On this episode we speak with Jennifer Pahlka, the founder and former executive director of Code for America and author of the new book Recoding America: Why Government Is Failing in the Digital Age and How We Can Do Better, as well as Dave Guarino, who recently left the Department of Labor after working on upgrading the unemployment insurance system. Both have a long history of working on public sector software systems and they explain why the problem is so tricky. This transcript has been lightly edited for clarity.
Key insights from the pod:
The complicated requirements for building public software — 5:09
What is the process for building government tech? — 6:17
Who gets to build on government projects? — 10:43
The compliance burden of public vendors — 15:37
Why public-sector decision making is so hard to change — 18:54
Why budgeting makes it difficult to build technology — 21:14
Lessons learned from the Covid UI debacle — 25:51
How Covid changed public sector thinking — 29:31
The challenge of public sector complexity — 35:14
Balancing resiliency vs. efficiency — 36:14
Why is it so hard to rebuild old systems? — 38:56
Why the public sector struggles to hire tech workers — 46:53
What are some particularly egregious public sector tech bugs? — 49:57
---
Joe Weisenthal: (00:10)
Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal.
Tracy Alloway: (00:15)
And I'm Tracy Alloway.
Joe: (00:16)
Tracy, you know what I felt was a really interesting thing in our recent debt ceiling episode that we did, was about all the coding it would take if the Treasury were to introduce a new kind of a bill.
Tracy: (00:27)
Yes. That was interesting to me too, because I think there's a perception out there that if Treasury wakes up tomorrow and decides to sell a new type of bond, they could just do it. But actually there are all these backend changes that would need to be made. And as you and I know from multiple conversations at this point, it feels like government technology can be a little bit clunky sometimes. Is that a fair way of putting it?
Joe: (00:54)
It is. I would say, in defense, and we'll go into this, corporate technology can be clunky. But that point reminded me that we talk a lot on the show, I think, about the sort of “built” economy. Like the CHIPS Act and battery investment and semiconductor and all these things that are like, “can the US build things again?” as a government, sate capacity, these ideas. But in the 21st century also building software and tech has to be part of that question.
Tracy: (01:25)
Right. So it's one thing to announce that you're going to do this new thing, but almost everything that we do nowadays comes with some sort of software requirement, like I'm sure there is a system for processing applications for CHIPS funding and things like that. And someone has to actually build that. So there's two questions contained in the “can America build things?” question. It's like, can it build the actual things and then can it build the software systems that would allow it to build the things?
Joe: (01:53)
And you know, in the pandemic period, we got sort of smacked in the face, I would say, by this realization that the software component is not trivial at all. And most notably it was like every state had some sort of problem with their unemployment insurance system and there was just such a surge in claims obviously in those initial, you know, really for that first year, just on the scale that no one had ever seen.
And it reminded people were like, oh yeah, 50 different state systems much with tech that was probably built by some contractor in, I don't know, the 80s or 90s and the people who knew how to run it have left. And you know, I guess eventually it got smoothed out but it sort of exposed like, okay, why did that happen? And what else is lurking out there that we don't really know how it works until it's broken?
Tracy: (02:38)
I'm sure this is going to end up being a classic episode where we mentioned COBOL quite a few times. It's inevitable, right? It's coming, you can feel it.
Joe: (02:45)
One day we're going to do it an actual, just like “what is COBOL?” episode where we just focus on that directly. But yes, we are going to talk about software and we're going to talk about it from the public sector perspective. What does it mean when the government tries to fix something?
What happens when the government tries to build something? What happens when the government tries to buy something? So some of these themes, we talked about them from a private sector perspective with Patrick Mackenzie earlier in the year, but more to do on this topic.
Tracy: (03:12)
Yeah, I'm excited. This is going to be a really good chance to actually ask about the decisions that are going into some of these things being built. I'm still floored every time I log into the Treasury Direct website that, you know, you have to click the little buttons, like the actual letters and numbers to type your password. It won't let you just type your password. How did that come to be? I have questions.
Joe: (03:39)
That is a great question! So we have, I think, literally the two perfect guests to discuss this. Two guests with extensive experience answering and working on exactly these problems. We're going to be speaking with Jennifer Pahlka. She is the founder of Code for America. She helped found the US Digital Service. She co-chaired the California task force on fixing its unemployment insurance system during the pandemic. And she's the author of a new book called Recoding America.
And we are going to be speaking with Dave Guarino, who's a number of things. Currently an independent consultant/researcher focused on this stuff. Founding engineer at Code for America helped create GetCalFresh, which allowed Californians to easily access SNAP. He has recently worked on unemployment insurance modernization at the Department of Labor. Also worked in California.
So two people with extensive experience on exactly these things. So Jennifer and David, thank you so much for coming on Odd Lots.
Jennifer Pahlka: (04:40)
Thanks for having us.
Joe: (04:41)
Absolutely.
Dave Guarino: (04:42)
Great to be here.
Joe: (04:43)
Um, now I want... Tracy, can I steal your first question?
Tracy: (04:46)
Yeah. Do it.
Joe: (04:48)
How does it happen that a government website doesn't let you type in something and you have to like click buttons that resemble a mini keyboard on a website? Like what happens when you see that, do you have, in your mind you're like, “Oh, I know exactly the meeting that caused this to happen?”
Jennifer: (05:02)
I think Dave has been in those meetings and has come out like with his hair on fire fuming. So I will let him answer that.
Dave: (05:09)
Well, I wasn't in that specific meeting. Maybe the best answer I can give is, and there's deeper issues behind it, is that the way government tends to build technology, the reason for that is no one wrote down specifically as a requirement upfront that people should be able to use the keypad and the numbers on their keyboard to enter the digits. And so someone's like, “well, I've met the requirements.” Like the requirements are that someone can do this, they can enter it. And the way they do that is they click in this somewhat, I suppose, insane way.
Joe: (05:46)
This is going to be a conversation where we're like banging our head on the desk a bunch, I feel like, as we listen these answers . Anyways.
Jennifer: (05:52)
Get ready.
Tracy: (05:53)
Okay. Well let me ask maybe a big picture question to start, but what is the process by which government software is developed?
Because you know, that answer just then it makes it sound like someone gets a mandate: We need a website, for instance, through which people can buy government bonds and then someone else goes off and I guess they execute it along the lines of the mandate that's been set.
Jennifer: (06:17)
I'll jump in on that one. I mean, first of all, it is changing. So just before we get everyone sort of banging heir heads on the table, there's a lot of movement in good directions, but what has been the process is exactly that. There's sort of this mandate, it often comes from legislation or regulation, that very frequently now like says there will be a website, they'll even put the URL in the legislation.
And then it kicks off what's a really long process that does exactly what Dave was saying. It is first centered around requirements gathering and that process of just requirements gathering can easily take 10 years. Now when it's something like healthcare.gov, you know, and they have a launch date that's three years out, they don't take 10 years, but they still gather sort of every imaginable requirement and they kind of throw it in one giant bucket.
And that's how the RFP is created. The request for proposal. That's what vendors bid on and it's sort of an undifferentiated reprioritized mess of everything everyone could think of. Plus things that have sort of pulled over from previous RFPs plus a bunch of compliance stuff around security compliance stuff around, well, various other things. Dave can maybe chime in, but what that isn't is product management, right?
There's no one saying “here's the important things this should do.” And as Dave said, there's never a requirement for it to just work. So then the vendor, I mean I'll skip several steps in the the middle where, you know, it takes a long time to bid it out and then there's like a protest because one vendor thinks the other vendor shouldn't have got it. And that delays it a couple more years. But then they go into development and their job, as Dave said, is just to like check all these boxes and it's thousands.
I mean when we were at the state of California working on unemployment insurance, they were actually about to put out a business system modernization RFP there actually I think to award it to a vendor and I think it had 6,700 requirements. We can check that. But that's really a normal number of requirements. And so they do all these things that where you can check a box, but what they don't do is actually check that it works.
So I mean maybe I'll just tell a quick story from the book. And you know, there was famously this application for veterans healthcare benefits at the VA that didn't work outside the building and they couldn't see it. So basically somewhere in the specs it had that this form needed to work on a very specific and outdated combination of Internet Explorer and Adobe reader.
And the reason no one knew that it didn't work inside the building is that's how all the computers in there were set up. But if you were outside the building and had any other possible combination of those two pieces of software, it literally wouldn't load. And so they had very few people applying for these benefits online and you know, veterans were really, really, really frustrated.
And it took a team going out there recording a veteran who had tried to do this dozens and dozens of times bringing that video back and showing it to the deputy secretary for them to be able to say, “oh, okay, actually there is something to be fixed here.” Okay you, you know, you can go ahead and make a new form. But up until then they said “Sorry, it's fine. We're looking at the requirements. The requirements have been met, there's technically nothing wrong.”
Tracy: (09:38)
That's amazing. It's also kind of crazy to think that people would've been looking at that and just been like, “Oh well I guess demand for veterans benefits is lower than we thought it would be,” but actually it was a tech issue.
Jennifer: (09:49)
I think what they said was demand for them doing it online is low. These veterans must not have access, which is absolutely not true.
Tracy: (09:56)
Or they can't figure out the computer when in fact it it was people...
Jennifer: (09:58)
Yeah, I mean they would tell them, people couldn't figure out the software, they would tell them and they told this guy Dominic, the veteran that they interviewed, they kept telling him it's user error. There's something wrong with you.
Joe: (10:06)
Can I ask a question about that bidding process? When I hear government goes out to bidders for a website, I imagine these like big sort of faceless offices in Rosslyn, Virginia where you know, people get like chopped salad for lunch and stuff like that. And there's like three or four of them and they sort of rotate who wins the bid and stuff like that. How competitive would a typical RFP auction or bidding process be? And how hard would it be for say, a more agile, innovative firm or someone wanting to do it differently to break into the pursuit of that contract?
Jennifer: (10:43)
That's changing a lot. So now you've got a set of vendors, I think there's about 35 of them in this thing called the Digital Services Alliance, which are smaller companies that work in a more sort of agile user-centered way. And you know, they're just a lot smaller than the Beltway Bandits. So they're still a tiny portion of the market, but you know, they're really viable businesses and people in government can contract out to them.
In fact, I think there's a mechanism by which you can put an RFP out just to those companies. But what has happened in the past is, Dave can pile onto this, if you were bidding out say, you know, benefit system and a state — like, pick your benefit — usually they would require in the RFP that the bidder had to have done a similar system in a different state in the same category. Which means that right there it limited to like three vendors. And so you were never going to get any disruptors because they were boxed out in the very first step.
Dave: (11:43)
Just to maybe jump in, that's exactly right. And I mean to some extent it's rational. If you are thinking about this as someone who's not a technologist, you run an agency and you're thinking ‘we need to do this very large project, it's going to take a very long time. We really need it to go right.’ It is hard to think about, why wouldn't I look for a vendor who's done this five, 10, 15 times”
The problem is, as Jen said there, well I think there's two problems in that. One is you've now very, very drastically narrowed your vendor pool. So there's a lot less competition. And two, you also may have too big of a project. Like really what level of traction are you trying to work towards? And is it the case that really you need someone or a vendor who's worked on this specific type of system in another entity that it works exactly the same way?
And all of those details of the exact same program, the exact same structure of agencies, probably not because probably it's about more generically software. But if you really focus only on the track record there, you kind of get stuck, as Jen was saying, in this [place] where there's only a small number of vendors who can meet those criteria even if you think they're rational.
And it's hard because the other thing I would say is as much as there are new vendors and maybe Jen can speak more of this, but it's also the case that if you have a new vendor that wants to work in a more agile way, wants to work in a more user-centered way, but they are bidding on an RFP that is structured in a very waterfall, top-down, meet all the requirements and that's what success looks like.
It's also not going to be successful because you're not giving them any space to say, “oh, well I know your requirement says a way to enter phone numbers using the key the mouse. But when we tested it with people, like people, it turns out, really don't like using the mouse to click numbers to enter digits, they prefer to use the keyboard. Could we change that requirement?”
And honestly, I mean that's an extreme example, but a lot of what Jen's book is about, what resonated with me in it, is that in doing these things and building software, you learn those details that you never could have gotten right upfront. And so if you make success defined by implementing everything as we understand it before we ever get started, you almost ensure that you are not going to get what you really wanted because you're not allowing any information after that cutoff point right when you might be getting to the point where in doing it you're going to learn a lot.
Tracy: (14:39)
I definitely want to ask you more about that top-down waterfall nature of how a lot of these software projects get commissioned. But before I do, I'm just curious about the actual vendors because I'm sort of imagining these like big faceless offices, but what types of vendors are we talking about?
And I'm trying to think how to phrase this kind of diplomatically, but I can imagine a company that is putting itself out bidding for government contracts on software development, maybe they're not as experimental or cutting edge as a software company that's somewhere in San Francisco and it's building new exciting products for the corporate world — although we've also done an an episode on how bad internal corporate tech can be. But it seems like it might be a little bit more, I hesitate to say boring, but sort of basic, playing it safe.
Jennifer: (15:37)
Well, you know, one thing that I will say, I hear both incredible frustration and even anger from folks that have to hire these companies. There's just like, there's no alternatives. This has gone badly, it will go badly. They kind of know that. Like I had this woman in a major state, I won't say which, when I said, “you know, your, your project's going to fail,” she said, “Do you think we don't know that? The last seven projects have failed.”
So there's this huge frustration, but there's also this real feeling that the company in San Francisco with its 40 people who make consumer software, like great. People love using that software, but they don't understand us. They're not going to actually understand the constraints of government. So we have to go with these big companies. And they're not wrong in the sense that the constraints of working with government are real.
There's a huge compliance burden. That 40 person company in San Francisco mostly does not want these jobs. There's so much that goes into getting the bid. You know, it's sort of famously said that these companies know how to get the work, not how to deliver on the work. That's not probably what a 40 person company wants to do. But you know, it's true that you have to deliver on stuff that is mind bogglingly complex.
Like when we're working in unemployment insurance, again during the pandemic, my colleague was talking with the claims processors week over week and we're trying to dissect it and figure out what's going wrong and clear this backlog and one of these guys keeps saying, “Well, I'm not quite sure about that answer. I'm the new guy. I'm the new guy.” And she finally says, “How long have you been here?” And he says, “I've been here 17 years. The guys who really know how this works have been here 25 years or more.”
So think about. You know, going from doing some simple cool, you know, tech app, you know, easy consumer app to trying to build or fix or improve upon a system that is so complex that it takes 25 years to learn how to process a claim.
Joe: (17:54)
Yeah.
Jennifer: (17:55)
That's sort of, I think, what needs to be on the table as part of this agenda is not just “can the tech be better?” But can we go back and simplify the accumulated like 90 years of policy and process that's making that so hard to make?
Joe: (18:09)
Well, why don't we sort of back up then, and again, I'll kind of steal Tracy's question because I had the same thought. What is it about government buying that creates these massive waterfall RFPs that create these constraints that the software developer would not necessarily have with a private buyer and puts government agencies in a position where they all say to you, “Of course we know it's gonna fail.”
Which again, presumably that actually probably does happen in the private sector too. But what are the dynamics that make these things like so hard to change and so hard to uh, you know, rip up, you know, come up with a new system?
Jennifer: (18:54)
Well, this is a good Dave question, but I'm going to start. You know, I really spent a lot of time reflecting on this. You know, I've had sort of three years since I stepped down from Code for America where I just like sat in a room and interviewed people and thought about it and thought about my own experiences.
And I think that there's a deep seated culture in government where the policy people are the important people. They do the important stuff and technology, digital is just part of implementation, which is not just the bottom of a software development waterfall. It's the bottom of a big rigid hierarchy in which information and power and insights only flows from the top to the bottom.
And so it's problematic in part because the people who are doing the tech are really just sort of downstream of everything else and the power and ability and willingness to step up and say “Hey, we probably shouldn't do those 6,700 requirements, we should probably focus on these 200, get that out the door and then, you know, add edge cases as as later.” There's no permission really to say that.
And if you compare that with say, I'll call it metaphysical Silicon Valley ‘cause I don't mean actually like “Silicon Valley,” like the people who write code started the companies, they're in power. They're at the top. Like compliance is below them in sort of the DC government hierarchy compliance is way above, right? Policy is way above and there's not this sort of build-measure-learn cycle that Dave was referring to where people are learning from each other as they're doing the work and that technology, you know, certainly the policy has an impact on the technology, but the building of the technology needs to have an impact on the policy.
That fundamental culture is something that needs to be looked at and called out and where it's not happening, where there's actual conversation, you know, dialogue beats directives, right? Where that's happening, we should be lifting that up and saying this is possible within government. Because that's the foundation, I think, of getting away from these mega projects that fail. But Dave, you're going to disagree with me on this and I'm going to love it.
Dave: (21:14)
I actually do think that's right. I think a big part of it flows downstream from budgeting. Meaning a lot of this, and again, everything that I think I and Jen say, like there's some degree of generalization here and there's always outliers and there's always exceptions. And of course there's also a lot of people trying to change these things.
That said, maybe the dominant mode is we are going to budget for a large project. We get one time funding to spend to do that project. So we’ve got to get everything right. We're only get this chance once every so often. We're going to replace everything. We're going to have this new system, it's going to be great. And the way that this paradigm breaks down is in what's called a design development phase, where you build the system, you design what it should do.
You think about that again, you're kind of making all these decisions up front, which I think is very counter to a lot of the, if you want to call it the Silicon Valley approach, but then you're building the system and then you enter what's called maintenance, maintenance and operation and it's supposed to be very, very little. You're not going to change much. It's just kind of keeping the system going.
And the problem is, modern software you kind of learn the most the second you've actually deployed to real users. That is the point when you are getting people saying, “Oh, this doesn't work.” Or you're getting an error or you're getting, “Hey, I tried to put in this address but I have a funky address and it won't let me do that.”
So all of a sudden you have all this information and unfortunately in sort of what gets called a big bang launch project where you just put it up and you're done and you're not going to touch it again, you now, almost immediately on day one can see potentially all of these issues. And a lot of people prepare for this in these kinds of projects. They're like, the first month is going to be be really hard. All of a sudden know all the things that were assumptions you made that you got wrong. But the model that you had that you were given is, “well, we're gonna do this one time.”
Now contrast that with actually going live is the first step and you're gonna have funding that's continuous and kind of flat over 10, over 20, over 30 years. And you're gonna have those people on staff and you're gonna continuously change the system as you learn more about how the users interacted with it. As you learn more about like what things you didn't get right upfront, did you, you made it possible to upload a certain type of document.
You accept PDFs and TIFF images. TIFF. This is a real example, but turns out you don't support JPEGs, I think iPhone proprietary default image format. So now all your mobile users are saying, “Hey, we can't upload.” Or a bunch of them are saying, “I can't upload documents.” It says I can't submit this file type right now. A lot of , you know, well-intentioned folks in government get to that situation and then they're stuck because they don't have the mandate or the funding to say, “now we're gonna change that.”
They're waiting for the next big project to replace the system and now support those new things. Whereas instead, if you have a model where you're saying we expect to continuously change this, it's a very, very different thing. And that's a little bit, that is more how Silicon Valley operates.
I mean this maybe is sort of hyperbole, but Google didn't build search and then lay off 90% of their staff and say “We're done. This is great.” You know, like “look, we got this search thing.” And I think that difference flowing downstream from how tech is budgeted for where you're budgeting for a one-time project and this comes from legislatures, you know, from congress, a one-time project versus we want to have ongoing funding because this is a living breathing system is a huge paradigm shift and has an uncomfortable aspect, which is, it may seem like it's costing more money because it's potentially dollars ongoing because it's staff not one time buy. But I think a lot of the dysfunction that we see probably flows down from that root cause.
Tracy: (24:59)
Okay, so here's a big question. If we can identify the cause of a lot of this dysfunction and if we can trace it back to this waterfall structure or the sort of top down mandates or maybe like theorists versus practitioners, so you have a politician who has a big idea and they want that big bang reveal of the software, they're going to expand benefits and it's going to come with a really cool website and everyone's going to be able to access them easily. And yet it leads to these problems that we've been discussing, why do we keep doing it?
Jennifer: (25:31)
That's a hard question...
Tracy: (25:34)
Well, what is the system that is keeping this way of doing things in place versus maybe, you know, after a few decades of developing these types of systems, why isn't someone going, well, actually the way we've been doing things hasn't really been working?
Jennifer: (25:51)
I think a lot of people have been saying that and that's why I say things are changing. And I will offer as evidence Covidtests.gov. Do you remember that one?
Beginning of 2021, The President announced that there would be this site where you could request tests. And I think he gave a date that was, I want to say, six weeks out. It launched a early, it launched in multiple languages. It took 11 seconds for me to use. Other people say eight seconds, 15 seconds. And your test showed up a couple days later. Like that sounds like it could have been really complex and like, okay, so it's much simpler than say signing up for healthcare.gov or whatever. But they could have made it way more complex. They could have said like, “let's gather every possible requirement and try to fill, you know, fulfill every possible requirement.”
But you had internal capacity. To Dave's point, there are people in-house who were saying, “Hey, we have been doing it away that hasn't been working. Let's try this way. “ And it's great. It serves as a fantastic example I think within government and outside government, right? So part of it is how people in government think what they believe and part of it is what the public thinks, right?
The public couldn't imagine something that easy, but we have to start imagining it and holding our government accountable to doing that. So I mean, I will say it is changing, but the belief that we aren't good at it, that we couldn't ever do like a Covidtest.gov drives politicians, pundits vendors to say, government's bad at this. And so we should outsource everything.
Now of course we're gonna have vendors, we are and we should have vendors, but there has to be enough internal capacity and competency to outsource well and to make those decisions. Like “hey, let's have this thing only ask like name, address, and like, you know, not ask for their health insurance, you know, numbers and not like do different numbers of tests per household.”
That's an internal capacity question that is really important and we don't have enough internal capacity for these kinds of decisions in government because we believe we're not good at it and we believe the only people who are good at out of these outside companies. Plenty of evidence that that's totally wrong.
Joe: (28:32)
I want to go back to internal capacity in a minute, but since you mentioned the Covid test website, it's sort of a good chance to pivot a little bit here. You know, Covid specifically seemed to open people's minds a little bit about how fast we can work and how much money we can spend, and obviously a historically fast pace of vaccine development that we've never seen prior to that.
And so it sort of had this brief moment where it sort of opened people's minds. And I'm not sure if it's closing again, but you know, since both of you worked on the California unemployment insurance system and generally I think almost every state on some level was seen as some debacle.
What was, I guess, the biggest eye-opening thing that going into that environment taught you or about the system and then what was, you know, what's the key sort of takeaway of like, “Okay, here's the thing that we can learn from this” that whether it's UI and other states or more broadly, okay, this is something that you learned that then can have extensible lessons?
Jennifer: (29:31)
So I think you're right. Covid had sort of twin impacts on our thinking. One was, “holy cow, we can actually do stuff.” And the other one was, “holy cow, we're really screwed.” Right? And there's evidence of both, which is really fair.
I mean, I think for me, you know, I'd been doing this for you know, 10 years when I got pulled into the unemployment insurance, I'll call it a rescue. We were clearing the backlog and I felt like nothing here can scare me. I've seen the operating of some pretty janky government agencies. But I actually really did learn a lot and was kind of shocked at how the unemployment insurance system worked.
And the metaphor that came to mind for me was everyone kept saying, “well, it's a system, obviously you can just fix it” and it isn't a system. And I think that's true of other things we think of as systems too. Like it is archeological layers of technology that sort of map to archeological layers of policy that didn't ever get designed.
It just accrued, right? So nobody ever goes back and says, “Okay, how is this going to work with this?” There's such little appetite for anything that's like backward looking that would actually update and make these layers work together well, it just isn't what people think it is. When you go look at it, you're just like, “Whoa, of course you can't do these things. Like this thing isn't connected in any meaningful way to this thing.” And like this was built in a certain era for a certain purpose and we just glommed stuff onto it when we needed to say add internet like let people apply online.
I mean the biggest thing was like, I think we haven't grappled with the fact that when these systems weren't quote unquote “designed.” We should say when they were started, because they really have never been designed and a lot of them and there's huge advantage to actually doing some design, but you went into an office and showed who you were, like you identify, you validated your identity by going in.
And then we moved online and we never figured out how to validate people's identity. And there were a number of problems in California with unemployment insurance. The fact that, you know, it took 25 years to learn how to do it meant that any claim that couldn't be processed automatically was a huge, huge bottleneck and you couldn't add claims processed do that. You would've to like go back in time 25 years, start new claims processors and have them ready to come online during a downturn, right? Like that's never going to scale.
But the other big problem was that the only went through the automatic sort of pipeline if they felt like they could verify your identity, but they weren't really doing any meaningful identity verification. And we, unlike most other industrialized countries, don't have a national system for that. Like we don't have a national technology platform for that.
But also, like, again, back to this all derives from like the legal and policy framework. We don't actually have a reasonable policy framework for identifying people and knowing who they are when we start to do a transaction. And that is causing problems everywhere. And there is interesting conversations about how to fix it. There's some stuff out there, but it's really contested and it, it's not just a problem for the pandemic, it's gonna continue to be a problem.
Dave: (33:16)
I mean, I'll just, and my vantage point was, was somewhat different… I was coming at it from a different angle. I think the major thing that I took away, especially with so much focus on technology and so much focus on COBOL mainframes, and that was, you know, that was the dialogue.
Tracy: (33:36)
We need to do like a COBOL drinking game or something. Take a shot.
Jennifer (33:41)
We need to time how long it took us to get to that word.
Joe: (33:42)
I think it's been 30 minutes.
Dave: (33:45)
You know, and the best part is that, I mean my next point was that's misplaced causation and yet I am the one who brought up COBOL. I mean I think, you know, for so much focus on, “oh, these tech systems are falling over, they're not good,” I think the number one lesson I learned was that, and I think this is true, but we don't tend to think about it, which is the technology systems are part of a larger sociotechnical system.
What I mean by that is, as Jen mentioned, you have staff who can do stuff manually and you have a system that might be able to do stuff in an automated way. One of the problems with programs that are so complex is if you have a massive spike of volume and your technology system isn't set up to just cover all those cases in an automated way, you try to throw bodies at it and you try to hire people. But if it takes a year and a half to train someone and like train someone to a relatively basic level with some of the complexity of these programs, you're not able to hire fast enough. And unfortunately...
Jennifer: (34:53)
Just a second. We were hiring really fast. We hired 5,000 people in California to help process these claims, but because they couldn't do anything, they were taking up the time of the experienced claims processors and we calculated that every single new hire the state made slowed processing down.
And one of the big things we did was actually just get them to reassign staff away. I mean some of it was just reassigned staff to other things like nobody was opening the mail, which was pretty critical. And yet opening the mail is something you can do if you just joined and have no background. But yeah, it was pretty frightening. Sorry to interrupt Dave. Just, yeah we were hiring, it's just that the hiring was having a perverse effect.
Dave: (35:40)
Well, and that's, I guess that's my point is that's exactly right. I guess the number one lesson I learned was if you look at the history of a program like unemployment insurance, but specifically UI, you have this thing where once every 10 years or so there's a major recession or in this case this was a pandemic, which was like a 10X recession, right? Because it was overnight as opposed to gradual. It was overnight. So much of the economy, so many of the working force labor force were just out of work immediately. So all of those claims came in simultaneously.
But we have this cycle of every 10 years or so we have recession or, I mean, I don't know, I'm not an economist, I couldn't tell you exactly when that is, but it happens in a recurring way. And then everyone gets frustrated with how difficult it is to get unemployment insurance because they're overloaded with volume and then the claims volume goes down, recession ends, and then honestly there isn't a lot of focus on, “okay, we're going to sort of put more money into the UI system so that we're ready for next time.”
But unfortunately you can't have resilient systems if you optimize for pure efficiency. In the down years, in the years when you have very, very low volume, if you are just going to cut staff down to very, very bare bones skeleton crew, it comes back to this point of not only are you not able to do preparation, if you don't have the funding to have staff, you can't even make the changes. That would be good. And you can't, when you get the money in a new recession, hire the people because you can't train them. And so we kind of get stuck in the cycle. So that was the number one from a systems modeling perspective, I took away with it from it.
Jennifer: (37:17)
I’ve got to add though, you know, we did invest several billion across states during the great recession or after great recession. So I think about half of the states technically modernized, then they did take advantage of pretty significant funds. None of those states did any better than average in this downturn. So I agree with Dave and I would argue that taking a different approach to quote unquote “modernization” to policy and process simplification to, you know, having goal driven modernization, like all of those things is equally important as investing during the downturn. Investing the way we have been doing it is not working.
Tracy: (38:02)
There are so many different threads to pick out of there, but just on that last point, you know, when you were talking about how a lot of these government systems were not designed to be that way, I kept thinking of a lot of the software used by the banks where, you know, you think of JP Morgan's software system, it wasn't designed to be JP Morgan's software system. It's the result of many, many acquisitions of other banks and, you know, every time they take over this business it gets sort of duct taped onto the rest of the system.
But I'm curious, is there a moment that you have seen in your careers at which someone says “this system is just irredeemable, we're going to have to start from scratch.” And what is the trigger point for making that decision versus reaching for the band-aids and the duct tape and just trying to make it work?
Dave: (38:56)
I mean, and I forget if someone else has mentioned this, I listened to the Patrick Mackenzie episode, maybe he mentioned this, but in general, a good rule of thumb is never rewrite a system from scratch that's working .
I mean other people might disagree with that, but the reason I don't have a good answer for sort of a a moment. Lots of people have tried to do that thing where they just rebuild it from scratch. And again, it tends to go poorly because, and this is the specific thing, the old system encodes so much tacit knowledge and so many of the many, many little details. And some of those details might no longer be relevant, but many of them might encode some really hard-learned truth about the world. Because at the end of the day, the software is just modeling the real world.
It's just saying, “okay, well, we have customers, they have addresses.” Okay, so maybe you have two address fields in your old system and you think, oh well let's just streamline that down to one. But then you launch your new system has one address and everybody calls you up and is like, “What are you doing?” The reason we have two addresses is because our billing department is over here and the customer facing address is different. So I think the ‘throw it away and start from scratch’ is an impulse everybody has, but it has a lot of risk as well.
And I think the better approach tends to be can you get to a place where you can start to incrementally change your systems as you go in small ways that are safe where it doesn't take it, you know, it's not like we have to test for six weeks any change before making it live. We can ship weekly, we can ship daily, we can ship multiple times a day. That tends to yield better things.
There are plenty of projects out there we're going to build this from scratch from the bottom up. Most of the ones I'm aware of don't have great launch stories. Maybe Jen disagrees.
Jennifer: (40:56)
No, they're still in that mode of like, get it all in once and then we're done and we have sort of maintenance and so they're pretty bad. But I will say two things.
One on a practical level, just because I was talking with a colleague on the way over here about this. There are great examples now of what Dave's talking about. Like not waiting for this big bang project but saying we are just going to be making this a little bit better every day. And one of them is New Jersey's Department of Labor. Their unemployment insurance response during the pandemic was quite good, but they've also learned from that and said we are not going to wait for some like big system to come that's going to fix all of our problems. We have a team now that can just incrementally fix things as we go.
And you know, eventually that will involve some bigger investments as they learn X, Y, and Z, but they're just not waiting. And it's just a great, great example. But I will say, Tracy, I think about what you're asking about all the time, I don't think this will ever happen, but if you do get a chance to sort of zero base it and start over you have to zero-base the policy.
Like could you do a new unemployment insurance system in California that encodes the 25 years of knowledge that those claims processors have? Well, I mean maybe AI will help you. Now I don't want to talk about AI, but like that's a problem. You know, I say in the book, there's this great team that's working on this Medicare problem and it's like there's nine different definitions of even like what a medical group is like.
Even the first question doctors are supposed to answer is wildly complex. And the policy, the delivery team is sort of fighting with the policy team and she's like, I get that it's complex. It has to make sense to a person and I like fight not over the tech, but out of like over collapsing these nine definitions into two, they want to get to one, they can't, at least they get to two.
And this idea that it has to make sense to a person. We've lost track of that. I mean of course you're frustrated when you apply for unemployment insurance. It's wildly complex in its policy and the process that accrued from those policies. So if you were going to start over, you can't start over with the tech. You would have to say, okay, let's figure out, let's look at this 90 years’ of accumulated policy and say what was this actually trying to do?
Can we kind of, you know, either go back and just rationalize all the changes, right? There's not one binder of regulations that covers UI. There's 90 years of memos that changed the previous memo that changed the previous thing that changed the previous thing and you could at least go back and condense that and like figure out all of the conflicts in it.
But it would make actually I think a lot more sense, and I will be clear, this is never going to happen, if you just said let's design the policy for an unemployment insurance system that makes sense in 2023. And then you could build tech for that.
Joe: (44:12)
There's so many different threads and places we could go. This such a fascinating conversation. I just have one more question and it sort of relates to something Jennifer, that you said earlier about like in government the tech people are like so secondary to the policy wonks and people designing these things and that that's a really big difference between government, public and private sector and you know, this internal capacity.
How much does that hinder? And I think this is another thing that Patrick Mackenzie brought up, but hiring and the challenge of public sector pay scales are not private sector pay scales. I have to imagine for a lot of these things he talked about a little bit, you know, and so dessert, you know what you've worked on some of these like sort of volunteer things that sort of bridge the private and public sector. But can you talk a little bit about that constraint of like even if like just the challenge getting like experienced talented people.
Jennifer: (45:07)
I'm glad you brought that up because it's a little bit of a soapbox I like to get on because we have, you know, like politicians, elected leaders who are so frustrated with delivery in technology and like they're generally, they're well-intentioned, but the sort of things they push on are kind of unhelpful.
You know, they kind of increase the risk aversion of the bureaucracy by yelling them about things and not recognizing the constraints on the bureaucracy. I call this the accountability trap, but there are things they actually could do. And fixing the civil service and civil service hiring rules is really, really important.
I do know what Patrick Mackenzie said on your show about pay scales and I don't totally agree with that. The biggest problem is not the pay, it's the time to hire. So if it takes, as is really common right now in federal government, nine months to make an offer, of course you lose everybody.
You know, when I started this work, there was not like a line of people out the door wanting with great tech and design skills wanting to work from government. Now there actually is and we can't get them in the door. I mean I know people who do wait nine months for that offer because they really want to help and the pay is fine. It's not fantastic, right? I think he was comparing to like Google whatever, but, you know, compared to a startup where yes, you have the hope of getting an exit, but you're probably not going to make like $500,000, right? As a programmer you're going to make what? Half of that? I mean we don't have to get into like exact numbers, but the pay is in the right ballpark for mid-level folks. It's not going to pay at the executive level.
But as he mentioned, at the executive level, if you've been at Google for 20 years, pay is not your biggest concern. I do want to push back a little bit on the notion that that is who is in government. There are some of those people, but there are far more people who just have good tech and design skills who just want to make a difference more than anything else. And we should be working on all of the constraints that are keeping that hiring process at nine months. Then we're like calling up the people who, you know, manage the Department of Labor in a state and yelling at them because they had had these failures like fix the environment in which they're working, give them that capacity, then you can hold them accountable, but they can't hire people right?
Dave: (47:38)
You also need to be able to hire people and you need them to be able to use their judgment to make higher order decisions than just ‘do we enter the phone number with the keyboard or by clicking on a virtual pad of numbers?’
It is also the case that some of these systems are the way they are because it's just Jen, this might be your phrase, it's like policy vomit. It's just the rules put onto a screen just like all the rules in statute put onto a screen. And so I think the thing I'd like to add is the whole point of product management is to arbitrate the trade-offs between user experience, compliance goals, technical cost of change, and technical trade-offs. And if you don't give those people, if you hire them, they don't have the ability to say, well maybe we need to simplify this to make it make sense to a person.
If instead the only response they get is what we need to put on the page exactly what's in statute then or whatever other compliance goals there are. Like, oh, we need to strictly follow this process. It's procedurally what it is. If those are counter to the outcome, you need a, a change in government where outcomes start to be outcomes in this area around technology start to sort of have a higher importance than the the strict process and the strict procedures. And I do think that that's a necessary corollary to bringing smart, capable people in because if you bring in an a superb product manager and they're just, they have no ability to sort of bring these trade offs up and make those calls on the ground, you're gonna get the same status quo I think.
Tracy: (49:31)
So I can't resist asking this question. I take all the points about here's what we need to do in order for this to change. But can I just ask, because both of you have that practical experience of doing this, what was the craziest thing that you saw during your respective tenures? What was something that just really surprised you, or shocked you? I'm back to the banging our head on the table a portion of this conversation.
Jennifer: (49:57)
I mean, I guess the thing that comes to mind just because it really made an impression on me and it's a negative that I want to sort of balance out with something positive, but I was working in the White House and had decided to do what we convinced the Veterans Administration to do what we call discovery sprint on the veterans benefit management system. And it was going really slowly. There was like huge latency and I got these two folks in to help out for just a couple weeks and just, you know, do an analysis of what was wrong before they go. You know, trying to figure, you can't solve the problem until you know what it is. So the first thing was we show up and the guy's like, ‘oh, I'm glad they sent the White House to, you know, verify that everything's fine now.’
And it turns out everything was fine because the leader had defined latency as under two minutes. Okay, so you've just defined the problem away. So people who are trying to process benefit applications, like if they clicked and it took one minute and 59 seconds for the application to respond, it was fine. No problem here.
But then I kept talking to the guy and I kept asking him questions about the system, like, “Well why is it designed this way? Why is it designed that way?” And he said, I've spent my career teaching people not to have an opinion on business requirements. The IT people should have no opinions there. And I said “why?” He said, “well, if they ask us to build a concrete boat, we'll build a concrete boat because then it's not our fault.” And I was just like, I felt like I'd been gut punched.
It was so hard to hear that. There were, I think at the time, 18 veterans a day committing suicide, most of whom couldn't get their healthcare benefits. And I just couldn't understand how someone could take that approach. But honestly, you know, in the 10 years since then, I've seen a lot of reasons why he felt like he could say that. And, you know, I don't want to blame him, I want to blame the system. But yeah, if you tell us to build a concrete boat, we'll build a concrete boat. What way is that to run a country?
Dave: (52:13)
I have a much more technical problem and please yell at me if this is too technically in the weeds. So at one point we, in my old work on SNAP, we observed a system, one of the official systems that was user facing. So external users, folks trying to apply for food stamps, you use it. And we noticed one day that it wasn't loading. And it wasn't loading in the following way. You would go to the website, it would have buttons and everything loaded, but then when you try to click on anything, it wouldn't work and it wouldn't work weirdly for about a minute and a half, then all of a sudden you click on things and we are like, what is going on here? This is bizarre. And we looked around and we did some debugging on the client side, meaning we opened up the web browser inspector and we said, “okay, what's going on?”
And what we found was this website was loading, I forget exactly the name, but it was like translations.js or translations.xml. And it was taking a minute and a half, everything else was loading fine, but this was taking a minute and a half and counter to best practice. It was, you know, blocking any interaction with the page until it was fully loaded and it was loading very, very slowly. And we looked at it and it’s a giant file. It was like, I don't know, it was like 50 megabytes.
It was a giant file, like what is going on here? Looked into it, and it was a translation of every page on the entire system into like eight languages. And we call the thing. And that was like, “okay, this is not great.” And there's ways to fix that. Like I said, you can let people interact with the page until before that's fully loaded, and that's fine. Also, you definitely shouldn't put all the translations for every page in your website on eight pages into a single file that you send to the client. That's just bad development practice.
But the thing that maddened me is we being goodhearted people who are not out to, you know, just sort of lob bombs and talk trash, we contacted the entity who ran it and we said “Hey, this is happening. We're seeing it consistently. This is the issue that's going on.” And they said, “No, no that's not happening. It's loaded totally fine for us.” And the reason is because, we diagnosed it down to one or two reasons. One was they'd already loaded the file once, so it was cached, and so it was on their computer. So every time they went to it, it was fine. And they're like, “oh yeah, it works for us.”
The other was, and, and this is was our better guess, is, their server that was serving this was on their network and actually sending it much more quickly. And they were just like, “well, it works on our network, it works totally fine.” And I don't know exactly what it was, but I think that is, the reason that's so maddening and it gets back to a little bit like Jen's point about accountability…
The truth is we can't fix problems that we don't see or don't recognize. And I do think that a big part of all of this is we have some expectations that websites should work, that technology should work. We also need to give more. We have to set an expectation, I think government wide, about what stuff working looks like. Like Jen gave an implicit one of a normal human should be able to understand this, that should be tested on every system.
In the e-commerce world, what's the number one metric you care about? It's from start to finish. How many people start a transaction and actually end up purchasing that conversion rate? That would be a really useful feedback loop to measure the success rate of users trying to do a transaction across every transaction in government, right? People applying for food stamps, people applying for whatever. If you saw massive drop off or massive disparities across systems, at least it would tell you, “oh, there's something we need to fix here.”
So I think there's another layer of all this. And that's my depressing story because eventually it got fixed and I think probably they figured out we were right like a week later, but it took a week, I think that's why in my mind, if we can get to better accountability and give people better, on the government side, better visibility into these issues with possible routes to solving them, they wanna solve them.
But right now we don't have a system that necessarily looks for those issues. And I think that would be a better form of accountability we have relative to, say, “did you follow every checklist, bullet point in policy?” Like, if we're getting bad outcomes while following that, then maybe something else needs to change and maybe there's a different form of accountability necessary there.
Jennifer: (56:44)
We should give a quick shout out to the President Biden's customer experience executive order that is trying to get agencies moving in that direction.
Joe: (56:56)
There is so much to talk about in so many different sub-versionss of this conversation we have. But that was an amazing conversation. Jennifer Pahlka, Dave Guarino, thank you both so much for our coming on Odd Lots sharing your time and insight and direct experience and all this stuff.
Jennifer: (57:11)
It was a delight. Thanks so much. Thank you.
Dave: (57:13)
Thank you.
Joe: (57:26)
Tracy, that was an extremely illuminating conversation, I found. I mean, it's like easy enough to say like, “oh, the government's bureaucratic and they don't pay enough and that's why they can't build software.” But there were, that was so much richer and more complex than I like appreciated.
Tracy: (57:41)
Right. The emphasis on incentive was really interesting and sort of fits into a lot of the discussions we have here. And also Jen's point about a lot of this emanates from the policy side. It's people trying to tick various boxes that have been set in stone by policy.
I’ve got to say one thing coming out of that though, I feel a newfound appreciation for being a journalist. And actually, you know, when you and I produce an episode or write an article, we sort of send it out in the world and then it's done and we can move on to the next thing. We don't have to go back, spend 10 years and refine it in response to customer feedback and various testing and things like that.
Joe: (58:21)
If we were Wikipedia editors we would have to be continually editing them for years. I thought that was really great. And again, Jen's point that you just brought up, it's this sort of reflection, like the real solution has to emanate from the policy level. I was actually getting drinks recently with a friend of mine who's a software engineer.
And he said there's this phenomenon. He said, it is called Conway's Law, every organization ships its org structure and this then it makes like so much sense, like the software is complicated because US politics is like complicated. 90 years of policy changes and Democrats and Republicans switching who's in power and etc. It's no wonder the ultimate product of that thing is going to not look like, you know, Google.com.
Tracy: (59:05)
Well, also the story of the guy who was like “it takes 25 years to really know how to file a claim.” You can imagine if you spend 25 years developing expertise in this one very specific skillset, you're not really incentivized to start changing that, right? Anytime soon. And so the system just kind of perpetuates itself. There's so many different directions. So many ways you can go with that conversation,
Joe: (59:29)
So many different ways the system is self-perpetuating. I learned a lot from that.
Tracy: (59:33)
Yep. Shall we leave it there?
Joe: (59:35)
Let's leave it there.
You can follow Jennifer Pahlka on Twitter at @pahlkadot and Dave Guarino on at @allafarce.