What We Didn't Predict, Can Still Hurt Us, An Interview with Internet Hall of Famer Paul Vixie
Well, thanks for coming on. It's fun to ... I mean, you're literally a legend, right? It's a hall of famer. Not everybody can say they're a hall of famer.
Hall of fame thing, plus three dollars will get you a cup of coffee at Starbucks.
But you have a pretty interesting story in terms of an education background, from dropping out of high school to getting your PhD. For those who are listeners who haven't heard that story, I'd love to hear it again.
Well, so high school for me started in, I don't know, '78 or so, and computers at that time were bigger, more expensive, slower, and less plentiful than they are now. So, in order to get access to one, you had to go to where they were and work out some kind of a deal with whoever owned that computer, usually lab tutor. I did that instead of homework, and was a really horrible student. Anyway, in 1980, my counselor told me that I was going to be a junior again for the upcoming school year, and I decided that the trend was not to my liking. The way I was earning a living at the time was not computers, I was earning a living pumping gas. I'd always envisioned myself graduating to someday being a tow truck driver, but I thought, "Alright, before I go check that out, I gotta see if this computer this is able to pay." It turns out, being a computer guy paid a lot better than driving a tow truck, so I did that.
Years later, I did start a PhD program at Keio University in Japan, and it took seven or eight years for me to get that done. At the same time, we were having new babies, I was starting companies. I had a lot of other stuff going on. Then the year after that, I was inducted into the Internet Hall of Fame, largely because of all I did that made my PhD take so long.
Well, when you finally got it, it seems like it went pretty well.
It's a funny path, and not one that I could repeat or that I'd recommend, but it's mine.
Yeah. It sounds like you've sort of navigated that practitioner to academic sort of leading research kind of world in a really interesting way. Having sat in a number of your talks, it seems like you're always having conversations at a global level, right? Sort of trying to push the community to do things. How did that happen? How did you get pulled to that part of the space?
Wow. All my life I wake up angry about something. If you do that, and you do it pretty much at scale, which is to say every day, eventually you start to want to know not so much why I'm angry about this, but where did this come from? Does it have any ancestors in common to the thing pissing me off yesterday? What that'll lead you to if you follow it, and I didn't have a choice, is to realize that the thing we live on is round, and no solution that is only going to help one part of that round thing is going to last long enough, or be sustainable, or make any real difference in human history.
As a result, I am one of the 10% of Americans who has a passport. I have traveled, I've seen how other people live, I've seen that not everything we do is the best way it can be done, and I have tried to apply some hierarchy to the things to be pissed off about, and there's just no way to do that without a global view.
What you're saying is you're pissed off at a global scale?
Yeah. Pretty much.
Yeah, I mean, you know, it's interesting I think. The internet come out of this sort of community of people who knew each other, and seems like it ... It was before I was born, so you'll tell me, probably, but there was sort of a collaborative spirit. It was a small community, and then this thing grew to a scale that I think maybe few of the people who originally came up with it ever could imagine. But all the sort of unintended consequences, sort of unanticipated issues that have arisen across that spectrum, and that initial everyone is here in this together community spirit, maybe some of the later people don't quite have that same communal ethos.
“ Even the people who did do well financially, you know, started companies that later went public or sold for a lot of money, the thing they were passionate about was changing the way communication worked, and making a lot of money was just a really beneficial side effect, as opposed to being the point in and of itself.”
I agree that a lot of the latecomers right now are sort of concerned about trying to cash in on something before whatever it is collapses. That's very extractive thinking, and that was not prevalent in the early days. Even the people who did do well financially, you know, started companies that later went public or sold for a lot of money, the thing they were passionate about was changing the way communication worked, and making a lot of money was just a really beneficial side effect, as opposed to being the point in and of itself.
Now, for those of you born later, it's maybe hard for you to understand that within the generation of people working on this kind of thing in the 80s, some of those were quite junior. I, for example. When I came to work at the Digital Equipment Corporation Western Research Lab in 1988, I was 25. I had always been a very big fish in whatever small pond I was in, and so I thought very highly of myself. I went to work in this lab that had 30 scientists, you know, PhD people, and I was very much the junior man on that totem pole, as low as you could go.I learned a great deal about what a real pond looks like, instead of the small pond I'd been swimming in up until then.
Even a few years ago, I was on a panel at a Hackers conference with Marshall Kirk McKusick and Eric Allman, who are two people from the BSD community. The organizers were probably born about when you were, so they think of all of us as BSD people. What I had to point out to the audience is, you know, when I came into this, Sendmail was already a thing, it was already a global presence in the world.I learned a lot about what I know about programming and C by reading Eric's source code and asking him questions, which he was kind enough to answer. And Kirk had written the Fast File System, and that was also as universal at the time. It was in every Sun computer made in the first decade, and all the BSD systems. Again, I learned what I know about file systems by studying his work that was five to ten years before my time.
Yes, at this point, a lot of people are going to lump us together. I always remember coming into this late and really feeling like I had missed the boat, and had to do a lot of work to prove to these people that I was useful. I think there was a lot that went on then that is not necessarily going on now, simply because there are millions of people doing it instead of dozens.
Yeah. Yeah, the scale of the community just makes it really hard for it to feel maybe like a community sometimes, right?
Yeah.
But, you know, I was bouncing around the internet and came across a LinkedIn article that you had written about that mentorship process, and you had your six questions, right? For those people who hadn't seen that, talk to me about that kind of piece, and how you think about kind of the mentor mentee relationship and whatnot, because it was, at least for me, really interesting.
That article was a little bit scary for me because it was written for management people, not technical people I don't have a lot to say to new technologists, other than to tell them, "Well, don't do this, because that worked poorly for me." So, the idea that I would have something innovative to say in a non-technical way felt like a bit of a phase change..
A lot of times you have people whose interests are theoretically alignable who can't get it done. They end up spending a whole bunch of time circling and misdirecting and just kind of not making progress, either for themselves or whatever organization they're in. So I crafted those six questions. The idea is to really invite somebody to speak plainly and to speak the truth and to create a safe environment for them to say something that maybe has been on their mind. It's not too different from what a lot of married couples learn to do, which is, you know, "Hey, honey. How have I been pissing you off? What is it that you have just given up talking to me about because I was never able to listen?" If we all try that, we'll probably hear something, and that's interesting because it means not so much that you have to dig for it, but that there are some things that you have to dig for If they have a list of things that they hate, and you are willing to give on every one of those points, you could create a lot of marital harmony if you would just talk about it, as opposed to sort of running through our lives with our hair on fire, as we do. So, that's what those questions were designed to do, to open a lane of communication to people who really should be talking and should be listening but, for whatever reason, aren't.
Yeah. I mean, that seems like at least a theme that I've heard in a number of your talks, this sort of, how do we get a community to align themselves around something that's maybe not directly in their own self-interest, sort of a tragedy of the commons problem at cosmic, global scale.
Yes.
Yeah, I mean, where ... Excuse this analogy, but every time I hear you I can't sort of help but think of Don Quixote, right? You seem like you keep tilting at windmills, right?
I do. I do, and I sometimes think of that as character flaw, but sometimes the windmill moves a bit.
Yeah. It's always this industry, so much to talk about the problems and what's broken, right? But maybe not enough attention of what has worked and what is working. Where have you seen maybe a windmill lean or shift just a little bit?
Well, I'm going to answer a slightly different question, which is, what works and what doesn't? I've started a couple of non-profit companies, couple of for-profit companies, some of them have done okay, some of them have been terrible failures. Generally speaking, the time that I launched the anti-spam industry with a company called MAPS, Mail Abuse Prevention System, and some technology called the RBL, which is used pretty much by everybody now. No one who listens to this webcast will ever receive mail that was not, in some way, subject to RBL lookups. If I had been a little bit more foresighted about it, I could have patented that and I'd be the road reflector family. But I didn't, right? I was trying to solve a problem, not make money.
But the problem I was trying to solve was to make spam harder to send, more expensive, less successful, and in that sense, I was building a wall. You may have noticed that you're still getting spam.It's my belief that without the RBL and without the industry that I inspired, you'd be getting more spam. It's possible that if you and everybody else was getting more spam, you'd do something about it. You'd get angry about it, and if we had angry people, torch bearing mobs, as it were, then maybe the problem could be finally solved. So, it may be by building the walls I built when I built them, all I did was to contain the problem.. So, I may have had the opposite impact on history that I planned to.
Now, in other cases, for example, starting the Internet Systems Consortium, which is the non-profit that Bind comes from, or starting the Palo Alto Internet Exchange, which was the first neutral commercial internet exchange. It was the first place that ISPs could come in this country and connect to each other in a neutral house where nobody was getting a circuit revenues from anybody. It was really groundbreaking. Nobody realized how important that would be. We hired the people who later left in disgust and funded Equinix, which has become the big player in that. But in the case of those two companies, I was not building walls, I was building roads. I wasn't making certain things harder, I was making, instead, certain things easier, and the impact of those has been far greater. My lesson, out of all that, is you should build roads, not walls.
That's such a interesting way of approaching a problem. Yeah.
Well, it's interesting, but it's also terrifying, because I actually had to do all of that. It feels now that I could spend a few hours with a pad of paper or a white board and work that out theoretically, and maybe there are people who can, but no, I actually have to go through the whole thing before I will learn the lesson.
Is it, I mean, even at the scale of a community that's millions, billions, right? Billions using it, millions working in it, is it still in many cases handfuls of people who are really kind of moving the needle, or ... When change happens, who's doing it, right? More than the practical, kind of. What's the community and what's that look like when it happens?
“He had a vision of a Unix-like operating system that would be a fairly open license that you wouldn’t have to pay AT&T for and that the community could take the biggest single hand in shaping its future. ”
I think the easiest case study for success in making a difference would be Linus Torvalds. He had a vision of a Unix-like operating system that would be a fairly open license that you wouldn't have to pay AT&T for and that the community could take the biggest single hand in shaping its future. I'm certain that a lot of people laughed when they heard what he wanted to do, but I think you've got at least two versions of Linux running on your body right now, and everybody listening to this, likewise. That's how prominent it's become. That's what a big deal it is.
Again, he was not building walls. He was building roads. He wanted to make certain things possible, and he's won that game. There are other examples where it took a lot of money, maybe, to get something done, and certainly Linux has a lot of money invested in it, but it's being invested by companies who accepted the code base, they accepted the culture, they accepted the license, they hired people from the Linux community to be their executives There are other cases where somebody has said, "I want to have a success like that, but I don't want to do it in a crowd sourced sort of community integrated way. What I want to do is just write a lot of checks and end up owning stuff." You know, maybe that is better than nothing, but look at a couple of the highly commercial versions of Linux, and they've got nothing compared to the market share that Red Hat has, having both a commercial and non-commercial version.
I look at Google as an interesting example. Of course, Alta Vista was really the first search engine of that kind, and that was part of Digital, but Digital Equipment Corporation was completely insane when it came to networking. They didn't understand what they had, so they kind of wrecked Alta Vista. Anyway, Google could not have come about if Linux hadn't come first, because they had to be able to hack on the bare metal and figure out how they were going to do things differently enough that no commercial provider would ever have given them enough access, or whatever. But they didn't have to start by building their own operating system. They could sort of join a community already in progress, and in fact, they've already put huge contributions into that community. They've paid back much more than they benefited. I think that's a better approach, is to figure out a way, and this goes back to what I was saying earlier about alignment of interests. A lot of the people that you compete with are potential fellow travelers, if you're creative.
Yeah. Yeah, and standing on the shoulders of giants, right? You know, and hopefully helping someone else up with you, right? From a historical perspective, I think it's ... Just to change directions ... Interesting sort of the, at least some of the explosive growth of the internet took place kind of in the early 90s and through the 2000s, right? When you sort of had a quieting of the Cold War. The interests from a geopolitical standpoint were maybe in a different place than they are right now. I've heard some people have begun to sort of think about that cyber warfare is becoming essentially another stage of the Cold War, with China and Russia sort of rising as powers and trying to battle with the US for global supremacy. When you think about that, do you get worried about ... When you think of a community that's working together, and how does that ... When you start to have competing factions at that level, what's your sort of ... I realize that's very much out of left field, but ...
I used to think I could change human nature, and I've given up on that. What that means is that any macro historical pattern will reissue. You're going to see old ideas applied to new technology inevitably. If you take a look at not just the United States' recent election, but recent elections in Germany, Ukraine, and what not, it's no longer really possible to trust democracy, because crowds of people can be misinformed through social media, and made to either abandon or adopt positions that are not the ones that they would come to if they were allowed to think peacefully on their own. In that way, the Internet has turned pretty much the whole of the human population into a potentially torch bearing mob, and we don't think clearly. We did that. We, the people who built whatever small piece of the Internet each of us built, made that possible, and we didn't have a plan for how we were going to keep this bad side effect from occurring.
Now, you mentioned the 90s, and I think there's a point that often gets missed because the Clinton family and the Clinton Foundation often takes credit for how great the economy was during their eight years in office. What I want to point out is that in 1993, the National Science Foundation decided to release the Internet. They were no longer going to fund it, they were going to do what was called the Commercialization and Privatization Effort, and it was in full swing by '94, and it was over by '96. I think that is why the 90s was a period of boom growth.
Because we were getting a lot more efficiencies, we were getting a lot more potential relationships, we were virtualizing a lot of things. That was the good part. The bad part is, when I was a kid growing up in San Francisco, you'd read in the newspapers every month or so, some senior citizen had got mugged on social security day, you know, because they got their check, they walk to the bank, and people wait outside the bank. Hey, you're a senior citizen on social security day, I'll bet snatching that purse is going to be profitable for me. So, those people were at those risks.
Now those people are not at the risk because it's much easier to steal that same money through a malware infection and a keylogger or a botnet one nickel at a time, or whatever you're going to, and you can do it from the privacy of your parent's basement, or wherever it is that you live to launch those attacks You don't have to be in the same country. You're not on the scene, you're not at risk of tripping or being chased by a good Samaritan, recognized. There's no risk, and the attack surface is effectively infinite. And we did that. We the people who did our small part to make the Internet possible did all of that without a plan for what the heck we were going to do now.
It seems like you're a student of unintended consequences.
Yes, I am.
How about now? What are you most ... What windmills are you tilting at, and what are you most concerned about?
The natural forces of the human economy and history and human nature are going to mean that this Internet of things deal is going to make all of us even less safe. We will have less privacy. We will have less certainty that the transactions that we are conducting are actual with the counter-party we thought they were, and we're going to put all of this type of electronics into cars and start to have a lot of self-driving cars without having really learned the lesson that all software has bugs. Or at least all software ever written has had bugs. We don't know if maybe some day there will be some that doesn't. But trend line indicates that all software will always have bugs, and that we just don't know what those bugs are at the moment we ship it. That means we're going to kill people with bugs.
When I've talked to people in the self-driving car industry, they say, "Well, right now we're killing people without software. It's humans driving the cars, and humans are incompetent." So when I ask the question, "So, your value proposition here is that you're going to kill fewer people in new ways." And they said, "Well, yeah," because they see the fewer, and I see the new ways. It's like, I am a pedestrian, and all the cars around me are going to have your bugs in them. I am a test subject in your laboratory, and my consent was not sought. Yet, that's how human history progresses. That's what I'm worried about, is this unintended consequences thing is bad by itself, it's worse when you square it, and it's even worse when you cube it, and we're going to cube it.
It sounds like you need to recruit more ... Now, you've been mentoring on the small scale, and probably on the large scale. For those people who are listening to this, reading this, what's the ... Where would you recommend that they go? How would you sort of direct them to follow a path somewhat like yours, and work for the good, at least in some measure?
Well, there isn't so much a path, but there is an attitude. Now, 100 years ago most American families knew where their food was coming from. Even if they weren't growing it themselves, they understood the process that they were outsourcing, and so they knew what was in their food. Now, that's gone away. We have outsourced all of that. Our food is brought to us and we don't really have any more than a fuzzy concept that we got from a movie some time or a story we heard about how that animal was raised and slaughtered or what had to get sprayed on that crop or how much diesel fuel went into bringing it to our table. That is where we go wrong. I'm not saying that we should all continue to grow our own food, but we should continue to investigate where it comes from so that we can be making informed choices about what we do.
If I could get people to investigate what they do and what they're benefiting, and make a decision about what future they would like to aim for, that would be huge. But at the moment, people hear what they want to hear, and they believe what they want to hear, and other people are very good at figuring out what that's going to be and delivering those messages. And a lot of us, sadly, just lap it up and do not think about the way that we are participating in really our own destruction or our own imprisonment.
Yeah. The silent majority, right? Sadly, for good or for evil, right?
People who cannot be bothered to understand what's going on but still vote.
Yeah. Well, this was great. Really fun. Thank you. Appreciate it.
Thanks.
Before we end, here's your soap box. Anything you want to talk about, mention, get people involved in? The floor is yours. Feel free to pitch whatever you've got or want to.
Okay. So, it's a two-parter. You cannot make a digital system, like a computer or a network, safer by making it more complicated. What that means is that, let's say you have a network of some kind and you add some new thing, maybe it's a firewall or some other security related technology. That's got software in it, and all software has bugs, and so there are some risks. You need to have some kind of an inventory about what you have and what versions you're running so that if a bug is found, you know that you need an update. That means that the point at which you write a check or flip out your credit card and get a new thing is not the end of your investment, it's the beginning your investment. If you don't, then the bad guys, who understand your technology a lot better than you do, because that's how their incentives are aligned, are going to take you for a ride. I think security and complexity don't go hand in hand. If you want both, you have to work really hard.
That, I think, is a good lead in to understanding why I'm doing the company I'm doing, Farsight Security. A lot of folks in the security buy side just can't be bothered. They don't have time.They have budget, they want to write a check to be safe. I don't think that's actually possible. I don't think you could do that, so I don't tell people that that's what we do. What I tell them instead is a bit of a tough love story. You have to understand what the threats are. You have to understand what your threat surface looks like, and then when something happens, you have to have enough clues inside your corporate DNA to be able to do an investigation. So, I sell tools and data feeds that facilitate that kind of understanding, that kind of investigation., and I am often criticized by a current customer, or sometimes by a prospect, who says, effectively, "Paul, you're trying to sell me a shovel, and I want to buy a hole." My point is that you probably can't solve your problem by buying holes, and I'm not good at lying to you and telling you that you can.
Yeah. You don't know where to put it. Where do you want me to dig it, right? I have to ask you that every day.
I think that investing in your understanding and in your ability to make enlightened followup actions after an attack, or even before an attack, is the best thing you can do. The people who invest that way might spend more up front, but they certainly save more on the back side.
Yeah. This is awesome. Anything else? Where can people find you if they want to follow up? We can throw links and stuff in the ...
Well, I am, of course, Paul Vixie on LinkedIn, Twitter, and Google, and Facebook. The company is FarsightSecurity.com.
And I am always excited to hear from people, to answer questions, to get involved in arguments about the best way forward. Yeah, pretty much if somebody wants to choose the road less traveled, they should be reaching out to us.
Paul, this is great. Thank you so much.
Thank you.
Great to sit down, literally with a legend.