# Dalton & Michael: Things that don't scale, the software edition

**Author:** Dalton Caldwell, Michael Seibel
**Type:** Video
**URL:** https://www.ycombinator.com/library/Ig-dalton-michael-things-that-don-t-scale-the-software-edition
**YouTube ID:** TCPjk8Tpb5c

---

Dalton Caldwell and Michael Seibel on software hacks that don't scale. Companies discussed include Google, Facebook,
Twitch, and imeem.

## Transcript

we'll get a founder that's like oh how do i like test my product before i launch to make sure it's gonna work and and i always come back and tell the founders the same thing like if you have a house and it's got full of pipes and you know some of the pipes are broken and they're gonna leak you can spend a lot of time trying to check every pipe guess whether it's broken or not and repair it or you can turn the water on and like you'll know like you'll know exactly the work to be done hey this is michael cybel with dalton caldwell today

we're going to talk about what does it mean to do things that don't scale the software edition in this episode we're going to go through a number of great software and product hacks that software companies used to figure out how to make their product work when perhaps they didn't have time to really build the right thing now dalton probably the master of this um is the person we work with a guy named paul buhit who infrinted this term the 9010 solution he always says

something like how can you get 90 of the benefit for 10 of the work always this is what he always puts onto people when they tell them it's really hard to build something or take too long to code it um he'll just always push on this point and you know founders don't love it right would you would you say that's a fair assessment michael um that's a fair assessment yes founders hated it tell the audience why it's worth listening to the guy like why does he have the credibility to say that to

people well pb is the inventor of gmail and as kind of a side project at google he invented something that 1.5 billion people on earth actively use and he literally did it doing things that don't scale so i'll start the story and then please take it over so as i remember it um pb was pissed about the gmail product a google product he was sorry the email product he was using and so

um google had this newsletter um product the first version of gmail he basically figured out how to put his email into this google groups ui and um as he tells the story kind of his eureka moment was when he could start reading his own email in this ui and then from that point on he stopped using his old email client and what i

loved about this is that as he tells the story every email feature that any human would want to use he just started building from that point and so you know he would talk to yc batch and he's like and then i wanted to write an email and so i built writing emails and if you know pb like he could have gone a couple days reading emails without replying at all so like he didn't need writing emails to start i remember him telling the first time he got his like co-worker like literally like his desk mate or something to try

to use it and his destiny is like this thing's pretty good it loads really fast it's really great the only problem is pb it has your email in it and i wanted to have my email and b was like oh okay well i gotta build that a forgotten solution and so then it started spreading throughout google and do you remember when it broke no what happened oh so he told the story where like one day pb came in late to

work which is you know knowing pb every day you know and everyone was looking at him really weird and they're all like a little pissed they got to his desk and someone came over to him was like don't you realize that gmail's been down like all morning and people was like no i just got to work i didn't know and so he's like trying to fix it trying to fix it and then his co-workers see him like grab a screwdriver and go to the server room and it was like they were like oh god why don't we trust pb with our email

like we're totally screwed and i think he figured out like there was a corrupted hard drive and i remember that point of story he was like he says and that day i learned that people really think email's important and it's gotta always work [Laughter] and like perfect cause i think the reason i think the reason he did it man is because he liked to run linux on the desktop and he didn't want to run outlook like the google like suits were trying to get

him to run outlook on windows and he was like i don't really want to win windows but yeah it was the dirtiest hack and as i recall in this you know final part of the story it was hard for him to get google to release it because they were afraid it was going to take up too much hardware and so there was all these there was all these issues where there's a good there was a decent chance i think it never would have been released well this part was that everyone thought gmail's like um invite system was like some cool like growth reality hack yeah like virality hack it's like oh you got access to gmail you got i

think four invites to give someone else and these are like precious commodities and it was it was another product it was just another version of things that don't scale they didn't have enough um server space so they had to build an invite system yes there was not an option to not but basically there was no option other than building an invite system it was not like genius pm growth hacking it was like yeah well we saturated this the hard drives are full so i guess we can't invite anyone else in gmail today

that's it that's it so you had another story about facebook early days that is similar in this light so so let me paint the picture um back when you started a startup a long time ago you had to buy servers and put them in a data center which is a special room that's air conditioned that just has other servers in it and you plug them in and they're it's they have fast internet access and so being a startup founder until aws took off

part of the job was to drive to the suburbs or whatever drive to some data center which is an anonymous warehouse building somewhere go in there and like plug things in and what was funny is when your site crashed it wasn't just depressing that your site crashed it actually until getting in your car like part of being a startup founder was waking up at 2am and getting in your car and driving to like santa clara because your code wedged the you have to

physically reboot the server and your site was down until you physically rebuild to the super so i'm just trying to set the stage for people so this was this was what our life was like okay and so my company i meme we had a data center in santa clara and there was a bunch of other startups there as well and so something that i like to do was to look at who my neighbors were so to speak there was never people there was just their servers and there'd be a label at the top of the rack and you could see their servers and you can see the lights blinking on the switch okay so this is what i was like and so

uh our company was in the data center in this data center in santa clara and then one day there's a new tenant oh a new neighbor so i look at it and the label at the top of the cage next to ours you know three feet away the label said the facebook.com and i remember being like oh yeah i've heard of this like cool like sounds good and they had these super janky servers i think there was maybe eight of them um when they first moved in and they were like super cheap they're like super micro servers um you know like the wires were hanging out

like it did you know i'm like cool but the the lights were blinking really fast okay and so what i remember was that there was labels on every server and the labels were the name of a university and so at the time one of them one of the servers was named stanford one of them was named harvard you know like and it made sense because i was familiar with the facebook product at the time which was like a college social network that was at like eight colleges okay so then i watched every time we would go back to the data center they would have more servers in the rack with more

colleges and it became increasingly obvious to me that the way they scaled facebook was to have a completely separate php instance running for every school that they copy and pasted the go the code to they would have a separate my sequel server for every school and they would have like a memcache instance for every school and so you'd see like the university of oklahoma you know might you'd see the three servers next to each other and the way that they managed to scale facebook was to just keep buying these crappy

servers they would launch each school and it would only talk to a single school database and they never had to worry about scaling a database across all the schools at once because again at the time hardware was bad okay my sequel was bad like the technology was not great if they had to scale a single database a single user's table to hundreds of millions of people would have been impossible and so their hack was the 9010 solution like pve used for gmail which is like just don't do it and so at the time if you were like a harvard student and you wanted to log in

you would it was hard-coded to the url was harvard.thefacebook.com right then like and so if you try to go to stanford.facebook.com it'd be like you know error like that was just a separate database and so then they wrote code so you could bounce between schools and it actually took them years to build a global user's table as i recall um and avoid this this hack and so anyway the thing they did they didn't scale is to copy and paste their code a lot and have completely separate database

instances and then talk to each other and i'm sure people that work at facebook today i bet a lot of people don't even know the story but like that's what it took that's the real story behind how you start something big like that versus what it looks like today so in the case of twitch all if not all like most of the examples of this came from this core problem and it's why i tell people to not create a live video site a normal website even a video site on a

normal day will basically have peaks and troughs of star of of traffic and and the largest peaks will be 2 to 4x the steady state traffic so you can engineer your whole product such that if we can support 2 to four x to steady state traffic and our site doesn't go down we're good on a live video product our peaks were 20x now you can't even really test 20x peaks you just experience them and fix what

happens when 20x more people than normally show up on your website because some pop star is streaming something and so two things kind of happened that that were really fun about this so the first hack we had was um if suddenly some famous person was streaming on their channel there'd be a bunch of dynamic things that could load like your username would load up on the page or our channel and the view count would load up and a whole bunch of other things that would basically hit our application servers and destroy them if

a hundred thousand people were trying to request the page at the same time so we actually had a button that could make any page on justin tv a static page all those features would stop working your name wouldn't appear the view cat wouldn't update like literally a static page that loaded our video player and you couldn't touch us we could just cache that static page and as many people as possible want to look at it now to them certain things might not work right [Laughter] but they were watching the video the chat worked because that was different system the video worked that was a

different system and we didn't have to figure out the harder problems until later later actually kyle and emmitt worked together to figure out how to cache parts of the page we'll make other parts of the page dynamic but that happened way way later dude let me give you a quick anecdote yes remember friendster before myspace yeah of course every time you would log in it would calculate how many people were two degrees of separation from you and it would it would fire off on my sql thread where you would log in it would look at your friends and it would calculate your friends and friends and show you a live

number of how big your extended network was and the founders you know john abrams he thought this was like a really important feature i remember talking about it guess what myspace is uh do things that don't scale solution was i mean if they were in your friends list you would say this is in your friends you know so is in your friends list and if it wasn't it'd say so and so is in your extended network there it is that was it that was the feature and so so friendster was like trying to like hire engineers and skill my sequel and their run into like too many threads on linux issues and like updating the

kernels and myspace was like uh so-and-so is your extended network that's our solution anyway carry on that but that's same deal so our second one was um it would always happen with popular streamers or second was if you imagine um if someone is really popular and there's a hundred thousand people want to watch their stream we actually need multiple video servers to serve all of those viewers so we basically propagate the original stream coming from the person streaming across multiple video servers until there was on enough video servers to serve all people who are viewing

the challenges is that we never had a good way of figuring out how many video servers we should propagate the stream to and if a stream would slowly grow in traffic over time we had a little algorithm that could work and like spin up more video servers and be fine but what actually happened was that a major celebrity would announce they were going on and all their fans would descend on that page and so the second they started streaming a hundred thousand people would be requesting the live stream bam video server dies

and so we were trying to figure out solution solution solutions and like how do we how do we model this how do we like there were all kinds of like overly complicated solutions we came up with and then once again kolinema got together and they said well the video system doesn't know how many people are sitting on the website before the video stream before it starts starts trying to start video but the website does all the website has to do is communicate that information to the video system and then it could pre-populate the stream

to as many video servers as they would need to and then turn the stream onto users so what happened now in this setup is that some celebrity would start streaming they would think they were live no one was seeing their stream while we were propagating their stream to all the video servers that are needed and then suddenly the stream would appear for everyone and would look like it worked well and like the delay was a couple seconds it wasn't that bad right but like dirty super dirty but it worked and and honestly that's going to be kind of the

theme of this whole setup right super dirty but it worked you had a couple of these in ime right yeah there was a couple that we had at i mean so one of them um so at the time again like to set the stage the innovation of showing video in a browser without launching real player no one here probably knows what that is but it used to be to launch a video it would launch another application in the browser that sucked and it would like crash your browser and you hated your life okay

so one of the cool innovations that youtube the startup youtube had before it was acquired by google was to play video in flash in the browser that required no external dependencies or just play right in the browser at the time that was like awesome like it was like it was a major product innovation to do that yeah and so we wanted to do that for music at i mean and we were looking at the tools available to do it and we saw all this great tooling to do it for video and so rather than rolling our own tools

that was music specific we just took all of the open source video stuff and hacked the other video code that we had so that every music file played on i mean was actually a video file it's a dot flv back in the day and it was actually a flash video player um and the entire it was basically of we were playing video files that had like a zero bit in the video field and it was just audio and we actually were transcoding uploads

into video files you know i'm saying like the whole the entire thing was was it was a video site with no video i don't know how to explain it um and it works and i do think this is a recurring theme is a lot of the best product decisions or ones made kind of fast and kind of under duress i don't know what that means but it's like when it's like 8 p.m in the office and the site's down you tend to come up with good decisions on this stuff so we had two more at twitch that were really funny um

the first one talking about duress um was our free peering hack so streaming live video is really expensive back then it was really expensive and we were very bad fundraisers that was mostly my fault and so we were always in the situation we didn't have enough money to stream as much video and we had this global audience of people who want to watch content and so we actually hired one of the network ops guys from youtube

who had figured out how to kind of scale a lot of youtube's early usage and he taught us that you could have free peering relationships with different isps around the world and so that you wouldn't have to pay a middleman to say sir video to folks in sweden you can connect to yourself your servers you you go i forgot what they're doing it saves you money and it saves them money that's what they wanted yeah and there were these massive like switches where you could basically like run some wires to the switch and bam you

can connect to the swedish isp now the problem is is that some isps wanted to do this free peering relationship where basically you can send them traffic for free they can send you traffic for free others didn't they they didn't want to do that or like they weren't kind of with it and so i think it was sweden but i don't remember some isp was basically not allowing us to do free peering and we were spending so much money sending video to this country and we're generating no revenue from it's like we couldn't make a dollar on advertising

and so we did is that after 10 minutes of people watching free free live video we just put up a big thing that blocked the video that said your isp is not doing a free peering relationship with us so we can no longer serve you video if you'd like to call to complain here's a phone number and email address and that worked and how fast did it take for that to work i don't remember how fast i just remember it worked and i remember thinking to myself it's almost unbelie like that isp was a

real company like we were like a website in san francisco um and and hey that worked and then the second one was translation so we had this global audience and we would like call these translation companies and we'd ask them like how much would it cost to translate our site into these like 40 different languages and they were like infinite money and we're like we don't have infinite money and so i think we stole the solution from reddit um we were like what happens if we just build a little website where our community translates

everything and so basically it would just like serve up every string in english and it was like served to anyone who came to the site who wasn't from an english-speaking country and was like do you want to volunteer to translate the string in your local language and of course you know people are like well what if they do a bad job translation i was like well the alternative is it's not in their language at all so let's not make the perfect enemy the good and i think we had something where like we would get three different people translated and like match but like that happened later we basically got translation for a whole product for free

um maybe to end because i think this might be the like maybe the funniest of them all tell the google story because i think this one's like the like really like so so look for the facebook story that was firsthand where i personally witnessed the servers with my own eyes so i'm 100 confident that is what happened because it was me right this google story is second hand and so i may get some of the details wrong i apologize in advance but i'll tell you this was related to me by someone that was there all right you ready so

look the original google algorithm was based on a paper that they wrote which you can go read page rank um it worked really well it was a different way to do search okay it worked they always didn't have enough hardware to scale it because remember there was no cloud back then you had to run your own servers and so as the internet grew it was harder and harder to scale google you still with me like there were just more web pages on the internet so it worked great when the web was small but then they kept having

more web pages really fast and so google had to run as fast as they could to just stay in the same place just to run a crawl and re-index the web was like a lot of work and so the way the work of the time is they weren't re-indexing the web in real time constantly they had to do it in one big batch process back in the day okay and so there was some critical point this was probably on the 2001 era again this is secondhand i don't know exactly what it was but there was some critical point

where this big batch process to index the web started failing and it would it would take three weeks to run the batch process it was like the you know reindex web dot sh you know it was like one script that was like you know and it started failing and so they tried to fix the bug and they restarted it and then it failed again and so the story that i heard is that there was some point where for maybe three months maybe four months i don't remember the exact details there was no new index of google

they had stale results so anyone any user of google they didn't know that you know the users didn't know this but a user of google was seeing stale results and no new websites were in the index for quite some time okay and so obviously they were freaking out inside of google um and this was the genesis for them to create mapreduce which they wrote a paper about which was a way to parallelize and break into pieces all the little bits of crawling and re-indexing the web

and um you know hadoop was created off of mapreduce there's a bunch of different software used and i would argue every big internet company now uses the descendants of this particular piece of software and then it was created under duress when google secretly was completely broken for an extended period of time because the web grew too fast but i think this is the most fun part about this story when the index started getting scale stale did google shut down the search engine did you like that's the coolest part like people

just didn't realize they didn't know and did they build this first again in terms of do things they don't scale did they build mapreduce before they had any users no like they basically made it this far by just building a monolithic product and they only dealt with this issue when they had to you know i think this is like such a common thing that comes up when we give startup advice you know we'll get a founder that's like oh how do i like test my product before i launch to make

sure it's going to work and and i always come back and tell the founders the same thing like if you have a house and it's got full of pipes and you know some of the pipes are broken and they're going to leak you can spend a lot of time trying to check every pipe guess whether it's broken or not and repair it or you can turn the water on and like you'll know like you'll know exactly the work to be done when you turn the water on i think people are always surprised that that's basically all startups do is just turn the water on

fix what's broken rinse and repeat and like that's how big companies get built it's never taught that way though right it's always taught like oh somebody had a plan and they wrote it all down and it's like never never and you earned the privilege to work on scalable things by making something people want first you know what i think about sometimes with apple is picture like wozniak hand soldering the original apple computer and like those techniques compared to like whoever it is works in

apple to design the airpods like it's the same company but like wozniak hand soldering is not scalable but you know they earned because that worked they earned the privilege to be able to make airpods now and because google search was so good they earned the privilege to be able to create super scalable stuff like like mapreduce and all these other awesome internal tools they built right yes but if they wouldn't put that stuff first it wouldn't be google man and so to wrap up kind of what i love about

things that don't scale is that it works in the real world right the airbnb founder's taking photos the doordash folks doing deliveries it also works in the software world right like don't make the perfect the enemy of the good just try to figure out any kind of way to give something somebody something that they really want and then solve all the problems that happen afterwards and and you're doing way better all right thanks so much for watching the video
