Mint Shutting Down, Project Catchup

• Intuit is apparently shutting down Mint, pushing folks to Credit Karma 
• Colin's school bus update

• CJ is continuing to build out a comms center for managing customer comms in a scalable way for Craftwork. SAML`
• Colin is working on the conference room booking app,
  • Likely use jumpstart for all stuff you need when you build something new
• CJ talks more about his new side project, Story Fuel 

• Colin is also learning a lot about sending/receiving audio over socket connections
• The ‘Enshittification’ of TikTok Or how, exactly, platforms die." from Cory Doctorow
• The Imminent Enshittification of the Internet from Gary Marcus

CJ: Welcome to build and learn. My name is CJ.

Colin: And I'm Colin, and we are back again this week to talk about what we're working on, what we're learning. And there's a little bit of a kind of upheaval in the finance app space this week. So we're going to chat about that a little bit as well.

CJ: Yeah, I've been, I have been a mint user for a super, super long time. But recently too, I've been frustrated that some of the things don't connect to it and I haven't been able to like manually manage those and so in my head I'm already sort of doing math and

Colin: So yeah, what's the, the news is that Mint is getting shut down by Intuit January 1st, 2024.

CJ: that's so wild. I feel like so many people use that for not, I mean, I don't know. The budgeting in it honestly isn't that good.

Colin: it's not,

CJ: But, like, the, just tracking kind of your overall financial health, like it gives you your credit score and it gives you an attempt at a net worth, like, estimate based on Zillow minus your loans or whatever for real estate, and I know it does like a pretty good job of tracking credit card balances sometimes, so, uh, yeah, I don't know, do you have, yeah, like, It sounds like you've been using this for a long time too.

Colin: yeah, I've been a user. I mean, I think from the very early days, I pulled up some of the dates just cause I was like, okay, how long ago have I been using this for? And, um, I think I probably, have been using it since 2006. So it launched in 2006. And there's some fun things that I want to dig into here, but how long have you been using it? Do you use any other tools?

CJ: It's the one I've been using the longest. I have also tried Personal Capital. And I have also tried for budgeting, we, right now we're using YNAB. But yeah, we've tried a bunch of stuff. Also just like Google, billions of Google spreadsheets. But, um,

Colin: Always a spreadsheet, right? Yeah, I, I use Mint. I use Credit Karma and I think I actually had wondered because Intuit bought both of them, right? Credit Karma got bought. More recently, and I was wondering why they would buy two of these things. And they are pushing folks towards credit karma, which is not a budgeting app. But you know, it kind of like watches your credit score. It watches if you own any, if you own a house, if you own a car, if you have loans, like it kind of gives you a sense of your that high level picture of your financial health. But it'll be interesting to see if Intuit pushes some sort of like credit karma budgeting app or something in the future. But yeah, I think it's a little bit different too in that Mint made money by recommending credit cards to people which is very similar to like NerdWallet or, or some of those sites.

CJ: Yeah, I feel like they, didn't they also like recently do something with taxes too, like you could, you could do your taxes through Mint or something, or is that just like an on ramp to TurboTax, or?

Colin: I think it's yeah, they, like, you can have your single Intuit login would log you into TurboTax, log you into Mint, any of those products, and I think be interesting to see, because I think there's something about how the IRS is rolling out, uh, the ability to do taxes without TurboTax in certain states, states without income tax, so Nevada, Texas, things like that would be good,

CJ: there's gonna be like, yeah, those new self file, like the IRS is gonna provide a free website where you can go and file your own taxes in the same way that you would through a paid service. Yeah, I... Was I mean, as soon as I read about this announcement, I was like, Oh, I wonder if that was related that now they're like the IRS is trying to offer. I don't know. It's so wild to me that the IRS doesn't already have that.

Colin: yeah, well, and the argument there is that Intuit has been lobbying against the IRS. they're being choices for a very long time. So yeah, I'm, this is an interesting timing for me cause I, I'm more of a passive user of Mint. I actually just started using co pilot money, which I really like. It's very, it feels like the linear version of all these tools. Very like sexy looking, but also just like has your investments, has your like debt. And asset ratios and everything in it, you can see your stocks, you can see your portfolios, you can see your budgeting. And I've never really had an actual budgeting practice of going through and categorizing things. So I'm trying that out in copilot. But like you, I have personal capital. I never really log into it. I have lunch money, which I really wanted to support because it's built by a single solo developer and it's an awesome tool, but it really only does the budgeting piece and I kind of want that like bird's eye view of everything. So

CJ: yeah. And you in the past have had like talked about building your own thing, right? Is

Colin: yeah, and I think that's, that's where it gets into like the detail, like it's so hard to do that stuff. Like we saw maybe. co ran into this too. And some of the tools that they're like plaid and modern treasury, it's just like looking at transactions. especially in the world of like Venmo and bank transfers and all this stuff. It's like really hard to auto categorize and do it. You kind of have to do it by hand. And the really interesting story. I just want to throw out some things here. We can chat about them before we talk about what we're working on is that Yodlee was the tech that ran And they, there's a story that we can find the link to where Yodlee was just like a contract, you know, they had a contract with Mint. They made like 4 million or so on that contract before they sold to, into it. And then there was this owner of a domain mint. com that they sold to Mint for equity. And so the owner of the Mint domain made more money than Yodlee did on that acquisition, just because I think it was actually like a, a cap, like a VC that owns the mint. com domain. And so they took Series A stock in exchange for the domain. anD that turned into either a stock and into it or actual dollars. I'm not sure which, but and just looking at these numbers, you know, we talk a lot about Bootstrapping and kind of indie co founding and things like that on, on the show. But they had, I guess, quote unquote, only raised 30 million, but like at the time when they were acquired, there were only 35 employees. And I think when you look at startups raising that much money today, you see them having typically a lot more employees. They're raising way more money, which makes it really hard to get acquired in today's market, just because you don't have a lot of. Acquirers who can buy things. And it's like, once you get into even handing like a first time founder, 25 million, it's like, how do you know how to spend that money? It's crazy.

CJ: Yeah, it also is interesting to think about, like, how Mint really didn't have a business model, if I can remember correctly, like, I feel like in the beginning, I was like, how is this free? And how, like, feeling like, oh, they've got to figure it out eventually. And then, After I want to say it was like after years of using it or something, then they started showing like all the credit card ads and trying to get you to buy stuff. Maybe, maybe I just didn't notice that when I first started using it. And maybe that was like part of their thing all along, but I was like, when is this going to not be free anymore? I just, and then the fact that they were able to raise money. I mean, yeah, I don't know. It's kind of surprising,

Colin: Yeah. I mean, they had about 140 million valuation right before they were acquired for 170 million. So they, they went up in value with that acquisition and the CEO worked it into it for a little bit and then pieced out and it looks like he's doing some AI stuff these days, which is cool. So he's, I haven't seen his name. In years and so it's just interesting to kind of go down that rabbit hole of like end of an era for sure it's cool to see

CJ: Yeah, interesting. Yeah, I guess I'm curious what the listeners are using to track personal finances, and if you have apps or whatever. I, I would love to try a bunch out if Mint is, since Mint is disappearing, so. Yeah, let us know.

Colin: definitely and I think we'll have to do that money episode we keep teasing at some point where we get a little bit more into the tools and tricks and things that we do. I think that'd be fun.

CJ: Yeah. One of the things that surprised me about Mint was that I could connect my like, Stripe stock account, which is run through Solidus, I think it's Shareworks, Solium, Solium, which is a Morgan Stanley thing, so maybe that's not too, too surprising, but that you could see, The value of your private stock in mint, which was kind of cool. It's like being able to see as that changes and as the valuation changes. And

Colin: Yeah, a little bit less so these days, like everything is just pure red in the stock market.

CJ: yeah, but it gives you a good idea of like. You know, how am I tracking towards, you know financial independence or, you know, how much margin do I really have in my life? Or should I really be like buckling down and getting, as Dave Ramsey would say, gazelle intense about my budgeting and like, you know, rat like, I don't know, eating rice every night or something, trying to thin out the, the lifestyle. But,

Colin: yeah, well, in the similar vein to that, an update from last episode is that I did not drop a bunch of money on a school bus. So I was outbid and I can't remember. I think I like. Emotionally, I just had to set a limit and walk away. And so I put four grand down as my limit and walked away and they sold for four grand, but not to me. So I think that means that somebody had a four grand limit put in before me. And it's okay. Cause I would have had to figure out how to go get the bus and drive it back. Right now. I would probably be doing that right now. I'm happy to not be doing that. So I will keep an eye out for another school bus. We'll see if, if I'm still interested when another bus comes up, then I feel like it's going to be another bit. Cause you know, it's a good, it's a good test of like, is this just a fixation of mine right now? Or is it something that I really want to do? And I did have to figure out like, In another, yet another spreadsheet do I, where do I store it? Where do I work on it? How and when do I find the time to do all that stuff? So I have some more time to figure that out. Can make a better informed decision, but

CJ: Nice. Yeah. Speaking of utility vehicles, I've had this idea. This is sort of like a, I don't know. If this started when I was burnt out when I was at Stripe or like when this onset happened But I have this weird desire to start like a leaf A leaf blowing company that like comes to your house And we'll just I don't want to like do the leaf blowing into the pile I just want to come over and like use the vacuum thing that like sucks up all the leaves into the truck and then drives away And so it'd be like an uber like some sort of like uber for a leaf pickup

Colin: I feel like this is a, the next craft work service.

CJ: Yeah, it could be. It definitely is like a Kraftwerk add on for sure.

Colin: Mike, Mike's already got you thinking about the future, future house things.

CJ: yeah, I mean, I, the It crossed my mind that a school bus would be a good solution for this because you can get like a towable leaf vacuum for like two grand and then you just like open up the back window and you have it all blow in there and Then you have some system for I don't know compacting and

Colin: You just make like a really, or like if you've seen a wood shops, they have the really big dust collectors. Right. And so you just fill up the entire school bus with leaves.

CJ: Yes, exactly. Yeah. And

Colin: Very brandable.

CJ: yeah, like, Oh, what's the, what is the by product? How could we sell like the leaves somehow as, you know, maybe somehow reprocess them into pellets that then go into pellet stoves or reprocess them into some sort of like mulch. Yeah. Compost. Yeah. Some other things. So

Colin: I'd be a stinky bus,

CJ: Right. Those like feed the bus or those, yeah. Like

Colin: biodiesel for the bus

CJ: Yeah. Yeah. So anyways, nerding out about whatever.

Colin: Anything we can do to get away from a computer these days. I think,

CJ: Yep. Yep. And also it, I only think about it in fall. Like right now we are literally making several dump runs per week with the truck full of leaves because I don't know. In New England there's, there are a lot of leaves.

Colin: definitely. All right. Well, there's, there's the school bus update. We'll get back into it. Maybe you'll have a school bus one day. We'll just have a couple of schoolies. But what are you what are you working on in front of the computer this

CJ: Yeah, we got a couple things going on. At Kraftwerk, we're still building out this comm center for customer customer facing comms, and I've been integrating with Twilio Flex. Which is like this drop in UI that lets you handle incoming phone calls. You can make outgoing phone calls with a little dial pad. You can do incoming texts, outgoing texts, and then also like a web chat widget all through their UI. And when I was testing it out and sort of validating this idea, I was like, Oh yeah, this is going to work great. And then we don't have to build all these different things ourselves. And at least for the desktop experience where agents can like mark themselves as available and then they can have these inbound support or sales requests routed to them. I thought this is a pretty sweet way to avoid a bunch of dev work. And so I went to go implement it for realsies and realized that in the dev experience, you can easily log into the console as an admin just by like offing into the Twilio admin interface. But in order to log in as like agents in real life, I wanted to have a single sign on experience so they didn't have to like go to Twilio and create yet another account. And so I started learning SAML, which is like. This SSO protocol that you can implement. It's what, you know, if you are not logging in with OAuth, but you're logging in with SSO, meaning you just open some corporate website that is probably somehow tangentially related to your big corporate company and it automatically logs you in, you don't have to put in anything like not even your like Google. Like. Email and password, then that's probably happening through SAML. It turns out there are a couple of different libraries in Rails for setting up your Rails application as either a SAML identity provider or a SAML service provider. And so in this case, Twilio is the service provider. So we're implementing like the identity provider side of that, which is, it's interesting. There's like a lot of new off concepts that I'm trying to pick up and learn. And all these different things about XML attestations and how you sign this thing or sign that thing. And there's instead of one URL, there's like 10 different or more like five

Colin: Sounds like OAuth to, to nth degree.

CJ: Yeah, it's sort of like OAuth. If it's like, I don't know, I've been trying to think of it as like OAuth and OpenID and JWT

Colin: Well, and you have like a directory usually too. So it's like with OAuth, it's like, oh, anyone with a Twitter account can OAuth. But now it's like, no, if you're not in this directory or LDAP or whatever it is, you're not allowed to, to OAuth.

CJ: exactly. And then it's also like, what roles do they have? And like, what are all these other like, really particular details that the service provider wants. So on the identity provider side, you're supposed to like, try to be as flexible as you need to be to work with all the service providers. By implementing like attributes that they all require. So maybe some of them need an email and other ones need a user ID and other ones need you know, their title at the company or the department they work in or whatever. So yeah, kind of digging through that, but that's been interesting to learn about for sure. So.

Colin: Yeah, we use, I think a lot of startups and companies use Okta. I don't know if Okta is like, is it technically SAML under the hood? It's probably a whole bunch of different flavors of different things.

CJ: Yeah, I think Okta, like Okta and Auth0 and a lot of these folks provide, like they are both IDP and service provider and like an OAuth platform

Colin: They're a bridge from OAuth to the providers for SAML. Yeah.

CJ: yeah. So Okta, they do have like tons of documentation though for testing and learning about SAML, which is pretty cool. So, uh, anyways, I didn't find any content on YouTube about this. So this is like, as soon as I figure this shit out, there's going to be some content because

Colin: for you. Yeah.

CJ: using, yeah. So. Working on that and yeah, what, what, what's going on with the, the conference room thing? How's the conference room booking software going for

Colin: Yeah. It's going good. I started, I just had like a lightning bolt the other night in my brain where I was like, I just need to work on this. And so I'm in that accountability group and we've got two more meetings, I think. So I got. A lot of the Google Calendar stuff working in React and Node, and I used Tailwind to do a little UI for it and I'm not doing, I'm doing like the iPad screen right now, I've just like, this room's either green or red, it's available or not available, and then you can tap on a button that will, Anonymous, like if you tap on it, it'll just book it for whoever tapped on it. There's no sense of who logged in and did it. We don't really need to know that. But we want to have the ability for like you to log in and book a room for tomorrow or for me to log in and book one for next week. And I'm starting to like, look at this node and react app. And I'm like, I do not want to do all this stuff from scratch. And so I think I'm going to move to jumpstart. Move it to rails because I don't want to build all that stuff. And then I think I was going to use react native, but I think I'd rather use turbo native and just play around with that to be able to handle the iPad. And then eventually like Android or whatever, if someone really wanted that, we have the iPads already. So it would just be, you know, an iPad app. I was planning on just doing it as a website for now too. Like it doesn't need to be fully native though. The Robin one has some cool hooks where it knows when our hours are and on the weekends and stuff, it actually dims the iPads. And so it hooks into the actual stuff. So I'm not sure if TurboNative gives us that, or if we have to like create a function in, in, you know, Swift or Objective C that TurboNative can call to do that stuff. Those are obviously like scope creep. To hell for sure. Like I don't need to have auto dimming iPads for this. So my goal for my presentation, my show and tell will be just that the iPad app turns from green to red when people have, when the room is actually booked. And there's like a little bit of admin, like you need to be able to create a few conference rooms and assign iPads to those rooms. And then that way each iPad knows which one it is. So that it knows like, okay, this is downstairs. This is the calendar that's associated. So you need to have the OAuth, like an admin OAuth in and give access to the calendars. And so you have this room, this calendar, this iPad.

CJ: Mm hmm. Sweet.

Colin: yeah, that's kind of where we're at right now. And we get the, we get to have the obligatory Chris Oliver shout out that way. We get to mention Jumpstart.

CJ: Yeah. Yep. Totally. Yeah, I jumpstart, I don't know, I've started leaning on it a ton for things. So it, yeah, I, I'm, I'm loving it.

Colin: If I stick the node way, it's like, okay, do I want to go figure out which ORM I'm going to use and build out my models? And no, I just want. To have user logins and all of that.

CJ: right. Yeah, I think the closest I've seen to something that was productive and kind of like got you started on the JavaScript side was remix, like the remix experience where they, and they have these things called remix stacks where I think Kent C Dodds maybe has like a repo or something, or maybe it's on the remix repo where you can say like, Oh, I want to create a new remix app from this stack. And it will have like auth already built for you. And it will tell you this, you're going to use Prisma and you're going to use fly to deploy and you're going to use whatever, and then you kind of just follow that path. But even with that sort of kickstart, I still didn't feel like it was as productive as you can be with Rails. And so

Colin: It's tricky because if, if all that stuff's been done for you, you need to figure out how they did it for you. Right. And in rails, it's like, okay, I know these patterns and I know which drawers to look in and JavaScript. You're like, Oh, that's an opinion way to do this. And I guess I'm going to now have this opinion if I don't want to swim upstream against all of the things that they recommend. So.

CJ: Right. And it's right now, I don't think remix is like the number one most popular thing. And so who knows if that will end up being supported long term and, or is next JS going to just continue to steamroll everything that happens over there. And

Colin: Right. Well, and I don't really get to write Ruby at my current job, so it'll be good to get back into that world just to stay connected to Rails and stay connected to Ruby because I am, I'm eyeing RailsConf for next year. Just as a, probably as an attendee, maybe I'll think of some talks to submit, but I'm already, I think two for two on the last two RailsConfs for doing a talk. So,

CJ: nice, nice. Yeah. Well, yeah, I don't know. It just, it also just, yeah, will make you happier if you get to write

Colin: yeah, totally. How's the story feel going?

CJ: Oh yeah. So I don't

Colin: know if we, I don't know if we used the name last

CJ: Yeah, we didn't use the name last time. So story fuel is a little side project I'm hacking on. The idea in the beginning was, Oh, I want to give I'm going to author a story, a fiction story, and I want to give it just like a little bit of a, you know, genre audience, short description, and then have it spit out like a really good starting point for me. And then use that to go do stuff. And what it's sort of morphed into is that I made it so that you can give it those things and it uses a bunch of open AI endpoints and it will write a story and you can sort of, uh, guide it a little bit, but it will write a story for you. And then I'm using 11 labs. So I'll convert the story into audio and then I'm using FFmpeg to like combine and concatenate all these audios into like one long audio. And then I use Descript and images from Dali to cut like a little video together. So it's like it's kind of like a little audio book with. The captions kind of flashing at you the descript way and I I'm going to start publishing all of the episodes on, uh, on like a podcast story fuel podcast. So I think the direction it's heading is that you'll be able to dear listener head over to. Some website eventually and suggest a story and then it may or may not get picked up by story fuel and converted into something that you can listen to or watch

Colin: of a cool interactive way, right? If you're subscribed to StoryFuel podcast and you submitted, like you have this, you know, creative community of submitting stories and then like yours gets turned into a story and shows up on the list and that's cool. Are you going to have a three, three books per day limit like Amazon?

CJ: So I, what I found was that it doesn't generate enough content for it to be like an audio book or like a, an actual book that you could like, yeah. So you still would need to like heavily, heavily edit it. Right now it, it comes out to like between 20 and 30 minutes of audio. Which is like perfect for a podcast, right? Like you want a podcast that's kind of bite sized. And so what I've been doing lately is sort of asking it to tell me stories that I'm interested in. So I'm like, tell me about a dystopian future where AI is taking over and like some hacker has to save the world or whatever, and then it spits out a bunch of stuff that is, I think is like kind of interesting. And so Yeah, it also like will develop the characters. So like one by one, it'll go through and add character traits and detail and all this stuff. And then I use those as input into the Dolly output in generation. So I'm like, Oh my gosh, like that's totally this character. You know, that's Nikki right there. Like, look at that. That's a wild

Colin: What'd you think of 11 labs,

CJ: Oh my gosh, it's really good. I've been, so 11 labs. Takes text and converts it into audio for you. Now, there are a lot of text to audio tools that you can use, but what 11 labs has surprised me with is that depending on the context. Of the, of the text, it will use different emphasis and it's not the same every time and it's definitely like kind of, I don't know if stable diffusion, it's like some sort of thing that they're trying to figure out, like, how would this reader read this sentence in this context, not just like, how would they say this specific word or phrase? And so it comes out a lot less robotic, I would say, and they have in the same way that. If you've played around with any of like the GPT playground or models, how you can control the temperature, which makes it like more or less creative or whatever, or like kind of more or less off the beaten path, you can control this like stability and some other parameters that go into the 11 labs prompt, and then it'll come out like more or less sort of emphasis on its own sort of accents and syllables and things yeah. So, yeah, it's been pretty sick. It's also been really fun to learn ffmpeg, like more of ffmpeg. It's such a, that is such a powerful little piece of software. It's just mind blowing what you can do with it. So my next little task that I have in mind for that is like, take the images that were generated from dolly. And use FFmpeg to cut a video instead of having to do it manually in Descript. sO the idea is like, oh, it's going to generate the story arc and all of the chapters of the story and all the content for the chapters and all the characters and all the places and then it'll generate images for the characters and the places and it'll keep track of their memories and then it'll, you know, generate this audio and then ultimately cut it all together and splice it into something that I want to watch. And the pipe dream and what I've been telling people is that like, in the future, I don't think people are going to be able to sell like generated content that is like, I'm not gonna generate like a book and sell it on Amazon. Instead, you'll build a platform where someone comes and says, I want to hear a story or see a movie about X, and that is going to be generated for you. Like in real time that and it's going to be one for one, you experience it one time and then it's gone forever. And it's right around the corner. And the fact that I'm just little old me hacking around, like a couple of tools on the weekend and able to put together enough that like, You can get pretty good fidelity or like pretty interesting stories just out of whatever open AI. So

Colin: Yeah, there's, there's this big wave of, I think Cory Doctorow's book, and he's making waves right now on the inshittification of the internet. And what I'm really curious to see is whether or not, like, I'm not sure if you, have you played around with asking it to do stories that are based on a theme of an existing story or anything like that?

CJ: I, I haven't just because I want to be able to like publish them and I don't want it to.

Colin: Be like a new hope.

CJ: Yeah, exactly. My kids are my kids. On the other hand, when we're playing around with it, they're like, do a one that's yeah, do one that's based on on Star Wars or like make the character based on Han Solo or do one about Pokemon. I'm like, no, like, I don't want to take anyone's IP. Like, I want it to be just some generic storyline that will be interesting. And I want new characters that I've never seen before and whatever. But,

Colin: But the, the thing, the thing there is when you're talking to it about making stories, like, it is basing it on what it knows about. Right. And so over time, do we get this general, like, reversion to the mean story, right, where it's like, it's now being trained on other AI stories, and it's garbage in garbage out. And I think that this idea of prompt engineering or storytelling is pretty powerful. Because if you can Okay. Have an idea for a story and then use, like when you think of like the Iron Man movies where he's kind of like riffing and having Jarvis pull in things. But he's still very much in the cockpit and very much pulling it all together. And it's pretty powerful because I think there's still gonna be that storytelling element and still having that un or we end up with a bunch of things that end up being the same. So we'll see how that pans out.

CJ: totally. So I reached out to Sam Bafa from that like works in the Reno Collective about his choose your own adventure platform. I don't know, I haven't talked to him in years, or and I don't have any updates on that. But I was like, Oh, another way that this could go is, as you're experiencing the story, whether it's in you're reading the written word, or you're listening, It would be sick if like you get to some point and you're like, actually, I don't like how this is going, just mix it up. And like from that point forward, the story is different. You know, like it can just like pick a different conflict or pick a different

Colin: Or choose your own adventure, right?

CJ: Exactly. It is literally

Colin: you a

CJ: own adventure. Yes, yes. And that's a, that's like totally another thing too is like when you, the way that it works right now is you give it this, you give it those like little pieces or whatever, and then it generates a story arc and some characters, and you have an opportunity to like edit the characters or edit the story arc at that point. And then you say, okay, now generate like a list of chapters that might happen within the story arc. And so then you can edit those chapters a little bit if you want. And then you say, generate, now generate the content for each of those chapters. And then you, if you want to do like the painstaking work of going in and editing all the content, you can do that. But yeah, so one, one thing is like, do a choose your own adventure version. The other thing was I, I thought about doing like a rap genius, like a rap genius style thing where like, In order to edit it, you sort of select a chunk of the text that you want to change and then give it a prompt that's like, make this more dialogue or, you know, you know, make this an alliteration or change this into like a joke or something. I don't know.

Colin: Yeah, did you, did you ever watch the Black Mirror Bandersnatch movie? It's,

CJ: no, I haven't.

Colin: it's a, it's a, it's not AI that I know of, but it lets you choose options as you're watching it. So like the, the show, you're watching the show or the movie and it stops and asks you like which one you would do.

CJ: okay. Okay.

Colin: is different based on it, but it's, it's not AI generated. And I think there's like a certain number of possible outcomes and story branches and stuff. So it's like playing a video game or like you know, Knights of the Old Republic style type thing.

CJ: Yeah, exactly. And in fact, like when I was thinking about doing the Choose Your Own Adventure version of this, My first thought was to use Dragon Ruby and like make it an interactive thing where you're kind of like, I don't know, maybe you become the protagonist and you have your little avatar that was generated by Dolly and you're just standing in a dark room type situation and it's reading off the text and then you can be like, no, I don't want to, or like, you know, more like, or kind of like a one player D& D situation where you're kind of like, Oh, I want to, I want this to happen. And you can type out, like, I want this to happen and then boom, the story changes. But what's interesting is like, I want to keep all of those characters memories. So I'm storing like a bunch of contexts in different tables in this giant Postgres database. And the idea is that I can use sort of use embeddings. With each of the memories. So then I can do like a vector search on all the memories and then embed the relative memories into the context when generating like the next section of the story. So I don't know.

Colin: Wow.

CJ: It's fun. It's like super, super nerdy. Like,

Colin: I was going to say, I need to sit down and learn more about embeddings and vectors and all this stuff. It's, that's pretty cool.

CJ: it's fun. It's super fun. And the tools are getting, I don't know, I feel like For a long time. I tried really hard. I was like, what is it? All this machine learning stuff. I really want to learn it and I don't get it. And a lot of the content I found was like how to implement and train your own models and whatever. And like, it was way over my head and it still is like all the mathy, like how the models actually work just. Escaped me. And to this day, if I watch anything about training or, you know, VLLM or like the, the new high speed stuff that makes it faster or whatever, I'm still just like, how do people figure this out? I don't get it. But when I'm using it and applying it like in actual applications, I'm like, holy smokes, like this is, this is fun. Like, it's pretty cool. So I don't know. It's

Colin: sounds like something you're working on and learning. It's, it's a

CJ: yes. Yeah.

Colin: we're always learning.

CJ: Yeah. Always learning. Always learning. Speaking of learning. Yeah. , you're doing some deep stuff over there with is that audio? Mm-Hmm.

Colin: it doesn't feel too deep when it doesn't work. But kind of shifting back to my actual job. We've been trying to review our PRs. We have like our, our docs are public and so we take PRs to the docs and I'm trying to close out some PRs from 2022 so that we can get a clean slate of 2023 PRs. We're down to like less than 40 PRs open, so we're going to try to get through those. But we, one of them was around this voice connections. And so if you've used Discord, you know, you can And those bots can easily send texts like texts. We can test all day long. I can look at it and be like, yeah, that coach would work. But when it comes to a bot joining, so a bot can join a voice channel and you can do all kinds of stuff with that. You can have it play a sound, you can have it do all sorts of things like that. You can think of like a lot of the zoom apps and bots and Slack apps and things like that out there is that you can have. You know a soundboard for instance, which is also a feature of discord now, but before we had that, you could do it as a bot or an app. And so I couldn't figure out whether or not I could even merge the PR without sitting down and actually doing this. And it's pretty easy to do if you use. Like one of the third party libraries, because a lot of the libraries have built around all the basic tenets of this. But when I try to build our bots, I try to use just pure JavaScript or pure Ruby or pure Python, whatever it might be. So that I really understand like what's happening. Cause in our docs, we're not documenting libraries, we're documenting. The APIs that the libraries are using. And so I've been working on sending and receiving audio over like through discord. And so what I found is it's very hard to debug because you're working with packets and buffers and like you can't just console log or use a debugger and say like when you've chopped up audio files into tiny pieces, you can't listen back to it. And know, like, is it right? And when you send it to discord, you have to also encrypt it. And so once it's encrypted, I'm like, I have no idea. And I was able to, like, I started looking around to see if there's like anything around debugging UDP and audio like tools, and I didn't really find anything. There's a dev tool company called deep gram that does. Text to speech. And I think both the other way and they accept UDP stuff. So I was starting to play with their, like, look at their docs and like play around with their tools, because I was like, if I can get my audio to show up inside of their tool and translate it or like caption it, then I know it works and I know my code's working because with this court, I'm sending it. And I just don't even know if it's just not encrypted, right. Or what's happening, but I did set up like a local. UDP like echo server, and I was able to unencrypted send a little Zelda sound effect from my code to that server and have it output as an mp3. So like I know that part is all working and so I'm probably getting tripped up on encryption. And I'll probably bug some engineers next week about like, tell me what's wrong with my code, do a little code review, but.

CJ: Is it So encryption is required or? Mm-Hmm.

Colin: So there's this handshake that happens. So you have to connect to our WebSocket gateway. And then from that WebSocket gateway, you then establish, like when you join a voice channel, you establish a UDP socket. And so you're managing all these states and sockets and reconnects and heartbeats and all this stuff. And so I was able to recreate all of that, which is good because I've never done any of that before in our, in Discord specifically. And so just knowing like how the handshake works and what events are sent and being able to see it so that when other people open other PRs, we can actually like have this little playground to verify, like, that's how it works. That's how it's working as documented or no, that's right. It's, it's broken or whatever that might be. But in that handshake, we give you a key to use for, and different types of. Like hash like modes so that you have, I think there's three different ciphers that it could be and then a type of key. And so you use that to send along in your headers and you do that encryption on your end so that way you're not just sending through on encrypted voice.

CJ: got it. Dang. That sounds interesting.

Colin: Yeah.

CJ: Yeah. Are

Colin: ideally, I mean, I don't know if I'm, I'm hoping we can get it done so that like maybe we have our own voice SDK or something at some point, because sometimes you just want the voice SDK or it's like to have to like dig into this low level stuff, um, doesn't make a lot of sense. And maybe an SDK from us can also be used as a reference for people to implement their own SDKs or whatever that might look like. So,

CJ: Yeah. Are you writing it with Ruby or

Colin: It's a JavaScript.

CJ: JavaScript? Okay, cool. Yeah, the gosh, just working with any kind of sockets is yeah, it's always a little bit harder than I expect once I first get into

Colin: Yeah. Well, and you're getting into also like moving into Opus streams and Aug streams and all these different like audio formats that we don't necessarily use on our computers. And so then I have to like export it to either Aug or mp3 so I can listen to it. And then I could, what was nice is that I'll be like, okay, export this. And it's like, yeah, this file's corrupt. You're like, okay, something's not right.

CJ: Right. Yeah. Interesting. Yeah. One thing that was kind of Tricky about the 11 labs integration, not necessarily tricky, but like, eh, I don't think they have a Ruby library. And so they just, you give them like an HTTP request with text, and then they just dump back like the audio in binary. And so, or whatever, like it comes back as just like, you know, unreadable blob of whatever. And so then I was like, I don't know, let's see if we write this to a file. Like if that ends up working and it just magically was like, cool. And then I was like, okay, now let's upload it to active storage. Boom. It just kind of like that part. I was surprised at how, how sort of a straightforward that part was, but,

Colin: I mean, You're not far from being able to take that story, getting the binary, and then you can broadcast it out to ice cast or something. And now you've got live stream.

CJ: yeah, gosh, that's interesting. Oh man. Okay. Yeah. But yeah, along that path, I was, I was similarly like, how the hell do you debug this? Right? Like you're getting back something and like, you just have to know that, oh, that's an MP3 file that I need to store somewhere.

Colin: I have been using chat GPT to try to debug some of this stuff. And I mean, it was kind of cool how normally we're used to getting texts back and there's these things called RTP headers that you have to add to the front of your packets when you send the audio. Like, it, like created this like visual representation of what an RTP header was. And I, I was mostly using it to be like, okay, I don't understand what this header's for, what is it comprised of? You're setting a lot of different bits and things. And so I was like, explain this to me. Okay, explain it to me like I'm 10. Explain it to me like I've never written code before. And it was pretty useful for that. You know, you still gotta check your homework and say like, 'cause half the time it's like, oh, you need this. And then I'm like, the docs say we don't need this. And it's like, oh, I'm sorry. Right. And thankfully these docs, I think, were, have not changed since before the cutoff window. Which I think we talked about on the show where the cutoff window was gone and I still see a cutoff window. So I don't know how that works to you.

CJ: I think you have to enable Bing search maybe or something like in, in like the pro GPT four dropdown, there's like four options now or something or they have, I don't know if you have to enable them or plug plugins in or something, but there's like the Bing search thing. I think that's part of it, but they also have like different models that have cutoffs at different dates. But man, I would say over the last two weeks, it's been less valuable. Like, I feel like the quality of the responses I've gotten over the last, definitely the last week is way lower than it was before that. And I don't know if they changed something, but I'm like, this response that you just said makes no sense at all. And it is not going to be,

Colin: of the internet. We should see if Cory Doctorow wants to come on our show.

CJ: yeah, totally.

Colin: If anyone knows Cory, hit us up. We should reach out. I mean, he's on a podcast tour right now talking about this and it's top of mind for sure.

CJ: let's do it. Should we wrap there?

Colin: I think that's, A full featured podcast.

CJ: There you go.

Colin: This was not generated automatically by Eleven Labs. This is, these are, we confirm humanity.

CJ: Yeah, that's this is our watermark of H generated, not C generated or something.

Colin: all of our, all of our quips and, and weird vocal tics and all, all, you get it all. So,

CJ: Yes. Yes. All right. As always, you can head over to buildandlearn. dev and check out the show notes that'll have links to all the resources and things we chatted about.

Colin: we'll see you next time.

CJ: See you next time. Bye friends. All audio, artwork, episode descriptions and notes are property of CJ Avilla, Colin Loretz, for Build and Learn, and published with permission by Transistor, Inc. Broadcast by