Building a Redis clone in Go - Part 11 (EXPIRE and TTL)

Ladies and gentlemen, welcome back to this Go Redis Clone Database Thingy series. In the previous parts, we built the... Let me just show you what we did. In the previous parts, we built the RDB checksum protection over here in the saveRDB function. And we also built the auth command to allow us to use password protection to secure our servers. So, that's what we did in the previous parts. In today's video, we're going to build the expire command, right? Expire and the TTL, time to live command. Super excited about this video because these commands are awesome, alright? They allow you to basically delete keys automatically after a certain period of time has passed, right? So these are super useful keys and I'm super excited to build these. Let's get started right now. first things first if we want to expire a key that means we need to store the expiry date and time right of the key so we know when to expire the key which means first of all we need to go into db.go over here and you can see that our store over here is just a map of strings right now instead of the values being strings it needs to be a custom struct that can store not just strings but also the expiry value and any other future like properties that we want to store in each key right so for that let's create let's go down here and create a new struct we'll call this key this will have a value v which will just be a string just like what we have right now and it'll have an expiry field so let's call this exp for expiry and this will be a time dot time object There we go. Now instead of the store being a value of string right this should just be a pointer to a key and over here when we initialize the database store instead of creating this instead of creating it as a string let's create it as a key pointer as such. So this is going to be a breaking change it's going to break a lot of our handlers over here that expect the value to be a string so we need to fix that so over here in the get function first of all change this to value dot v in the set function as well over here instead of the so over here we trying to set the value over here which is the string right so instead of doing that let's just create a key struct assign v to the val variable and there's no need to set an expiry over here because the set command doesn't have an expiry field right an argument called expiry um save that go down here and in the bg save command we have another error and this is happening because the copy of the database is still um holding values of string so fix that by just changing this to a key pointer and change the same thing in this db copy field as well in the app state. Change this to a key pointer and there we go. What is this? In call to maps.copy it does not satisfy map kv. Okay we'll see that. But first of all let's continue fixing this and change this to a key pointer and initialize it. There we go. So all done, awesome. And it looks like that error is gone as well over here in naps.copy. Okay, so now our database instead of holding values of strings, it holds a value of key, right? And each key will have a string which will be the value and an expiry field which will tell us when to expire this key. And obviously in the future as we add more features to this project, we can add more properties to each key like last accessed or something last modified right and so on and so forth right we can add all sorts of different things over here and just you know improve this project even further right adding even more features but for now this is fine now one thing I don't like right now if I go over here to the set command handler is that we have to create this key struct over here and I don't like this because this is kind of like a like this key is kind of like a private variable and we're just directly creating this over here which I don't like. So what I want to do is I want to have down here I want to create a pointer receiver for the database let say database pointer over here and add a set method and this can just take the i guess the key and the value which will just be a string and doesn't need to return anything right now and this can just grab the database store the key all right and just set that to a new key where the value will be this we very variable that we pass in and we can change this over here as well and just call the DB dot set function and pass in the required arguments you might think this is not a huge change but let me just explain first of all I just like that we're creating the key struct over here instead of out there because it just feels better i don't know like only the database struck itself should be able to you know interact with the key struct and modify it and so on and so forth right i want the handlers and all the outside functions to just use like this um these public functions that we declare on the database right so the database will have this kind of like a public api a bunch of functions that it can that the outside handlers can use and we just let them use that and not you know directly initialize internal structs like key and all that right so that's the first thing and the second thing is we are going to add a bunch more features to this set command in the in a couple future videos all right? Stuff like database transactions, stuff like memory management. So what happens if the server memory is, you know, it runs out, right? And we have to remove a bunch of keys and so on and so forth. So we're going to add a bunch of more features in the next couple of videos. And a lot of those features are going to be in the set command handler right over here. and I don't want to you know just clutter this this command handler with all of that extra code so we just do all of this in the set command over here in the database right in a pointer receiver over here I hope that makes sense now let's move on by the way guys sometimes I feel like I just rambling at the camera and not really explaining things properly I don know if that the case but I just feel that way because so like if any of these things seem too complicated to you or if you feel like I not explaining this correctly just please let me know in the comments or something so I can try to fix it and you know just get better at explaining and teaching these concepts right Because a lot of times I feel like I'm just rambling endlessly and not really explaining things properly and succinctly, right? so yeah help me out over there if this makes sense to you then great if it doesn't let me know in the comments and i'll try to fix that um let's move on so we can now create the actual command handlers so let's go over here and say expire equals expire and ttl equals ttl there we go go down here and declare both of these functions this will take a command or sorry a client value and state just like all of the other command handlers and we'll return a value there we go let's declare the TTL command handler as well let's say client value and this state app state there we go and return a value pointer there we go awesome so let's start with the expire command the expire commands works very very simply what you do is you basically run the expire command followed by the key name that you want to expire followed by the number of seconds that you want to expire it after so over here I'm saying expire the key one after 30 seconds right or after 300 seconds or after 60 seconds or 70 seconds whatever right so that's how it works it takes two arguments the key name and the expiry value in seconds so let's grab the arguments first of all there we go let's also make sure that we only get two arguments over here no more no less otherwise return an error of type error there we go and this will just be error in what invalid number of arguments for expire command there we go so now let's grab the key which will be the first argument as a bulk statement string and then the expiry value in seconds which will be the second value as a bulk string. Now since this is a bulk string we need to convert it into an integer so we know how many seconds to expire the value after right. So let's do that as well. So first of all let's just declare a variable expiry seconds and an error value as well and run the strcon.atoy function. I don't know how to pronounce this I just call it a toy and pass in the expiry variable over here make sure to handle the error as well we always want to handle the errors let's print the error as well we're actually not even bothered to print it we don't really need to print it let's just return a value type error and the error will be error invalid expiry value right there we go so now we have the expiry seconds now we can actually lock the database for reading and by the way you can see that i did all of these other things up here and i didn't lock the database right so just a quick little lesson for you if you're doing any sort of concurrency programming and stuff right when you're using things like mutexes and stuff right and you have multiple go routines or channels like multiple functions accessing the same stuff right in your program you want to make sure that you only lock the mutex when you're actually using the shared variable right so the shared variable in our case is this store in the database struct right the actual database right if we're not using this we don't need to lock the mutex at all right because if i if i lock the mutex up here what happens is i'm doing all of these things over here right and they're going to take time to execute all of these all of this code is going to take time to execute and i'm not even using the database store during that time right which means other clients will have to wait for an unnecessary period of time right so yeah just a quick lesson in concurrent programming always lock the shared variable exactly when you need to use it and then let it go exactly when you're done right no need for any extra processing there if there is any extra processing you do it after unlocking the shared variable anyway see how much you learning in these videos right anyway let unlock it as well there we go and then in between over here we want to grab the key And let say this will be the key Let's also add an OK over here. So use the comma OK idiom. If not OK, so if the key doesn't exist in the database, we should return an error over here as well. error invalid key right and the key does exist then we just need to grab the key.exp and assign a time.duration to it so let's say time.now to get the current time and then add the extra time to it so let's say time.second multiplied by time.duration and pass in the expiry seconds. So let me just explain what's going on over here. We use this time dot duration struct to convert the expiry seconds which is an integer into a duration right. Then we multiply that duration by time dot second and then we grab the current time and just add those seconds into the current time all right. And then we're basically done. We can unlock the database after that and return a success value over here. So we'll say return value type integer and by the way the expire command returns its values in integers so if the command was successful we'll return a one otherwise up here i made a mistake instead of returning an error if the key doesn't exist we return an integer of value zero and now the expire command is done all right nothing complicated over here we just make sure we get two arguments we grab both of those arguments, we convert the expiry into seconds, we lock the database, grab the key, and we assign the expiry seconds to the key.expiry field by converting it into a time.duration, right? Then we just unlock the database, return a success value, and we're done. Now let's do the TTL command as well. So the TTL time to live command is basically going to tell how much time is left right how many seconds are left until a given key is expiring so this command will only take one argument let's grab the arguments list make sure that there only one argument in the list otherwise we return an error oh I forgot to actually add the one over here there we go type will be error and error value will be error invalid number of arguments for TTL command there we go now assuming we get the correct number of arguments the first and only argument will be the key so let's grab that by saying arg0.bulk then we can just lock the database again for reading unlock it down here and let's grab the key using the comma okay idiom again let's say db.store pass in the key and also grab the expiry over here which will be key.expiry and now we're done with the database stuff right we've grabbed the key we don't need to like keep the database locked after this point right because we've already grabbed the values that we need to use now we can unlock the database and do all of the other processing that we need to do without locking the database okay so first of all if the key doesn't exist then the TTL command expects that we return an integer. Alright, type integer with a value of minus 2. Alright, so that's one thing. Now what happens if a key isn't even set to be expired, right? So all of the keys that we've set up to this point are persistent and when I say persistent I mean that they don't have an expiry value set, right? So what happens if they don't even have an expiry value set? Then we need to return minus one an integer of minus one but how exactly do we tell which values are set to be expired or not because in go and you'll know this if you use go but in go every single struck every single field every single data type right has a default value so for strings that can be an empty string for an integer it can be zero for errors it's nil right and similarly this time time.timestruck has it has a default value as well and that is the unix timestamp which i believe is the 1st of january 1970 or something some super old time right what we need to do is we need to compare the expiry field over here and see if that matches that unix time epoch and see if it matches that old january 1970 unix time right if it does that means that it's using the default time field right the default time struck which means that we haven't actually set an expiry value on this all right so and by the way if you're confused by what I mean I actually just mean this the Unix timestamp epoch or epoch or however you pronounce this right this is basically the number of seconds that are passed since January 1st 1970 all right now the default value of this time struck is going to be January 1st 1970 we just need to compare that right to tell which keys are set to expire and which keys are persistent and are never going expired. So let's do that. I'm just gonna go over to main.go and up here let's just declare a variable constant polyunix timestamp epoch or epoch I don't even know how to pronounce this and I'll just paste in the value over here directly. If you want to get this value programmatically just do something like time dot time and just do unix over here so create an empty time dot time struct and just call the unix time yet the unix method which will return the unix timestamp which will be this value over here anyway let's save all of that go back over here and down here if the expiry if the keys expiry value let's convert it to the Unix timestamp equals the Unix timestamp epoch I'm just gonna call this epoch because I don't know how to pronounce this if both of these values equal right that means that there is no expiry value set and this key is persistent so let's return a value of type integer and return minus one over here alright so that's all the error handling done now now we can actually actually grab the number of seconds left. So let's say expiry seconds. I'm going to use the time.until function over here and this will just return the duration until the time t which we pass in over here which will be the expiry. So this will return how much time is left until the expiry right. Now we want to get the seconds over here, which will also be a duration or actually a float64, sorry. We want to convert this float64 into an integer. So let's do that and then just return the value over here as a type of integer. And the value will just be expiry seconds. So how many seconds are left until the expiry? Okay, that was a lot of coding. Now let's actually test this out and see if it works, right? So let's stop the redis server that's currently running. There we go and run the go server Okay, so there's an error over here error decoding Rdb file Yeah, this is because we added this new key struct instead of just using string values. So let's open up Rdb and and oh yeah actually this is happening because the old rdb file over here expects string values but we added this new key struct so just delete this right because there's no way to recover this anymore there we go um restart the server now and let's open up a client over here as well and by the way this is not going to actually expire and delete the keys right now because we haven't added that functionality but we can still test this out and I believe I've already implemented password per section so this is not gonna let me yeah see so we need to authenticate ourselves first there we go now let me just grab the okay yeah so we do have a name key over here let's set this to expire in five seconds so let's say expire name five it gave me the integer one value let's grab the expiry over here and now it's giving me zero so I guess the time is already passed awesome now it just keeps increasing the time over here right negative 13 negative 15 and so on and so forth because we're not actually deleting the key right now but you can see it is it is still working if I try setting a new key let's say key 1 equals I don't know ABC right grab it it's working let expire it after let say five seconds let TTL it as well two seconds left one second left zero the key has been expired And now it just going to continue counting because we don't actually delete the key, right? So let's add that. Let's clear everything over here as well. Now, first of all, in the TTL command, let's say if the expiry seconds is less than or equal to zero, right? Let's quickly lock the database. Unlock it down here. Let's delete the key. And by the way, for deleting as well, let's create a helper method over here. Let's create a pointer receiver. Database. Delete. And this will just take a key which can just be a string I guess. And this will just, where is it, yeah, it will use the built in delete function which will take the database store and the actual key that we want to delete, there we go. So let's call that over here as well, pass in the key and then let's just return a value over here of type integer and return minus two because if this key is expired we're going to delete it right and we if we delete it that means it doesn't exist anymore if it doesn't exist then we just return minus two just like we do up here right so that's all done um let's do the same thing up in the get command as well all the way up here so if the value doesn't exist then we already return nil right let's also do let's get the expiry value convert it into unix and say if it doesn't equal the timestamp epoch over here right so it does have an expiry value set it's not persistent and we say time.until value.expiry converted into seconds is less than zero let's say less than or equal to zero then we can delete this key So again lock the database run the delete command over here the delete function pass in the name which is the key name that we get over here Unlock the database again and return a value of type null. There we go. Awesome. So now this should be 100% complete. Let's open it up again. Let's run a client. let's authenticate ourselves and now we can try setting a key one again there we go let's grab it it's working let's expire it after let's say seven seconds there we go let's grab the ttl of it as well and there we go whoops okay so there's an error over here invalid memory address or nil pointer dereference okay so yeah we broke something what did we what did we miss over here okay i've got it so the the problem over here is that we're not checking the okay value on time right we're checking the okay value too late so we grab the key over here right and if the key doesn't exist then we should check the okay value up here all right and the reason is because otherwise even if the key doesn't exist we're still trying to get to grab the exp value the expiry value from the key right now if the key doesn't exist if it is nil then we can't actually access this expiry value right it's just going to crash which is exactly what it's doing over here and giving us this you know runtime error of a nil pointer dereference right because key is nil and we're trying to dereference it and grab the expiry value. So in order to do that we first need to make sure that the key actually exists. So let's do that over here. Now let's clear everything. Run everything again. There we go. Run a client. Authenticate myself as well. There we go. Set the key one to ABC again, expire key one after five seconds, run the TTL command, see how much time is left, one more second and there we go. Awesome. So now it just keeps returning minus two no matter how many times I run this which just means that the key doesn exist anymore I can try to get it as well which should return nil and there we go awesome right if I try to get it it just returns nil because the key has been deleted or expired from the server awesome so with that we have completely built the expire under TTL commands and our server can now hold keys temporarily and expire them automatically after some time awesome now there is still one issue which i'm not actually going to solve because it's not a big deal um the way redis actually expires keys is one it's kind of a lazy way it doesn't actually expire them exactly when the expiry value or the expiry time is you know is hit right uh the way Redis expires keys is in two ways all right it basically has these checks in commands like get and TTL and stuff right which run and make sure that the key does exist and it hasn't expired right and that's what we're doing right now so if you run commands like get or something else Redis will check if the key has expired or not right that's one thing and the other thing is redis also has a background process that runs every i don't know 100 milliseconds or something or one or two seconds i don't know but it repeats every you know interval of seconds or something and it checks a sample of keys like i don't know 50 or 100 keys and see if any of those expire or not all right this is the thing that we're not going to actually build right now i might build this in a future part of this series but not gonna do it right now but the way this actually just works is it basically and by the way we are already doing this um checks in these commands up here so we don't need to discuss that but basically the background process method is it basically just runs a background process every like second or every 500 milliseconds or something like that right and We'll check only a small sample of keys not all the keys in your database because that would just be terrible for performance Right because you're trying to check every single key in the database every one second. Let's say right. That's just terrible for performance So what it does it is it samples like I don't know 50 or 100 keys checks their expiry values if they fire then it deletes them then after another second it samples another 50 or 100 keys and so on and so forth and it just keeps going like that every one second or every interval right I might build this or I might not either way the expiry and the TTL command works we have a completely functioning server that can now expire commands as well and we are done with this part of the series awesome I've been recording for a very long time 44 minutes so far I'm trying to make shorter videos just because they're easier to watch for people but there's just so much so much to do in these projects so much code to write so many things to discuss so many things to teach you guys right and it just ends up being such a long video so I'm trying to do what I can to make these shorter but But yeah, it's just, it is pretty difficult. But yeah, thank you for watching this one. Like, share, subscribe, comment and reach out. And yeah, I'll see you in the next video, which will be about, let me just see what comes next after this. Yeah, so after this, we're going to build the AOF rewriting function. I'll explain what that is in the next part of the video. then we're going to build the database transactions which is super cool I'll explain what those are as well and then finally we're going to look into eviction policies that are used to manage memory in our database server and I will explain that as well when the time comes until then thank you for thank you for watching this one and I will see you in the next video bye bye