Building a Redis clone in Go - Part 11 (EXPIRE and TTL)

Ladies and gentlemen, welcome back to this Go Redis Clone Database Thingy series. In the previous parts, we built the... Let me just show you what we did. In the previous parts, we built the RDB checksum protection over here in the saveRDB function. And we also built the AUTH command to allow us to use password protection to secure our servers. So, that's what we did in the previous parts. In today's video, we're going to build the EXPIRE command, right? EXPIRE and the TTL, time to live command. Super excited about this video because these commands are awesome, alright? They allow you to basically delete keys automatically after a certain period of time has passed, right? So, these are super useful keys, and I'm super excited to build these. Let's get started right now. First things first, if we want to expire a key, that means we need to store the expiry date and time, right, of the key. So we know when to expire the key. Which means, first of all, we need to go into db.go over here. And you can see that our store over here is just a map of strings right now. Instead of the values being strings, it needs to be a custom struct that can store not just strings, but also the expiry value and any other future-like properties that we want to store in each key, right? So for that, let's create, let's go down here and create a new struct. We'll call this key. This will have a value V, which will just be a string, just like what we have right now. And you'll have an expiry field. So let's call this exp for expiry, and this will be a time.Time object. There we go. Now, instead of the store being a value of string, right, this should just be a pointer to a key. And over here, when we initialize the database store, instead of creating this, instead of creating it as a string, let's create it as a key pointer as such. So this is going to be a breaking change. It's going to break a lot of our handlers over here that expect the value to be a string, so we need to fix that. So over here in the GET function, first of all, change this to value.V. In the SET function as well, over here, instead of the... So over here, we're trying to set the value over here, which is the string, right? So instead of doing that, let's just create a key struct, assign V to the val variable, and there's no need to set an expiry over here because the SET command doesn't have an expiry field, right, an argument called expiry. Save that. Go down here, and in the BGSAVE command, we have another error, and this is happening because the copy of the database is still holding values of string. So fix that by just changing this to a key pointer, and change the same thing in this DB copy field as well. In the app state, change this to a key pointer, and there we go. What is this? In call to maps.Copy, it does not satisfy map K V. Okay, we'll see that. But first of all, let's continue fixing this, and change this to a key pointer and initialize it. There we go. So all done. Awesome. And it looks like that error is gone as well over here in maps. Okay, so now our database, instead of holding values of strings, it holds a value of key, right? And each key will have a string, which will be the value, and an expiry field, which will tell us when to expire this key. And obviously, in the future, as we add more features to this project, we can add more properties to each key, like last accessed or something, last modified, right, and so on and so forth, right? We can add all sorts of different things over here and just, you know, improve this project even further, right, adding even more features. But for now, this is fine. Now, one thing I don't like right now, if I go over here to the SET command handler, is that we have to create this key struct over here, and I don't like this because this is kind of like a... Like, this key is kind of like a private variable, and we're just directly creating this over here, which I don't like. So what I want to do is I want to have down here, I want to create a pointer receiver for the database. This pointer over here, and add a SET method. And this can just take the, I guess, the key and the value, which will just be a string, and doesn't need to return anything right now. And this can just grab the database store, the key, alright, and just set that to a new key where the value will be this value variable that we pass in. And we can change this over here as well and just call the db.set function and pass in the required arguments. You might think this is not a huge change, but let me just explain. First of all, I just like that we're creating the key struct over here instead of out there because it just feels better. I don't know, like, only the database struct itself should be able to, you know, interact with the key struct and modify it and so on and so forth, right? I want the handlers and all of the outside functions to just use, like, these public functions that we declare on the database, right? So the database will have this kind of like a public API, a bunch of functions that the outside handlers can use. And we just let them use that and not, you know, directly initialize internal structs like key and all that, right? So that's the first thing. And the second thing is we are going to add a bunch more features to this SET command in a couple future videos, alright? Stuff like database transactions, stuff like memory management. So what happens if the server memory is, you know, it runs out, right, and we have to remove a bunch of keys and so on and so forth? So we're going to add a bunch of more features in the next couple of videos, and a lot of those features are going to be in the SET command handler right over here. And I don't want to, you know, just clutter this command handler with all of that extra code. So we just do all of this in the SET command over here in the database, right, in the pointer receiver over here. I hope that makes sense. Now let's move on. By the way, guys, sometimes I feel like I'm just rambling at the camera and not really explaining things properly. I don't know if that's the case, but I just feel that way because... So, like, if any of these things seem too complicated to you, or if you feel like I'm not explaining this correctly, just please let me know in the comments or something so I can try to fix it and, you know, just get better at explaining and teaching these concepts, right? Because a lot of times I feel like I'm just rambling endlessly and not really explaining things properly and succinctly, right? So yeah, help me out over there. If this makes sense to you, then great. If it doesn't, let me know in the comments, and I'll try to fix that. Let's move on. So we can now create the actual command handlers. So let's go over here and say `expire = expire` and `ttl = ttl`. There we go. Go down here and declare both of these functions. This will take a command or, sorry, a client value and state, just like all of the other command handlers, and we return a value. There we go. Let's declare the TTL command handler as well. We say client value and the state app state. There we go, and return a value pointer. There we go. Awesome. So let's start with the expire command. The expire command works very, very simply. What you do is you basically run the expire command followed by the key name that you want to expire, followed by the number of seconds that you want to expire it after. So over here, I'm saying expire the key one after 30 seconds, right? Or after 300 seconds, or after 60 seconds, or 70 seconds, whatever, right? So that's how it works. It takes two arguments: the key name and the expiry value in seconds. So let's grab the arguments first of all. There we go. Let's also make sure that we only get two arguments over here, no more, no less. Otherwise, return an error of type error. There we go. And this will just be error invalid number of arguments for expire command. There we go. So now let's grab the key, which will be the first argument as a bulk statement string, and then the expiry value in seconds, which will be the second value as a bulk string. Now, since this is a bulk string, we need to convert it into an integer so we know how many seconds to expire the value after, right? So let's do that as well. So first of all, let's just declare a variable expiry seconds and an error value as well, and run the strconv.Atoi function. I don't know how to pronounce this. I just call it Atoi, and pass in the expiry variable over here. Make sure to handle the error as well. We always want to handle the errors. Let's print the error as well. We're actually not even bothered to print it. We don't really need to print it. Let's just return a value type error, and the error will be error invalid expiry value, right? There we go. So now we have the expiry seconds. Now we can actually lock the database for reading. And by the way, you can see that I did all of these other things up here, and I didn't lock the database, right? So just a quick little lesson for you. If you're doing any sort of concurrency programming and stuff, right, when you're using things like mutexes and stuff, right, and you have multiple goroutines or channels, like multiple functions accessing the same stuff, right, in your program, you want to make sure that you only lock the mutex when you're actually using the shared variable, right? So the shared variable in our case is this store in the database struct, right? The actual database, right? If we're not using this, we don't need to lock the mutex at all, right? Because if I lock the mutex up here, what happens is I'm doing all of these things over here, right? And they're gonna take time to execute. All of this code is going to take time to execute, and I'm not even using the database store during that time, right? Which means other clients will have to wait for an unnecessary period of time, right? So yeah, just a quick lesson in concurrent programming: always lock the shared variable exactly when you need to use it, and then let it go exactly when you're done, right? No need for any extra processing there. If there is any extra processing, you do it after unlocking the shared variable. Anyway, see how much you're learning in these videos, right? Anyway, let's unlock it as well. There we go. And then in between over here, we want to grab the key, and let's say this will be the key. Let's also add an `ok` over here. So use the comma ok idiom. If not okay, so if the key doesn't exist in the database, we should return an error over here as well. Error invalid key, right? And if the key does exist, then we just need to grab the key.exp and assign a time.duration to it. So let's say time.Now to get the current time and then add the extra time to it. So let's say time.Second multiplied by time.Duration and pass in the expiry seconds. So let me just explain what's going on over here. We use this time.Duration struct to convert the expiry seconds, which is an integer, into a duration, right? Then we multiply that duration by time.Second, and then we grab the current time and just add those seconds into the current time. All right? And then we're basically done. We can unlock the database after that and return a success value over here. So we say return value type integer. And by the way, the expire command returns its values in integers. So if the command was successful, we'll return a one. Otherwise, up here, I made a mistake. Instead of returning an error if the key doesn't exist, we return an integer of value zero. And now the expire command is done. All right, nothing complicated over here. We just make sure we get two arguments. We grab both of those arguments. We convert the expiry into seconds. We lock the database, grab the key, and we assign the expiry seconds to the key.expiry field by converting it into a time.Duration, right? Then we just unlock the database, return a success value, and we're done. Now let's do the TTL command as well. So the TTL, time to live command, is basically going to tell us how much time is left, right? How many seconds are left until a given key is expiring. So this command will only take one argument. Let's grab the arguments list. Make sure that there's only one argument in the list. Otherwise, otherwise we'll return an error. Oh, I forgot to actually add the one over here. There we go. Type will be error, and error value will be error invalid number of arguments for TTL command. There we go. Now, assuming we get the correct number of arguments, the first and only argument will be the key. So let's grab that by saying `arg zero dot bulk`. Then we can just lock the database again for reading, unlock it down here, and let's grab the key using the comma ok idiom again. Let's say `db.store`, pass in the key, and also grab the expiry over here, which will be `key.expiry`. And now we're done with the database stuff, right? We've grabbed the key. We don't need to, like, keep the database locked after this point, right? Because we've already grabbed the values that we need to use. Now we can unlock the database and do all of the other processing that we need to do without locking the database. Okay, so first of all, if the key doesn't exist, then the TTL command expects that we return an integer, all right? Type integer with a value of minus two. All right, so that's one thing. Now, what happens if a key isn't even set to be expired, right? So all of the keys that we've set up to this point are persistent. And when I say persistent, I mean that they don't have an expiry value set, right? So what happens if they don't even have an expiry value set? Then we need to return minus one, an integer of minus one. But how exactly do we tell which values are set to be expired or not? Because in Go, and you'll know this if you use Go, but in Go, every single struct, every single field, every single data type, right, has a default value. So for strings, that can be an empty string. For an integer, it can be 0. For errors, it's nil, right? And similarly, this time.Time struct has a default value as well, and that is the Unix timestamp, which I believe is the 1st of January 1970 or something, some super old time, right? What we need to do is we need to compare the expiry field over here and see if that matches that Unix time epoch and see if it matches that old January 1970 Unix time, right? If it does, that means that it's using the default time field, right? The default time struct, which means that we haven't actually set an expiry value on this. All right, so, and by the way, if you're confused by what I mean, I actually just mean this: the Unix timestamp epoch, or epoch, or however you pronounce this, right? This is basically the number of seconds that have passed since January 1st, 1970, all right? Now, the default value of this time struct is going to be January 1st, 1970. We just need to compare that, right, to tell which keys are set to expire and which keys are persistent and are never going expired. So let's do that. I'm just going to go over to main.go, and up here, let's just declare a variable, constant, call it unix timestamp epoch or epoch. I don't even know how to pronounce this, and I just paste in the value over here directly. If you want to get this value programmatically, just do something like time.Time and just do Unix over here. So create an empty time.Time struct and just call the Unix time, get the Unix method, which will return the Unix timestamp, which will be this value over here anyway. Let's save all of that, go back over here, and down here, if the expiry, if the key is expiry value, let's convert it to the Unix timestamp equals the Unix timestamp epoch. I'm just going to call this epoch because I don't know how to pronounce this. If both of these values equal, right, that means that there is no expiry value set, and this key is persistent. So let's return a value of type integer and return minus one over here. All right, so that's all the error handling done. Now, now we can actually grab the number of seconds left. So let's say expiry seconds. I'm going to use the time.Until function over here, and this will just return the duration until the time T, which we pass in over here, which will be the expiry. So this will return how much time is left until the expiry, right? And now we want to get the seconds over here, which will also be a duration, or actually a float64, sorry. We want to convert this float64 into an integer. So let's do that and then just return the value over here as a type of integer. And the value will just be expiry seconds. So how many seconds are left until the expiry? Okay, that was a lot of coding. Now let's actually test this out and see if it works, right? So let's stop the Redis server that's currently running. There we go, and run the Go server. Okay, so there's an error over here: error decoding RDB file. Yeah, this is because we added this new key struct instead of just using string values. So let's open up RDB, and, yeah, actually, this is happening because the old RDB file over here expects string values, but we added this new key struct. So just delete this, right? Because there's no way to recover this anymore. There we go. Restart the server now, and let's open up a client over here as well. And by the way, this is not going to actually expire and delete the keys right now because we haven't added that functionality, but we can still test this out. And I believe I've already implemented password protection, so this is not gonna let me. Yeah, see? So we need to authenticate ourselves first. There we go. Now let me just grab the... Okay, yeah, so we do have a name key over here. Let's set this to expire in five seconds. So let's say `EXPIRE name 5`. It gave me the integer one value. Let's grab the expiry over here, and now it's giving me zero. So I guess the time has already passed. Awesome. Now it just keeps increasing the time over here, right? Negative 13, negative 15, and so on and so forth, because we're not actually deleting the key right now. But you can see it is still working. If I try setting a new key, let's say `key 1 = abc`, right? Grab it, it's working. Let's expire it after, let's say, five seconds. Let's TTL it as well. Two seconds left, one second left, zero. The key has been expired, and now it's just going to continue counting because we don't actually delete the key, right? So let's add that. Let's clear everything over here as well. Now, first of all, in the TTL command, let's say if the expiry seconds is less than or equal to zero, right? Let's quickly lock the database, unlock it down here. Let's delete the key. And by the way, for deleting as well, let's create a helper method over here. Let's create a pointer receiver, Database.Delete, and this will just take a key, which can just be a string, I guess. And this will just, um, where is it? Yeah, it'll use the built-in delete function, which will take the database store and the actual key that we want to delete. There we go. So let's call that over here as well, pass in the key, and then let's just return a value over here of type integer and return minus two. Because if this key is expired, we're going to delete it, right? And if we delete it, that means it doesn't exist anymore. If it doesn't exist, then we just return minus 2, just like we do up here, right? So that's all done. Let's do the same thing up in the get command as well, all the way up here. So if the value doesn't exist, then we already return nil, right? Let's also do, let's get the expiry value converted into Unix and say if it doesn't equal the timestamp epoch over here, right? So it does have an expiry value set. It's not persistent. And we say time.Until(value.expiry).Seconds() is less than zero. Let's say less than or equal to zero. Then we can delete this key. So again, lock the database. Run the delete command over here. Here, the delete function, pass in the name, which is the key name that we get over here. Unlock the database again and return a value of type null. There we go. Awesome. So now this should be 100% complete. Let's open it up again. Let's run a client. Let's authenticate ourselves. And now we can try setting a key one again. There we go. Let's grab it. It's working. Let's expire it after, let's say, seven seconds. There we go. Let's grab the TTL of it as well. And there we go. Whoops. Okay, so there's an error over here. Invalid memory address or nil pointer dereference. Okay, so yeah, we broke something. What did we miss over here? Okay, I've got it. So the problem over here is that we're not checking the `ok` value on time, right? We're checking the `ok` value too late. So we grab the key over here, right? And if the key doesn't exist, then we should check the `ok` value up here, all right? And the reason is because otherwise, even if the key doesn't exist, we're still trying to grab the expiry value from the key, right? Now, if the key doesn't exist, if it is nil, then we can't actually access this expiry value, right? It's just going to crash, which is exactly what it's doing over here and giving us this runtime error of a nil pointer dereference, right? Because key is nil and we're trying to dereference it and grab the expiry value. So in order to do that, we first need to make sure that the key actually exists. So let's do that over here. Now let's clear everything, run everything again. There we go. Run a client, authenticate myself as well. There we go. Set the key one to ABC again. Expire key one after five seconds. Run the TTL command, see how much time is left. One more second, and there we go. Awesome. So now it just keeps returning minus two no matter how many times I run this, which just means that the key doesn't exist anymore. I can try to get it as well, which should return nil. And there we go. Awesome, right? If I try to get it, it just returns nil because the key has been deleted or expired from the server. Awesome. So with that, we have completely built the expire and TTL commands, and our server can now hold keys temporarily and expire them automatically after some time. Awesome. Now, there is still one issue which I'm not actually going to solve because it's not a big deal. The way Redis actually expires keys is, one, it's kind of a lazy way. It doesn't actually expire them exactly when the expiry value or the expiry time is hit, right? The way Redis expires keys is in two ways, all right? It basically has these checks in commands like GET and TTL and stuff, right? Which run and make sure that the key does exist and it hasn't expired, right? And that's what we're doing right now. So if you run commands like GET or something else, Redis will check if the key has expired or not, right? That's one thing. And the other thing is Redis also has a background process that runs every, I don't know, 100 milliseconds or something, or one or two seconds, I don't know. But it repeats every, you know, interval of seconds or something. And it checks a sample of keys, like, I don't know, 50 or 100 keys, and sees if any of those expire or not. All right, this is the thing that we're not going to actually build right now. I might build this in a future part of this series, but I'm not going to do it right now. But the way this actually just works is it basically, and by the way, we are already doing these checks in these commands up here, so we don't need to discuss that. But basically, the background process method is it basically just runs a background process every, like, second or every 500 milliseconds or something like that, right? And we'll check only a small sample of keys, not all of the keys in your database, because that would just be terrible for performance, right? Because you're trying to check every single key in the database every one second, let's say, right? That's just terrible for performance. So what it does is it samples, like, I don't know, 50 or 100 keys, checks their expiry values. If they expire, then it deletes them. Then after another second, it samples another 50 or 100 keys, and so on and so forth. And it just keeps going like that every one second or every interval, right? I might build this or I might not. Either way, the expiry and the TTL command works. We have a completely functioning server that can now expire commands as well, and we are done with this part of the series. Awesome. I've been recording for a very long time, 44 minutes so far. I'm trying to make shorter videos just because they're easier to watch for people, but there's just so much, so much to do in these projects, so much code to write, so many things to discuss, so many things to teach you guys, right? And it just ends up being such a long video. So I'm trying to do what I can to make these shorter, but yeah, it's just, it is pretty difficult. But yeah, thank you for watching this one. Like, share, subscribe, comment, and reach out. And yeah, I'll see you in the next video, which will be about, let me just see what comes next after this. Yeah, so after this, we're going to build the AOF rewriting function. I'll explain what that is in the next part of the video. Then we're going to build the database transactions, which is super cool. I'll explain what those are as well. And then finally, we're going to look into eviction policies that are used to manage memory in our database server, and I will explain that as well when the time comes. Until then, thank you for, thank you for watching this one, and I will see you in the next video. Bye-bye.