Building a Redis clone in Go - Part 20 (Finish INFO & fix AOF memory)

Hello there, guys. Welcome back to the Go Redis Clone series, and welcome to the final part of this series! Alright, in the previous part, we implemented the INFO command over here. Implemented is the wrong word. We actually just implemented like half of the command. We're going to finish this in this part of the series, and then we're going to basically just end this project because I think we've built enough of this already. We've built so many different features in this project, and I think I'm ready to move on. So before I actually continue this, we have to implement the persistence category and the general category. Before I do that, let me just go up here and declare the INFO command handler. And then down here, let's create a command handler for it. Now, to actually get the INFO output, this is going to be a string. We want to have another method over here, and we can just call this something like `print`. This is going to be another pointer receiver, and we can just call this something like this. This can also take the app state, and it's going to return a string. There we go. So first of all, let's just call `info.build` over here and pass in the state. And this will just build the info object, add all of the information that we're having over here, and then let's craft the message that we're going to return. So let's say `message` string, and it's going to be just an empty string, obviously. Then let's create a subfunction over here and call it `printCategory`. Alright, this is going to actually print a category in the INFO struct over here. So you can see we have five categories over here, right? The server, client, memory, persistence, general, and all of these are going to be printed in the same way, right? So there's no need to duplicate the logic. Let's just create a `printCategory` function over here, and we can just pass in the actual category, and it will just print it for us. So this is going to take a couple of arguments, which will be the header, which is a string, and then the actual data, right, which will be a map of string of string, and it will return a string as well. And also make sure to add the colon over here since we're declaring a new variable. I don't know why I forget to do that sometimes, but yeah. So first of all, let's just create a string over here and add the actual header. Let's say hash. This is how we declare a heading in this output. This is how Redis actually displays headings in the INFO command output. And let's print a string over here, which will be the header, and let's also add a new line after this. Now we can basically just loop over the entire map, write the data in this category, and then we can just say the `s` string. We can append to it. We can say `fmt.Sprintf`, print two strings over here separated by a colon, right? The first will be the key, and then the value, and then also add a new line after this. Finally, let's return the string with one more new line over here. And that's all we need to do. Now we can just say something like `message` and then append to the message. And say, just call the `printCategory` function. Pass in the header, which will be server. Pass in the data, which is `info.server`. Then just copy-paste this three times. Second, the category will be client. The data will be in info. And then finally, memory and info. And then at long last, let's return the message to the caller. And that's all we need to do. Now, let's actually create this INFO struct. We need a place to store the INFO struct, right? So let's do that in app state as well because we store everything in app state, apparently. Let's create an INFO struct over here. This will be a pointer, and let's go down here in `newAppState` and call `newInfo` over here. And yeah, that's all we need to do. So back in the handlers over here, now we can say something like `state.info.print` and pass in the state. This will return the message that we want to send. Then we can just return a value of type bulk. The output of the INFO command is a bulk string, and the bulk value will just be message. And we can actually test this out as well. If I run the server, and then if I run a client over here, I can run the INFO command, and it's going to return all of this information over here, right? Awesome! So now let's configure the other two categories, persistence and general stats. Let's go down here to the build function and say `info.persistence`. This will be a map again. First, the `rdb_bgsave_in_progress` field. This just tells you if a BG save is currently in progress, and I think we already tracked this in state. Yeah, we do in state.bg save running. Then we also have a field called `rdb_last_save_time`, so basically the last time that the RDB file was saved, right? And we don't actually track this right now, so let's actually do that. Let's open up the app state file over here, and let's just create a new struct up here. Call it `rdbStats`. There we go. This will be a struct, and let me just name this something better, `RdbStats`. There we go. And this is going to have two fields: the `RdbLastSaveTimestamp`, which will be an int64, and the `RdbSaves`, so the number of times that the RDB snapshot has been created, which will be an integer. Let's attach this to the app state by saying `rdbStats` equals `RdbStats`. Let's do the same thing for append-only AOF files as well. Let's say `aofStats`. This will be another struct, and yeah, before I actually write all of these things into the INFO struct, let's just create all of the actual structs. Let's just track all of the data that we need to track, and then we'll just add it all to the INFO struct over there. For the AOF stuff, we don't really need to track a lot. We just need to track the number of times that we have done an AOF rewrite, alright? Let's attach that to the app state as well. Let's say `aofStats` is `aofStats`. There we go. Now finally, this is all we need to do to track the, what is it, the persistence category. But for the general category, we want to have another struct called `GeneralStats`, and this will have a couple of fields. First of all, the total connections received, which will be an integer, and the total commands processed, which will also be an integer, and then the expired keys, which will be an integer, and also the evicted keys, which will also be an integer. Let's add this down here as well. Let's say `generalStats` is `generalStats`. Add it all to the app state, and down here in the `newAppState` function, where we create the actual app state, let's just create an empty object for all of these different stats, right? Like this. Okay, awesome. All done! Now let's actually start tracking all of these things, right? So for the, let's start with the RDB stats. We want to go over to the RDB save function, which is in the RDB file over here. There we go. And every time we save the RDB file, we want to update the last save timestamp and the number of saves. So let's go down here to the very end over here and say `state.rdbstats.lastsave` timestamp is `time.Now().Unix()`, and the `state.rdbstats.rdbsaves` just increment this one time. And now we're tracking all of the RDB stats. Now, for the AOF rewrites field, we can go over to `handlers.go` again, in the BG rewrite AOF file, or the command handler. We can say down here, `state.AOFStats.AOFRewrites`, and just increment this by one, right? So each time we rewrite the AOF file, we can just increment the rewrites by one. Now, for the total connections received field over here in general stats, we can go over to `main.go` and the `handleConn` function, where we increment the client count. And down here, we can just say something like `state.generalStats.connected` or the total connections received, and just increment this, right? And there's no need to decrement this because this is the total connections field, right? Not the current connections. And for the commands processed, you can probably guess this. We can just go over to `handlers.go`, up here in the handle function, which receives every single command. We can say down here, `state.generalStats.totalCommandsProcessed` and increment it. Now, for the expired and evicted keys, both of these operations happen in the `db.go` file, right? So we can track it over here. First of all, in the `tryExpire` function over here, this is where we actually expire an item. We can pass in the app state over here. App state. There we go. And if the key should expire, and if we delete it, then we can just say `state.generalStats.expiredKeys` and increment it over here. And down here, where we call this `tryExpire` function, we make sure to pass in the state. We're not actually receiving the state in this get function, so let's add it over here as well by saying state and app state. Now, back in `handlers.go`, in the get function over here, where we call `db.Get`, we want to pass in the key name as well as the app state. There we go. And now we can basically track the number of expired keys. Finally, we just need to track the evicted keys, and we evict our keys in this `evictKeys` function up here. So for this one, we can just say over here in the `evictUntilMemCreate` function, we can basically return how many keys were deleted. This can just be an integer, so this will return an integer. Every time we delete a key, we can... And first of all, let me just create an integer up here. We call it `n`. And every time we delete a key, we can... And first of all, let me just create an integer up here. We call it `n`. And every time we delete a key, we just increment `n`. And then at the end, we'll just return this `n` value. And then over here, every time we call this `evictUntilMemFreed` function, this is going to return the number of keys that were evicted. So we can assign these as such, and we can just say `state.generalStats.evictedKeys +=` so add to this evicted keys field, and then just add the evicted keys that we received from this `evictUntilMemFreed` function over here. And now we're tracking all of these things properly. So let's add these to the info struct at long last. Let's say `rdbLastSaveTime` is `fmt.Sprint(state.rdbStats.lastSaveTimestamp)`, and `rdbSaves` equals `fmt.Sprint(state.rdbStats.rdbSaves)`. There we go. And AOF enabled will be `fmt.Sprint(state.conf.AOFEnabled)`. We track this already in the config. We also have one more field over here called AOF rewrite in progress, and I don't think we tracked this yet. State... Yeah, we don't track this yet. So let's add this as well to the app state. So kind of like we tracked the RDB BG save running up here in the app state, we can say something like AOF rewrite running or something like that, and this can just be another boolean. And in the info struct over here, we can just return the rewrite running boolean over here. Now, to actually track this, we want to go to the `handlers.go` function again, the `handlers.go` file again, and in the BG rewrite AOF file over here. This is actually where we rewrite the AOF file, right? So every time we call this rewrite function, we can say up here `AOFRewriteRunning = true`, and as soon as the rewrite is finished, we can say that `AOFRewriteRunning` is false. And you could even do something like `AOFRewriteRunning = true` up at the top of the function, and then just create a defer function that would basically say AOF running equals false, and then call that up here, right? So as soon as this goroutine ends, this function will run, and `AOFRewriteRunning` will become false. Awesome. Of course, both approaches are valid. You can track it down here as well in the rewrite function, or you can do it on a per-function basis, right in the entire function. Yeah, finally, let's track the AOF rewrites over here in the info struct as well. Let's say `fmt.Sprint(state.aofStats.aofRewrites)`, and that's all we need to do. Awesome. So that's all for the persistence category. Now, for the general category, let's say `map[string]string`, total connections received, `fmt.Sprint(state.generalStats.totalConnectionsReceived)`, total commands processed, `fmt.Sprint(state.generalStats.totalCommandsProcessed)`. See, this is super simple stuff, right? The hard part is basically just tracking all of these stats across your entire program, but returning this to the user, that's super easy, right? So we can say expired keys and evicted keys over here is `fmt.Sprint(state.generalStats.evictedKeys)`. Just copy-paste this. This will finally be expired keys, and over here, change it as well to expired keys. There we go. And then down here in the print function, let's add both of these categories: persistence, which will take `info.persistence`, and general, which will take `info.general`. And I think that's all we need to do. Let's rerun the server. `go run .` There we go. Start a new client, run the INFO command on it, and yeah, awesome. So this is working. We get the used memory and everything else. I can try to, I guess, set a new key to, let's say, XYZ. I guess if I try to get the INFO command again, you see that the commands processed is three now. Connections received one because we're only using one connection right now. Connected clients is also one. That all makes sense. We get the process ID, the executable, the config file, uptime in seconds, and all that. We get the used memory. We get the used memory peak as well. I can actually delete a key, like delete key one, and then grab the INFO again. And you'll see that the used memory peak is 201, but the used memory right now is 106. Awesome. And I wonder what keys are still being stored over here. Okay, so the new key is being stored. What is its value? Okay, a large number. Let's try expiring that key, right? So let's say expire new key in three seconds. Let's grab the TTL of this key, and it's already expired. Awesome. So let's grab the INFO again. And, okay, so it didn't actually increment this. Why not? Okay, so the evicted keys works, but the expired keys doesn't. I wonder why. I guess we're not tracking it correctly right now. Let's go back to DB in the try expire function. Oh, never mind. This is because in the handlers, in the TTL handler, instead of calling the try expire function, we're trying to expire it on our own, right? So we shouldn't be doing that. We should just be running DB expire over here, pass in the key name, which I guess is K. The item, which is over here, it's key. Let's rename this key to item just so it makes sense. And then, in the try expire call over here, let's pass in the item and then let's pass in the app state. This will return a Boolean, which will tell us whether it was expired or not. Then we don't need all of this logic over here. We can just say if expire, return all of this. We actually do need the expiry in seconds. Let me get that back. Where was it? Yeah, this stuff. Okay, so let's actually track the expiry seconds. There we go. But to actually try expiring the key, we should call DB.tryExpire because that is what will actually track the stats as well. We're done with this. Let's run the server again because I just want to make sure that this is actually tracking all of the keys properly. Let me see what keys we have right now. Okay, we have a bunch of keys. Let's say expired name in three seconds. Let's grab it. TTL, and there we go. It expired. Let's grab the info, and you will see that one of the keys was expired. Awesome! So we're tracking all of the stats perfectly now, and that is the final thing that I wanted to implement in this server. Now, before I actually end this project, I want to fix one last bug in the project, which is in aof.go over here. You'll recall that the AOF synchronization process over here doesn't actually respect the maximum memory limit of the server, right? So you can see over here that we're using 395 bytes, and even all the way up to 493 bytes, right? When we sync the AOF records, even though the maximum memory of this server in the configuration is 256 bytes, right? So the AOF sync is not respecting the maximum memory limit. And the reason it's not respecting it is because we're passing in an empty config over here. Now, to fix this, we can pass in the actual maximum memory and the eviction policy and stuff directly in this sync function. The reason I'm not passing the actual config object is because then it would also have stuff like AOF enabled and AOF file and F sync and stuff. And then every time you try to sync it, it will just add to the AOF records again and again, right? So every single value you restore from the AOF file, it would just add it to the AOF file again because the config would have all of these fields enabled, right? Which is why we set the config to a blank struct in the first place, right? Anyway, the sync function should take a bunch of new arguments now, and they are max memory, which is an int64, eviction policy, which is of type eviction, and memsamples, which is an int64. There we go. And we can assign this to the config object over here. We can say max memory is max memory, eviction is eviction policy, mem samples is mem samples. There we go. And I guess this is an int, so we can convert this back to an int in the argument as well. Awesome! Now, let's actually pass these in the main.go file up here. Where are we calling this? Over here. Okay, so this is going to take the maximum memory from con.maxmem, the eviction policy as well, and then the mem samples. All right, so we're only grabbing the configuration directives that we need to respect the memory limit. We're not grabbing all of the other configuration variables over here. So now that we have this, let's stop the server and run it again. And yeah, awesome, it works. So you can see in the logs over here, previously we were not respecting the memory limit, but now when we sync the AOF records, you can see that the memory increases as we add more keys. But as soon as we cross the 256 bytes memory limit, it starts evicting keys even from the AOF backup over here, right? It starts evicting keys and always makes sure to stay below the 256 bytes memory limit, right? So this works as well. The AOF file, the AOF functions respect the memory limit of the server as well. And yeah, now we're done with this, and this is turning into a very long video. I've been recording for more than 30 minutes. This is going to be a very long video. But yeah, that's the final video for this project. And with this, we are done. It's been a very interesting, remarkable journey. I've been building this project for the last 20 videos. So yeah, so many cool things that I built in this project. So many awesome new features. So many new things that I learned about Redis, about databases, about how all of these things work under the hood, right? Behind the scenes, you just don't even know about these things, right? Like you use Redis, you use Postgres, you use MySQL, you use all of these other tools, right? All of these databases to store your data. And you don't even know how they work under the hood, right? All of the different, like, features that they have, all of the different design decisions that they have to make to make sure that they're delivering a performant server, right? All the different trade-offs that they have to consider to make sure that this database system that they're building can run and scale up to thousands and tens of thousands of, you know, operations per second. And all of those wonderful things, right? There's just so much that goes into this, into projects like these, right? And what I built over here is very simple, right? Compared to actual Redis, compared to all of the other database systems out there, what I've built over here is very, very simple and bare-bones, right? You would think that this is a giant project, right? But yeah, it's nothing compared to all of the other database systems out there that are actually running in production, actually powering enterprise software, right? But yeah, this has been a very fun project. Hopefully, if you were following along the entire journey, you learned a bunch of things yourself as well. And you're now a much better programmer, just as I am. And yeah, thank you for following along this entire series. It's been a very fun journey. Let's see what comes next, right? I'll probably build another cool project in Go, or I might actually start learning C or Rust or something like that. I've been very interested in learning C for a while and just becoming much more used to low-level programming and how systems and hardware works under the hood, right? So I might actually build a new project in C or just continue coding in Go for a while. But we'll see. Again, thank you for watching this video. Thank you for following along with the whole series so far. Hopefully you learned something. Hopefully you got some value out of it. And yeah, I'll see you in the next video, which will be about who knows what, because I have no idea what comes next for the channel after this project. But yeah, I guess we'll see. Thank you for watching, and see you in whatever video I film next. Bye-bye!