Building a Redis clone in Go - Part 7 (SAVE and BGSAVE)

Okay guys welcome back to this go redis database clone series. This is I believe it's part 7 of this series and in the last part we basically finished adding the concurrency features to the entire server right to enable it to accept multiple connections at the same time and Also, we added a bunch of new commands like delete exists and keys. All right now in this in this part We're going to implement the save and the BG save the background save commands But before we do that We're also going to do a bit more housekeeping and just fix a bunch more Issues and bugs and just kind of like improve this server a bit right in a bunch of different places So let's do that. And by the way, as you can see, I'm not wearing glasses today. I broke my glasses frame and so I'm gonna have to like just squint at the screen to to make sense of what I'm writing. So if you see me doing that, you know the reason. So first let's just open up the rdb file. I want to make a few more improvements over here. So as you can see over here in the save rdb function, we are using this gob protocol to encode the database store. The problem is this is not like thread safe. This is not concurrency safe. We have a mutex in our database struct right so we need to use that over here as well and I'm going to lock it just for reading not for writing because we're only we're only reading the database store in this like function call right so that's the first thing now let's go down or let's actually go back up here and you can see I'm passing in a bunch of flags over here right when I want to save the RDB file and create the actual file I am creating it if it doesn't exist already and I am opening it in writing mode right not in reading mode so this is fine but I also want to do one more thing I want to add one extra flag over here which is the OS trunk right so this stands for truncate and what this is basically going to do is it's going to guarantee that every single time you open this file the previous contents of the file are going to be overwritten right so we always keep the latest snapshot of the database not some old data that you know just it's just lingering around for no reason right so we always want to truncate the entire file every time we save the database snapshot so that we always have the latest snapshot of the database now down here in the sync RDB function we We don't need to use os.open file. This is unnecessary. One, because these permissions aren't going to be really necessary because the files are already going to be created, right? We can use os.open over here because right now what we're doing is we're basically just creating the file if it doesn't exist, right? But there's no need to create an empty RDB file just to synchronize it, right? Because look, this is the syncRDB function, not the saveRDB function. This function is just going to open up the RDB file and synchronize it with the database But if the file doesn't exist then this create flag is going to create it right and then open it in reading mode But there's obviously no point in creating an empty file because there's nothing to synchronize right so Because of that let's just use OS dot open over here instead of using the OS dot Open file method right so remove this and there we go Now also one last thing that I want to do is you can see over here that We do make sure to close the file right when we're done with this whole function by using the deeper keyword over here, right? But what happens if there's an error in opening the file, right? If there's an error then it's going to print this log statement and then it's just going to return Without ever reaching this defer call right? So that's a problem what we should do instead then is to just call f dot close over here again and that's going to just make sure that the file actually gets closed even if we don't reach the deferred down here right and with that we are done with all of the housekeeping that I wanted to do this close should just like fail gracefully and silently if the file is already closed so I think it's safe to call this again over here as well right but yeah now we're done with all of this general improvements that I wanted to make now we can actually get started with implementing the save and the BG save command so let's do that so let's get started over here by adding the save command and this will run the save function which we need to create down here let's say save this will take a value pointer and a pointer to the app state and it will return a value pointer there we go now the save command what this is going to do is it going to save the rdb file right and this is going to take a config but before i pass that in the return value for this is going to be a value of type string all right and the string value will just be an okay so just a general okay success message all right and of course passing the um state.config in here and this should work now just to make sure that this is working what i'm going to do over here is i'm going to delete this um rdb file all right and you can delete this aof file as well and then let me just make sure that the original redis server is not running there we go and then let's run this and see if it works all right okay so it says error opening rdb file no such file or directory that's because the sync rdb function is running and we change this open file to the open method right so it's not going to create an empty file for us so there's really no reason to worry about this error right but anyway let's open up a client over here and you can see the connection was accepted let's run the save command and we got the okay success message and over here you can see the rdb file now this should be empty i think because there's nothing yeah see we didn't actually add any um data to this to this database so the rdb file is empty let's add a bunch of keys like name um age i'm still 19 but i am going to be 20 in like a couple weeks so yeah let's try running the save command again now and there we go now you see the keys that i just set over here right now they all got saved to this rdb file right so the save command works this is awesome now the problem with this save command right is it requires a mutex lock right so if I show you the RDB function over here again you can see that we just added these read and write or sorry not not right but just these reading locks right the problem is that this will block the server from running any like set commands right so if you have a client that's running the save command the server has to lock the database for reading and now since this is a read write mutex you can have multiple multiple readers at the same time so that's great but anytime you want to write new data to this database all of these readers will have to finish first right and so what ends up happening is if you want to write new data to the to the server to the database right and if some client is running the save command you cannot write any new data Until this lock is freed and this lock will be freed when you are finished with encoding this database So if you have potentially a very large database, right a very large key value store then this This encoding command can take a very long time to finish right? And while it finishes you can't run any write commands any set commands, right? So this is a problem and in a production setting Redis doesn't actually recommend using the save command. All right, so What you should do instead is use the BG save command right the background saving command and what that does is it basically just like Handles all of the RDB snapshot processing in the background While still allowing the main server to run and accept commands like set and all these other commands, right? So that let's implement that let's go over to the handlers File again and add the BG save over here There we go now we're going to implement this Quite differently from how Redis does it because and let's add this over to the handlers list as well So the way Redis handles the background save command is it uses something called c o w copy on write. So this is like a memory optimization algorithm that the operating system provides us and this is not available in Go sadly. Since Go is a garbage collected application, a garbage collected language right, it has a GC, a garbage collector and it handles memory management for us automatically using this garbage collector so we don't have access to this memory optimizing algorithm right which means we're not gonna be able to implement true background saving right because go just doesn support that if I was using C or something to write this whole server then yeah I could probably use that but um with go we just can do that so we gonna sacrifice a lot of like performance over here to implement this background save um command but um yeah it is what it is so the first thing i need to do right is go back to main.go and go down here to app state and we want to add a few more fields over here. I want to add the background save running field which is a boolean and I want to add the db copy which is a map string of strings. Awesome. Next we want to go over to the rdb file and instead of accepting the config over here right we want to accept the state instead and i'll explain why in a second but let's fix these reference errors as well there we go now the reason we want to accept the state over here in the save rdb function instead of the config is i want to be able to access the database copy and i want to access this bg save running boolean to tell whether a background save is running or not all right now let's implement this let's let's say that if the background save is running then we're going to save the database copy all right which is going to be in the state dot DB copy field right and since this is a copy since this is not the main database we don't need to use the mutex over here right so that that gives us a bit of a performance improvement over here and if there's no background save running then we can just use the actual database value over here right to encode and write the database snapshot so basically if the background save command is run we're going to encode the database copy otherwise we're going to encode the actual database store right let's also add a log statement up here just so we know what's going on so let's say saving DB to RDB file let's just say that and now let's go down over to the or actually up over to the init RDB trackers function and this needs to accept the config instead of the or this need to accept the state instead of the config why because we need to pass in the state to this rdb function over here right just save rdb function down here so pass that in solve this reference error as well state.conf.rdb there we go now let's go over to main.go and we need to pass in the state over here as well instead of the config there we go so now this is all done right now let's go back over here and let's fix this as well save our DB and pass in the state now in the background save command handler over here right first of all let's just check if a background save is already running right and if it is already running then we need to return an error so let's say the type will be error and the error message will just be error background saving already in progress awesome now we want to copy the entire database right into a new variable and then pass that in as the database copy that we want to save in the snapshot so let's do that let's create a copy first of all I'm going to declare a value a variable C which is going to be the copy I'll use the make function and I'm going to make a map of strings just like our database and the length or the size of this map is going to be the size of the database store that we have right then what I want to do is I want to lock the database for reading and I want to run the map dot copy function we want to copy into C and we want to copy from the database store and once this copying process is finished we can unlock the mutex and move on then inside the state we can say BG save running equals true because the background save process is now going to be started we can also say that the DB copy equals C then we want to run a go routine down here which is going to run the save RDB function passing in the state. What we also want to do is we want to add a defer function over here and what this will do is once the RDB saving process is entirely completed right it's going to set the BG save running back to false and the database copy back to nil let's just say. So this go routine is now going to handle all of the RDB saving processes entirely on its own right The main thread is free to return a value to the client and we just going to return another success message just a simple string with the text of okay let's restart the server now and see if everything works let me try setting some new keys like job youtuber I don't know hobby coding so you set some new keys over here now let's try to run the BG save command and there we go we got the ok success message the go routine was launched to save the RDB file and there we go we can see the hobby keys over here the job key the youtuber value everything right it's all in here so the BG save command works as well obviously you can see that both commands in this case the save and the BG save command both of them run instantly right because this is such a tiny database store right we only really have like four or five keys over here right because I'm just testing this out right now in a production setting where you have millions and millions of keys or even thousands of keys that's where you can actually see the performance benefits of using the BG save command over the save command right usually when it comes to performance and just you know like trying to trying to test whether one approach is more performant than the other you really need a very large sample size right you need a very large sample size to actually test whether that performance benefit works at scale or not right but yeah we're basically done with this part of the video we implemented the save and the BG save commands and we did a bit bit of a housekeeping across the codebase we made a bunch of tiny improvements here and there right and yeah again it sucks that we can't actually implement the real BG save command in go because it's a garbage collected language right it doesn't give us the same memory optimization algorithms that other languages like C have access to right like the copy on write algorithm that I told you about this is a actually spent some time like studying this algorithm and it is very very like smart I would say but it just like we just can't access it and go sadly so that sucks and we're just gonna have to live with this BG save command implementation now it's not really that big of a like improvement over here because we still need to lock the database to make an entire copy of the entire database right so that's still that is still like a very performance heavy operation but the benefit here is that once the copy is made we can free up the database the mutex right and the database is free to serve other clients meanwhile in this go routine we can handle the encoding process and everything by using the database copy that we made and so we won't need to lock the database while we encode it and write it to the file right because we're here we're just writing the database copy right so we won't need to lock and unlock the database again right so there's that benefit for sure but other than that yeah kind of a small optimization in this case but still anyway that's enough of me dragging out the outro in the next part we're going to we're going to build a couple more very simple commands the DB size command which is and let me just show you over here we're going to build the DB size command which will basically tell you the number of keys in the database right so the size of your database we're also going to create the flush DB command which will basically just delete your entire database all right we're going to implement both of these commands we're also going to implement a like a checksum protection in the RDB snapshots so checksums are basically like this like we can use something like a shot 256 or something hashing algorithm right we can create a hash of the bytes the raw bytes that we are writing to the file and the raw bytes that we encode from the database and you can just see both of the checksums that they have match right and if they don't match that means there's a mismatch between the database and the file that we write right so stuff like this just helps us to keep both things in synchronization basically so we're going to implement these three things in the next part of the of the series thank you for watching this part and stay tuned for the next one like share subscribe comment all those wonderful things and thank you so much for watching and I will see you in the next video until then bye bye