Building a Redis clone in Go - Part 12 (Rewriting AOF files)

Welcome back guys, we're going to continue building this go redis clone. In the previous video we built the expire and the TTL commands over here. And these will just allow us to expire keys after some time, right? Not gonna go too deep into them again. In this video we're going to build the BG, let me just add it over here. BG rewrite AOF command. There we go. And what this is going to do is it's going to rewrite our AOF file over here and just make it more compact. So let me first show you the issue over here that we have. Right. Let me delete everything that we have right now. Let's comment this out. And there we go. Let's run the server. I just want to show you what the problem is right now before I actually solve it. Right. And let me just before I actually run this server. let me disable the password protection because I just hate having to enter the password every time I run the client and let's set the name key to let's say Hassan alright so you can see that it backs it up in the AOF file right now Now what happens if I change it to Michael and it backs it up again, right? And if I change it again to, I don't know, Elon, how's that? You should be able to see a problem over here, right? Every time we make a change, it gets backed up into the AOF file, which is great. But at the same time, when we actually synchronize this to the database, right? going to first set the name key to Hassan then to Michael and then finally to Elon the thing is that the final value of the name key is the string Elon right and all of these other set commands before it are completely unnecessary right so the server is literally just executing these set commands for no reason because the final value is just going to be Elon over here right and that is the problem with this AOF function with this AOF data persistence method over here right because it just records all of these set commands and they can become outdated as we override different keys with new values right so if I run the server again the value of the name key should be Elon right not Hassan not Michael but Elon okay which means we don't really need all of these other set commands up here that is what the rewrite AOF command is going to fix right so the BG rewrite AOF command is going to run in the background that's why BG right and it's going to rewrite the AOF file to remove all of these old and outdated set set commands and just have the latest version of them right and it's just going to make our whole AOF file much much smaller much more compact and much easier to read and use and it's going to improve the performance of the server because it doesn't have to execute all of these extra unnecessary commands to backup and restore the database. Alright that's what we're going to build and now that you're all up to speed let's get started. So let's uncomment this and down here create the command handler for this. Let's say bg rewrite aof. Just like all the other command handlers this is going to take the client and the value and the state. There we go. And it will return a pointer to a value. Now since this is supposed to run in the background we should start a go routine over here and run this over here. And let's first of all lock the database. Unlock it down here and create a copy of the database call it cp again it's just cp um and this is going uh what is it going to take let me just yeah uh it's going to take the maps.copy um actually not not not that it's going to take the make command the make function i don't know why i call functions commands and commands functions I'm just fucked in the head but we're going to make a map with keys as strings and values as key pointers there we go and this will have a length of what is a DB dot store then we're going to use maps dot copy to copy the entire database into this CP variable and once this is is done we can unlock the database and carry on and by the way we can change this lock to a read lock and down here as well to a read unlock because we're only reading the database values right we're not modifying them in any way so that's done now we can actually just run the state.aof.rewrite function which doesn't exist yet we need to write it over here but this function is basically going to rewrite the AOF file, open it up, and just rewrite all of the set commands for us, right? So let's call that over here. This will take the copy that we create over here, and then we can just return a value up here of type string and we can say background AOF rewriting started awesome so now let's actually create this rewrite function and it'll take a map of string and a key pointer there we go so first of all what we want to do is we want to reroute all of the existing AOF records to the to a memory buffer basically so this this AOF file is going to be busy now right because we're going to rewrite it but what happens if the if one of the clients that's running sends a set command right like set name has done or something right that's that new command still needs to be recorded in the AOF file right and in order to do that since we're going to be using the file because we're rewriting it right the file is going to be busy we want to reroute all of the future basically all of the future set commands to a memory buffer that we can then write to the aof file later on when we're done rewriting it so let me just add a comment over here as well future set commands to buffer and instead of calling this set commands Let's just call this AOF records. All right, let's create a buffer There we go and the AOF writer will just be a new writer to the buffer over here Awesome. Now, let's clear the file contents, right? because we want to delete everything in this file right now and Rewrite it so clearing the file content is going to be a two-step process process. First of all we want to run the aof.f.truncate function and just add zero over here. What this is going to do is it's going to change the file size right to zero and it's going to return an error. So let's grab that as well and make sure to handle it over here. If there's an error let's just log it let's say aof rewrite truncate error and then log the error over here and return so that's the first thing this is going to shrink the aof file to zero bytes right delete all of the data inside the aof file then the next thing we want to do is we want to seek the aof file to zero bytes all right so basically we've talked about byte cursors in a previous video as well. Basically we want to make sure that the byte cursor, the point where we start reading or writing to a file, right, is at the beginning of the file. So we do that with this seek function. Make sure to handle the error as well. Let's say AOF rewrite, seek error, and then just log the error over here and return. Why is this giving me an error? Okay, because it returns two values. So let's just ignore the first value by using an underscore. There we go. And now we've cleared the file contents and the AOF file is going to be completely empty. Now we can start the rewriting process. So what we can do now is create a file writer and create a new writer over here and pass in the AOF file as such. This new writer will just create a writer struct which is what we created in one of the earlier parts of this series and what this will basically allow us to do is we can serialize all of the values that we pass in into resp strings and write them to the file directly all right so with that done now we can loop over the entire database copy that we get over here right now pay attention how exactly does Redis send commands to the server it sends them as an array of both strings right so let's create that first of all we need to create the command right which will be a value of type bulk because all redis commands are a set of bulk strings right an array of bulk strings so the command will be where is it bulk set right because all of these are going to be set commands right then we want to create the key which will be a value of type bulk again and this will be the k variable over here right then the value again type bulk and this will be v dot v so this v is a key struct over here right and inside it we have another v variable this v field which will be the actual string value over here right right here so now that we have all of these now we can create an array which will be a value of type array and the array will just be an array of values will pass in the command the key and the value there we go and now we can actually just write this over here by saying f writer dot write and pass in the array as a pointer This is going to write the command by converting it into a REST protocol message, right? And then write that to the file. Now, once all of the keys and values in the database copy have been written, then we're just going to flush the output just once, right? And this is going to make sure that the bytes that we write actually end up on the file on disk And now the entire rewriting process is complete I actually going to add a comment over here as well Rewrite all set commands or actually just write all set commands to file All right. Awesome. Now we're done with the entire rewriting process. Now we should reroute all of future AOF records from the buffer back to the AOF file over here. So let's say reroute future AOF records back to file. There we go. And let's just say AOF dot writer equals new writer and pass in the AOF file over here. And with that, we are done with this rewrite function completely. Now, we're basically done with all of this, but there is one more issue to solve. And that issue is that we have actually not been using the correct code to parse the REST strings over here. So the read array method and the read bulk method, all of these that we created in, I guess, the first or second part of this series have been wrong. All right. I'm sorry. I just messed up over here. And I did not use the, I just didn't build these in the correct way. So let me let me first show you what the problem is and then I'll explain what it is and then we'll solve it All right step by step Ready CLI open up a client. Let me just show you what happens if I run BG rewrite AOF It's going to return this strconf error over here right this parsing error in the server you can see right the problem over here, let me just explain is is this bg rewrite aof command is 12 characters right you can see 1 2 3 4 5 6 7 8 9 10 11 12 it's 12 characters why is that a problem because our read array method and our read bulk method can only support strings of less than 10 characters why because we're directly accessing the first byte in the buffer over here to grab the length of the array and the length of the bulk strings. Why is that a problem? Because we're only accessing one byte over here, right? And one byte can only handle one character, right? So one, two, nine, right? It can't handle two values, two bytes, like 10 or 11 and so on and so forth, right? So that's a problem. And we need to fix that. Otherwise, Otherwise we'll just never be able to use strings or values of more than 9 characters, right? 10 characters and beyond. So let's fix that. So first of all, let's just create a helper method over here. Call it readLine. And this is going to take a reader which will be a buffIo.Reader as such. And it will return a string which will be the line that it reads and an error. let's say line equals r dot read string and pass in the delimiter which will be backslash n. This will also return an error so let's make sure to handle that as well and by the way instead of using double quotes over here let's use single quotes so it doesn't give us an error over here. Awesome now if there is an error let's return an empty string and the error otherwise let's return the line and nil and instead of returning the line directly what we're going to do is we're going to use the strings dot trim suffix um let me import this package over here dot trim suffix function and we're going to trim the backslash r and backslash n the new line characters at the end and this way we won't have to remove these new line characters on our own right this is just going to return us the line without these new line characters so it's going to be very very useful for us now first of all in the read array method let's create a new buff io reader by saying buff io dot new reader passing the reader that we get over here so basically we're gonna get a reader in the read array method and we're going to convert it into a buffered reader using this method. Alright then read the first line by calling the read line function and passing in this reader. Make sure to handle the error as well. If there is an error let's just return it and let's fix this as well. So go up here to the read line method and instead of passing in instead of accepting a buffio.reader let's accept a pointer to buffio.reader. awesome now let's add a bunch of error handling over here let's say if the first line of the the first character in the line is not the star symbol which is the symbol for arrays uh use single quotes over here guys i don't know why i keep forgetting that if this is the case then we can just return a new error over here and say expected array and got something else right now we don't need to use this buffer anymore we can remove all that and instead of using this um in the strconf.xoe call over here right instead of only accepting the first byte we can just say line and then just accept everything after the first character right so if you remember what rest strings look like right it going to be first of all the array symbol which is star then the length of the array let say it 3 and then backslash R backslash N right what we doing now is we reading the first line using this read line method we're trimming the backslash R backslash N so we can remove them over here right all that's left now is the star symbol and the number right the length of the array so we ignore the first character Which will be the star symbol and then everything after that is just going to be the length of the string, right? so now we can just use this function over here and Even if the length of the array is 10 elements or 20 elements or 500 elements. It's not gonna matter It's gonna read it properly this time everything else is perfectly fine over here But we need to make the same changes in this read both matter as well. So it can accept strings of of greater length than 10 characters right now this read bulk method is only being called in this um v dot read bulk over here in this read array method right this should accept a buffered reader as such and let's change this to above i o dot reader as well over here now we don't need to read these directly we can just say line error and just use the read line function that we wrote pass in the buffer make sure to handle the error as well and let me just log the error over here because we don't actually return anything over here error in read bulk and just log it over here and return and you can return an empty value doesn't really matter there we go and then we just need to instead of passing in this buffer and the first byte we can just say line ignore the first character which will be the type of the buffer type of the bulk string which will be the dollar sign and just grab everything after that which will be the length of the buffer string there we go now instead of using reader dot read over here and let me just rename this buffer to buff just like that what we can do is we can use the io.readFull method which will basically just read and keep reading until we fill the buffer right so what we can do is we can say underscore error io.readFull we can pass in the reader which is going to be r and then the buffer which is going to be buff over here make sure there's no error if there is any we can print it and return value over here by the way this is not like best practice to not return the errors like this i'm just doing it because i know this is going to work right and this is kind of like a practice project but in in a production scenario what you should do is you should return the error value so that the caller in this case the read array method right can use it to basically just do whatever it wants right whether that's rerunning the server, retrying the function or something, or just crashing or whatever, right? But we should return an error value over here so that the caller can use it in whatever way seems best, right? I'm not going to do that over here because this is just a practice project, but I'm just saying it's best practice to return the error. Anyway, now we're done with all of this. Let's run the server again and see if everything went well. Let's run the bg rewrite aof command again. And it says background a of rewriting started Let's go over here and you can see it actually did rewrite everything for us, right? So all of the set commands for the name key are now over written. Let's actually try setting the name value again Let's say name one Then let's say name two and name three. Alright, so the latest value should be named three, right? Let's say BG rewrite AOF again And it removed all of them and only gave me name three over here, right? Which is the latest value. So that works as well. The bg rewrite AOF command works. And we can now get a much more compact, much more shorter and cleaner AOF file that doesn't take up as much space, that doesn't take too much time to synchronize with the server. And it's just so much easier to maintain, right? So yeah, that's it. We built this that's the end of this part of the of the series. Thank you for watching so far make sure you like share and subscribe and comment and all of those wonderful things that drive up the YouTube algorithm for me and Thank you for following along with this series so far that it's been like I think this is part 12 of the series So 12 videos on just this one project so far. So if you're still watching this, thank you Thank you. Just thank you so much for supporting the series and watching it for so long right hopefully you're learning new things just as i am and hopefully you're getting some sort of value from this video series right i'm going to continue this for as long as i can but uh yeah so far so good thank you for watching like share subscribe and i will see you in the next video in which we are going to build database transactions that's going to be a very kind of complicated topic but also a very interesting and fun topic to just work on and build right so yeah stay tuned for that and i will see you there