Auto-generated transcript okay guys welcome back to this video series this is part four of the go redis database clone series whatever you want to call it um in the in the last video we added concurrency support for this server so we could listen to where is it uh so we could listen to multiple clients at the same time and the way we did that is by using this mutex over here right to lock and unlock access to the database depending on which client is accessing it at the moment right and secondly we also built this config file and the parser for the config file right so we can actually grab all of these different fields and their values and based on that we can set up data persistence right speaking of which that is what we're going to do in this part we're going to build data persistence now before i actually get into the code and everything right we we need to talk about the different methods that Redis gives us to store data. So let me just open up a little notepad over here so I can take notes. There's basically two ways that Redis allows us to store data, right? One of them is AOF, which is append only. Append only file. And the other one is RDB, right? And I forgot what this stands for, but who cares, right? What this basically does is it basically takes the entire database from memory and stores it in a file in bytes, right? Both of these methods work a little bit differently, right? So let me explain RGB first, since that is probably very natural, right? It's a lot more natural to think of it this way. So what RGB does is every time you ask Redis to take an RGB snapshot, what it does is it basically takes all of the keys and values in your memory, right? And it's going to convert them all into bytes, right? And then it's going to save them all in the database file, which over here we've set to backup.rdb, right? So it's going to store them in backup.rdb, or whatever you call this file in your config, right? That's what it's going to do, all right? And there's basically two ways to ask Redis to create a database snapshot in RGB format, right? One of them is automatic. You can use this save keyword and add a bunch of values after that. So over here, if 900 seconds pass, and within those 900 seconds, one key changes in the database, then we will save the database in an RGB snapshot, right? So you can use these save commands over here to automatically create RDB snapshots based on a bunch of conditions, right? Or you can actually use the save command and obviously we haven't built this yet, right? But you can actually use this save command and it will manually create a database snapshot and save it in RDB format, right? So those are the two ways that we can ask Redis to create an RDB snapshot, right? And as for the append only file, what this does is it basically creates an AOF file which is basically just a bunch of REST strings, right? So you remember how REST strings look like, right? I don't really have an example over here but basically, where is it? Yeah, so strings like these, right? You remember the REST protocol strings that we were using in the earlier parts of this video, right? Basically what AOF does is it basically records the set commands, right? Converts them into, let me just write it all down as well, records each set command as a REST string, right? And every time you set a new value, it's going to append that to the AOF file, right and that's why it's called append because it appends to the AOF file right so what you'll basically do is you'll use the database like normal and what Redis is going to do behind the scenes is it's going to grab each of the set commands that you send convert them into a resp string and append that resp string to the AOF file now to restore data from this file what Redis is going to do is it's going to take all of the REST strings, right, from the file, parse them, right, and then just rerun all of these set commands that we have in this append only file, right, and what will end up happening is it basically is going to create a complete copy of the earlier database that you had, right, by running all of these set commands, right, so So let me just show you a quick example. Let me run the Redis CLI. If I say something like set name Hassan, right? And set age to 19. What it's going to do behind the scenes is it's going to convert these commands into REST strings, save that to an AOF file, right? And then when I run the server again, it's going to rerun all of those set commands from the AOF file to create basically a whole replica of this database, right? That's how AOF files work and that's how RDB files work. We explained that in previously. So yeah, we're going to build both of these. But first, for this part of the video, we're only going to build AOF file because you can imagine there's a lot of complexity in this whole process and we need to, it's just going to take a lot of time to build both of these. So I'm going to split them into two parts. And for this part I'm only going to build append only files right hopefully I explained everything correctly but even if I didn't as long as you follow along in the code you should have some idea of what's going on alright so let's create an AOF dot go file and by the way if I if something is unclear or if I'm not explaining anything correctly then please leave a comment below it helps me improve as well and I'll make sure to whatever it is you're having trouble I'll make sure to explain that again in the next part or just reply in a comment or something. So If you're having any trouble with my explanations, please let me know Let's add package main over here and then let's create an AOF struct. Alright, this is going to take a couple things First of all, it's going to take a writer which will be a pointer to the writer struct that we have over here. Alright second it's going to take a file which is going to be a pointer to os.file and thirdly it's going to take a config which is going to be a pointer to the config object that we created over here up here all right now let's create a function called new aof to actually create this struct and this will return a pointer to the aof object this will need to have the config parameter over here argument and then let's create an aof struct right and pass in the config over here now let's actually um create and open the aof database file right so we'll say the file path is path.join right and first of all we need to grab the database directory right this data folder from the config and then we need to join that with the aof file name backup that we have over here all right and then we can open up this file so let say f and er equals os first of all we need to pass in the file path which will be fp second of all we need to pass in the flags which will be um create so So create the file if it doesn't exist. Append so only open it in append mode not write mode right because we don't want to overwrite the previous stuff right. And open it for reading and writing both at the same time alright. As for permissions I'm going to say 0644. What this means is the owner will have read and write permissions and everyone else will have the read permissions right so only the creator of this file which is us can write to this file right so those are the permissions that we're setting over here with 0 6 4 4 let's make sure to handle the error as well let's say cannot open this file path and then just return the AOF right and if we can open this file successfully then we should just assign the writer to this new writer function and pass in the file and as for this F field we're just going to pass in the file then finally we can just return this AOF object so that's the new AOF function complete now in order to actually in order to actually report the AOF records every time we run the set command we need to actually have some sort of a global state object that can hold all of these other different objects right so to do that let me go back over here to main and create an app state struct alright so let's call this app state this will be a struct it will hold the config file right or the config object and the AOF object as well all right then then let's create the new app state function this will take the configuration object that we read and return an app state pointer let's say state equals app state and pass in the configuration file over here and if the AOF mode is enabled then we should say state dot AOF equals the new AOF function and pass in the config over there as well and then we can just return this state over here as a pointer and now what you can do is we can actually store this configuration that we read in a conf file create a new app appState from this new appState function, pass in the configuration over here, and we can then pass in this whole state to this handle function down here. Right, and then let's go over here and allow it to accept this state pointer. There we go. And we're also going to pass this state into every single handler. So to do that first, update the type over here to accept an app state pointer as such. And then we just need to add that to every single handler over here. So in get, let's add it over here, right? In set as well, and then also in the command over here. So now we have the app state everywhere. Now we can actually, in the set command, we can actually use this state variable to record the command in the AOF file. So in this set command, once we've actually stored the value in the database, right, let's say if state.conf.aofenable, so if the AOF is actually enabled, right, then what we can do is we can say state.AOF.W the writer.write right and passing the value that we get in the set command over here right and this is why I'm using this writer field over here right if I show you AOF again you might think I could just use this OS.File object and just write the raw bytes right but the problem there is that I actually have to convert them into a REST format as well, right? This writer struct that we created before, this can already convert values into REST format, right? So we can just reuse this, pass in the value that we get from the client and just pass it in into this writer and it will automatically convert it into a REST format, which we can then write to the file, right? That is why I'm using this writer struct over here in this AOF file as well. Let's save this and save the main file as well. And let's just rerun the server over here, right? And see what happens if I run a set command. Let's set name to Hassan. It says okay. Let's go over here to the data and you can see the backup.aof file was created. Now, there's nothing in here right now because it's not flushing the output to the database, to the file, right? So let me just run this again, set age to 19. And you can see it's not really flushing the output right now, right? And the reason for that is that this is actually a buffered writer, right? So the data that we're writing right now, it's just being stored in the buffer and we actually need to flush it out to actually store it. So let's figure that out next. And let's also add a log statement over here and just say saving AOF record, right? Just so we know what's going on, right? Now, we all know that the messages that the Redis client sends, right? They are arrays, right? And we're not really handling arrays over here, right? We're just handling strings, bulks, errors, and null values, right? We need to handle arrays as well so we can deserialize and serialize arrays into and out of the REST protocol format so that we can write them to the AOF file. So before I do that, let me just move all of this up into a serialize function. All right. This will be a pointer receiver to the writer. We'll call this deserialize. It's going to take a value pointer. All right. And it's going to return a reply, which will be a string. There we go. We don't need this declaration up here anymore. Let's add a default case as well. Let's say log.println invalid type received, right? And just return. Now let's handle the array data type over here, right? So this is actually going to be a lot simpler than you might think at first. Let's say reply is equal to fmt.sprintf, right? Now, you remember that the arrays in REST format, what they have first is the symbol, right? The star symbol, right? So let's add that. And then the length of the array, right? So let calculate that as well using this line function We say lenv and then we just need to add the new line characters over here right There we go and after that the rest is simple We just need to parse the individual elements in the array right. So to do that we can just say we can just use a loop over here. I'll say for sub which is the sub value range v.array. So grab every single element inside this array and then we can just append this to the reply and we can call the deserialized method on this again right on the sub value turn this into a reference and that's all we need to do so basically we're just using recursion over here to parse every single value inside the array right and whatever we get over here from the deserialized function we're just going to append that to the actual reply and in the end we're going to get the full array deserialized into rest format awesome now over here we can just say reply equals w.deserialized and just pass in the value over here right and that's all we need to do and over here in this return statement let's just return the actual reply and if none of this works then just return an empty string there we go or you could just also return reply here as well it's just going to be an empty string again if none of these cases match right and with that we are done with this writer struct so let's run this whole thing again right and this time you should see some output in the AOF file let's say set name to Hassan. If I can spell my name correctly, there we go. And there we go. You see, we're getting this resp format array over here. Let me try setting my age as well and there we go. We get the age over here as well. Awesome. Now there's a couple more things to do over here. First of all we need to move this flush to its own method all right so let's do that as well let's say this will be another pointer receiver right and there we go the reason I'm doing this is I only want to flush the output from the buffer to the file on certain conditions so let me show you the configuration file again right see this f-sync keyword over here It sets every second right now, right if I go back to config and show you the fsync modes If I set this to every second, right? I want the rider to flush every one second, right? If I set this to always I want the riders of flush every time I write something to the file, right? That could be Multiple times per second, right? That could be every hour or something, right? That depends, right? And if I set it to no, I don't want the flush function to run at all, right? I want to let the operating system handle when to flush the output from the buffer to the file I don't want to manually do it in any Circumstances, right? So that's what these three modes mean, right? And in order to configure every single one of these modes We need to move this whole flush thing into its own method so we can interact with it separately from the right method All right, so that's why I move this to its own method now Let's go back to the set command over here and handle the flushing of this output right now We're writing the set command to this AOF file, but in order to flush it We need to first see what the actual mode is, right? So if the fsync mode equals always right Then that means we should flush the output every single time we write to the AOF file, right? So let's do that all we need to do is say state dot a left dot writer dot flush there We go which means that now if I set this Where is it f sync to always right and? Then I run it again There we go And now it's not Running the client because it's not flushing the output back. So let me explain what's happening over here because I think it's a learning moment Since we move this flush to its own method right every time we write to the connection you can see over here that We're using this writer struck in the actual connection as well, right? So if I don't flush the output it just stays in the buffer and the client never receives that output, right? The client isn't receiving the output and so we're not getting any interface over here, right? So we need to fix that and in order to do that What we can do is go and go into this handlers file and go into the handle function up here This is where we're writing the reply to the connection, right? But we're not flushing it out to the actual output, right? It just stays in the buffer if it stays in the buffer It's never gonna be sent to the actual client to the actual connection if they don't receive the output they can't really do anything with it, right? So let's flush this right here. Every time you're trying to write to a network connection, you should always flush the output after every single message, right? Just to make sure that it gets sent. But when it comes to writing to system files like backup.aof or something like that, then you can use a buffered writer to, you know, for performance reasons, right? So you can write a lot of data to the buffer and then sync all of that to the file all at once, right? So that's why we're using a buffered writer. But for network connections, you should always flush the output every single time you write it, just so that the network can, so that the connection or the client on the other side of the network can receive the message. So let's run this whole thing again. And there we go. This time it works, right? This time we're flushing the reply every single time we write to it so it works let's set the name to hassan again and you can see it gets set over here as well let's set the age to i don't know 69 and you can see it gets set over here as well because the aof mode fsync mode is set to always this is going to get updated the file is going to get updated every single time um here every single time I run a set command. Alright, so that works. Next we just need to handle the every second mode, right? So if this is set to every second then we don't really need to do anything in the set command. We just need a simple go routine that runs every one second and flushes the output, right? And we can do that over here in this new app state function. What we can say is if AOF is enabled and if the fsync mode is set to every second right then we can run a go routine over here this is going to have a time dot ticker all right for one second because we want this to run every single second and then before anything else let's just create a dipper to stop this ticker once the go routine ends right always make sure to stop tickers, close files and connections and stuff like that. Just so you don't have any unused resources lying around in your application. And then we can just use a simple range loop to listen to this channel and every one second we can flush the writer. There we go Alright So now this is set to every second If I run the entire server again and if I open up redis cli let me just remove everything in here right just so we're starting fresh let's run the cli let's set the name to hassan and you can see there's a very minor delay in writing this file right before it was instant right before if i set anything over here if i run the set command it will just save it instantly right but if i run this now there's a bit of a delay right there's a bit of a one second delay over here because what's going on is it's running this go routine and every one second it's flushing the output to the file right so if i run the set command it stays in the buffer and every one second it's flushed out from the buffer to the actual file right so with that we've also configured the every second mode over here right lastly we just have the no um mode which basically we don't really we don't really need to do anything about this because if the f-sync mode is set to no then we basically just write to the buffer and we let the operating system handle when to flush that output to the buffer so we don't need to do anything with this right so we don't even need to bother with implementing this because there's literally nothing to implement right we basically just shift that responsibility off to the operating system so now we've configured all three f-sync modes and with that we've basically configured all of aof right now we actually need to synchronize all of these records from the file to the um to the database right so let me remove all of this let me run the server again and the client let's set the name to hassan and let's set the age to 19 all right i'm going to stop over there clear everything and now let's create this synchronize function and before we can even do that let me first go over to the value file uh which is over here and over here in this read array method i want to return an error all right so this reader.read method it's going to return the number of bytes read and an error message we don't really need to worry about the bytes so i'll ignore that but we do need the error field right so if we do get an error over here then i want to return that error over here as well instead of printing the error i just want to return it right and if there is no error then I just want to return nil all right there we go let's make sure that everything is still running let's run a server run um let's try getting name obviously it's nil right now because we haven't built the synchronize function but the server works so now let's go back over here to the AOF file let's run let's create the synchronize function um this will be a pointer receiver to the AOF object and we'll call it sync. This will be an infinite loop first of all. We'll create a value struct right and then we will run the read array method. The reader can just be the AOF file. Now let's handle the error as well. If the error is the end of file error that means we've reached the end of the file then we can basically just stop right and if it's some other error then we can just print the error over here unexpected error while reading aof records and print the error after that and then just uh return right or you can just break over here it's the same thing and then we need to create an app state so i'll call this blank state and just create a new app state with an empty config all right because i don't want to use the current configuration that we have over here in this in this aof struct right because this configuration is going to have stuff like aof enable aof backup and f-sync and this is going to mess around with the set commands right so we want to use a completely blank app state that doesn't have aof enabled and then we can just call the set command pass in the value as a pointer and pass in the blank state as well there we go and that's basically all we need to do in order to sync this right let's go over to the main function up here and let's write a log over here and say syncing AOF records all right and then let's just run the state dot AOF dot what is this sync method over here right and let's only run this if can't dot AOF is enabled all right there we go awesome now let's run this and you can see first it reads the config file then it syncs the AOF records and now it's listening on this port right so let's run the ready CLI again let's try to get name and you can see this time I'm not getting nil I'm getting Hassan right let's try to get the age as well and I'm getting 19 over here perfect right so this is successfully synchronizing all of the records from the AOF file into the database over here right even if I close the client and if I run this whole server and client again I'm still gonna get the exact same values right and of course I can change these let's say I set the age to 20 I exit right and then I run the whole thing again let me try getting age and now it's 20 right so AOF records are being stored over here in this file and they are being synchronized with the database with the server every time we run the server over here right awesome so that took a while let me see how much time that took um we've been recording for like 40 minutes over here i'll probably cut this a lot and it'll probably end up being like a 30 minute video but you can see even implementing aof data persistence right that took like 30 minutes right this is why i wanted to split the AOF and RDB data persistence methods into two videos. Alright, so in this video we implemented AOF. We did the whole thing and now it's successfully working and everything. In the next video we're going to implement RDB data persistence and hopefully that will be a bit simpler, right? So yeah, good progress so far. This project is coming along very nicely. I'm gonna try and keep building new features in this and continue this series for as long as possible because I Am seeing a lot of interest in this in this video series These videos are getting a lot more views than my regular videos. So clearly there's a lot of interest here So I'll try to keep this series going for as long as I can just building new features and all that good stuff, right and Yeah, thank you for supporting the series. Thank you for watching like subscribe And if you want to work with me or something get in touch or whatever my website will be linked in the description below once again, thank you for watching and Stay tuned for the next part of this video series, which will be coming out in like one or two days So stay tuned. Thank you for watching and I will see you in the next video