Auto-generated transcript Okay guys, good morning and welcome back. Before we get started, I just got back from a gruesome leg day at the gym. So if I sound a bit low energy right now, you know why. Anyway, we're going to continue building this GoRedis database clone dinghy that we've been building for the last, I don't know, five videos or so. Let me just quickly recap what we did last time. we built this AOF append only file data persistence method right we will disrupt a couple of methods this backup dot a web file which stores the set commands that the client sends in REST format and then it just uses this sync method to read them into this value struck and then just run them and create a new database replica from this backup file so basically we implemented this method of data persistence. Now what we're going to do today in this part is we're going to implement the RDB method of data persistence and RDB is basically it's a lot more of what you really think when you think of data backups like what RDB does is it basically just takes the entire database and like converts it into bytes perform some optimizations to conserve space and everything And then just stores that into a database file on your disks So it's a lot more natural to think of RDB as backups than it is to think of AOF, right? So anyway, that's what we're going to build right now. So first things first, let's just create a file out here and Just call it RDB dot go Package main and then let's create a couple of things over here So let's think about this a bit. When we want to implement RDB data persistence, right, we need to think of a couple things. We need to at least have these. We need to somehow track the amount of seconds that have passed. Right. So we know how many keys have changed. And we also need to track how many keys changed within those seconds. Right. So we need to track both of these things. Now, this would obviously depend on the language you're using. but in Go this is very very simple because we can use channels and in particular tickers from the time package to implement this whole thing right so for convenience sake let me just create a snapshot tracker over here a struct I'll call this snapshot tracker and this will basically just hold all of the RDB snapshot data right we'll give it some fields keys which will be an integer this will be the number of keys that have changed in within these seconds right within this interval we'll give it a ticker as well which will be a time dot ticker and then we'll give it the RDB right which will be the actual RDB snapshot now let's create a new function for this right this will take the RDB snapshot and just return a pointer to this struct. Keys are going to be 0 to start with because no keys have changed obviously. Ticker will just be a new ticker right and the duration can just be the seconds from the RDB struct right so something like this and the actual RDB field will just be this right and by the way this should be a pointer not a struct not an object there we go wonderful so now we have this snapshot tracker struck now we do have multiple RDB snapshots over here right so we should create a list to store all of them and I'll just call this trackers and this will just be a list of snapshot trackers and I will initialize this as such then let's create a function I'll call this initRDBTrackers this will create the actual trackers for us and we will run this at the start of the server to initialize all of these things right and this will take the configuration object right so first of all let's just run a loop right and this will loop over the rdb snapshots in the configuration dot rdb field and then it will create a bunch of trackers right so let's create a tracker pass in the rdb and then let's append this to the trackers list right and change this to a pointer up here awesome now we're creating snapshot trackers and adding them to this trackers list right now we also need to run these trackers on this interval of seconds right so the best way to do that is to just use a go routine right so it can run without blocking the main thread right we're going to call first of all the ticker dot stop method just so you always remember remember to close this channel when we're done with it right then we just loop over this ticker the channel in here until it closes right and we say if the trackers keys are greater than or equal to the tracker dot RDB dot keys changed, then we should run the saveRDB function, alright? And we will create this in a bit, right? But basically, so basically what we're doing over here is we saying if the keys that have changed in this interval are greater than or equal to the keys that we set in this configuration over here right the keys change field then we should save the RDB database right and every time this ticker runs we want to set the trackers keys to zero right now what we're going to do is go down here and create a bunch more functions so let's create an increment function we'll call this incrementRdbTrackers, alright? And this is just going to loop over every single tracker in the trackers list, alright? And we're going to say, c.keys . So we'll run this every time we change a key, right? And this will just increase the number of keys that are changed over here, alright? And then when the interval of seconds runs in this go routine over here and the keys changed are greater than or equal to the one we set in our configuration over here, right? Then we will save the database. It's as simple as that. All right. So now let's actually create the saveRDB function. SaveRDB. And we are going to need the configuration file over here as well. So there we go. pass that in over here as well where we're calling it now first of all let's just grab the file path right and that will be pad.join configuration.dir which is the data directory over here right and where are we there we go then let's join this with the rdb file name and then let's open up this file using os.open file. Alright, pass in the file path. The flag will just be ocreate, which means create the file if it doesn't exist and open it for, where is it, writing only. So we don't need to open it up for reading right now, just for writing. The flag will just be zero or the permissions will just be 0644 which basically means owners have read and write permission all right and everyone else has read permissions that's it make sure to handle the error as well we don't want to skip this under any circumstances. Let's just say error opening rdb file and print the actual error over here and return. And also make sure to close the file at the end by using this defer keyword. Okay, so now we have to figure out how we're going to actually store this um this database into the rdb file now in actual redis it's a lot more complicated than what we're going to do over here i'm just going to implement a very simple method to store the data but in redis in the actual database right what they do is they basically convert the database that they have in memory into bytes and they perform all sorts of different optimizations to conserve space and all sorts of other things, right? We're not going to do all of those things as it would make this function really, really complicated. What we're going to do is we're going to use gob, right? Now this is Go's native binary serialization format, all right? So if you've used something like Python and you've used pickle in Python, you'd know what I'm talking about. But basically this is exclusive to Go and what this basically will do is it will convert any object that we give it right any Go object and it will convert it to a bunch of bytes and we can write those bytes into the file right so this is a very simple and also pretty performant way to to store our data we need to pass in a writer and that can just be the actual file right and then we need to run the encode method and pass in an object we'll just pass in the database store as a reference and of course make sure to handle the error as well error saving rdb file all right and just print the error over here and then return now what this is going to do is it's going to encode the store that we have in memory over here right convert it into bytes and then just write it to the file over here so that's basically all we need to do for this save rdb function now finally to make this work we need to go over to handlers over here and in the set function if the rdb is enabled right and we can tell if it's enabled by just running len state.conf.rdb right so basically if we have a bunch of rdb snapshots in the configuration, you know, stuff like this, this save keyword stuff, right? If we have any of these, that means that we've enabled RDB, right? That makes sense. So if this length of this list is greater than zero, right? Then we know that RDB is enabled and we can then just run the increment function that we built, right? And that's basically all we need to do, right? So every time we run a set command, it's going to increment all of the RDB trackers that we have over here right it'll say that one key changed right and then in this go routine right every time the interval of seconds pass right so something like 900 seconds pass or 300 seconds pass depending on what we have in the configuration right every time that interval of seconds passes this for loop will run and it will check if the number of keys changed are greater than or equal to the number of keys that we set in the configuration here And if so then it will just save the RDB snapshot right simple as that so let try this out and make sure that this works so far right let me just stop the currently running Redis server and then let's run the actual server over here and before I do that let's also add a bunch of logs let's say save RDB file over here all right let's add that and over here we can add another log just to show you guys what's going on let's say log dot print F he's changed right and also he's he's required to change let's say for lack of a better name right I don't know what to call this one but okay now let's print the tracker keys over here and and then the tracker.rdb.keys change over here, all right? And you'll see what's going on. Now, just for example sake, I'm going to remove both of these from the configuration, right, and just say save five, three. So if in five seconds, three keys change, then we should save the database, right? Otherwise not. So let's run this and see what happens. and let me open up another terminal over here and run the CLI and wait before okay so I forgot to actually run the initialize function over here at the top right we should have done that as well let's make sure that the RDB is enabled again if the length of the this list this RDB list over here is greater than zero right if it's greater than zero then rdb is enabled and if so let's run this init function and pass in the configuration over here right i forgot to do this and so it didn't work before okay so this is not what we want it's it's calling the channel way too many times um okay so i think we made a slight mistake over here instead of this um duration instead of just passing in this duration directly we need to multiply it with time dot second so this turns into a second and not i don't know milliseconds or whatever it's nanosecond or whatever it's being represented as right now so do that save everything and then run it again and there we go now let's run the client as well and you can see the logs over here from the go routine it's saying that zero keys have changed right now and at least three keys need to change in order to save the database file, right? And you can see it's running this go routine every five seconds because that's what we've set in the configuration over here. So let's try to set a couple of keys over here, right? And you can see it's saying one key has changed over here, right? But the next time this runs, it's going to reset. See it's back to zero. So three keys need to change in five seconds to save this database file just as we've said in the configuration here. So let me just do this real quick. Right there we go. And we've changed five keys in three seconds. So that should save the RDB file. And you can see over here we have a backup dot RDB file. If I open this it's going to be in bytes in the gob format which we configured over here. down here, right? And yeah, that's basically all we need to do to implement data persistence using RDB, right? Based on the conditions that we set over here, everything is working fine. And as soon as the amount of keys that we need to change are actually changed, the system, the server is going to take the RDB snapshot and save it into the disk over here, right? Awesome. Now, the last thing we need to do is we need to synchronize this to the database on server start right so to do that let's create a sync rdb function this is again going to take the configuration file right or the configuration object and it's going to first of all create the file path we'll say path.join cont.directory, cont.rdb file name and then let's open up the file. This time we're going to open it up in read-only mode, right? So again, create it if it doesn't exist and open it in read-only mode with the same permissions as before, right? So owner has the read and write permission and everyone else gets the read permission all right make sure to handle the error error opening our DB file and just log the error over here and return if we get an error make sure to also close the file right now let me show you a cool new trick the benefit that we get by using this gob encoder right goes native binary serialization encoder is that we don't need to like do any mental gymnastics to create a new database object over here so let me show you what I mean basically when we were using the AOF records right we had to store them in REST format and then we had to convert them back into a value struct right and only then could we like interact with it and execute it right execute these commands on the server and so on and so forth right but since this is in gov format which go understands natively we can do something like gov.new decoder right pass in the file as the reader and run the decode method and we can directly pass in the database.store field as a reference. Again, this is going to return an error. We just need to handle that error over here. And you can say error decoding rdb file and return. and you can see since we're using the this gop format which go natively understands we can literally synchronize the database in just one line right so that's pretty cool now finally let's go over to main.go right and let's just run sync rdb over here all right and pass in the configuration so if the rdb is enabled then we're going to synchronize the rdb database with the server on server start right so in order to make this example um more like easier to see i'm going to clear the entire aof file so this doesn't interfere with the database right and over here you can see the only thing i've set is the name keyword to hassan right so let's um close everything down restart the server and see if the if this name keyword is going to get backed up right let's run a client let's try getting the name and you can see we got it right even though the AOF file is completely cleared right this is coming from the RDB file over here right and if I try to get any other field wait I I guess I set the oh wait never mind yeah so I set the age as well I totally forgot about that let's try getting another field I don't know um job or something yeah that's nil so basically we set the name field and the age field over here it got backed up to the rdb file and then we just load that and synchronize it on server start over here right in main.go and we get all of these fields back so that is all we needed to do and with that we are done setting up the rdb persistence method i'm going to try this one more time just to be just to be safe right try getting the name that works try getting the age that works try getting literally anything else right it's not going to work right so yeah that's that works perfectly right and now of course you can go and change this save keyword in the configuration to something more realistic like something like I don't know 300 seconds like five minutes and 20 keys or something right you can do that but just for the sake of example right because nobody wants to wait 30 300 seconds to test these things out right so I set this back to five and three but yeah that's all we needed to do to implement RDB data persistence now we have a OS data persistence and RDB data persistence both running at the same time so yeah that's perfect I'm gonna end the video over here. In the next part we're going to set up a couple more commands right so so far we have the this command command and get and set but this is obviously not enough right we need to set up a couple more commands so in the next part we're going to set up the delete command the exists command which is going to tell us if a key exists or not in the database and the delete command will basically just delete it right and finally the keys command which is basically a way to which is basically a way to filter keys in the database so basically you run something like I guess I can show you the real implementation of this in database in redis right the video is over by the way you can leave if you want this is just the little outro section over here right so now we're running the actual ready server not the not the one we're building over here alright so if I do something like set name to a Sun right and then I try something like keys name right it's going to return the matching key names right so basically keys is a command where you can pass in a pattern right something like let's say and a.m. and then any given character it's going to return all of the matching key names right that's where it's going to do I can also do something like n and then just a wild card which means any number of characters after this right and then just run that and it's going to return that field again right I can also probably do something like this right and this basically means n and then any given number of characters and then the character which matches this name key which is why it returns this so we're going to build these three commands in the next video the delete command the exists command and the keys command right so stay tuned for that and thank you for watching this one like subscribe share comment all those things and I will see you in the next video thank you for watching and bye bye