Building a Redis clone in Go - Part 5 (RDB Data Persistence)

Okay, guys, good morning and welcome back. Before we get started, I just got back from a gruesome leg day at the gym. So if I sound a bit low energy right now, you know why. Anyway, we're going to continue building this GoRedis database clone dinghy that we've been building for the last, I don't know, five videos or so. Let me just quickly recap what we did last time. We built this AOF (append-only file) data persistence method. Right, where we just write a couple of methods that back up this `backup.aof` file, which stores the set commands that the client sends in REST format. And then it just uses this sync method to read them into this value struct and then just run them and create a new database replica from this backup file. So basically, we implemented this method of data persistence. Now, what we're going to do today in this part is we're going to implement the RDB method of data persistence. And RDB is basically, it's a lot more of what you really think when you think of data backups. Like, what RDB does is it basically just takes the entire database and, like, converts it into bytes, performs some optimizations to conserve space and everything, and then just stores that into a database file on your disk. So it's a lot more natural to think of RDB as backups than it is to think of AOF, right? So anyway, that's what we're going to build right now. So first things first, let's just create a file out here and just call it `rdb.go`, package main. And then let's create a couple of things over here. So let's think about this a bit. When we want to implement RDB data persistence, right, we need to think of a couple things. We need to at least have these, we need to somehow track the amount of seconds that have passed, right? So we know how many keys have changed. And we also need to track how many keys changed within those seconds, right? So we need to track both of these things. Now, this would obviously depend on the language you're using, but in Go, this is very, very simple because we can use channels and, in particular, tickers from the time package to implement this whole thing. So, for convenience's sake, let me just create a snapshot tracker over here, a struct of all this snapshot tracker. And this will basically just hold all of the RDB snapshot data, right? We'll give it some fields: `keys`, which will be an integer. This will be the number of keys that have changed in within these seconds, right, within this interval. We'll give it a ticker as well, which will be a `time.Ticker`, and then we'll give it the RDB, right, which will be the actual RDB snapshot. Now, let's create a new function for this, right? This will take the RDB snapshot and just return a pointer to this struct. Keys are going to be 0 to start with because no keys have changed, obviously. Ticker will just be a new ticker, right? And the duration can just be the seconds from the RDB struct, right? So something like this. And the actual RDB field will just be this, right? And by the way, this should be a pointer, not a struct, not an object. There we go. Wonderful. So now we have this snapshot tracker struct. Now we do have multiple RDB snapshots over here, right? So we should create a list to store all of them. And I'll just call this `trackers`. And this will just be a list of snapshot trackers. And I will initialize this as such. Then let's create a function. I'll call this `initRdbTrackers`. We'll create the actual trackers for us, and we will run this at the start of the server to initialize all of these things, right? And this will take the configuration object, right? So first of all, let's just run a loop, right? And this will loop over the RDB snapshots in the configuration dot RDB field, and then it will create a bunch of trackers, right? So let's create a tracker, pass in the RDB, and then let's append this to the trackers list, right? And change this to a pointer up here. Awesome. So now we're creating snapshot trackers and adding them to this trackers list, right? Now we also need to run these trackers on this interval of seconds, right? So the best way to do that is to just use a goroutine, right? So it can run without blocking the main thread, right? We're going to call, first of all, the ticker.Stop method, just so we always remember to close this channel when we're done with it, right? Then we just loop over this ticker, the channel in here, until it closes, right? And we say if the tracker's keys are greater than or equal to the tracker.rdb.keysChanged, then we should run the `saveRdb` function. All right, and we will create this in a bit, right? But basically, so basically, what we're doing over here is we're saying if the keys that have changed in this interval are greater than or equal to the keys that we set in this configuration over here, right, the keysChanged field, then we should save the RDB database, right? And every time this ticker runs, we want to set the tracker's keys to zero, right? Now, what we're going to do is go down here and create a bunch more functions. So let's create an increment function. We'll call this `incrementRdbTrackers`. All right, and this is just going to loop over every single tracker in the trackers list. All right, and we're going to say `t.keys` increment. So we'll run this every time we change a key, right? And this will just increase the number of keys that are changed over here. All right, and then when the interval of seconds runs in this goroutine over here, and the keys changed are greater than or equal to the one we set in our configuration over here, right, then we will save the database. It's as simple as that. All right, so now let's actually create the `saveRdb` function. `saveRdb`, and we are going to need the configuration file over here as well. So there we go, and pass that in over here as well, where we're calling it. Now, first of all, let's just grab the file path, right? And that will be `path.Join` configuration.dir, which is the data directory over here, right? And where are we? There we go. Then let's join this with the RDB file name, and then let's open up this file using `os.OpenFile`. All right, passing the file path. The flag will just be `O_CREATE`, which means create the file if it doesn't exist, and open it for, where is it, writing only. So we don't need to open it up for reading, right now just for writing. The flag will just be zero, or the permissions will just be `0644`, which basically means owners have read and write permission, all right, and everyone else has read permissions. That's it. Make sure to handle the error as well. We don't want to skip this under any circumstances. Let's just say, "Error opening RDB file," and print the actual error over here and return. And also make sure to close the file at the end by using this defer keyword. Okay, so now we have to figure out how we're going to actually store this database into the RDB file. Now, in actual Redis, it's a lot more complicated than what we're going to do over here. I'm just going to implement a very simple method to store the data. But in Redis, in the actual database, right, what they do is it basically converts the database that they have in memory into bytes, and they perform all sorts of different optimizations to conserve space and all sorts of other things, right? We're not going to do all of those things, as it would make this function really, really complicated. What we're going to do is we're going to use gob, right? Now, this is Go's native binary serialization format, all right? So if you've used something like Python and you've used pickle in Python, you'd know what I'm talking about. But basically, this is exclusive to Go, and what this basically will do is it will convert any object that we give it, right? Any Go object, and it will convert it to a bunch of bytes, and we can write those bytes into the file, right? So this is a very simple and also pretty performant way to store our data. We need to pass in a writer, and that can just be the actual file, right? And then we need to run the encode method and pass in an object. We'll just pass in the database store as a reference, and of course, make sure to handle the error as well. "Error saving our DB file." All right, and just print the error over here and then return. Now, what this is going to do is it's going to encode the store that we have in memory over here, right? Convert it into bytes and then just write it to the file over here. So that's basically all we need to do for this saveRdb function. Now, finally, to make this work, we need to go over to handlers over here, and in the set function, if the RDB is enabled, right, and we can tell if it's enabled by just running len(state.conf.rdb), right? So basically, if we have a bunch of RDB snapshots in the configuration, you know, stuff like this, this save keyword stuff, right? If we have any of these, that means that we've enabled RDB, right? That makes sense. So if this length of this list is greater than zero, right, then we know that RDB is enabled, and we can then just run the increment function that we built, right? And that's basically all we need to do, right? So every time we run a set command, it's going to increment all of the RDB trackers that we have over here, right? It'll say that one key changed, right? And then in this goroutine, right, every time the interval of seconds pass, right, so something like 900 seconds pass or 300 seconds pass, depending on what we have in the configuration, right, every time that interval of seconds passes, this for loop will run, and it will check if the number of keys changed are greater than or equal to the number of keys that we set in the configuration here. And if so, then it will just save the RDB snapshot, right? Simple as that. So let's try this out and make sure that this works so far, right? Let me just stop the currently running Redis server, and then let's run the actual server over here. And before I do that, let's also add a bunch of logs. Let's say, "Save RDB file" over here. All right, let's add that. And over here, we can add another log just to show you guys what's going on. Let's say log.Printf("Keys changed") and also "keys required to change." Let's say, for lack of a better name, right? I don't know what to call this one, but okay, now let's print the tracker keys over here and the tracker.rdb.keysChange over here, all right? And you'll see what's going on. Now, just for example's sake, I'm going to remove both of these from the configuration, right, and just say save 5 3. So if, in five seconds, three keys change, then we should save the database, right? Otherwise not. So let's run this and see what happens. And let me open up another terminal over here and run the CLI. And wait, before... Okay, so I forgot to actually run the initialize function over here at the top, right? We should have done that as well. Let's make sure that the RDB is enabled again, if the length of this list, this RDB list over here, is greater than zero, right? If it's greater than zero, then RDB is enabled, and if so, let's run this init function and pass in the configuration over here, right? I forgot to do this, and so it didn't work before. Okay, so this is not what we want. It's calling the channel way too many times. Okay, so I think we made a slight mistake over here. Instead of this duration, instead of just passing in this duration directly, we need to multiply it with time.Second so this turns into a second and not, I don't know, milliseconds or whatever. It's nanosecond or whatever it's being represented as right now. So do that, save everything, and then run it again. And there we go. Now let's run the client as well, and you can see the logs over here from the goroutine. It's saying that zero keys have changed right now, and at least three keys need to change in order to save the database file, right? And you can see it's running this goroutine every five seconds because that's what we've set in the configuration over here. So let's try to set a couple of keys over here, right? And you can see it's saying one key has changed over here, right? But the next time this runs, it's going to reset. See, it's back to zero. So three keys need to change in five seconds to save this database file, just as we've set in the configuration here. So let me just do this real quick, right? There we go, and we've changed five keys in three seconds, so that should save the RDB file. And you can see over here we have a backup.rdb file. If I open this, it's going to be in bytes in the gob format, which we configured over here down here, right? And yeah, that's basically all we need to do to implement data persistence using RDB, right, based on the conditions that we set over here. Everything is working fine, and as soon as the amount of keys that we need to change are actually changed, the system, the server, is going to take the RDB snapshot and save it into the disk over here, right? Awesome. Now, the last thing we need to do is we need to synchronize this to the database on server start. So to do that, let's create a syncRDB function. This is again going to take the configuration file, right, or the configuration object, and it's going to first of all create the file path. We'll say `path.Join`, `config.Directory`, `config.RDBFileName`, and then let's open up the file. This time, we're going to open it up in read-only mode, right? So again, create it if it doesn't exist and open it in read-only mode with the same permissions as before, right? So owner has the read and write permission, and everyone else gets the read permission. All right, make sure to handle the error. Error opening our DB file, and just log the error over here and return if we get an error. Make sure to also close the file. Right now, let me show you a cool new trick. The benefit that we get by using this Gob encoder, right, Go's native binary serialization encoder, is that we don't need to do any mental gymnastics to create a new database object over here. So let me show you what I mean. Basically, when we were using the AOF records, right, we had to store them in REST format, and then we had to convert them back into a value struct, right? And only then could we interact with it and execute it, right? Execute these commands on the server and so on and so forth, right? But since this is in Gob format, which Go understands natively, we can do something like `gob.NewDecoder`, right? Pass in the file as the reader and run the `Decode` method, and we can directly pass in the `database.Store` field as a reference. Again, this is going to return an error. We just need to handle that error over here. And you can say, "Error decoding RDB file," and return. And you can see, since we're using this Gob format, which Go natively understands, we can literally synchronize the database in just one line, right? So that's pretty cool. Now, finally, let's go over to `main.go`, right? And let's just run `SyncRDB` over here, all right, and pass in the configuration. So if the RDB is enabled, then we're going to synchronize the RDB database with the server on server start, right? So in order to make this example more, like, easier to see, I'm going to clear the entire AOF file so this doesn't interfere with the database, right? And over here, you can see the only thing I've set is the name keyword to Hassan, right? So let's close everything down, restart the server, and see if this name keyword is going to get backed up, right? Let's run a client. Let's try getting the name, and you can see we got it, right? Even though the AOF file is completely cleared, right? This is coming from the RDB file over here, right? And if I try to get any other field, wait, I guess I set the—oh, wait, never mind. Yeah, so I set the age as well. I totally forgot about that. Let's try getting another field. I don't know, job or something. Yeah, that's nil. So basically, we set the name field and the age field over here. It got backed up to the RDB file, and then we just load that and synchronize it on server start over here, right, in `main.go`, and we get all of these fields back. So that is all we needed to do, and with that, we are done setting up the RDB persistence method. I'm going to try this one more time just to be safe, right? Try getting the name. That works. Try getting the age. That works. Try getting literally anything else, right? It's not going to work, right? So yeah, that works perfectly, right? And now, of course, you can go and change this save keyword in the configuration to something more realistic, like something like, I don't know, 300 seconds, like five minutes, and 20 keys or something, right? You can do that. But just for the sake of example, right, because nobody wants to wait 300 seconds to test these things out, right? So I set this back to 5 and 3. But yeah, that's all we needed to do to implement RDB data persistence. Now we have AOF data persistence and RDB data persistence both running at the same time. So yeah, that's perfect. I'm gonna end the video over here. In the next part, we're going to set up a couple more commands, right? So far, we have this command, command, and get and set, but this is obviously not enough, right? We need to set up a couple more commands. So in the next part, we're going to set up the delete command, the exists command, which is going to tell us if a key exists or not in the database, and the delete command will basically just delete it, right? And finally, the keys command, which is basically a way to filter keys in the database. So basically, you run something like—I guess I can show you the real implementation of this in the database in Redis, right? The video is over, by the way. You can leave if you want. This is just the little outro section over here, right? So now we're running the actual Redis server, not the one we're building over here, all right? So if I do something like `SET name Hassan`, right, and then I try something like `KEYS name`, right, it's going to return the matching key names, right? So basically, `KEYS` is a command where you can pass in a pattern, right? Something like `nam` and then any given character. It's going to return all of the matching key names, right? That's what it's going to do. I can also do something like `n` and then just a wildcard, which means any number of characters after this, right? And then just run that, and it's going to return that field again, right? I can also probably do something like this, right? And this basically means `n` and then any given number of characters and then the `e` character, which matches this name key, which is why it returns this. So we're going to build these three commands in the next video: the delete command, the exists command, and the keys command, right? So stay tuned for that, and thank you for watching this one. Like, subscribe, share, comment, all those things, and I will see you in the next video. Thank you for watching, and bye.