Building a Redis clone in Go - Part 13 (Database Transactions)

Welcome back, guys. We're going to continue building this Go Redis clone. In the previous part of this series, we built the rewrite function over here in the AOF struct in order to rewrite our AOF file to take up less space, to remove all of the outdated old set commands, and just turn it into a much more compact and minimal file backup, right? In this video, we're going to build database transactions. So let me just first walk you through how they work in Redis and then how we can build this, right? So we're going to... Let me just close our current server and start the official Redis server. There we go. Awesome. Now, let's run the Redis CLI. Now, this is using the Redis server that's actually built by Redis, not by us, right? Not this project. What we can do is, in order to start a database transaction, we run the MULTI keyword, right? And this is going to return OK to tell us that the transaction has been started. And you can see over here we get this TX over here, right? Which just tells us that we're inside a direct inside a database transaction. Now we can run any kind of commands like key 1 XYZ and key 2, I don't know, DEF or something, right? Just assigning random variables and random values over here. Every time you run a command, it's going to say queued, queued, right? So it's just going to queue all of the commands that you run. And it's not going to actually execute them until you run EXEC, right? And then when you do, it's going to return the output of every single command like this in an array, right? And if you want to discard the transaction, if you don't want to execute the commands, you can run the DISCARD command, which it's not going to work unless you're inside a transaction using MULTI, right? But you can use that to discard a transaction if you don't want to execute the commands, right? So that's what we need to do. We basically need to implement command handlers for the... Let me go, let me go. Where is it? To handlers. There we go. Let me go up to the handlers list and add all of these. We basically need to create command handlers for the MULTI command, the EXEC command to execute, right? And by the way, there is a package in Go named exec, so we're not going to use this keyword. I'm just going to add an underscore before it, just so it doesn't conflict with the os/exec package, right? That's built into Go. And we want to handle the DISCARD command as well. There we go. I was going to write Discord instead of DISCARD. Okay, there we go. So we want to implement command handlers for these three commands. But before we do that, we also want to create a transaction struct that can hold all of the commands that we want to execute, right? Because remember, we want to add all of these commands inside a queue and then execute them all together, right? So let me just save that and go back to the DB struct over here. And down here, we're going to create a transaction struct, right? This will have a bunch of commands, which will be a list of pointers to TXCommand. This will be transaction command. Add struct over here. There we go. And then let's create the TXCommand struct as well. This will basically have a value, which will be the actual command that the client sends to us, and then also a handler, which will be a handler, basically a command handler function. The same as what we've been writing over here, right, in the handlers file. And yeah, that all we need. Let's also create a new transaction function over here to create new transactions. This will return a pointer to the transaction. And there we go. Awesome. Now, that's all we need to do. Now, let's implement these three commands over here. So let me uncomment these and create the command handlers for all of these over here. See, client V is the value, and then state is the app state. There we go. And it will return a value. Let me just copy-paste this three times, right? And this one will be called EXEC. Notice the underscore so it doesn't interfere with the exec package, right? And this one will be DISCARD. Now, before we actually implement any of these, we want to be able to store the database transaction in some sort of global state variable, right? And the best way to do that is to just add it to the app state over here. Let's go into the app state, create a transaction variable. I'll just call this TX for transaction, and this will hold a transaction pointer as such. And now we can actually start to implement this. So in the MULTI command handler, first of all, let's just make sure that there is no transaction currently being used, right? So if the state.transaction is not nil, if there is already a database transaction that's being used, right, we should return an error, TypeError, and Redis does not allow multiple transactions to be nested together, right? So we'll say error MULTI calls cannot be nested, all right? We'll say that. And if there is no transaction currently running, then we'll just say the state transaction equals a new blank transaction. And then just like you saw over here in the official Redis server, it's going to return a success message and OK string, so we can do that as well. Just return OK over here, and that is all we need to do for the MULTI command. As soon as we run this, it's going to start the start the new transaction, and then we can just queue our commands and so on and so forth. Now, let's also implement the DISCARD command over here. And first of all, let's just check if there is already a transaction or not, all right? If there is no transaction, then DISCARD doesn't make sense, right? Just like you saw over here, if we try to run DISCARD without using MULTI first, without being inside a transaction first, then it returns this error over here, right? DISCARD without MULTI. So let's return that over here as well. If there is no transaction running, we return an error, which will be error DISCARD without MULTI, right? There we go. And if there is a transaction, then we just say state.transaction equals nil, so there's no more transactions, and then we return an OK success message, just like this. There we go. That's the MULTI and the DISCARD command handlers all built. Now, before we implement the EXEC execute command handler, let's first go up to the handle function over here, right? So we've already implemented the MULTI command, which is going to start the transaction and help us to queue all of the upcoming commands, right? But how exactly do we queue them? Right? We do them in the handle function over here. So we're going to go down here after the invalid command handlers have been checked, after the authentication stuff has been checked. Down here, we're going to say if the state.transaction does not equal nil, so there is a database transaction running, and the command that we received does not equal EXECUTE or the command does not equal DISCARD. So basically, the only two commands that can, you know, change the transaction state, basically execute or discard the transaction. If the command that we receive isn't this, then we need to queue that command inside the transaction. Right? So what we're going to do is we're going to say `TxCommand` equals this new `TxCommand` struct we created. The value will just be the `v` variable over here that we receive in the handle function, right? And the handler will be the handler over here, up here. Right there we go. This handler that we get from the handlers list, we'll pass it in over here. And then we just append this transaction command into `state.transaction.commands` by saying append and then just append a pointer to it over here like this. Then we want to write the success message to the client. So we'll say type string, and the string will be queued, just like you saw over here, right? Queued, queued, right? Flush it and return because we don't want to actually execute the handlers. We just want to queue them all up over here like this, right? Now all of the commands that we sent to the server after running MULTI are going to be queued up in this transaction until we run EXEC or DISCARD. So we've already implemented DISCARD over here, right? Now let's implement EXEC. So first of all, just like DISCARD, if there is no transaction, then we should return an error over here. And we can call this error `EXEC without MULTI` because if there is no MULTI command, if there is no transaction that's being used currently, then what are we even executing, right? So make sure to check that. And then we can say replies. So the reply to every single command, you can see that when we run EXEC over here, it's going to execute every single command that we queued up, right? And it's going to return an array, and each of the element inside this array is going to be the output of every single command that we queued up, right? So let's do that. We want to create a list of replies, which will be the reply that we send from every single command in the queue. So let's run the make command over here. This is going to be a list of values. The length of this is going to be the length of the `state.transaction.commands` list, right? And then we can just loop over this. We can say `for index, command range state.transaction.commands`, and we can say `reply` equals `cmd.handler`, which will be the handler function. We can pass in the client, the value and the state. And by the way, instead of passing in the value like this, we want to pass in the `command.value` over here, right? From this command variable over here in the loop. This will return the reply from this command handler, and then we can assign that to the replies list using this `replies[i]` and just say reply. There we go. Awesome. The reason, by the way, that I do this direct assignment using the index over here instead of using the append function is performance, right? We can already create a list of values, right? And the length of this list can already be, like, received from the state commands list, right? That the state transaction commands list. The reason I use this direct assignment approach instead of the append function is performance. Basically, if we try to append every single reply to this replies list, it just becomes very bad for performance because you basically reallocating and growing the capacity and the length of the array in every single append call, right? And that just not a good way to handle these kinds of things, especially when we know what the size of this array is going to be. We know that the number of replies is going to be equal to the number of commands inside this transaction, right? So we can pre-make a list of replies, and the length of it we can already pre-allocate to the state transaction commands, right? And then we can just directly assign them over here. And this is just a much more performant way to create this list rather than using the append function, which will have to reallocate and reassign all of these different, different elements again and again, right? So hopefully that makes sense. That is why we use this direct assignment approach instead of the append method. But yeah, now we can build the final reply, and the final reply will just be a value of type array. And the array in here will be, will just hold the replies. We can say that the state transaction is now nil because we've executed it, and then we can return the reply as a pointer. And that's all we need. Now let me shut down the currently running Redis server over here. There we go. And run my own server. Awesome. Clear everything. Start a client. Let's say MULTI to start a transaction. It returns an okay, and it gives me this `tx` keyword in the bracket over here, which tells me that I am inside a database transaction now. I can run a bunch of commands. Let's say key one equals hello and key two equals world, right? And you can see that it's going to queue both of them up, right? I can also say GET K2 and GET K3, right? K3 doesn't exist, but basically we can still queue it all up. And I can say EXECUTE, and it's going to return the output of both of these, right? So first of all, we set the keys K1 and K2. It gives us an okay success message for that. We try to then get the K2 key, which is world. Then we try to get K3, which doesn't exist. So it returns nil, right? Awesome. So this works. I can try another transaction, and this time I can say SET name to Hassan again, and I can then just discard it, right? So let's say I don't want to make these changes. I can run DISCARD. It's going to say okay, and it's not going to run those commands for us, right? And yeah, that's how database transactions work inside of Redis. This is actually a lot simpler than how most database systems run database transactions, right? Transactions in things like MySQL or Postgres or something, right? They're a lot more complicated than what Redis does over here. Transactions in Redis are actually very simple. So yeah, I hope you learned something from this. We've built database transactions here now, and yeah, thank you for watching. Hopefully you learned something. Like, share, subscribe, comment, all those wonderful things. In the next part of this series, we're going to look at, I believe, let me check. Yeah, we're going to look at eviction policies and memory management, and how we can track the memory usage of our database and how we can basically manage it to make sure that we don't run out of memory. And if we do, we can remove a bunch of keys to make more memory, right? And so on and so forth, right? So we're going to look into all those super complicated details of databases, right? That's going to be very fun. But yeah, thank you for watching this one, and I will see you in the next video. Bye bye.