Syncing projects with Google Drive - Part 9 (Concurrent uploads and progress bars)

Welcome back, boys. We're back in the DriveSinker project. Quick recap. In the previous video, we built a CLI, a basic CLI to interact with this program. And the only command I built so far is this sync command that's going to just sync all of our projects to Google Drive, right? In this video, we're going to expand on this a bit more and also add a couple more features to the actual sync process over here. First of all, I don't know if I mentioned this in a previous video, but obviously this program is going to upload multiple different projects, multiple different folders simultaneously, right? And so we need to, first of all, move this entire upload process inside a goroutine so we can do all of the, we can upload all of the different folders simultaneously, right? Not one by one. So we need to move this whole process inside a goroutine. That's the first thing. And secondly, I also want to be able to track the upload progress, right? And so we need to build the functionality to do that as well. And we're going to use channels to implement that. And we also need like some sort of an interface or view, right, where we can show the upload progress of all of the different folders that we have, right? And for that, I'm going to just go with a bunch of progress bars, right, that are going to show the progress of every single upload, right? So yeah, there are a couple of different stuff we need to build in this video, and let's get started. So first of all, before we do anything else, let's start tracking the upload progress, right? So to do that, let's create a model in `internal/models/types.go` over here. I'm just gonna call this `UploadProgress`. This will be a struct, and it will have a couple of fields. First of all, a pointer to the folder object, the folder that this upload progress is tracking, right? And then the current file, which will be a string. This will just be the file name. And I also want to track the total files in this folder, which will be an integer, and the uploaded files, which will also be an integer. And these two, these two properties is what we will use to determine the progress, right? Out of, out of a hundred percent, right? We're going to use these two numbers to calculate the upload progress. Next thing we should do is we should go over to sync the command over here, not the API slash sync, the `cmd/sync` file over here. And in the `UploadFolder` function over here, first of all, let me just add a channel over here. They should accept a channel, which will be a pointer to `models.UploadProgress`. The reason I'm using a pointer is just to save memory over here because otherwise I'll have to send a brand new upload progress object every single time I want to send something in this channel, right? Which is why I'm using a pointer instead of like a regular object. And maybe we can also name this better. Let's just call this instead of a regular channel, let's call this `progressChannel`. I think that's a better name to use over here, right? And now we can go back over here to `sync.go` in the `cmd` package, and we can create a channel. Let's say `make(chan models.UploadProgress)`, and we can pass that in over here. Whoops, I made a mistake over here. Go back to `UploadFolder` over here. This is, we need to add a `chan` keyword over here to make this valid. And now it's fine. So what I did was, instead of adding the `chan` keyword, I just defined this `progressChan` variable as a pointer to the upload progress model instead of a channel that can hold that pointer. So fix that, and yeah, no more errors. And now let's make this whole thing a goroutine as well. So to do that, let's create a wait group up here, `sync.WaitGroup`. There we go. And we can move this whole thing to a goroutine. Let's define a function over here. This function will run the `api.UploadFolder` function, and once that's done, it'll just say `wg.Done`, and then let's call this function down here. And then outside of the loop over here, let's just run `wg.Wait` to wait for the entire wait group to finish executing. And let's also add the number of folders over here. So let's say `wg.Add` out here, outside the loop, and let's just get the length of the `api.config.foldersList` over here and just add all of that to the wait group. And then we'll just upload the folders inside this goroutine over here, and then we'll just wait for all of the goroutines to finish down here. Right now, let's do a quick test just to make sure that we haven't broken anything so far, and then we'll, you know, continue building this. Okay, so apparently my token is expired, and it's not automatically refreshing it for me. So let's take a quick look at this. Quick detour, guys. Let's open up the OAuth file over here, and this error is coming from this `GetClient` function down here, where we try to generate a token from token source object over here, and that fails, right? So it says token has been expired, and we do handle this error down here where we test the access to the API, but this `GetClient` function is called way before that. Like in the, like we call the `GetClient` function in this `CreateDriveService` function over here, right here, `GetClient` function. You can see the function call. So we don't even get a chance to test the access to the API over here. So I'm not sure why I'm getting this error over here instead of further down, but we can handle this. All we need to do is let say if `strings`, let's just add a quick check over here. Let's say error. I also need to add a call to `strings.Contains` over here. Shouldn't forget that. `strings.Contains`, `strings.ToLower`, `error.Error`. And then this contains "token has been expired." Then what we should do is we should just run this `GetTokenFromWeb` function, right? And that should be the, you know, that should be the way we get the token instead of trying to get it from the file. If it's just expired, then just get it from the web, right? And yeah, just call this, and otherwise, if it's another error, then we just run this log function over here to log the error, and this should work now. Let's see. There we go. Now it gives me the authorization URL over here. I'm just going to go and quickly authorize my account. Okay, done that, and here is my code. That's weird. It says token is expired again. That's really weird. Let me run the program again. Okay, so now it says the token is valid. Okay, so let's note that for the future. For some reason, there's a bug in this program where even though we refresh the token, for some reason, it still says it's expired and it makes us refresh it again. But if I run the program again, like just close the program down and run it again, then it works. So my guess is that it's just not saving the token to the file correctly, and if that's the case, we need to fix that probably in some future part of this series because right now, now I'm trying to focus on the sync command over here. So let's not get distracted. So the upload process works. Nothing to change over there. Let's continue the, let's continue building the sync command. Where am I? Okay, there we are. Okay, so this is all done in a goroutine now. And I can add a bunch of different folders to the config over here. Where's the config? Yeah, I can add a bunch of different folders for a bunch of different projects over here, and they will all sync, you know, simultaneously because we're doing all of this inside a goroutine now. So that's great. But I also want to be able to track the progress, right? I don't want these log statements over here because frankly, they're just a bit difficult to read, right? Especially when you're trying to upload multiple different projects. Like if you have 12 different side projects that you got in your programming folder, they probably have like hundreds of files in total, right? You're not gonna scan these logs to, you know, figure out whether the upload was successful or not. So let's build some progress bars. Before we do that, though, we do need to move all of these logs to a separate, you know, a separate output over here, right? So we don't clutter our terminal with all of these logs. So the first thing we need to do before we build the progress bars is create a logging.go file out here in the root of the project. I'll just add package main over here, and all I really want to do is create a function called configure logger. And what I want to do is just redirect all of the logs in the program right now to a log file that we can analyze if we ever want to debug the application, right? So this is not going to take any arguments, but it is going to return a pointer to os.File, and this will be the file that we log the output to, right? So let's say file and an error equals os.OpenFile. I'll just call this drivesinger.log. It'll take a bunch of flags, which will be os.O_APPEND, so append to the file instead of rewriting it. O_CREATE, so create the file if it doesn't exist already. And open this in write-only mode. All right, not for reading, only for writing, because we're not going to be automatically reading the logs in this application, right? We're only going to be writing to them. So that's that. And the permission for this will be 0644. This means that the owner of the file, which is us, is going to have, let me just add comments over here as well, the owner of the file will have read-write permissions, so we'll be able to read and write to this file, and everyone else in this system is going to have read and read permissions over here, right? So anyone can read this file, but only we, the owner of the file, can write to it, which is fine in my opinion. Let's handle the error as well. Let's just say log.FatalF, error opening log file, and then print the error over here and add a new line as well. You'll notice that I'm using log.FatalF to crash the entire program if we can open the log file because obviously, if we can't even open the log file, then there is a critical error in this system and we need to fix that, and there's no point in running the program any further, right? So I use FatalF to just crash the entire program in case we can't open the log file. Let's also, actually, no, let's not defer the file close over here. We'll do that in the main function. Let's just do log.SetOutput to f, so all of the log statements that we print in our, in our program will be redirected to this file over here. And let's also clean up the formatting of the log statements a bit. Let's say log.SetFlags and pass in a bunch of flags. So Ldate, I hope I'm pronouncing this right. Yeah, there we go. Ldate, Ltime, and Lshortfile. So this is going to print the date, the time, and the file name over here in every single log statement. And then let's just return the file over here. And then let's open up main.go. Where is main.go? And right here, after we connect to the drive, and before we run the CLI, right between over here, let's say configure logger, and this will return a file. We just say that f, and we say defer f.Close. Now, just in case you didn't understand the reason I'm using f, defer f out here in the main function is because the file will be closed when the main function finishes executing. And the main function will finish executing when the entire program has been closed, right? Now, if I did the same thing in configure logger over here, right? Like I was about to do before I caught myself, if I said defer f.Close in the configure logger function over here, what do you think will happen? What's going to happen is this configure logger function is going to configure our logger, set the output to the file, set the flags and everything, return the file, and then it's going to close the file, which means that now every time we try to log anything to our log file, we're just going to crash because the file is already closed, right? Because this function finishes executing. So that is why we do not close the file in this function, and we return the file pointer so that we can close it ourselves in the main function over here when the program finishes executing. That's why we return the file. That's why we defer and close it in the main function. So now all of our log statements should go to the log file, which means that the only thing that should print in the terminal is anything that we say is like using the fmt package, right? If I do something like fmt.Println and then print something over here, that should print in the terminal. If I'm doing anything with the log package, it should go directly to the log file. So let's run this and see what happens. Okay, so it does print these two logs over here, and the reason it prints them is because these are coming from the connect to drive function over here, which runs before we configure the logger. So I can obviously move this up a bit as well. In fact, let's just do exactly that. But you'll notice that all of the logs after that are not even showing up here, and it created this drive singer dot log file for me over here, which, if I open up, has all of the upload logs. It's telling me all of the folders and files that uploaded, it skipped, and so and so forth, right? Okay, so we've redirected all of our logs, and the output over here should be empty now. This statement up top over here, drive singer video is equal to this path, this is coming from, I believe it's coming from the config file over here. So let's remove that print statement as well because we don't really need it anymore. Yeah, this one over here. Let's remove. Move this, and now we should have basically zero output over here, right? Zero output in the terminal, which means now we can actually start to build the progress bars. Now, to do that, I'm going to create a new package over here. I'm going to call this utils, and this will just house a bunch of utilities for this project. We'll create a file in here called progressbar.go, add the package utils, and this will have a function called render progress bar. We are going to export this function, so make sure that the R over here is capitalized. It's uppercase, so we can actually export it. And this will take an argument called progress, which will be a pointer to models dot upload progress, all right? And it's going to return a string, which will be the actual progress bar. Right now, first of all, let's just grab the files over here. Let's just grab the data over here. So uploaded will be progress dot uploaded files, and total will be progress.total files. And let's get the progress of the upload in a percentage out of 100 by doing uploaded divided by total multiplied by 100. And these should be float64 because the uploaded and total are going to be integers, right? But we want to be precise in our calculations over here. So I'm going to convert them to float64 over here as well. There we go. And while we're at it, let's also set the bar width, which I'm just going to set to 50 for now. This is the amount of characters that we're going to have in the progress bar, right? And next thing we need to figure out is how many of these characters are going to be filled and how many are going to be empty. Basically, how much progress has the upload made, and how much progress is still left, right? So let's calculate that. I'll say filled percent divided by 100 multiplied by the bar width, and this will need to be converted to a float64 because the percent is going to be upload64. Also, this whole thing should be converted to an integer. Why? Because we are going to use this to print the characters, right? The characters in the progress bar. And we need that to be an integer, right? Because you can't, let's say I have a character called C, right? You can print 16 Cs, right? You can print a float64 of a character. You always need an integer value that tells you how many characters you going to print, right? So we convert this to an integer, and we also need to figure out how many characters to leave empty, right? How much progress is still left in this upload. To get that, we can just say bar width, which is the total progress in characters, minus the filled, which is the progress that's been made already in characters, and then we can actually print the bar. So this will be an fmt.Sprintf. We'll print a string over here, and this is going to take a ton of arguments. So first of all, a string, which, and I'll just format it down here like this. It's going to take a string first, which will be the progress.folder.name. So which folder we're currently uploading. And then it's going to take, inside these square brackets over here, it's going to take two more strings, which will be the strings.Repeat function. We can use this. We'll say strings.Repeat a hash sign. This will be the progress character over here, and we'll repeat it for the filled amount over here, right? And we'll do the same thing down here with empty, and the empty character will just be a dash over here, right? Now, I also want to print the total upload percentage, and I want that to be in two decimal places. So we say percentage dot 2f, and then two more percentage signs after that. And then we'll print the percent over here. Then I also want to upload the, not upload, but I also want to display the current file that's being uploaded. So we'll say uploading, and then print another string over here. And let's also wrap this in quotes. So I'll wrap this whole thing in double quotes, and of course, make sure to escape the quotes as well. And down here, let's add the `progress.currentFile` string so we print that as well. And then finally, let's print two digits over here, separated by a slash. This will be the `progress.uploadedFiles`, so how many files we've uploaded, and `progress.totalFiles`, so how many files we still have left to upload, right? And then we just return this bar string over here, and that is all we need to do to render this progress bar. So now that we have this whole thing done, let's go back to sync the command over here and let's actually use this. Now, since this is a, since we allow multiple folders to be uploaded at the same time, we want to be able to show multiple progress bars at the same time. And the problem with this is going to be that each progress bar is going to overwrite the other one, right? Because, like, the only way to make this work is to clear each progress bar and rewrite that specific line of the progress bar, right? So let me just build, let me just write out the code for this, and I'll explain what this means. First of all, let's just create a utility function over here called `renderBars`. This will take `bars`, which will be a string list over here. It won't return anything, and we'll just have a loop over here. Let's say `for i` is less than the length of the `bars` list, and then I'm going to print some fancy terminal stuff over here: `\033[F`. And what this is going to do is it's going to move the cursor up one line, right? Move the cursor up by one line. Next thing I'm going to do is I'm going to do `\033` another square bracket, and this time `K`, and this will clear the entire line. So what I'm basically going to do is I'm going to move the cursor up to that specific bar which I'm going to overwrite, and then I'm going to clear that entire line and then rewrite the progress bar so we can actually make it look like it updating in real time, right? Once we cleared all of that, we can just say `for _bar range bars fmt bar`. And actually, instead of print, let's do `Println`. So we actually see one bar per line, right? Okay, so now we have this function. Let's actually use this. First of all, before anything else in this run function over here in the sync command, let's create a list of bars by saying make a string list that has the length of `api.config.folders`, right? And then let's actually update each of these progress bars inside of a goroutine. So let's run another, let's create another goroutine over here, call this down here, and we'll say `for progress range channel`. This is the channel over here, right, that we're creating and passing into the `api.uploadFolder`. We'll range over it, and we'll create its progress bar by saying `utils.renderProgressBar`, and we'll render the, or we'll pass in the progress over here. And now we have the progress bar for this specific upload, right? Then we just say `bars[i] = bar`. Now, the reason I'm updating the bars directly by index instead of doing something like appending to it is because I want to update that specific progress bar of that specific folder, right? I don't want to append to it because that just makes it harder to overwrite that specific bar, right, in the terminal. Now, this `i` variable, we don't really have it right now because we're ranging over the folders, so what I'm going to do instead is I'm going to convert this whole loop over here into a regular loop. Let's say `i = 0`, `i` is less than the length of `api.config.folders`. There we go, and `i` increment over here. And then let's say down here, `f` will be `api.config.folders[i]`. Wonderful, and that's literally all we need to do. Now, one last thing I did forget to actually send data to this channel in the `uploadFolder` function, so let's do that as well. The very first thing I want to do is, right up here in the top of the function, I'm going to do `defer progressChan` and run the close function on it because I want to close this channel as soon as the upload finishes, right? So very first thing, let's just do that. And then down here, just below the subfolders over here, let's create a progress variable over here, which will be `models.UploadProgress`. This will be that struct over here. Remove this star symbol. Remove this asterisk over here. I don't know why I added that. This will have a couple of fields. First of all, the folder, which we can just pass in. It'll take the total files, which we don't currently have right now. We'll calculate that in just a second. But for now, let's just pass in `totalFiles` zero and `uploadedFiles` also zero. And then let's send this whole object in the progress channel. Now, you'll notice that I'm not passing in the... I'm not defining the `currentFile` over here, right? This `currentFile` string. And the reason is because we haven't started the upload process, right? So we're not uploading any file right now. So the current file is just going to be empty for now, right? So that's the first progress symbol that I'm going to send, right? Now we need to calculate the total files over here. So let's do that as well. I'm going to do that in a new function up here. Let's just say `func countTotalFiles`. This video is probably going to be very long, by the way, but bear with me because this is a lot of features that we're adding. This will take the ignore list over here, which will be a string list, I guess, and then `fp`, which will be a string. All right, and it's going to return, I guess, an integer. And then let's just say `var totalFiles` is an integer, and then use the `filepad.WalkDir` function, passing the `filepad`, and then just define the function over here. First of all, the reason I'm accepting this `toIgnore` list is because we don't want to count every single file inside the folder, right? We want to ignore files that are in the Git folder, for example, that are in the node_modules folder, for example, that are in all of these folders that we're ignoring, right? We want to make sure that we ignore all of those files since we're not even going to upload them, right? So there's no reason in counting them. So that is why I accepted this `toIgnore` list up here. And I'm just going to copy this whole stuff in the upload folder function and paste it here as well, just so we ignore all of those files and folders that we want to ignore. This `folder.Path` should just be `fp` again, and then once we've ignored all of that stuff, we can just say if `d` is not a directory, if this current file is not a directory, then we should just say `totalFiles++`, right? Otherwise, or actually, we don't really need an else condition over here. Let's just say return nil for no error over here. And then finally, out here in the function, let's just return `totalFiles` over here, right? I'm not counting directories because there's nothing to upload in just a directory, right? The directory is the folders. They're just names, right? They're just labels for, you know, objects. The actual upload data is all inside files, right? Because, yeah, so that is why I'm ignoring the directories and only counting actual files. So now we're actually counting the files. Let's, over here, let's say `totalFiles` equals `countTotalFiles`, passing the `toIgnore` list, which should be defined way earlier. So let's move this entire `toIgnore` functionality up here, just below the subfolders, the definition over here. And then let's also pass in the `folder.Path` over here. And now we have the total files, and you can just pass them in over here in the progress channel. So that's the initial progress notification that I'm sending through this channel. Now, down here where we upload files, I'm going to say, just before we actually upload the file, I'm going to say `progress.CurrentFile` equals, let's say, what is the file name over here? I guess `d.Name`, and then let's send that to the progress channel. There we go. And down here, I know that this is returning the function. I'll fix that in just a second. Down here, when the file upload finishes, we can say `progress.UploadedFiles`, and we can increment this, and then we can send that in the progress channel as well. There we go. And since this is returning over here, this code will not run. So let's return, let's remove this return, and this upload folder function, upload file function, is going to return an error. So let's handle that error. If the error does not equal nil, then let's return the error, right? Otherwise, we're basically golden over here. And let's just run this progress stuff over here down here, and then let's return nil for returning some sort of an object over here so this `filepad.WalkDir` doesn't complain at us, because this does accept an error in the return case. So we make sure to return something down here, right? Anyway, with all of that done, I think we should be done with this whole thing. So let's just try and run this whole thing and see what happens. And by the way, before we, let me also just remove everything in the DriveSyncer log over here. Okay, so it's uploading all of the files, but I don't see any progress bar being printed. I wonder why. Oh, oh, we're so retarded, guys. I forgot to actually call this function. So down here in the loop, in this goroutine over here, let's just run `renderBars`, passing the bars over here, and now it should work. I can't believe I forgot to do something so basic. Now we should see something. Then there we go. Awesome! Okay, so now we have a progress bar over here. It's gonna tell us how much we've uploaded, the percentage of the upload progress. It's gonna tell us the current file we're uploading, and also how many files we've uploaded so far and how many are left to go, right? Amazing. Let me actually add another project to this whole program over here. Let me see what I even have. Okay, so let's add the—you guys remember my Go Redis project? Yeah, let's add that as well. So basically, if you don't know, I built a Redis database clone in Go. There's a whole playlist of that on my YouTube channel. There's like 20 videos of it, so if you haven't checked that out, go check that out. But let's add this second project over here as well, and let's run this and see what happens. There we go. We have two progress bars now, and as you can see, they're uploading files simultaneously because it's in a goroutine, so one upload is not going to hold back the other one. And, yeah, this whole thing works. Amazing, absolutely amazing. So now we can have, like, as many projects as you want in this config. I can have 20 different projects, and they're all going to upload simultaneously, and they're going to show me the upload progress over here, so I don't have to worry about it. Okay, quick bug over here. You'll notice that the program exited before this whole, before the first progress bar over here for DriveSync video came to 100%, right? The upload did finish. The upload was successfully finished, right? The problem is that the goroutine finished before the actual, what can I say, before the progress bar was actually rendered up to 100%, right? So I think one way we could fix this is we could do something like, we could call `wg.Done` in this goroutine as well. Or actually, no, we couldn't do that. Okay, so we do need to fix that part, but this video is getting extremely long, and I'm just going to end it over here because, yeah, this is getting extremely long. We have this tiny little bug to fix, that the rendering of the progress bar doesn't actually wait, or the upload process doesn't actually wait until the rendering of the progress bar reaches 100%. We need to fix that little bug. And, yeah, other than that, this whole thing works. We have progress bars now. We can upload multiple folders simultaneously. Productive video. Yeah, I'm gonna end the video over there, guys. If you like this, then go ahead and press like and subscribe. Share the video, comment if you have any questions. This was a very long video. I'm sure some of you have some questions about what I did and everything. Feel free to ask them, feel free to ask questions, and I will make sure to answer them in the comments. And thank you for watching, and I will see you in the next video.