Auto-generated transcript Welcome back boys. We're back in the DriveSinker project. Quick recap. In the previous video, we built a CLI, a basic CLI to interact with this program. And the only command I built so far is this sync command that's going to just sync all of our projects to Google Drive, right? In this video, we're going to expand on this a bit more and also add a couple more features to the actual sync process over here. First of all, I don't know if I mentioned this in a previous video but obviously this program is going to upload multiple different projects multiple different folders simultaneously right and so we need to first of all move this entire upload process inside a go routine so we can do all of the we can upload all of the different folders simultaneously right not one by one so we need to move this whole process inside a go routine that's the first thing and secondly I also want to be able to track the upload progress, right? And so we need to build the functionality to do that as well. And we're going to use channels to implement that. And we also need like some sort of an interface or view, right? Where we can show the upload progress of all of the different folders that we have, right? And for that, I'm going to just go with a bunch of progress bars, right? That are going to show the progress of every single upload right so yeah there's a couple of different stuff we need to build in this video and let's get started so first of all before we do anything else let's start tracking the upload progress right so to do that let's create a model in internal slash models slash types dot go over here I'm just gonna call this upload progress this will be a struct and it will have a couple of fields. First of all, a pointer to the folder object, the folder that this upload progress is tracking, right? And then the current file, which will be a string, this will just be the file name. And I also want to track the total files in this folder, which will be an integer, and the uploaded files, which will also be an integer. And these two properties is what will use to determine the progress right out of out of 100 right we're going to use these two numbers to calculate the upload progress next thing we should do is we should go over to sync the command over here not the api slash sync the cmd slash sync file over here And in the upload folder function over here first of all let me just add a channel over here. They should accept a channel which will be a pointer to models.upload progress. The reason I'm using a pointer is just to save memory over here because otherwise I'll have to send a brand new upload progress object every single time I want to send something in this channel, right? Which is why I'm using a pointer instead of like a regular object. And maybe we can also name this better. Let's just call this instead of a regular channel, let's call this progress channel. I think that's a better name to use over here, right? And now we can go back over here to sync.go in the CMD package and we can create a channel. Let's say make channel of models.upload progress and we can pass that in over here. Whoops, I made a mistake over here. Go back to upload folder over here. We need to add a channel keyword over here to make this valid. And now it's fine. did was instead of adding the channel keyword I just defined this progress Chan variable as a pointer to the upload progress model instead of a channel that can hold that pointer so fix that and yeah no more errors and now let's make this whole thing a go routine as well so to do that let's create a weight group up here sync dot weight group there we go and we can move this whole thing to a routine let's define a function over here this function will run the API dot upload folder function and once that's done it'll just say WG dot done and then let's call this function down here and then outside of the loop over here let's just run WG dot wait to wait for the entire wait group to finish executing and let's also add the number of folders over here so let's say WG dot add out here, outside the loop, and let's just get the length of the API.config.folders list over here, and just add all of that to the wait group and then we just upload the folders inside this goroutine over here and then we just wait for for all of the go routines to finish down here right Now let do a quick test just to make sure that we haven broken anything so far And then we'll, you know, continue building this. Okay, so apparently my token has expired and it's not automatically refreshing it for me. So let's take a quick look at this. Quick detour, guys. Let's open up the OAuth file over here. And this error is coming from this get client function down here where we try to generate a token from the token source object over here. And that fails, right? So it says token has been expired. And we do handle this error down here where we test the access to the API. But this get client function is called way before that. Like in the like we call the get client function in this create drive service function over here Right here get client function. You can see the function call So we don't even get a chance to test the access to the API over here So I'm not sure why I'm getting this error over here instead of further down, but we can handle this All we need to do is let's say if strings dot to lower let's just add a quick check over here let's say error dot error I also need to add a call to strings dot contains over here shouldn't forget that strings are contains strings are too lower error dot error and then this contains token has been expired then what we should do is we should just run this get token from web function right and that should be the you know um that should be the way we get the token instead of trying to get it from the file if it's just expired then just get it from the web right and uh yeah just call this and otherwise if it's another error then we just run this um log function over here to log the error and uh this should work now let's see there we go Now it gives me the authorization URL over here. I'm just gonna go and quickly authorize my account. Okay done that and here is my code. Hmm that's weird. It says token is expired again. That's really weird. Let me run the program again Okay so now it says the token is valid Okay so let note that for the future for some reason there a bug in this program where even though we refresh the token for some reason it still says it expired and it makes us refresh it again but if i run the program again like just close the program down and run it again then it works so my guess is that it's just not saving the token to the file correctly and if that's the case we need to fix that probably in some future part of this series because right now i'm trying to focus on the sync command over here so let's not get distracted so the upload process works nothing to change over there let's continue the um let's continue building the sync command where am i okay there we are okay so this is all done in a go routine now and i can add a bunch of different folders to the config over here where's the config yeah i can add a bunch of different folders for a bunch of different projects over here and they will all sync you know simultaneously because we're doing all of this inside a go routine now so that's great but i also want to be able to track the progress right i don't want these log statements over here because frankly they're just a bit difficult to read right especially when you're trying to upload multiple different projects like if you have 12 different side projects that you've got in your programming folder they probably have like hundreds of files in total right you're not gonna scan these logs to you know figure out whether the upload was successful or not so let's build some progress bars before we do that though we do need to move all of these logs to a separate you know a separate output over here right so we don't clutter our our terminal with all of these logs So the first thing we need to do before we build the progress bars is create a logging.go file out here in the root of the project. I'll just add package main over here and all I really want to do is create a function called configure logger and what I want to do is just redirect all of the logs in the program right now to a log file that we can analyze if we ever want to debug the application right. So this is this is not going to take any arguments but it is going to return a pointer to OS dot file and this will be the file that we log the Output to right. So let's say file and an error equals OS dot open file. I'll just call this drive singer Dot log it will take a bunch of flags which will be OS dot append So append to the file instead of rewriting it. Create. So create the file if it doesn't exist already. And open this in write only mode. All right. Not for reading, only for writing. Because we're not going to be automatically reading the logs in this application. Right. We're only going to be writing to them. So that's that. And the permission for this will be 0644. This means that the owner of the file, which is us, is going to have. let me just add comments over here as well. The owner of the file will have read write permissions so we'll be able to read and write to this file and everyone else in this system is going to have read and read permissions over here right. So anyone can read this file but only we the owner of the file can write to it which is fine in my opinion. Let's handle the error as well. Let's say log.f error opening log file and then print the error over here and add a new line as well you'll notice that i'm using log.f to crash the entire program if we can't open the log file because obviously if we can't even open the log file then there is a critical error in this system and we need to fix that and there's no point in running the program any further right so i use fatal f to just crash the entire program in case we can't open the log file. Let's also, actually no, let's not defer the file close over here. We'll do that in the main function. Let's just do log.set output to f. So all of the log statements that we print in our program will be redirected to this file over here. And let's also clean up the formatting of the log statements a bit. Let's say log.setFlags and pass in a bunch of flags. So lDate. I hope I'm pronouncing this right. Yeah, there we go. lDate, lTime, and lShortFile. So this is going to print the date, the time, and the file name over here in every single log statement. And then let's just return the file over here. And then let's open up main.go. Where's main.go? And right here, after we connect to the drive and before we run the CLI, right between over here, let's say configure logger And this will return a file We just say that F And we say defer F Now just in case you didn understand the reason I using f defer f out here in the main function is because the file will be closed when the main function finishes executing And the main function will finish executing when the entire program has been closed, right? Now, if I did the same thing and configure logger over here, right? Like I was about to do before I caught myself. If I said defer f.close in the configure logger function over here, what do you think will happen? What's going to happen is this configure logger function is going to configure our logger, set the output to the file, set the flags and everything, return the file. And then it's going to close the file, which means that now every time we try to log anything to our log file, we're just going to crash because the file is already closed, right? because this function finishes executing. So that is why we do not close the file in this function and we return the file pointer so that we can close it ourselves in the main function over here when the program finishes executing. That's why we return the file. That's why we defer and close it in the main function. So now all of our log statements should go to the log file which means that the only thing that should print in the terminal is anything that we say is like using the fmt package right if i do something like fmt.println and then print something over here that should print in the terminal if i'm doing anything with the log package it should go directly to the log file so let's run this and see what happens okay so it does print these two logs over here and the reason it prints them is because these are coming from the connect to drive function over here which runs before we configure the logger so i can obviously move this up a bit as well in fact let's just do exactly that but you'll notice that all of the logs after that are not even showing up here and it created this drive sinker dot log file for me over here which if i open up this has all of the um upload logs it's telling me all of the folders and files that uploaded it skipped and so on and so forth right okay so we've redirected all of our logs and the output over here should be empty now uh this statement up top over here drive singer video is equal to this path this is coming from i believe it's coming from the config file over here so let remove that print statement as well because we don really need it anymore uh yeah this one over here let remove this and now we should have basically zero output over here right zero output in the terminal which means now we can actually start to build the progress bars now to do that I'm going to create a new package over here I'm going to call this utils and this will just house a bunch of utilities for this project we'll create a file in here called progress bar go at the package utils and this will have a function called render progress bar we are going to export this function so make sure that the r over here is capitalized it's uppercase so we can actually export it and this will take an argument called progress which will be appointed to models dot upload progress all right and it's going to return a string which will be the actual progress bar right now first of all let's just grab the files over here let's just grab the data over here so uploaded will be progress dot uploaded files and total will be progress dot total files and let's get the progress of the upload in a percentage out of 100 by doing uploaded divided by total multiplied by 100 and these should be float 64 because the uploaded and total are going to be integers right but we want to be precise in our calculations over here so i'm going to convert them to float 64 uh over here as well there we go and while we're at it let's also set the bar width which i'm just going to set to 50 for now this is the amount of characters that we're going to have in the progress bar right and uh next thing we need to figure out is how many of these characters are going to be filled and how many are going to be empty. Basically how much progress has the upload made and how much progress is still left right. So let's calculate that. I'll say filled percent divided by 100 multiplied by the bar width and this will need to be converted to a float 64 because the percent is going to be a float 64. Also this whole thing should be converted to an integer why because we are going to use this to print the characters right the characters in the progress bar and we need that to be an integer right because you can't let's say i have a character called c right you can't print 16.7 c's right you can print um a float 64 of a character you always need an integer value that tells you how many characters you going to print right So we convert this to an integer and we also need to figure out how many characters to leave empty, right? How much progress is still left in this upload. To get that we can just say bar width which is the total progress in characters minus the field which is the progress that's been made already in characters and then we can actually print the bar. So this will be an fmt.sprintf. We'll print a string over here and this is going to take a ton of arguments. So first of all a string which and I'll just format it down here like this. It's going to take a string first which will be the progress.folder.name. So which folder we're currently uploading and then it's going to take a inside these square brackets over here. It's going to take two more strings which will be the strings.repeat function we can use this we'll say strings.repeat a hash sign this will be the progress character over here and we'll repeat it for the filled amount over here right and we'll do the same thing down here with empty and the empty character will just be a dash over here right now i also want to print the total upload percentage and i want that to be in two decimal places. So we'll say percentage dot 2f and then two more percentage signs after that. And then we'll print the percent over here. Then I also want to upload the, not upload, but I also want to display the current file that's being uploaded. So we'll say uploading and then print another string over here. And let's also wrap this in quotes. So I'll wrap this whole thing in double quotes. And of course, make sure to escape the quotes as well. And down here, let's add the progress.currentfile string. So we print that as well. And then finally, let's print two digits over here, separated by a slash. This will be the progress.uploaded files. So how many files we've uploaded and progress.total files. So how many files we still have left to upload, right? And then we just return this var string over here. And that is all we need to do to render this progress bar. So now that we have this whole thing done, let's go back to sync the command over here and let's actually use this. Now since this is a since we allow multiple folders to be uploaded at the same time, we want to be able to show multiple progress bars at the same time and the problem with this is going to be that each progress bar is going to overwrite the other one right because like the only way to make this work is to clear each progress bar and rewrite that specific line of the progress bar right so let me just build let me just write out the code for this and i'll explain what this means first of all let's just create a utility function over here called render bars this will take bars which will be a string list over here it won't return anything and we'll just have a loop over here let's say four uh i is less than the length of the bars list and then i'm going to print some fancy terminal stuff over here backslash 033 square bracket f and what this is going to do is it's going to move the cursor up one line right move the cursor up by one line next thing i'm going to do is i'm going to do backslash 033 another square bracket and this time k and this will clear the entire line. So what I'm basically going to do is I'm going to move the cursor up to that specific bar which I'm going to overwrite and then I'm going to clear that entire line and then rewrite the progress bar so we can actually make it look like it's updating in real time right. Once we've cleared all of that we can just say for underscore bar range bars fmt.print bar and it actually instead of print let's do print ln so we actually see one bar per line right okay so now we have this function let's actually use this first of all before anything else in this run function over here in the sync command let's create a list of bars by saying make a string list that has the length of api.config.folders right and then let's actually update each of these progress bars inside of a go routine so let's run another let's create another go routine over here call this down here and we'll say for progress range channel this is the channel over here right that we're creating and passing into the api.upload folder we'll we'll range over it and we'll create its progress bar by saying utils dot render progress bar and we'll render the or we'll pass in the progress over here and now we have the progress bar for this specific upload right then we just say bars i equals bar right now the reason i updating the bars um directly by index instead of doing something like appending to it is because i want to update that specific progress bar of that specific folder right i don't want to append to append to it because that just makes it harder to overwrite that specific bar right in the terminal now this i variable we don't really have it right now because we're ranging over the folders so what i'm going to do instead is i'm going to convert this whole loop over here into a regular loop let's say i equals zero i is less than the length of api.config.folders there we go and i increment over here and then let's say down here f will be api.config.folders i wonderful and that's literally all we need to do now one last thing i did forget to actually send data to this channel in the upload folder function so let's do that as well the very first thing i want to do is right up here in the top of the function i'm going to do defer progress chan and run the close function on it because i want to close this channel as soon as the upload finishes right so very first thing let's just do that and then down here um just below the subfolders over here let's create a progress variable over here which will be models dot upload progress this will be that struct over here uh remove this star symbol remove this asterisk over here i don't know why i don't know why i added that uh this will have a couple of fields first of all the folder which we can just pass in it'll take the total files which we don't currently have right now we'll calculate that in just a second but for now let's just pass in total file zero and uploaded files also zero and then let's send this whole object in the progress channel now you'll notice that i'm not passing in the i'm not defining the current file over here right this current file string and the reason is because we still we haven't started the upload process right so we're not uploading any file right now so the current file is just going to be empty for now right so that's the first progress um symbol that i'm going to send right Now, we need to calculate the total files over here. So let's do that as well. I'm going to do that in a new function up here. Let's just say func countTotalFiles. This video is probably going to be very long by the way but bear with me because this is a lot of features that we adding This will take the to ignore list over here which will be a string list I guess and then fp which will be a string all right and it's going to return I guess an integer and then let's just say var total files is an integer and then you use the filepad.walkzer function passing the filepad and then just define the function over here. First of all, the reason I am accepting this to ignore list is because we don't want to count every single file inside the folder, right? We want to ignore files that are in the git folder, for example, that are in the node modules folder, for example, that are in all of these folders that we're ignoring, right? We want to make sure that we ignore all of those files since we're not even going to upload them, right? So there's no reason in counting them. So that is why I accepted this to ignore list up here. And I'm just going to copy this whole stuff in the upload folder function and paste it in here as well just so we ignored all of those files and folders that we want to ignore this folder dot pack should just be FP again and then once we've ignored all of that stuff we can just say if the if D is not a directory if this current file is not a directory then we should just say total files plus plus right otherwise actually we don't really need an else condition over here let's just say return nil for no error over here and then finally out here in the function let's just return total files over here right i'm not counting directories because there's no like there's nothing to upload in just a directory right the directories the folders they're just names right they're just labels for you know objects the actual upload data is all inside files right because yeah so that is why i'm ignoring the directories and only counting actual files. So now we're actually counting the files. Let's over here, let's say total files equals count total files, passing the to ignore list, which should be defined way earlier. So let's move this entire to ignore functionality up here, just below the subfolders, the definition over here. And then let's also pass in the folder.path over here and now we have the total files and you can just pass them in over here in the progress channel so that the initial progress notification that i sending through this channel now down here where we upload files i going to say just before we actually upload the file I going to say progress dot current file equals let's say what is the file name over here I guess D dot name and then let's send that to the progress channel there we go and down here I know that this is returning the function I'll fix that in just a second down here when the when the file upload finishes we can say progress.uploaded files and we can increment this and then we can send that in the progress channel as well there we go and uh since this is um returning over here this code will not run so let's return uh let's remove this return and this upload folder function upload file function is going to return an error so let's handle that error if the error does not equal to nil then let's return the error right otherwise we're basically golden over here and let's just run this progress stuff over here down here and then let's let's return nil for um returning some sort of an object over here so this filepad.walkdirt doesn't complain at us because this does accept an error in the return case so we make we make sure to return something down here right anyway with all of that done i think we should be done with this whole thing so let's just try and run this whole thing and see what happens and by the way before we let me also just remove everything in the drive sinker log over here okay so it's uploading all of the files but i don't see any progress bar being printed i wonder why oh oh We're so retarded guys. I forgot to actually call this function So down here in the loop and just go routine over here Let's just run render bars passing the bars over here and now it should work I can't believe I forgot to do something so basic now. We should see something then there we go awesome Okay, so now we have a progress bar over here It's gonna tell us how much we've uploaded the percentage of the upload progress I'm gonna tell us the current file we're uploading and also how many files we've uploaded so far and how many are left to go Right amazing. Let me actually add another project to this whole Program over here. Let me see what I what I even have. Okay, so let's add the You guys remember my go redis project. Yeah, let's add that as well. So basically if you don't know before I built a redis database clone in Go. There's a whole playlist of that in my in my youtube channel. There's like 20 videos of it so if you haven't checked that out go check that out. But let's add this second project over here as well and let's run this and see what happens. There we go. We have two progress bars now and as you can see they're uploading files simultaneously because it's in a routine so one upload is not going to hold back the other one and yeah this whole thing works amazing absolutely amazing so now we can have like as many projects as you want in this config I can have 20 different projects and they're all going to upload simultaneously and they're going to show me the upload progress over here so I don't have to worry about it okay quick bug over here you'll notice that the that the program exited before this whole before the first progress bar over here for DriveSync video came to a hundred percent right the upload did finish the upload was successfully finished right the problem is that the that this go routine finished before the actual what can I say before the progress bar was actually rendered up to 100 right So I think one way we could fix this is we could do something like, we could call wg.done in this goroutine as well. Or actually, no, we couldn't do that. Okay, so we do need to fix that part, but this video is getting extremely long and I'm just going to end it over here because yeah, this is getting extremely long. We have this tiny little bug to fix that the rendering of the progress bar doesn't actually wait, or the upload process doesn't actually wait until the rendering of the progress bar reaches 100%. We need to fix that little bug. And yeah, other than that, this whole thing works. We have progress bars now. We can upload multiple folders simultaneously. Productive video. Yeah, I'm gonna end the video over there, guys. If you like this, then go ahead and press like and subscribe, share the video, comment. If you have any questions, this was a very long video. I'm sure some of you have some questions about what I did and everything. Feel free to ask them. Feel free to ask questions and I will make sure to answer them in the comments. And thank you for watching and I will see you in the next video.