Auto-generated transcript testing testing testing 1 million 2 million 3 million testing all right back in the drive syncer project in the previous video we built the progress bar functionality right here we basically made all of the uploads concurrent and we created these progress bars over here so you could actually see the upload progress right now the next major feature i want to implement is um a much more better and performant sync algorithm. Basically, let me just show you, where is the sync API over here? Yeah, here we go. Okay, so basically, when we upload a project folder right now, what we do is we delete the previous folder completely, and we upload the entire folder again, right? And this is, it works, it gets the job done. And to be fair, I don't think you're gonna run into any Google Drive limits over here, because their API limits are actually very generous. But at the same time, I just don't like this because it's really wasteful, right? You have to upload every single file all over again every time you try to sync a new project, right? Or even an old project, right? Which is not good, right? So what I wanna do is instead of deleting the previous folder, I want to implement a much better upload algorithm that can basically just skip the files that have not been changed and only re-upload the changed files, right? The files that we've actually modified, right? And that's a lot of work, right? It's gonna take about three or four videos to actually implement that entire thing. So stick around and we'll get started on that in this video. Before we do that though, I want to fix one feature in the progress bar that we built last time. So let me just run a test upload over here so you can see what I mean, right? let's just try uploading something right now. By the way, let me also zoom in so it's easier for you guys to see. Okay, so it needs to grant access. So I'm going to go do that real quick. Okay, here's my code. Why didn't that work? Okay, so there's still some issue with the OAuth implementation where it just exits, crashes the first time, but it runs the second time. And there we go. It's actually uploading now. Okay, so there's these two projects that we're uploading right now I'm gonna let them finish and then we'll see what the problem is basically the progress bar doesn't hit a hundred percent right and Google is telling me that some hacker logged in yeah that was me basically these progress bars are not going to hit a hundred percent progress right even though the upload finishes it it doesn't wait for the progress bar to reach 100% and so it looks like it didn't finish properly right so that's what we need to fix now and there you go see the go redis video reaches 100% but this drive sync video project this only reaches 94.44% and it looks like it's unfinished right even though the entire project was uploaded successfully it looks like it was unfinished so we're gonna fix that and the solution to this is extremely simple extremely simple we just need to double the size of our weight group over here so we have this sync dot weight group object over here and its length is the number of folders that were uploading right let's double this let's just say multiply it by 2 and what we're gonna do now is also call WG dot done in this go routine that handles rendering the progress bars so here's what's going to happen right it's gonna upload the folder to Google Drive and then it's gonna say WG dot done and it's also gonna render the progress bar all the way to 100% and then it's gonna call WG dot done and so both of these operations are going to complete and the program isn't going to like shut down prematurely anymore this is all we need to do now let's just test this out run the entire project again actually by the way before we before we run this I do want to add one more thing let's go to the main.go file over here and right up top let's just add an empty line over here let's just say fmt.println and we don't need to print anything I just want to have a new line over here why because you see over here it just started the upload process and it erased the previous line over here right and that just looks weird this whole thing looks like one single command right even though it was two commands it It just erased the line above. And that just makes it look weird. So to fix that, I'm just adding an extra line over here. So that's the one it's going to erase. This is what it's going to look like actually. If I run this. Okay. That was not supposed to happen. It just erased my terminal line again. That was not supposed to happen. Okay, so apparently I'm not so good at this stuff. But the progress bar is going to reach 100% in this case. Hopefully. Hopefully I didn't mess that part up. But let me just move that down a bit and see what the hell I did wrong. Yeah, there we go. See, both of the upload progress bars reach 100% now. So that issue has been fixed completely. Awesome. Okay, so with that fixed, let's move on to building the new upload algorithm that I was talking about in the beginning of the video. To do this, we first need to figure out which files are actually changed and which are unchanged, right? We're going to upload the files that actually changed and we're going to ignore the files that are not changed, right? And so we need some sort of a tracker to figure out which Which files were supposed to upload and which files we don't need to upload so inside this API package Let's create a new file and call it tracker dot go It's gonna have package API and it's gonna implement a bunch of functions. First of all, let's create a struct how do you create a struct again type tracker struct there we go I have such a short memory guys it's gonna take a couple of fields first of all the base folder ID this in our case is just the programming folder where we're storing all of our projects right it's gonna be a string and it's also gonna take the drive which is a pointer to models dot drive and it going to take the folder which is a models dot folder pointer also it going to take the folder ID which is which is going to be a string and this is the ID of the folder on Google Drive right this is the base folder ID on Google Drive and this is the project folder ID on Google Drive right finally it's going to take a bunch of nodes and this is going to be a map all right it's It's going to be a map to another struct that we'll create down here. I'll call this node struct. There we go. We don't actually need to implement this struct right now because we're not going to use it right now. This is going to be in a future video, probably the next video after this. For now, let's just finish the tracker struct. This node's key is going to be a map of strings and node pointers. Basically, this node struct is going to track every single file's changes inside that folder. And we're going to use this to figure out which folder or file we're supposed to upload and which ones we're supposed to not upload. Now, let's create a new tracker function over here. New tracker. And this will take the drive, which is models.drive pointer, the folder, which is a pointer to models.folder. and it's going to take the base folder id which is going to be a string and it's going to return a pointer to the tracker object this tracker will be a tracker will will assign the parameters over here drv that's the drive this is the folder and then the base folder id there we go and then it's going to return a reference to that tracker there we go now let's actually start tracking the changes right the very first change that we need to track is whether the project itself has been uploaded before or not right and we can tell this by checking whether the folder of that project exists inside our base folder or not if it does not exist then that means that we've never uploaded this project before and that we should just upload every single file right we don't need to add any any more checks after that because if the if the project folder doesn't exist at all then we know that the entire project needs to be uploaded, right? So we can skip the checks after that. So let's create a function over here. This will be a pointer receiver for the tracker struct. We'll call this project exists. It doesn't need any arguments and it's going to return a Boolean and an error message. Now, quick recall, the way we search for files in the Google Drive API is to first write a query, right? This is what we've done all this time, right? So let's do fmt.sprintf let's say mime type is gonna be a string over here and that string is going to be the folder mime type so that we're actually looking for folders right and percentage as in parents and this is going to be the base folder id if i can type that out over here tracker.basefolderid, there we go, and trash equals false, right? We don't want to search for folders that are in the trash folder, right? Now let's do if response and error equals t, which is a tracker, .drive, .api, .files, a lot of stuff over here, .list, and then .q, and here we'll pass in our query dot fields and here we'll pass in we basically need two fields over here right the id and the name just so we can match this folder and see if it exists right dot page size which is going to be one because we only expect this folder to like we only expect one folder with this name and these parameters right if there are multiple matches then that probably means we fucked up somewhere right ideally there should only be one folder with this name inside this folder. And now let's run this API call by saying do and then error equals nil. And there we go. Now do note that I'm using error equals nil over here, not error does not equal to nil. And in most of the Go code that you read, the you're gonna read error does not equal to nil, right? That means that an error exists, right? If you actually focus on the conditional over here. Error equals to nil, that means there was no error, right? So what I'm basically saying over here is run this API call. And if there was no error, then do all of the stuff I'm going to write over here, right? Now we need a switch statement. So switch length of r.files. That's going to be our condition. First case zero, which basically means that there is no folder in our base folder that matches this project name, right? That means we can just return false. the project does not exist and we can return nil for the error this means that we're going to upload the entire project all over again right case one and here we can return true and nil one more thing we can also do over here is we can assign the folder id in the tracker object over here so we can say r dot files and select the first element in the files and say id and then a default case over here and if it's not zero and it's not one then that then that means that it's larger than one right so multiple matches found so here let's return an error let's return i guess false put a boolean value and let's say errors dot new multiple project folders found and by the way The only reason I'm adding this default check over here is on the 0.001% chance that the Google API returns some weird error over here, right? Because this is impossible to reach. In fact, I'll actually comment this out as well. Impossible to reach because of page size 1. we added this page size one call inside of our inside this um this entire api call where we're calling the google drive api we added this page size equals one parameter as well right so it is impossible to get more than one files in the response right so this default case is largely useless it's not going to ever execute unless the google drive api breaks and it just you know starts returning gibberish right So in most cases this should not even run right In most cases you would just run case zero and case one Anyway so that all if the error was nil right Now, if there was an error in this API call, then we'll just return false and return the error, let's say. That's all we need to do for this one. Now, one final function that we're going to implement is the getCheckSome function. So let's say functracker, this is another method receiver. We'll call this get checksum and it will take an os.file object parameter or object pointer and it will return a string which will be the hash checksum of the file and an error message if there is one, right? Let's say hash equals shard256.new. shahs of 56 is the hashing algorithm that we're going to use and then let's say if underscore um error equals io dot copy and then let's copy to the hash from the file that we have open and say error does not equal to nil then we should just return an empty string over here along with the error message right otherwise let's grab the hash the hash checksum over here by saying hash dot sum and pass in nil over here because we don't need to append anything and then we also need to convert this into a string because this is not going to be a string at first it's going to be a byte slice and so to convert this to a string we'll say hex dot encode i believe that's what you say yeah there we go encode to string and pass in a bunch of bytes over here like this sum variable and then return nil for the error and that's all we need to do so these are the three functions that we're going to actually use in the tracker right now we're going to implement a couple more in future videos but for now this is good enough for us let's actually implement this let's integrate this in our api in our upload algorithm by going to the api slash sync.go file over here first thing i'm going to do is i'm just going to comment out the entire delete previous folder function over here or in fact not comment this out but actually go all the way up to the upload folder function over here and this is where we're actually calling the delete previous folder function so let's comment this out and if i do that if i comment this out then we're not going to delete the previous folder and we're going to have a bunch of duplicates in our projects right which is fine for now we're going to fix this in future videos but now we do need to fix a couple things first of all let's go down to the filepad.walkdir function you know that the this walkthrough function is first of all going to create the project directory right because that's the first path that this this function is going to access right it's going to be the root path of this project so we can say something like if the current path that we're on matches the folders path the project folder that we're uploading then we can do a couple of things here first of all let's just go up over here and just below the subfolders let's create a tracker object call new tracker and pass in the parameters over here let's pass in drive the folder and the base folder id which is going to be base dot id there we go and then let's see if the project already exists or not so i'll say project exists and an error message over here and call tracker dot project exists right and this will return a boolean and an error just like we implemented it and first of all let's just handle the error before we do anything else i'll say log.fatal f to crash the entire program i'll print the folder name first and foremost we'll say folder.name over here as the first argument checking if project exists already and then we'll print the error message along with a new line so let's add the error message over here and that's all we need to do cool now let's go back down over here in the filepad.walk there first of all if the project already exists then i want to let's just add a log first of all project folder already exists assigning id so how we implemented subfolders over here right uh where is it is where is we we basically add the id of the subfolder in that subfolders map over here right so So this happens automatically every time we create the subfolder. But since we're not deleting the project folder anymore, it's going to exist already. We need to make sure that we actually assign its ID to the subfolders list, right? So I can say over here subfolders and the key will be the folder.path, the project folders path, and the ID will be tracker.folderID, right? And this will get assigned as soon as we call the project exists function over here. it's going to assign this t.folder id value over here once it finds the project right and so we can just use that and if the project folder does not exist already then we should create the project folder and by the way before anything else let's also return nil over here because there was no error right and let's go up outside of this function and write the create project folder function this is just gonna be create project folder there we go it's gonna take the drive it's gonna be models.drive the base folder ID which is gonna be a string and the name which is going to be a string right the project folders name and it will return a drive dot file pointer and an error message and this is going to be a very short function literally just one line it's going to return create subfolder the function that we implemented several videos ago we'll pass in the drive the base folder id this will be the parent of course and then the d over here uh we need to pass in the fs.dir entry i'm just going to change this because i only have a name over here so that's what we're going to use let's go to the subfolder function down here instead of using this fs.dir entry let's just use the folder of the name Because we literally only need the name of the folder over here and nothing else, right? So let's just use that and let's also fix the error that this is going to cause in our code over here There we go And over here in the way we actually calling the create subfolder function Just use G dot name instead of name right pass in the actual name of the folder now that we done that let implement this let use this create project folder function that we just implemented i'll say response and error equals create project folder pass in drive uh there we go and then base.id and also the name which will be folder.name i guess if there's an error if error does not equal to nil let's say um logs.fatal f again fail to create project folder and then print the error message along with a new line there we go and of course make sure to assign the id of this folder the google drive id of this folder to the subfolders map that we're using over here so that we can actually upload all of the files inside this folder right so let's say sub folders I'm just gonna keep that in the video subfolders folder dot path and response dot ID there we go and return nil so we don't execute all of the other code below this right and that should be all we need to do now this is what the what my Google Drive looks like right now this drive sync or video project over here that we're actually uploading and let's let's actually run the entire upload process again let's see which folders am i uploading okay these two let's run this again and see what happens see if the stuff works or not now spoiler alert i don't think most of this is going to work i'm pretty sure we just broke the entire upload process we're going to fix this in future videos it's going to take like like i said in the beginning guys this is going to take like two or three videos to actually implement the entire um algorithm for synchronizing all of this stuff and it looks like it just failed already awesome let me try running this again I don't know what happened over there okay so there's some errors in the log file over here but I don't understand them one bit so I'm just gonna delete these folders over here instead of actually trying to fix the error I'm just gonna empty my entire Google Drive and then we're going to run the same process again and it still doesn't work nice okay so it's failing to find the parent folder where is this error message let's do a quick search okay right here fail to find error prints fail to find parent folder okay so this happens over here okay and it's failing to find the project folder okay we're gonna do a bit of live debugging over here okay guys uh before i actually try to solve this error let me go to main.go and uh let's not redirect all of our logs to a log file since that's just since i want to see them in the terminal now whilst I'm debugging this this entire application and also in sync.go in the command over here let's uh let's not call render bars because I don't want to see the progress bars anymore not while I'm debugging this application so let's run this again and it's still gonna fail but at least it's going to show us the logs right so it says project folder already exists and it's failing to find that print folder. Most likely it's just not assigning the ID properly. So let's go fix that. Oh wait wait wait wait wait. I figured it out. Oh my goodness. So the problem is that I'm using folder.pack over here in the subfolders list over here as a key. I'm using the folder.pack instead of folder.name. This is supposed to be the folder.name. That's how we filter that's how we map these subfolders by their by the folder name not by the folder path oh my goodness I can't believe I've messed all of that up let's run this again and this time it's going to actually work I can't believe I wasted like ten minutes trying to debug this application yeah this time it's working awesome oh my goodness how long is this video again yeah see we're approaching 40 minutes of recording again I I try so hard to not make such long videos but then the projects i build are also so complicated and all this stuff happens right so hopefully you have the attention span to watch this entire video because um i'm trying my best but the videos just get so long you know anyway this entire process finished and here are the project folders why is there nothing in here over here okay there we go oh okay all right so as you see there's a ton of duplicates over here right every single file got uploaded like twice this is because we're not deleting the inside folder anymore right so the upload process works but it's a lot more broken and messed up now right in future videos or in the next video probably we're going to fix this and track every single files changes so that we only upload the files that have been changed and we delete any other files right we we leave the unmodified files unchanged we upload the new files and the changed files and any single file that we delete from our local folder we should delete that from google drive as well right so we're going to implement all of that stuff in the next video or the one after that for now we're done with this video we've built the a basic tracker over here and we've implemented we've integrated with our upload algorithm. I'm gonna stop the video there guys because this is getting extremely long, 40 minutes just of recording and then I have to edit this entire thing as well. It's gonna take forever. So I'm gonna stop over there. Hopefully you learned something new over here because this project is starting to become like a really great useful project especially for me. Like the only reason I'm even building this is because I personally am going to use this because I'm the kind of guy who especially for personal side projects I just forget to push everything to github and I just you know like just code locally and just forget to push it to github especially when it's a personal project especially when it's a side project that I'm just working on that nobody else is going to really see I just forget to upload all of those things to to github so with this project this drive syncer project that i'm building i can just automate that entire process for me right even if i don't upload to github this program can just run automatically and synchronize changes with my programming projects to google drive right so that's going to be awesome this program is going to be awesome once it's finished but uh yeah thanks for watching i'm going to stop the recording now see ya